creating-openlineage-extractors

SKILL.md

Custom OpenLineage extractors for unsupported Airflow operators and complex lineage scenarios.

Two approaches: add OpenLineage methods directly to operators you own (recommended), or create custom extractors for third-party operators you cannot modify

Extractors intercept operator execution at three points: before execution for static lineage, after success for runtime-determined outputs, and optionally after failure for partial lineage

Register extractors via airflow.cfg or environment variable; on Astro, custom extractors are automatically picked up and lineage appears in the Lineage tab

Includes patterns for SQL operators, file transfers, and dynamic outputs determined at runtime; handle optional properties and avoid circular imports by importing inside methods

Creating OpenLineage Extractors

This skill guides you through creating custom OpenLineage extractors to capture lineage from Airflow operators that don't have built-in support.

Reference: See the OpenLineage provider developer guide for the latest patterns and list of supported operators/hooks.

When to Use Each Approach

Scenario
Approach

Operator you own/maintain
OpenLineage Methods (recommended, simplest)

Third-party operator you can't modify
Custom Extractor

Need column-level lineage
OpenLineage Methods or Custom Extractor

Complex extraction logic
OpenLineage Methods or Custom Extractor

Simple table-level lineage
Inlets/Outlets (simplest, but lowest priority)

Important: Always prefer OpenLineage methods over custom extractors when possible. Extractors are harder to write, easier to diverge from operator behavior after changes, and harder to debug.

On Astro