Data Lineage

Overview

Data lineage is a map of where data comes from and where it goes. Nodes are datasets, jobs and runs; edges show reads and writes. It answers two questions: what feeds this table, and what breaks if I change it.

How it works

Each job run emits an event naming its inputs and outputs. A collector merges events into a graph.

raw.orders --\
              job: build_orders_clean --> silver.orders --> job: daily_revenue --> gold.revenue --> dashboard
raw.refunds -/

An OpenLineage-style event contains the job name, run id, state, input datasets and output datasets, plus optional facets such as schema and row counts. Column-level lineage comes from parsing SQL to map output columns to source columns, which supports tracing a tagged personal data column to every downstream use.

Typical uses: before dropping a column, query all downstream consumers; during an incident, walk upstream from the broken table to the first failed or late run; for compliance, list every dataset derived from a regulated source and verify access controls.

Pitfalls

• Partial coverage: a script that writes outside instrumented systems creates blind spots. Track coverage and treat unknown edges as a risk.
• Treating lineage as documentation written by hand, which decays. Capture automatically at run time.
• Over-trusting column-level results from SQL parsing when queries use dynamic SQL, UDFs or select star.
• Ignoring run-level context: static graphs show possible flows, runs show what actually happened.
• Exposing lineage without access control may reveal sensitive dataset names.

Key takeaways

• Emit lineage from the tools, not from people.
• Use it for impact analysis, debugging and privacy audits.
• Combine static dependencies with per-run events.
• Measure coverage so gaps are visible.