Data Lineage & Integration Patterns

The data model is the contract. The lineage is the audit trail. The integration pattern is the translation between systems. The operating model is what keeps the data architecture current as the systems evolve.

On this page
  1. Top of page
  2. What data lineage and integration are
  3. The patterns
  4. State of the practice
  5. Questions we get asked
  6. Patterns we design our practice to avoid
  7. Evidence & references
  8. What to read next

§ 01 ·

What data lineage and integration are

Data lineage is the audit trail of where data came from, what was done to it, and where it went. The lineage is the evidence that the data is what the consumers think it is. The lineage is also the evidence that the data is what the regulators require it to be. The lineage is the data architecture's equivalent of the threat model: a shared model of the system that the rest of the conversation uses.

Data integration is the discipline of moving data between systems without losing fidelity, latency, or meaning. The integration pattern is the translation between the source system's model and the target system's model. The pattern is one of a small set: extract-load, extract-transform-load, change data capture, event-driven replication, request-response API, shared database. Each pattern has its own tradeoffs; the right answer depends on the latency, the consistency, and the coupling.

Two failure modes recur.

  • The post-hoc lineage. The data lineage is documented after the system is built, which produces a finding list, not a model.
  • The convenience pattern. The integration pattern is chosen by the convenience of the source system, not by the requirements of the target system, which produces a coupling that is hard to undo.

The right design is to design the data model and the integration pattern as part of the system design, not as an afterthought.

§ 02 ·

The patterns

The integration patterns that have held up.

ELT (extract-load-transform)

The pattern of extracting data from the source, loading it into the target, and transforming it in the target. ELT is the right choice when the target is a data warehouse or a lakehouse, and the transformation can be expressed in SQL. ELT pushes the transformation to the data, which is the right place for it.

CDC (change data capture)

The pattern of capturing changes in the source system and applying them to the target system. CDC is the right choice when the target needs to be eventually consistent with the source, and the latency requirement is seconds-to-minutes. CDC is the right pattern for operational replication.

Event-driven replication

The pattern of publishing events from the source and consuming them in the target. Event-driven replication is the right choice when the consumers are many and the consumers' needs are different. The event is the contract; the schema is the integration.

Request-response API

The pattern of calling a service in real time to get the data. Request-response is the right choice when the consumer needs the current state, not the history, and the latency requirement is milliseconds. The API is the contract; the schema is the integration.

Shared database

The pattern of two systems reading from the same database. The shared database is the right choice when the two systems are tightly coupled (e.g., the same bounded context) and the alternative is worse coupling. The shared database is the right choice rarely; the cost is the coupling.

The pattern is the tool; the contract is the architecture. Pick the pattern that fits the latency, the consistency, and the coupling requirements. Document the choice. The choice is reviewed when the requirements change.

§ 03 ·

State of the practice

5
Common integration patterns (ELT, CDC, event-driven, request-response, shared database).
Industry observation
~60-80%
Of a typical data engineering effort is data wrangling, not the integration itself.
Industry observation (Anaconda State of Data Science, others)
1
Right answer to which integration pattern: the one that fits the latency, consistency, and coupling requirements.
Industry observation

§ 04 ·

Questions we get asked

Depends on where the transformation belongs.

  • ELT fits when the target is a data warehouse or a lakehouse and the transformation is expressible in SQL. The data goes to the warehouse, then the transformation runs in the warehouse.
  • ETL fits when the transformation is domain-specific and the warehouse is not the right place to run it. The transformation runs in the pipeline, then the data goes to the warehouse.

The right choice depends on the warehouse, the transformation, and the team that owns each.

With explicit, machine-readable versioning and a contract registry. The schema is versioned (semver or hash). The version is in the message (header or payload). The registry is the source of truth for current and historical schemas. Breaking changes trigger new versions; additive changes do not. Deprecations are announced and timed.

Three controls.

  • A watermark (the latest event timestamp the consumer has processed); the watermark is durable and versioned.
  • An out-of-order policy (the consumer accepts events with timestamps older than the watermark, up to a configured tolerance).
  • A replay path (the consumer can rebuild its state from the source of truth when the watermark drifts).

The watermark is the contract; the out-of-order policy is the failure mode.

The lineage is current if the conversations that use it are current.

  • If the data quality conversation is referencing the lineage, the lineage is current.
  • If the regulatory conversation is referencing the lineage, the lineage is current.
  • If the architecture conversation is referencing the lineage, the lineage is current.

A lineage that is not referenced is a lineage that is not current.

§ 05 ·

Patterns we design our practice to avoid

A lineage that is documented after the system is built is a finding list, not a model. The model has to be in the same review process as the design. When the design changes, the lineage is part of the change.

An integration pattern chosen for the source's convenience is a coupling that is hard to undo. The pattern fits the requirements of the consumer: the latency, the consistency, the coupling. The source's convenience is at most one input to the choice, not the primary one.

A schema that is undocumented is a schema that is consumed by accident. The schema is a contract. The contract is documented. The contract is versioned. The contract is in the registry. A schema that is not in the registry is not really a schema.

Data quality that is checked at the warehouse is data quality that has already travelled. The check belongs at the producer, where the data can be corrected. The warehouse check is a backstop, not the primary control.

A pipeline that is owned by nobody is a pipeline that nobody fixes when it breaks. The pipeline has an owner. The owner is on the hook for the data freshness, the data quality, the schema evolution, the incident response. The owner is named.

§ 06 ·

Evidence & references

Public frameworks and writing that inform our practice.

Designing Data-Intensive Applications
Kleppmann, O'Reilly, 2017

The book that made distributed data systems a discipline. The treatment of event logs, change-data-capture, stream processing, and consistency models is the most rigorous available.

Fundamentals of Data Engineering
Reis & Housley, O'Reilly, 2022

The lifecycle framing. The book treats the data engineering practice as a discipline with a lifecycle (ingestion, transformation, serving, ownership), not a tool catalogue. The framing has held up.

Data Mesh
Zhamak Dehghani, O'Reilly, 2022

The federated data ownership argument. The book's contribution is the framing of data as a product, owned by the team that produces it. The book is opinionated. The framing is useful even when the specific architecture does not fit.

Change Data Capture
Debezium project, current

The reference implementation of CDC. The project is the most production-tested open source CDC implementation. The architecture (source connectors, Kafka, sink connectors) generalises beyond the project.

Read the Change Data Capture reference
Apache Kafka
Apache Foundation, current

The event log reference. The pattern (event log, consumer group, offset) generalises beyond the project. The pattern is the contribution, not the project.

Read the Apache Kafka reference

§ 07 ·

What to read next

Related elaborations on this site, plus the conversation to start.

Building a data architecture?

Weighing a data integration initiative, scoping a data warehouse, or trying to make an existing data architecture more than a pipeline documented after the fact? Useful at the boundary between the data model and the integration pattern. A short conversation is the right next step.

Learn more
© NOETRONIQ Contact