data lineage example (describe relationship between a table and a view on that table)

[suiluj]

hello,

currently I am describing a few tables and views via ODCS (version 3.1.0)

How can I describe a data lineage relationship between a source table and a view on that table.

The view filters the original table data and only contains a sub set of the columns.

I noticed that you can describe relationships between columns but this is not a foreignkey relationship (the only possible value at the moment).

I found some links describing data lineage plans:

• https://bitol.io/bitol-announces-odcs-v3-1-0-stronger-smarter-and-stricter/
  • RFC-0011: Enhanced Lineage
• https://github.com/bitol-io/tsc/blob/main/rfcs/0011-lineage.md

Are there any example available?

my goals

• reference existing column/properties in schema (so there is no duplicate column definition)
• make relationship between table and view visisble
  • this could mean a general relationship between table and view with a description about the tranformation
  • column level data lineage would be nice to have

Perhaps this is already possible but I did not understand the spec enough?

[sakshipatil-hue]

@suiluj To achieve your goals in ODCS v3.1.0, here is the recommended architectural approach:

1. Data Lineage (Table to View)
   With the implementation of RFC-0011 in v3.1.0, ODCS shifted towards course-grained data contract lineage. If your source table and view exist in separate data contracts, you should use the new lineage: block at the root level and reference the upstream table's contract under inputDataContracts.
   ODCS explicitly avoids modeling complex transformation logic or fine-grained column-level lineage inside the contract itself, deferring that operational lineage to tools like OpenLineage (which you can link via outputDatasets).

2. Avoiding Duplicate Column Definitions
   If your source table and your view are defined within the exact same ODCS YAML file and you want to avoid rewriting the schema for the subset of columns, you can use standard YAML anchors (& and *) to reference the original properties.

3. Relationships Array
   You are spot on regarding the relationships field. It is designed to model structural associations (like foreign keys across domains) rather than data transformation lineage between tables and views!

[suiluj]

@sakshipatil-hue thanks for the detailed explanations and hints. 👍