I am looking for the process of establishing data lineage in our DGC. I found documentation that points out that this function is a cloud base solution only. We are currently using an on premise environment. If this is not true and some you are using an on premise environment. Please let me know the resource you use to get data lineage into the DGC. I want to write out the process for our stewards.

Data lineage process/procedure | Product Q&A

fredhedrington

OP4 years ago · Edited

@chandra.papudesu Hi Chandra, thank you for the great deals on this issue. Greatly appreciate your time.

Kind regards,

Fred H.

Chandra Papudesu

·4 years ago · Edited

@fred.hedrington Collibra’s Data Lineage product automates the process of extracting the technical lineage from the most common SQL based data sources, ETL tools and BI tools. The technical lineage is presented along with the transformations in a specialized viewer and the data catalog (DGC) is updated with the summary lineage for more business-friendly consumption. This product is only available for our cloud customers.
For on-prem customers, they can continue to use our REST APIs to create assets and relationships within DGC , but the technical lineage features will not be available.

Chandra Papudesu

·4 years ago · Edited

@fred.hedrington Collibra’s Data Lineage product automates the process of extracting the technical lineage from the most common SQL based data sources, ETL tools and BI tools. The technical lineage is presented along with the transformations in a specialized viewer and the data catalog (DGC) is updated with the summary lineage for more business-friendly consumption. This product is only available for our cloud customers.
For on-prem customers, they can continue to use our REST APIs to create assets and relationships within DGC , but the technical lineage features will not be available.

noor

·4 years ago · Edited

@chandra.papudesu
May I know how does the Snowflake lineage harvester treat this scenario -

I have 3 schemas in Snowflake where data passes from one schema to another without any transformations in between. There is a metadata driven/parameterized stored procedure that does it.

Will lineage harvester be able to link tables/columns across these 3 schemas by reading their DDLs or it needs the ETL code (in this case the stored procedure or SQL based input) to understand the data movement so it can build links…

The stored procedure or even SQL input I have in this case may not provide much insight for the harvester to read it & understand the technical transformation details