Technical Lineage - any plan to read lineage from db procedures? | Product Q&A

Technical Lineage - any plan to read lineage from db procedures?

Hello,
My company is already using technical lineage harvester and we think it is a great tool to show the data lineage.
Some of our data transformations are performed via db procedures (we are using Snowflake), and we realized that data transformations happening in procedures is not covered by technical lineage harvester. Any plan to include this capability in the future?

Best Regards,
Razayu

razayuyusup

Posted 5 years ago · Edited 1 year ago·Last reply 4 years ago

2 comments

noor

·4 years ago · Edited

Hi Razayu,
May I know how does the Snowflake lineage harvester treat this scenario -

I have 3 schemas in Snowflake where data passes from one schema to another without any transformations in between. There is a metadata driven/parameterized stored procedure that does it.

Will lineage harvester be able to link tables/columns across these 3 schemas by reading their DDLs or it needs the ETL code (in this case the stored procedure or SQL based input) to understand the data movement so it can build links…

noor

·5 years ago · Edited

Yes, on my side too, transformations happen through SPs. I just now noticed that SPs are not supported to parse lineage from.

I also realized that none of the systems on my side are supported

SAP DI, Azure Data Factory, Databricks (Python), Snowflake SPs