Data lineage automation using openlineage framework

Hello! It’s definitely possible to hack, especially with custom lineage - the transformation code is just a string, so you can append anything you want.

We have an open ideation ticket - we’d love votes but even more so - your use case. Where do you want to see that information? what do you want to be able to do with it? e.g. you want jobs start time, job end time, and whether or not it failed. I assume that’s for the most recent run, not the entire history.

What sources are you using (e.g. Oracle, Matillion, Tableau, custom lineage) and where would that information come from?

https://productresources.collibra.com/ideation-platform/?id=PINT-I-60

@jean-luc.garnier.safrangroup.com @tom.kuppens.telenetgroup.be @miguel.guillen

Hi Sheeri,
Thanks for you message. We still need to refine what and how we are going to manage the operational lineage intergration in Collibra.

From an overall perspective, having two places to manage lineage is Collibra is not very efficient. The DGC has a powerful semantic layer that can connect everything. The techlineage engine is powerful in getting the lineage out of the source systems and getting into the techlin module. Both very cool! But then it stops being cool, because the integration between the techlin module and the DGC is not ideal at the moment, to mention a few issues:

• Custom tech lineage is not well integrated : [Custom Technical lineage] https://datacitizens.collibra.com/forum/t/custom-technical-lineage/2627)
• techlin deletes do not detele col-col relations in the DGC techlin updates/deletes do not update/detele col - col relations in the DGC – Support Portal (collibra.com)
• some lineage comes in via the DGC (nodegraph Qlik lineage, and we have some homebrew lineage parsing going-on ) and is not available in the techlin module.

Because of the descriptive nature of operational lineage and the issue mentioned above, I think the logic place to have that information in the DGC. We would use it to propagate the operational information on the actual jobs to everybody who is interested in it. The use case are plentiful e.g. if using a table, you want to know its provenance but also the jobs that it’s depended on (via the table provenance), and if these jobs ran without hiccups. How the information is compartmentalized and managed is open for discussion.

We are doing Qlik Sense lineage (Nodegraph), SPSS modeler lineage (own development), Oracle Lineage (techlin Oracle), Datastage Lineage (techlin Oracle hack because the Datastage method does not give good lineage). We are using a couple of other inhouse job frameworks and looking into phasing out Datastage.

Thanks,
Tom KUPPENS

Yes, good news thanks for your feed back

What about manta solution please ?

While it is tempting to bring in operational metadata into Collibra, we should be asking “who” does “what” with the information. Some of the operational metadata that report on freshness of data, usage, data quality, etc. does make sense to bring into Collibra, but you have to ask why “operational lineage” when even “definitional lineage” across an enterprise portfolio of applications is often challenging to represent in a data owner/steward friendly manner. We have Collibra well established as our enterprise metadata and data governance tool. But we are now recognizing needs for a product platform metadata repository (we selected LinkedIn DataHub, but there are other options) which sources definitional metadata from Collibra but also operational metadata from the platform, primarily for data engineer and data ops use. A federated architecture for metadata is necessary for enabling enterprise wide semantic consistency while also supporting product agility.

Hi Thomas,
Interesting!

We are also at the point where we understand that Collibra is not enough in terms of lineage management. Like I describe in one of my earlier posts in this thread, there are many thing we struggle at the moment, and our go solution to it mostly: let’s get the data outside Collibra and do it ourselves. E.g. reporting: Give me a list of tables that are used in a certain dashboard?, or In which reports in this table used?. We recently managed to get operational lineage into Collibra, but the engine and ingestion is built outside Collibra.

So, I’m really curious what your experience is with Datahub, and the other options you had a look at!

Best,
Tom Kuppens
[email protected]