"Where clause" in column-to-column source/target lineage?
I have the situation where IF a certain condition occurs THEN a particular column in a table X is used to derive the value of another column in table Y.
Linking the two columns in a source / target lineage relationship is not a problem, but what are people doing to document the condition that is necessary for this lineage to exist? Where is the best place to keep this in Collibra?
andrewmarsh
Posted 3 years ago · Edited 1 year ago·Last reply 3 years ago
3 comments
tomkuppens1
·3 years ago · EditedHi @andrew.marsh.telus.com,
I’m working on this case where we have a big table that contains - let’s say - all transactions. But the lineage of where these transactions come from are dependent on the transaction type. Further upstream, the views and other ETL jobs are consuming these transactions and they are filtered based on the transaction type id. So lineage for this table is particular tricky. With regex’s we are able to extract these filters from the queries and we use this information to create a virtual lineage layer on top the existing lineage in Collibra. In other words, a table or consuming asset is tagged with a specific set of transaction type id, which allows our data folks to know what the provenance of the transaction data is.
This solution is not ideal at all, but it helps our data engineers and analysts understand the data landscape a little bit better. Once we have our ETL lineage under control I plan to work towards a solution where we can have end-to-end lineage for these type of tables, but this means hefty customization and coding.
Tom
sophiemcenteemorris
·3 years ago · EditedHi @andrew.marsh.telus.com ,
We have a community set up specifically for things like this.
It allows us to capture details about the relationships and lineage between assets.
navaratanramesh
·3 years ago · Edited@andrew.marsh.telus.com May be you can use the transformation attribute on mapping specification / field mapping.