Custom Lineage

Hi Team,

Does the custom lineage support table level lineage instead of column level lineage ?
This is required because metadata which we receive doesn’t have a source and target column mapping.

When we run with out column details, lineage harvester expects column details else it will give a error.

Thanks & Regards,
Naveen

naveenjayaram

Posted 3 years ago · Edited 1 year ago·Last reply 3 years ago

8 comments

tomkuppens1

·3 years ago · Edited

Great tips . Great value!
So I added custom table level lineage of existing (in our data cat) tables, stitching went well (they light up all yellow) and visualisation in the lineage module looks good. But the techlin source-target relations are not introduced into the DGC for this type of lineage. Anyone has a solution for this?

Thanks!
Tom

@naveen.jayaram @rcraig.agloan.com @said.rahmani

tomaskissik

·3 years ago · Edited

Hi Tom,

harvester unfortunately won’t add relationship between tables if the asterisk workaround is used. If you want a relationship to be created, in technical lineage you must specify relationship between columns that exist in the catalog.

tomkuppens1

·3 years ago · Edited

Hi Tomas, thanks for the confirmation!
I’m working on a workaround where we create the source-target relations ourselves on table level when importing custom table level lineage. Conceptually it should work, but I hope it will not trigger any artifacts in the DGC diagram visualisation or other unexpected side-effects.
Thanks,
Tom

saidrahmani

·3 years ago · Edited

@naveen.jayaram
Instead of
“name”: “",
“type”: “string”
Put
“name”: “*”,
“type”: “column”

naveenjayaram

OP3 years ago · Edited

Thanks @kristen.freer @said.rahmani

I tried with * and i was able to export but i am not sure is it the correct way.

Below is my custom json

{
“version”: “1.0”,
“tree”: [
{
“name”: “ProductDB”,
“type”: “database”,
“children”: [
{
“name”: “product”,
“type”: “schema”,
“children”: [
{
“name”: “Sales_read”,
“type”: “Table”,
“leaves”: [
{
“name”: “",
“type”: “string”
}
]
},
{
“name”: “Sales_tx”,
“type”: “Table”,
“leaves”: [
{
“name”: "”,
“type”: “string”
}
]
},
{
“name”: “Sales_sink”,
“type”: “Table”,
“leaves”: [
{
“name”: “",
“type”: “string”
}
]
}
]
}
]
}
],
“lineages”: [
{
“src_path”: [
{“database”: “ProductDB”},
{“schema”: “product”},
{“table”: “Sales_read”},
{“column”: "”}
],
“trg_path”: [
{“database”: “ProductDB”},
{“schema”: “product”},
{“table”: “Sales_sink”},
{“column”: “*”}
],
“mapping”: “Sales_tx”,
“source_code”: "SELECT Country_Code, Order_id, Item_type, Sales_Channel, Order_Priority, Unit_Cost, Unit_Price\n FROM ${Sales_read}\n "
}
]
}

Am i doing it correct ?

Thanks & Regards,
Naveen

saidrahmani

·3 years ago · Edited

The workaround is to define a column with “name”:“*”
This workaround simulates table-level lineage. It also can be used to show indirect lineage; however, it will show up in the diagram as a solid line, not a dashed line, and there is no need to select the “Show indirect lineage” setting.

Roger Craig

·3 years ago · Edited

@naveen.jayaram I’m not sure as I’ve only done a single proof of concept with custom lineage at the Column level. Nevertheless, if I was in a similar position and could not find a way to exclude columns, I would just give each table a single column with the same name as the table. Granted, the column would not stitch to anything, but that shouldn’t be a problem.

Of course, I’m not sure what the end result will be, but it’s something to try.

Alexandra Jorgenson

Admin3 years ago · Edited

@naveen.jayaram One of our Solution Architects suggested that a workaround may be to declare a column named “*” ?