Custom Technical Lineage Challenges
Summary: Pavan Shukla is facing a challenge with creating a custom technical lineage between systems, having ingested metadata via Edge and attempting to synchronize lineage and metadata JSON files. Elliott Diehm suggests ensuring there are no spaces around the '>' in column names and questions whether the Activity and Edge jobs completed successfully. Pavan Shukla confirms successful synchronization. Grant Rollerson adds that the assets need to be in the same system for the lineage to work.
I've recently ingested metadata from a source via Edge and established the technical lineage. Now, I added another system with a manual file upload and need to create a technical lineage between the two. I've attempted to create a custom technical lineage utilizing cloud storage with lineage.json and metadata.json, synchronized as per the documentation. However, the technical lineage still doesn't display. Here’s a snippet of the lineage.json and metadata.json. Could anyone share effective strategies for handling custom technical lineage issues like this? What are some common pitfalls when setting up custom lineages, and how can they be overcome? Any experiences or insights would be greatly appreciated!
metadata.json
{
"version": 3,
"application_name": "IRIS_TO_ADS",
"asset_types":{
"Column":{"uuid": "00000000-0000-0000-0000-000000031008"},
"Table":{"uuid": "00000000-0000-0000-0000-000000031007"},
"Database":{"uuid": "00000000-0000-0000-0000-000000031006"},
"Schema":{"uuid": "00000000-0000-0000-0001-000400000002"}
}
}lineage.json
[
{
"src": {
"nodes": [{"name":"IRIS", "type": "Database"}, {"name": "XUNMDTAINI", "type": "Schema"}],
"parent": {"name": "POLICY_TRANSACTIONS", "type": "Table"},
"leaf": {"name": "accounting_brokerage_amount", "type": "Column"},
"props": {
"fullname": "IRIS.XUNMDTAINI.POLICY_TRANSACTIONS.ACCOUNTING_BROKERAGE_AMOUNT",
"domain_id": "019f6411-bdf9-756e-92e7-80792ea2892a"
}
},
"trg": {
"nodes": [{"name":"qa_lum_dp", "type": "Database"}, {"name": "policy", "type": "Schema"}],
"parent": {"name": "policy_transactions", "type": "Table"},
"leaf": {"name": "accounting_brokerage_amount", "type": "Column"},
"props": {
"fullname": "qa_lum_dp>qa_lum_dp>policy>policy_transaction>accounting_brokerage_amount(column)",
"domain_id": "019e02ab-4179-733d-8c9b-1bab208a7297"
}
}
}
]
Grant Rollerson
·3 weeks agoremember they need to live in the same System (parent)
Elliott Diehm
·3 weeks ago · EditedPavan,
You shouldn't have to put the fullname or domain_id in the lineage.json file. Just make sure that there are no spaces around the ">" in the full name of the columns. Did the Activity and Edge job complete successfully? Does the Technical Lineage status show that the synchronization was successful?
Elliott
Pavan Shukla
OP2 weeks ago · EditedYeah, Synchronization was successful
Pavan Shukla
OP2 weeks agoGreat, it worked. I updated the fullname in source with > as separator without spaces.