5 Messages
Unable to process SQL files retrieved from 'DBT Lineage to Collibra integration v1.2.3' in Lineage Harvester
The broad steps followed were as follows:
- The ‘DBT Lineage to Collibra integration v1.2.3’ retrieves SQL files from dbt.
- These SQL files were placed in a directory for Lineage Harvester to consume.
- Updated Lineage Harvester config file to include these SQL files by adding a new source with type as “ExternalDirectory”,
- Lineage Harvester reads the SQL files and prepares the zip file and places it in ‘output’ folder of LineageHarvester for further processing, and subsequently fails, showing error as :
ERROR harvester.Harvesting [src=Dbt] - harvester.error.ShowcaseClientError: Showcase returning client error:
File ‘CUSTOM-LINEAGE/lineage.json’ not found in ‘f21e9910-70b9-49e8-a65b-cf3fccb78d81.zip’.
So have following queries:
- Is the lineage.json file expected to be created by ‘DBT Lineage to Collibra integration v1.2.3’? If not, then how would it be created?
- For Lineage Harvester to ingest the SQL files, we have set source type to “ExternalDirectory”, is this correct or should it be something else?
alessandrofeoli
2 Messages
2 years ago
Having the same issue on my side. @bhupinder.parmar.underarmour.com did you ever get an answer?
0
0
bhupinderparmar1
5 Messages
2 years ago
@alessandro.feoli.moodys.com the issue at my end was resolved. The type for the source needs to be set to “SqlDirectory” instead of “ExternalDirectory”. Hope this helps.
0
alessandrofeoli
2 Messages
2 years ago
It did, thank you! This is how my lineage-harvester.config looks for anyone with the same issue:
“sources”: [
{
“type”: “SqlDirectory”,
“mask”: “*.sql”,
“dialect”: “redshift”,
“collibraSystemName”: “dbt”,
“database”: “dev”,
“schema”: “DefaultDBTSchema”,
“extractLog”: true,
“extractQueries”: true,
“id”: “dbt”,
“path”: “lineage”,
“verbose”: true,
“recursive”: true,
“deleteRawMetadataAfterProcessing”: false
}
]
0
0