Collibra Community

Lineage & Visualization

Created Nov 18, 2023

0 members

6 discussions

The Lineage & Visualization topic in the Collibra Community is a dedicated space for users working on understanding and building visual representations of data lineage. It covers practical questions around tools like Edge and Lineage Harvester, how to stitch lineage across complex systems, and best practices for creating clear, end-to-end views of data flow across an organization.

Andrew Southcombe
7 months ago
Repository for lineage harvester
Our internal policies mean we are only allowed to download software in one of two ways, 1) our Artifactory instance or 2) via a request to our helpdesk. It seems lineage harvester is only available to download from collibra's website via Lineage harvester 2024.08 - Collibra | Product Resource Center and the download link resolves to a random cloudfront URL. This means we can only install it after requesting from our helpdesk and they have a security scanning process to go through which can take a couple of weeks. This will make it painful for us to keep it up to date. Is there anyway of downloading it either from a fixed URL root, or from a repository that we can mirror on artifactory? (preferred).

3
Asif Rasul
1 year ago
Lineage understanding
Say, the IT landscape has heterogeneous datasources (Oracle, Talend ETL, SQLServer, PowerBI) and data flows in that order. Will Collibra be able to stitch automatic lineage? What is the mechanism that Collibra uses for automated lineage stitching?

0
Andrew Southcombe
1 year ago
Lineage from file to table
We have managed to run the Edge lineage capability to harvest column to column lineage within Snowflake. Our next step is go back to the source and try to model lineage between the original source file and the target Snowflake table. In general our source files are held in an ADLS2 data lake and then get loaded in via snowpipe.

We are planning to use the ADLS2 connector to harvest file metadata into the catalog. Once we have done that, is there a recommended way to show the technical lineage from file to table? For example, should we create custom lineage and submit to Edge? Or can we directly create relations between a file and the table it was loaded to?

0
Constance Barton
1 year ago
Dependent Sources for Technical Lineage for SQL Server
We are converting our SQL Server lineage from Lineage Harvester to Edge. We want to get end-to-end lineage for Database A to Database B to Database C. Each Database has a separate Source ID, connection, and lineage capability… We accomplished this in the Lineage Harvester by using the “useSharedDbModel: true” in our Lineage Harvester configuration file.

In the Edge “Technical Lineage for SQL Server” configuration page, there is a setting called “Dependent Sources”. This is located in the “Advanced Properties” section. The description says “List of Source IDs this source has a dependency on. Click Add Property to add each Source ID.”. The Collibra documentation does not describe this property.

I suspect that this property “Dependent Sources” is the way Edge accomplishes the same thing as the Lineage Harvester “useSharedDbModel” setting? If so, how do I properly configure the “Dependent Sources” property?

I have 3 databases and 3 Technical Lineage for SQL Server capabilities (one for each database). Do I need to set the “Dependent Property” value in each capability to mention the other dependent databases? For example:

Database A - list Dependent Sources as Database B and Database C
Database B - list Dependent Sources as Database A and Database C
Database C - list Dependent Sources as Database A and Database B

Any insight will be appreciated.

1
Pinky Pattnaik
1 year ago
Informatica PowerCenter Technical Lineage using EDGe
Hi ,

Has any one tried this technical lineage of Informatica and Power Center with Collibra through edge capability
Informatica PowerCenter Technical Lineage using EDGe

7
Manjunath Narayanawamy
1 year ago
Relationships created for technical lineage
we are in the process of building the technical lineage for our assets (Tables and columns), we don’t have the license for lineage harvester feature rather we use different application for our lineage use case.

To build the integrated view we are planning to ingest brief lineage into Collibra.
Q) May i know what is the relations I should be creating for lineage for table to table and column to column.
Q) If in-case we don’t the relation already existing, can we create the relations with the same name or do we need to create with specific details such as UUID.
Q) Hope we don’t need to additional license to visualize the lineage harvested else where and ingested into Collibra.

Appreciate any comments on this topic.

1

Repository for lineage harvester

Lineage understanding

Lineage from file to table

Dependent Sources for Technical Lineage for SQL Server

Informatica PowerCenter Technical Lineage using EDGe