Collibra Community

ETL Integrations

Created Nov 18, 2023

0 members

4 discussions

The ETL Integrations topic in the Collibra Community is focused on connecting Collibra with ETL tools like Talend, Snowflake, and ADF. It’s a space where users troubleshoot setup challenges, share insights on technical lineage, and explore integration best practices—especially for automating data flow visibility within the Collibra platform.

pramodanachimath
3 months ago
Challenges with Building ADF Technical Lineage due to Collibra Inability to Access AKV-Stored Credentials
We are working on build the ADF technical lineage and have created the necessary connections, capabilities and scanned the technical lineage. However, the process is failing to build the lineage due to an issue with the data sources in the pipeline.

The connection string details are stored in the Azure Key Vault (AKV), and Collibra is unable to retrieve this information, which is preventing it from identifying the correct server and database names.

As per our organization's policy, we store credentials in the AKV, and currently, Collibra does not have the capability to read date from the AKV.

Has anyone encountered a similar issue and found a solution to proceed with this?

1
andrewsouthcombe
8 months ago
dbt incremental models with the Snowflake technical lineage connector
We have recently deployed the snowflake technical lineage capability via Edge. We're using a JDBC connection in SQL-API mode. We have a number of teams using dbt for transformation and we've noticed that technical lineage doesn't always work for incremental models. I'm not sure but this mainly seems to affect dbt versions > 1.1. I can't see any obvious differences in the SQL generated so I wondered if anyone else had come across this issue?

I'm aware of the specific dbt connector but this will be really onerous for us to deploy as we have quite a decentralised org structure so I'm hoping to get the snowflake lineage connector working in as many use cases as possible.

1
manojkumar_bushigampala
10 months ago
talend etl integration
Hello,

I was working on the Talend ETL Integration and successfully able to establish the connection.

When I trigger the integration after dumping the .xml files at the folder and provided the path to the folder at

"trigger.file.observer.directory.path=" with in the application.properties file and the spring boot integration works but not reading the data from the .xml files and not uploading into collibra.

I dont see any error aswell.

Could someone help me on this.

Thanks,

Manoj.

0
manojkumar_bushigampala
1 year ago
Talend ETL to Collibra Integration v1.0.8 (Spring Boot)
Hello @spring-team.collibra.com
I am following the documentation which was in the Collibra marketplace for the installations for Talend ETL to Collibra Integration. Do i need to create separate VM for the installations or Can do it in my local machine.

Thank you,

0

Challenges with Building ADF Technical Lineage due to Collibra Inability to Access AKV-Stored Credentials

dbt incremental models with the Snowflake technical lineage connector

talend etl integration

Talend ETL to Collibra Integration v1.0.8 (Spring Boot)

Challenges with Building ADF Technical Lineage due to Collibra Inability to Access AKV-Stored Credentials

dbt incremental models with the Snowflake technical lineage connector

talend etl integration

Talend ETL to Collibra Integration v1.0.8 (Spring Boot)