Collibra to redshift connector

Hi, Data citizens,

Today I have an interesting discussion with my colleague. Currently, alongside other connectors, we are using the connector to ingest metadata from Redshift to Collibra. And we were thinking if it is possible to do it vice-versa and have a connection from Collibra to Redshift. I was searching through Collibra documentation and I haven’t found any info related to a use case like this.
Does anybody have experience with it or give it a try to build such a connector?

Thank you

jakubjaniga

Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago

3 comments

Paulo Taylor

·4 years ago · Edited

Hi @jakub.janiga,

The Redshift connector is fully technically supported by Collibra. Please feel free to create a suppport ticket if you’re still blocked.

Best regards,
Paulo

arthurburkhardt

·4 years ago · Edited

@jakub.janiga

The official solution is the Collibra Insights: it’s a star-schema optimized for analysis, but users have complained about its incompleteness and I believe a redesign is underway.
You can create a connector based on the outputmodule, that would allow you to extract deltas from collibra and pipe into redshift on a regular (e.g. daily) basis.

Quite comprehensive, but documentation is not 100% and it can be difficult to write queries. e.g. you can query all attributes of java objects, but they are not all documented. e.g. users.lastLogin.
Make sure to implement pagination to avoid platform instability.
Consistency is not guaranteed (every page is a new transaction, state can change between transactions) => You need to account for those in your pipeline. e.g. if you fetch attributes then assets, you might fetch attributes related to assets that have been deleted in between calls, leading to an inconsistent state.

You can download the daily repo backup, restore into a postgres server, and query the data directly from the replicated collibra postgres database.

Alexandra Jorgenson

Admin4 years ago · Edited

@spring-team.collibra.com Is this something you could chime in on or no?