Fivetran to Collibra Integration (Python)

Hi,

Currently our organisation our using Collibra and we want to ingest our Fivetran connection data into Collibra using the “Fivetran to Collibra Integration (Python)” script from the Collibra market place. However, we have a hard to time to ingest the information into the correct place in Collibra. When running the ingestion script, we get an error when putting in the “–dest_sys_uuid” and we are unsure what dest_sys_uuid we should use. Would like some help and clarification about this.

andazhang

Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago

10 comments

ralphosmond_rimorin

·4 years ago · Edited

That is too early for me. How about 9am EST or 3PM CET?

andazhang

OP4 years ago · Edited

9am EST works perfectly. Let me invite you

/Anda

andazhang

OP4 years ago · Edited

Hi Ralph,

Thank you ! I will check with our Data Gov Team once they are back from vacation.
Have you managed to check if this is some incompatibility issue?

/Anda

ralphosmond_rimorin

·4 years ago · Edited

Hi Anda,

I believe it should still work… Would you be open to a call to review your configurations and mock data?

Ralph

andazhang

OP4 years ago · Edited

Hi Ralph,

Yes, that would be great. Are you able to join on a call 9:00 - 10:00 tomorrow CET?

/Anda

ralphosmond_rimorin

·4 years ago · Edited

Hi @anda.zhang.eqtpartners.com,

Thanks for reaching out. The parameter –dest_sys_uuid is the ID of the System asset linked to the Fivetran destination database catalogued using the JobServer or Edge. Basically you need to catalog first the target or destination database and then link it a System Asset, which is created automatically with Edge. So make sure you have the following assets System -> Database -> Schema -> Table -> Column for the destination database and then use the System asset id as -dest_sys_uuid.

cc: @kristen.freer @antonio.castelo

andazhang

OP4 years ago · Edited

Hi Ralph,

First of all thank you for your message. I am quite new to Collibra so would need some further clarifications.

What does it mean to “catalog first the target or destination database” and then link it to a System asset? And with Edge, do you mean the web browser?

From what I see, we have created mock assets like you described: System -> Database -> Schema -> Table -> Column. What when I run the code I get the following error:

Traceback (most recent call last):
  File "/Users/anda.zhang/Downloads/pyi-template-fivetran-2.0.0/fivetran_collibra_ingestion_job.py", line 44, in <module>
    fivetran_to_collibra_facade.extract_fivetran_assets_then_import_to_collibra_by_destination()
  File "/Users/anda.zhang/Downloads/pyi-template-fivetran-2.0.0/tasks/fivetran_collibra_facade.py", line 565, in extract_fivetran_assets_then_import_to_collibra_by_destination
    self.build_destination_asset_import_command(importRequest, destination)
  File "/Users/anda.zhang/Downloads/pyi-template-fivetran-2.0.0/tasks/fivetran_collibra_facade.py", line 266, in build_destination_asset_import_command
    destination_system_assets = self.query_system_assets(system_uuid=self.destination_system_uuid, system_location=destination['config']['host'], db_name=destination['config']['database'], schema_name='FOR_QUERY_SCHEMA_EXPECT_TO_RETURN_EMPTY')
KeyError: 'host'

It seems like, it can not find the sys_uuid.

ralphosmond_rimorin

·4 years ago · Edited

Hi Anda,

About the Mock assets, you cannot just mock the data unless you are familiar with the asset identifiers and relations ingested with the Catalog feature of Collibra. All of these follows some naming convention of the identifiers for example for the Column, it will have a full name format something like (SystemName > SchemaName > TableName > ColumnName). Please reach out to the data team (or Data Governance team within your company) that is working with Collibra or the CSM (Customer Success Manager) assigned to your EQT Partners to get some help about Edge or Collibra Catalog.

I am seeing this error, looks like something related to the parsing of data being returned by the Fivetran API Destination endpoint. I will run a test on Monday and see if there are some incompatibility issue between the recent Fivetran API and this code.

KeyError: ‘host’

Thanks!

Alexandra Jorgenson

Admin4 years ago · Edited

@spring-team.collibra.com, is this something you could help @anda.zhang.eqtpartners.com with? Also, which Integrations forum should I move this post to?

Antonio Castelo

·4 years ago · Edited

Hi Kristen, yes. We can help with that. @ralph.rimorin.collibra.com has built the integration. He should be able to help. Let me reach out and get back to you.