Unity Catalog to Collibra integration – metadata and lineage tracking | Product Q&A

Unity Catalog to Collibra integration – metadata and lineage tracking

Hi team,

I would like to get metadata and lineage from Unity Catalog into Collibra and was just curious if anyone here has made this integration to work ( I have the documentation and the package template from Collibra).

Would like to understand more as I’m having difficulties of understanding the different components and how they work and are interrelated to each other and for it to work with our intended cases.

hafizamzar_hussain

Posted 3 years ago · Edited 1 year ago·Last reply 3 years ago

23 comments

anieissac

·3 years ago · Edited

hi @antonio.castelo/@mehjabeen.sultana.teck.com,
Can you please advise if you have step by step details of the integration? I find the document is bit hard to follow and it doesnt mention prerequisites for running the integration
regards
Anie

Antonio Castelo

·3 years ago · Edited

Hi, can we meet sometime next week? my email address: [email protected]. I’m in CEST…

Mehjabeen Sultana

·3 years ago · Edited

I am able to get the swagger page.
Can 3pm EST / 9pm CEST tomorrow work? and if you can share your email, I can send the invite. Thanks

Mehjabeen Sultana

·3 years ago · Edited

unityRunCMDLog.txt (24.7 KB)
Hi Antonio, I am also trying to deploy the jar. it gives me “internal server error”. I have attached the log.
the Springboot2.6.6 is included in the maven dependencies. Can you please share any intuitions on this error. THANKS.

Antonio Castelo

·3 years ago · Edited

the log doesnt tell me much. Possibly you have the app set to refresh every 5 secs and you have issues in accept status which are missing attributes… Check all your register unity catalog resources issues and move all ‘Accepted’ issues to something else. The app takes any issues in accepted status and processes that… you may have some issues with missing attributes or something…

If not that then we need to make sure you’re using the right version…

Antonio Castelo

·3 years ago · Edited

do we have a better way to communicate?

Mehjabeen Sultana

·3 years ago · Edited

Hi Antonio, I dont have any assets from unity in Collibra at the moment. I am trying to establish the integration and test any ingestion.UnityRunAsJavaApplication.txt (17.8 KB)
I tried to run as JavaApplication from eclipse and have attached the logs if that can be helpful.
I am happy to get on a working session with you. Thanks

Antonio Castelo

·3 years ago · Edited

can we get on a zoom call or something? it would be faster. I’m in CEST but could find time… Meanwhile, change the application.properties file, comment or delete the scheduler property so the app doesn’t poll work every 5s. If you have it all setup including the cma, you should have a simple workflow used for test and demo purposes called somewhat like “Register Unity Catalog Resources” in the Global create (the big + sign). Give it a go. Will ask you where you want to keep the metastore, catalog and the remaining schema assets. Choose a domain. Then give it a go. See if it shows you the catalogs. BTW… you need to make sure the workflow configuration points to the right url… here’s mine as example

After you chose the catalog it will present the available schemas, chose one and submit. It will create an issue you can find in the helpdesk app. Once at this point you need to tell the app to refresh… you can do it using the swagger ui that goes with the app. The doc tells you where (/swagger-ui.html). Here’s mine Don’t worry about the json doc. Leave it as is and run… The app should find the issue and do what it needs to be done… Let me know how far this goes for you

Mehjabeen Sultana

·3 years ago · Edited

Thank you Antonio. I was able to import the cma. Appreciate your help.
Can you please share the column changes that need to be done manually?
Do we have any documentation for knowing the workflows in detail?

Antonio Castelo

·3 years ago · Edited

the changes are listed in the document… all you need is a few additional attributes. Make sure the names are correct.

something like this. Check the document for details.

Mehjabeen Sultana

·3 years ago · Edited

looks like the problem is with complex relation - field mapping and foreign key mapping. attaching the import xls and logs. thankssbi-template-unitycatalog-2.0.2.xlsx (33.3 KB) dgc.txt (27.1 MB)

Antonio Castelo

·3 years ago · Edited

this is it. So… simplest way is to create a new cma file without the Column asset, import everything else and add the column changes manually. I’ve seen you have version 2.0.2… Let me try recreate the 2.0.2 without the column asset and get back to you. Any way I share the new cma file with you if it fails to upload here?

Antonio Castelo

·3 years ago · Edited

how can i share the cma file with you? let me know if you can reach it here https://drive.google.com/file/d/1kItDmk97CfrMerfL1xODi0eLok8rS1yi/view?usp=sharing

Mehjabeen Sultana

·3 years ago · Edited

Hi @spring-team.collibra.com, can you please help to fix the permission issue - systemResourceNoPermissionUpdate ?

Antonio Castelo

·3 years ago · Edited

ok… so this is apprently more common than expected… For some reason there are a few instances out there that have some attributes set as system attributes for some reason. The export mecanism that builds the cma file pushes all the asset assignments including those which have not changed and creates a checksum that invalidates the cma file if changed… System attributes cannot be changed. This makes things harder. We need to understand which is the conflicting asset, remove it from the cma file and have that asset changes done manually. Meanwhile, we are already looking at ways to fix the export mecanism and change the cma file so we exclude those violations… Can you tell what is the conflicting asset?

Mehjabeen Sultana

·3 years ago · Edited

Hi @antonio.castelo , thank you for sharing the insights.
there are 6 workflows and system configurations that will apply when we ingest data from Unity. There is a cma file included in the package and when I try to import that into DGC at my end, it gives me the error message :com.collibra.common.api.exception.ApiAuthorizationException: com.collibra.common.exception.AuthorizationException: systemResourceNoPermissionUpdate

I am not sure of what the permission and on what area of DGC I need update permission

Best regards,
Mehjabeen

Mehjabeen Sultana

·3 years ago · Edited

Hello,

While trying to import the cma file for the configurations, it fails with below error message.

2022-10-13 11:48:06.020 [pool-53-thread-1] ERROR c.c.d.m.a.i.c.impl.MigrationApiImpl - error during Migration [trace_id=e9b84d4e5d65e9c3be3b399da3d5b404, trace_flags=01, span_id=8d1d589a31630f43]

com.collibra.common.api.exception.ApiAuthorizationException: com.collibra.common.exception.AuthorizationException: systemResourceNoPermissionUpdate

at com.collibra.dgc.core.api.ApiExceptionTranslationHelper.translateToApiException(ApiExceptionTranslationHelper.java:70)

at com.collibra.dgc.core.api.aspect.ApiExceptionHandlingAspect.mapExceptions(ApiExceptionHandlingAspect.java:39)

at jdk.internal.reflect.GeneratedMethodAccessor559.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.base/java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)

at com.sun.proxy.$Proxy751.changeComplexRelationType(Unknown Source)

I have system admin access. I have tried to create a scope and import and still same error. I want to know the root cause of the error message.

What permissions do I need for this to work?

Best regards,
Mehjabeen

Antonio Castelo

·3 years ago · Edited

Yes @anie.issac . The app uses unity catalog’s lineage tracking api to collect table and column lineage and produce a json flies per schemas. With the distribution we provide a simple python script that takes all those files and produces a single json file with all lineage found. This json file is then pushed to our technical lineage harvester who will do the rest.

anieissac

·3 years ago · Edited

hi @antonio.castelo,
can you please confirm whether the integration is able to export metadata and lineage from unity catalog? I’m looking for column level lineage for tables in databricks. can you confirm if its possible to achieve?
regards
Anie

hafizamzar_hussain

OP3 years ago · Edited

Hi @antonio.castelo, appreciate the reply and thanks @kristen.freer for helping us to connect.

Our team has quite a limited understanding of this integration:- how it works, how to set it up etc for our use case.

As I understand, there are a few things that needed to be setup that includes the workflow and the integration package. On the workflow, we are quite unsure of the workflow definitions need to be setup and on the integration package we are having issue when trying to package it using our own configurations based on the given template.

Are there any step by step resources that we can refer to, apart from the technical documentation to troubleshoot this? And it would be great if we could connect over a call on this if possible.

Antonio Castelo

·3 years ago · Edited

Hi… Briefly, all you need is the jar and the application.properties file. Unless you want to change and compile something, once you have configured the application properties with both the collibra and the databricks credentials you are good to go. All you need is your collibra username and password and your databricks personal access token.

For example:

java -jar sbi-template-unitycatalog-2.0.2-SNAPSHOT.jar —spring.config.location=./application.properties --add-opens=java.base/java.nio=ALL-UNNAMED

The document also explains how to encrypt the credentials with jasypt so they are not left in the clear in the file.

Antonio Castelo

·3 years ago · Edited

Hi @hafhussain.deloitte.com , yes., this is working. We have a few customers already using this some of which they managed to configure and run by themselves. What exact difficulties are you having?

Alexandra Jorgenson

Admin3 years ago · Edited

Unity Catalog to Collibra integration - metadata and lineage tracking -...

Creates, updates and represents existing Unity Catalog metastore, catalog, schema, table and column resources in Collibra.

@spring-team.collibra.com, can you possibly help with this?