Azure Purview to Collibra Integration - cma file installation not working

@spring-team.collibra.com, is this something you can help @indranil.chakraborty1.metlife.com with?

Hello @indranil.chakraborty1.metlife.com,

Thanks for pointing this out.

We are currently looking into this and will keep you updated.

Hello @indranil.chakraborty1.metlife.com,

The Purview to Collibra integration is highly customisable. It is up to the user to pick and choose what kind of entities should be synced from Purview to Collibra and down to which level. Therefore, it makes more sense for users to choose themselves the asset/attribute/relation types they need (OOTB or otherwise), rather than having one CMA file containing many custom asset/attribute/relation types which might not be useful for most users. Having said that, certain OOTB asset/attribute/relation types are pre-set in the properties file by default, though they can still be configured by the user.

Regarding the error when importing the CMA file, can you please try creating the following attributes from the Collibra instance, settings page:

• Size (collibra.attribute.size)
• Version (collibra.attribute.version)
• Nickname (collibra.attribute.nickname)

Additionally, can you please also try creating the following relation (in case it does not already exist):

• Power BI capacity Workspace (collibra.relation.powerbi.capacity.workspace)

Finally, can you please update the respective properties from the Purview integration, application.properties file? Thanks

As a side note, just to let you know that these are the resources that are created by the CMA file. The Purview integration also supports other properties that are by default commented out in the properties file.

@spring-team.collibra.com - Thank you team for explaining the information in detailed. The Azure Purview integration is started working fine. After importing some metadata from Purview to Collibra, i could see for the tags the below error is coming up with multiple runs. Why it doesn’t give consistent behavior? Any suggestion to fix the below issues:

C:\Users\ICHAKR~1\AppData\Local\Temp\1\loadFile7228438158363517241
java.lang.RuntimeException: {“type”:“MESSAGE”,“message”:“An asset cant have the same tags assigned twice.”}
at com.collibra.marketplace.azure.purview.util.CollibraImportUtil.checkJobStatus(CollibraImportUtil.java:231)
at com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent.genericImportCall(CollibraGenericImportComponent.java:123)

Another error:

C:\Users\ICHAKR~1\AppData\Local\Temp\1\loadFile4659971217203693464
java.lang.RuntimeException: {“type”:“MESSAGE”,“message”:“An asset with signifier ‘mssql://xxxxxx:xxxx/MSSQLSERVER/DMSKSQLT/dbo/tbaddressdev’ already exists for domain ‘MSSQL Table’.”}
at com.collibra.marketplace.azure.purview.util.CollibraImportUtil.checkJobStatus(CollibraImportUtil.java:231)
at com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent.genericImportCall(CollibraGenericImportComponent.java:123)
at com.collibra.marketplace.azure.purview.FullSyncProcessor.start(FullSyncProcessor.java:1084)
at com.collibra.marketplace.azure.purview.controller.EntryPointController.fullSyncTriggeredByApiRequest(EntryPointController.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Hello @indranil.chakraborty1.metlife.com,

Thanks for the update. Will check regarding the encountered import errors and get back to you.

@spring-team.collibra.com - it looks to me duplicated assets are coming up in the import json which is causing this issue.Do you have any insights to fix the issue please

@spring-team.collibra.com - looks like batch size parameter is being used in the import api which is deprecated in the latest DGC version 2022.06 . And i think it is causing the duplicated assets in the import json . Is there any way to fix it?

Hello @indranil.chakraborty1.metlife.com,

Thanks for the update.

Can you please (as a private message) share the latest loadFile that the integration created in a temporary folder just before it failed?

The path of this loadFile can be retrieved from the last log message starting with the message “Temporary Import Load File located at ::”. and it contains the Collibra Import API payload that is failing. Thanks

@spring-team.collibra.com - please check the above temp json file which is causing the issue

@spring-team.collibra.com - do you have any update on this?duplicate assets are coming up in the import files which is causing the issue.

@kristen.freer or @spring-team.collibra.com - I have responded to you via email . After replacing the source code , compiling and running, i am still getting the duplicate asset error in the import json file.

Regards,
Indranil

@kristen.free or @spring-team.collibra.com - I have replied to you on your latest email. Please let me know in case you didn’t receive it.

@spring-team.collibra.com - any update on the mentioned issue please?

Yes @kristen.freer. I have sent the details to @james.scicluna through email. Waiting for the response.