Azure Purview to Collibra Integration - cma file installation not working
Hi Team,
The .cma file provided as part of this integration(https://marketplace.collibra.com/listings/azure-purview-to-collibra-integration/#) are not working during installation on DGC version 2022.06 as well as 2021.09.
Can you please check and let us know with the resolution
indranilchakraborty
Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago
24 comments
indranilchakraborty
OP4 years ago · Edited@spring-team.collibra.com @kristen.freer - can we extend the functionality to include azure synpase to harvest metadata and classification information. Can we also harvest lineage part of purview to collibra?is that something in your roadmap?
Regards,
Indranil
springboot_team
·4 years ago · EditedHi @indranil.chakraborty1.metlife.com,
The extended functionality you requested is currently not on our roadmap.
However, you can post feature requests on the ideation portal (https://productresources.collibra.com/ideation-platform/), which will then be considered for our roadmap. You can also ask your CSM (Customer Success Manager) to push forward your feature requests internally.
Thanks.
indranilchakraborty
OP4 years ago · EditedThe hotfix is working fine. Thanks to @spring-team.collibra.com.
indranilchakraborty
OP4 years ago · EditedYes @kristen.freer. I have sent the details to @james.scicluna through email. Waiting for the response.
springboot_team
·4 years ago · EditedHi @indranil.chakraborty1.metlife.com
Thanks for sending the details. Responded again via a private message. Please let us know if the new fix solved the issue you are facing.
Thanks
springboot_team
·4 years ago · Edited@indranil.chakraborty1.metlife.com confirmed that the issue has been resolved.
Cause:
The duplicate assets issue was caused due to Purview’s API paged search request returning the same entities on different pages. For example, the following two search queries on the POST /search/advanced endpoint are both returning the same table in their result (even though the search pages defined by the offset and limit do not overlap):
{“filter”: {“and”: [ { “entityType”: “mssql_table”, “includeSubTypes”: false } ] }, “keywords”: “*”, “offset”: 50, “limit”: 50 }
{“filter”: {“and”: [ { “entityType”: “mssql_table”, “includeSubTypes”: false } ] }, “keywords”: “*”, “offset”: 100, “limit”: 50 }
Resolution:
A hot fix in the Purview Connector was implemented to make sure that any duplicate search results are removed (even though the Purview API should in principle not return duplicates on different pages).
integrationdev_rohit
·4 years ago · Edited@spring-team.collibra.com : Could you please share the HOT fix with me also I am also getting the same error.
springboot_team
·4 years ago · EditedHi @rohit.chandra.1,
Just shared with you via a private message the Purview Connector hotfix.
indranilchakraborty
OP4 years ago · Edited@spring-team.collibra.com - any update on the mentioned issue please?
Alexandra Jorgenson
Admin4 years ago · Edited@indranil.chakraborty1.metlife.com Did you check your Community DM as @james.scicluna from @spring-team.collibra.com mentioned that he private messaged you there.
indranilchakraborty
OP4 years ago · Edited@kristen.free or @spring-team.collibra.com - I have replied to you on your latest email. Please let me know in case you didn’t receive it.
indranilchakraborty
OP4 years ago · Edited@kristen.freer or @spring-team.collibra.com - I have responded to you via email . After replacing the source code , compiling and running, i am still getting the duplicate asset error in the import json file.
Regards,
Indranil
indranilchakraborty
OP4 years ago · Edited@spring-team.collibra.com - do you have any update on this?duplicate assets are coming up in the import files which is causing the issue.
springboot_team
·4 years ago · EditedHello @indranil.chakraborty1.metlife.com
Responded via a private message.
Thanks
indranilchakraborty
OP4 years ago · Edited@spring-team.collibra.com - please check the above temp json file which is causing the issue
indranilchakraborty
OP4 years ago · Editedsametaggedassignedtwice.txt (1.0 MB)
Summaryjava.lang.RuntimeException: {“type”:“MESSAGE”,“message”:“An asset cant have the same tags assigned twice.”}
springboot_team
·4 years ago · EditedHello @indranil.chakraborty1.metlife.com,
Thanks for the update.
Can you please (as a private message) share the latest loadFile that the integration created in a temporary folder just before it failed?
The path of this loadFile can be retrieved from the last log message starting with the message “Temporary Import Load File located at ::”. and it contains the Collibra Import API payload that is failing. Thanks
indranilchakraborty
OP4 years ago · Edited@spring-team.collibra.com - looks like batch size parameter is being used in the import api which is deprecated in the latest DGC version 2022.06 . And i think it is causing the duplicated assets in the import json . Is there any way to fix it?
indranilchakraborty
OP4 years ago · Edited@spring-team.collibra.com - it looks to me duplicated assets are coming up in the import json which is causing this issue.Do you have any insights to fix the issue please
springboot_team
·4 years ago · EditedHello @indranil.chakraborty1.metlife.com,
Thanks for the update. Will check regarding the encountered import errors and get back to you.
indranilchakraborty
OP4 years ago · Edited@spring-team.collibra.com - Thank you team for explaining the information in detailed. The Azure Purview integration is started working fine. After importing some metadata from Purview to Collibra, i could see for the tags the below error is coming up with multiple runs. Why it doesn’t give consistent behavior? Any suggestion to fix the below issues:
C:\Users\ICHAKR~1\AppData\Local\Temp\1\loadFile7228438158363517241
java.lang.RuntimeException: {“type”:“MESSAGE”,“message”:“An asset cant have the same tags assigned twice.”}
at com.collibra.marketplace.azure.purview.util.CollibraImportUtil.checkJobStatus(CollibraImportUtil.java:231)
at com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent.genericImportCall(CollibraGenericImportComponent.java:123)
Another error:
C:\Users\ICHAKR~1\AppData\Local\Temp\1\loadFile4659971217203693464
java.lang.RuntimeException: {“type”:“MESSAGE”,“message”:“An asset with signifier ‘mssql://xxxxxx:xxxx/MSSQLSERVER/DMSKSQLT/dbo/tbaddressdev’ already exists for domain ‘MSSQL Table’.”}
at com.collibra.marketplace.azure.purview.util.CollibraImportUtil.checkJobStatus(CollibraImportUtil.java:231)
at com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent.genericImportCall(CollibraGenericImportComponent.java:123)
at com.collibra.marketplace.azure.purview.FullSyncProcessor.start(FullSyncProcessor.java:1084)
at com.collibra.marketplace.azure.purview.controller.EntryPointController.fullSyncTriggeredByApiRequest(EntryPointController.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
springboot_team
·4 years ago · EditedHello @indranil.chakraborty1.metlife.com,
The Purview to Collibra integration is highly customisable. It is up to the user to pick and choose what kind of entities should be synced from Purview to Collibra and down to which level. Therefore, it makes more sense for users to choose themselves the asset/attribute/relation types they need (OOTB or otherwise), rather than having one CMA file containing many custom asset/attribute/relation types which might not be useful for most users. Having said that, certain OOTB asset/attribute/relation types are pre-set in the properties file by default, though they can still be configured by the user.
Regarding the error when importing the CMA file, can you please try creating the following attributes from the Collibra instance, settings page:
Additionally, can you please also try creating the following relation (in case it does not already exist):
Finally, can you please update the respective properties from the Purview integration, application.properties file? Thanks
As a side note, just to let you know that these are the resources that are created by the CMA file. The Purview integration also supports other properties that are by default commented out in the properties file.
springboot_team
·4 years ago · EditedHello @indranil.chakraborty1.metlife.com,
Thanks for pointing this out.
We are currently looking into this and will keep you updated.
Alexandra Jorgenson
Admin4 years ago · Edited@spring-team.collibra.com, is this something you can help @indranil.chakraborty1.metlife.com with?