Azure Purview to Collibra Integration - cma file import error | Integrations

Azure Purview to Collibra Integration - cma file import error

Hi Team,

The .cma file provided as part of this integration (https://marketplace.collibra.com/listings/azure-purview-to-collibra-integration/)throwed an error during import.
Can you please help me to solve this issue?

Thanks

shanthisubramanian

Posted 3 years ago · Edited 1 year ago·Last reply 3 years ago

17 comments

shanthisubramanian

OP3 years ago · Edited

Hi Thank you,

I am trying to ingest metadata (schemas, tables, columns from SQL Server using SpringBoot).Schemas and tables were ingested.But i cant able to ingest colums. The code throwed “errorMessage”: “RuntimeException: {“type”:“MESSAGE”,“message”:“You are not allowed to add, edit or remove an attribute whose type is read only.”}”.} Is that the reason for missing columns?. Can you please help me to fix this?
Thanks

springboot_team

·3 years ago · Edited

Hi @shanthi.subramanian.transport.nsw.gov.au,

The value that you need to check is located in the src/main/resources/application.properties.

In this file, there should be a property that has the following key: purview.rest.api.base.path.

Thanks

shanthisubramanian

OP3 years ago · Edited

Hi @spring-team.collibra.com,

Now i am getting the "404 not found " error.Here is the error message,

2022-12-14 14:56:03,117 [https-jsse-nio-8443-exec-7] INFO com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Preparing and posting to Purview advanced search for azure_sql_server starting at 0 and limit to 1000 with:
{filter={and=[{entityType=azure_sql_server, includeSubTypes=false}]}, keywords=*, offset=0, limit=1000}
2022-12-14 14:56:03,292 [https-jsse-nio-8443-exec-7] ERROR com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - 404 Not Found: [no body]

—Which means the server is not able to find the requested resource.Even when i trigger manually via Postman api by providing the guid,i received the same error.Am i missing something?Can you please advise?Below are the REST API connection details.I am suspecting the host name.

purview.rest.api.host=####-####-####.purview.azure.com
purview.rest.api.port=443
purview.rest.api.base.path=/api/atlas/v2

When i check the endpoint details via azure purview properties,i found the endpoint ended with /catalog

which means ####-####-####.purview.azure.com/catalog.

Is that something i have to look ?

Thanks

springboot_team

·3 years ago · Edited

Hi @shanthi.subramanian.transport.nsw.gov.au,

We don’t think the issue is with the host name because the request was received by the server (otherwise, you wouldn’t have received a 404 response). But the issue could be due to an incorrectly set base path.

The advanced search Purview API URL is constructed by the integration as follows:

https://${purview.rest.api.host}:${purview.rest.api.port}${purview.rest.api.base.path}/search/advanced

The pre-populated value of purview.rest.api.base.path is /api/atlas/v2, because the integration uses the Apache Atlas based API of Purview (as opposed to Purview’s REST API which uses /catalog…)

shanthisubramanian

OP3 years ago · Edited

Thanks so much. This is very helpful.

Can you please help me to solve this issue?Do i have to modify the code or the application.properties REST API details?

Thanks,

shanthisubramanian

OP3 years ago · Edited

Thank you.I will check

springboot_team

·3 years ago · Edited

Hi @shanthi.subramanian.transport.nsw.gov.au

The “You are not allowed to perform this action on ‘Azure Datalake Gen2 Resource Set” error message means that the integration wasn’t able to perform an operation on the domain “Azure Datalake Gen2 Resource Set”.

Therefore, you need to give the Collibra user used by the integration all the necessary rights to perform any operation on the domain “Azure Datalake Gen2 Resource Set” (as well as all other domains used by the integration, which are all contained in the community collibra.azure.purview.community). In particular, this Collibra user needs to be able to create and delete any domain within the community collibra.azure.purview.community as well as import assets in any of these domains.

To do so, you need to add the necessary responsibility/ies on the community collibra.azure.purview.community for the Collibra user used by the integration.

shanthisubramanian

OP3 years ago · Edited

Hi Team,

I dont have a permission to access Azure Data Lake.I am getting below error.Can you please guide me how to disable the one i dont have a permission?

java.lang.RuntimeException: {“type”:“MESSAGE”,“message”:“You are not allowed to perform this action on ‘Azure Datalake Gen2 Resource Set’.”}

Thanks,

shanthisubramanian

OP3 years ago · Edited

Hi Team,

Sure,I will send

Thanks

shanthisubramanian

OP3 years ago · Edited

Thank you, suddenly i am receiving below error.

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
I checked the keystore.jks file .Its all good. I dont know why did i get this suddenly.

I tried disabling the server.ssl.enabled property to false and i used http via postman. Still i am receiving the same error.

Thanks,

springboot_team

·3 years ago · Edited

Hi @shanthi.subramanian.transport.nsw.gov.au,

Did you re-deploy the application after setting server.ssl.enabled to false? You need to do so for the change to take effect. It is not possible to get a certificate error when triggering the application via HTTP (as opposed to HTTPS) with server.ssl.enabled set to false. Or maybe this error is referring to another certificate (Collibra’s or Purview’s).

Can you please provide us (via a direct private message) with your full log messages (removing any sensitive data you might have in the logs)? This way we will be able to determine what is failing and why this is happening.

Thanks.

shanthisubramanian

OP3 years ago · Edited

Thank you.

I got a below error:

com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - 403 Forbidden: “{“error”:{“code”:“Unauthorized”,“message”:“Not authorized to access account”}}”

Is read only access enough to access the account?

Thanks,

springboot_team

·3 years ago · Edited

Hi @shanthi.subramanian.transport.nsw.gov.au,

Yes, read access is enough for the type of Purview API requests performed by the integration.

The integration performs only these two kinds of API requests:

POST /search/advanced - searches for Purview entities based on a search query request body. Only retrieves limited details on these entities.
GET /entity/bulk - retrieves details of Purview entities identified by their GUIDs, which are passed as query parameters on this request.

shanthisubramanian

OP3 years ago · Edited

Hi,
Thanks for the reply. I did comment out these three attributes.
I did not upload the cma file. Do i still have to create any assets/attributes/relations manually? I manually created the community and scope.
1)When i was trying to create a build via mvn command, I received the below error
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder ‘collibra.asset.file.schema’ in value “${collibra.asset.file.schema}”.

2)When i deployed through eclipse IDE, the JVM is running. So i was trying to trigger the integration via postman API, but i got the "Client Credential null "error.

Can you help me to fix this?

Thanks,

springboot_team

·3 years ago · Edited

Hi @shanthi.subramanian.transport.nsw.gov.au

You don’t need to create any assets/attributes/relations manually. Though any attribute you remove/comment out from the properties file will not be imported in Collibra. You can also remove/comment out assets and relations from the properties file, as long as the corresponding Purview entities are not synced (i.e. their upsert.level properties are set to NONE).

Re. Question 1): Since you are syncing File Schema assets, you need to set an asset type in the property collibra.asset.file.schema (this is not set by default). You can set any asset type as long as its root parent asset type is “Schema”. So, you can set this to “Schema” itself or any custom asset type under “Schema”.

Re Question 2) The API is secured via Basic Authentication, meaning that a Username and Password need to be provided to trigger the integration via API. The username and password are set in properties trigger.api.username and trigger.api.password. You can set these in Postman in the Authorization tab as explained here: https://learning.postman.com/docs/sending-requests/authorization/#basic-auth

springboot_team

·3 years ago · Edited

Hi @shanthi.subramanian.transport.nsw.gov.au ,

This CMA file was create more than a year ago. We can confirm that it is now not compatible with the latest version of Collibra.

Instead of using this CMA file, you can manually create the 3 custom attributes contained in this CMA file (which are pre-set in the application.properties file) and change the IDs of these attributes in the application.properties file. The 3 attributes are:

collibra.attribute.size (Kind: Number; Integer)
collibra.attribute.version (Kind: Text)
collibra.attribute.nickname (Kind: Text)

Alternatively, if you’re not interesting in mapping these 3 Purview attributes into Collibra, you can comment out these 3 properties,. By doing so, you won’t need to create any custom attributes at all.

Note that the integration supports many optional attributes, assets and relations. By default, these are all disabled since their properties are all commented out in the application.properties file. You can uncomment the ones you’d like to use and set their ID (you can use either OOTB or custom ).

Alexandra Jorgenson

Admin3 years ago · Edited

@spring-team.collibra.com, is this something you could help @shanthi.subramanian.transport.nsw.gov.au with?