AWS Lake Formation Bidirectional Colibra Integration
HI Team
We have recently introduced “AWS Bidirectional Collibra Integration” and configured everything as per the documentation and updated “aplication.properties” file to reflect the correct Domain ID, Domain and community Name. While Integration from Collibra to AWS shows no issues but Sync from AWS to Collibra is failing with the below error:
org.springframework.security.web.session.SessionManagementFilter@6d24ffa1, org.springframework.security.web.access.ExceptionTranslationFilter@74cadd41, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@6ca0256d]
2022-11-22 15:57:43,383 [main] INFO org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 14 endpoint(s) beneath base path ‘/actuator’
2022-11-22 15:57:43,434 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler [“http-nio-0.0.0.0-8080”]
2022-11-22 15:57:43,456 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path ‘’
2022-11-22 15:57:43,489 [main] INFO com.collibra.aws.lakeformation.Application - Started Application in 10.42 seconds (JVM running for 11.96)
2022-11-22 15:57:43,493 [scheduling-1] INFO com.collibra.aws.lakeformation.scheduler.ScheduledSyncProcess - Job Running:Sync From Aws
2022-11-22 15:57:43,500 [scheduling-1] INFO com.collibra.aws.lakeformation.service.IntegrationProcessorService - Start Integration From Collibra To Aws
2022-11-22 15:57:43,811 [scheduling-1] INFO com.collibra.aws.lakeformation.service.IntegrationProcessorService - Integration From Collibra To Aws Completed
2022-11-22 15:57:43,811 [scheduling-1] INFO com.collibra.aws.lakeformation.service.IntegrationProcessorService - Sync From Aws to Collibra
2022-11-22 15:59:22,494 [scheduling-1] INFO com.collibra.aws.lakeformation.service.IntegrationProcessorService - Integration Processor Start Integration
2022-11-22 15:59:28,565 [scheduling-1] ERROR com.collibra.aws.lakeformation.scheduler.ScheduledSyncProcess - Sync Scheduler Error {“message”:“The Collibra Platform job did not complete successfully.”,“failingObjects”:[],“collibraPlatformApiError”:"{“id”:“6669444e-6b94-416a-8921-cc7bc2876292”,“createdBy”:“4152cbda-1670-4cf8-8c18-68a313ba516b”,“createdOn”:1669132765467,“lastModifiedBy”:“4152cbda-1670-4cf8-8c18-68a313ba516b”,“lastModifiedOn”:1669132767690,“system”:false,“resourceType”:“Job”,“name”:“Import”,“type”:“IMPORT”,“userId”:“4152cbda-1670-4cf8-8c18-68a313ba516b”,“visibility”:0,“progressPercentage”:100,“cancelable”:true,“startDate”:1669132766113,“endDate”:1669132767689,“state”:“ERROR”,“message”:"{\“conflicts\”:{\“Line 1 contains the following errors:\”:[\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”],\“Line 3 contains the following errors:\”:[\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”],\“Line 4 contains the following errors:\”:[\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”,\“No attribute type matches the specified criteria.\”],
The same error message keeps on repeating. Please have a look and let me know if extra details are required.
springboot_team
·3 years ago · EditedHi @mohammad.sufiyan.ansari,
Thanks for the update.
To determine what the issue might be, can you please do the following?
REGION("Region", "11a20804-f651-40f4-a00f-80a108409b99"),), copy it (e.g.11a20804-f651-40f4-a00f-80a108409b99) and search whether it exists in the Resource ID column (from step 3)In case there is a/are missing attribute type/s, can you please search whether their name (e.g.
Region) exist in the Collibra > Attributes > Name column? Thanksmohammadsufiyan_ansari
OP3 years ago · EditedLatest update
We used the CMA file to import the Objects and used their Corresponding Resource Ids for Assets, AttributeTypes and Relationships. Still getting the same error.
mohammadsufiyan_ansari
OP3 years ago · EditedHI @spring-team.collibra.com , can you please help here?
springboot_team
·3 years ago · EditedHi @mohammad.sufiyan.ansari,
This issue happens when the integration uses an attribute type that is not found on your Collibra instance. If you have used the CMA file, then the file should have imported this attribute. However, the new attribute is not created if your Collibra instance contains a previous attribute with the same name. Since the integration uses attribute IDs, there will be a mismatch.
Can you open the following file inside the source code under src/main/java/com/collibra/aws/lakeformation/util/ CustomConstants.java and check the IDs under the AttributeType enum all exist in your Collibra instance by going to the settings>attributes page, please.
Thanks
mohammadsufiyan_ansari
OP3 years ago · EditedHi @spring-team.collibra.com
Thanks so much for your response. We have not used CMA file for the import of the attributes which explains that the IDs were not present in the Collibra instance. As a workaround, I created the attribute types in Collibra which were there in the AttributeType enum list in the file CustomConstants.java. I updated the CustomConstants.java (After having a backup of this file) file with the new IDs generated for the newly created Attribute Types. I ran the integration and its still failing with the same message. Is there other locations as well where these Attribute IDs are being looked up? Any pointers will be highly appreciated.