38 Messages
Spring boot Integration to AWS Lake
Hi Team,
For AWS Lake to Collibra Bidirectional Integration
We have set up the Spring Boot Integration library with Collibra Integration Library and when trying to run the package we are getting the below error
com.collibra.marketplace.library.integration.exception.CollibraIntegrationLibraryException: {“message”:“There are one or more validation errors.”,“details”:“The Relation Key (:SOURCE) Asset ID ‘’ is not a valid UUID - it cannot be empty.”}
Can you please help us on this.
Community_Alex
678 Messages
•
14.4K Points
3 years ago
@spring-team.collibra.com, can you help with this error?
0
0
springboot_team
368 Messages
3 years ago
Hello @vani.nithya.indla
Thanks for trying the AWS Lake Formation to Collibra integration.
Regarding the error encountered, can you please confirm that the asset/attribute/relation types defined com.collibra.aws.lakeformation.util.CustomConstants enum were updated according to your Collibra instance resource IDs?
1
springboot_team
368 Messages
3 years ago
Hello @vani.nithya.indla,
Thanks for the update.
Regarding the assets in the Databases domain having obsolete status, the integration contains logic that marks assets that were not part of the last sync as obsolete.
Can you please confirm that the databases and their related information was included as part of the last sync?
1
0
vaninithya_indla
38 Messages
2 years ago
Hi @spring-team.collibra.com
Thanks a lot for your assistance
We are trying to sync assets from Collibra to AWS and when trying to do that we mapped below properties to operational domains
We have created a tag under “Create-update -delete-Tag” Operational domain and mapped the UUID to “Collibra.domain,tag-to resource” configuration property
we are getting the below error when running the integration
“ERROR 13916 — [ scheduling-1] c.c.a.l.scheduler.ScheduledSyncProcess : Sync Scheduler Error Invalid UUID string:”
Can you kindly help us on this ?
Also there are three operational domains but only two properties are mapping to operational domains , Please advise us on this
0
0
springboot_team
368 Messages
2 years ago
Hello @vani.nithya.indla
Thanks for the update.
Regarding the error encountered, can you please confirm that the ID of the respective domain is set for the following properties?
Meanwhile, regarding the documentation, just to let you know that will be updated accordingly, thanks.
1
0
springboot_team
368 Messages
2 years ago
Hello @vani.nithya.indla,
Thanks for the update.
Can you please provide the stack trace for the following error?
Additionally, can you please confirm that (1) all the domains IDs in the application.properties file and (2) the com.collibra.aws.lakeformation.util.CustomConstants IDs are set and exist on your Collibra Cloud instance?
Regarding the CustomConstants class, the ID of each asset type, attribute type and relation type should be defined. Example:
Also, just to note that a CMA file located in the src/main/resources/collibra_exports folder can be used to set up the required Collibra instance metamodel.
Moreover, can you please confirm that the AWS database is not empty? In case it is, you could also try disabling the AWS Data Catalog and AWS Glue services by setting the following property in the application.properties file:
1
0
springboot_team
368 Messages
2 years ago
Hi @vani.nithya.indla,
Thanks for your feedback, we will make sure to update the CMA file accordingly.
Also please note that the integration is creating a
and
Could you please confirm whether these relations are visible from the diagram view of the Database asset page?
Thanks
1
0
springboot_team
368 Messages
2 years ago
Hi @vani.nithya.indla,
Manually creating the AssignLFTag asset to perform an assign action can be a bit problematic since you have to replicate exactly the asset types/relations and attributes that the integration is expecting. That is why we are currently developing a set of workflows that help automate this process. I have attached the workflow with this message, please feel free to use it. They will be released with the next version of this integration.
To be able to use this workflow, import the whole zip file to the Collibra Cloud Settings page > Workflows > Definitions > Upload a file. Once uploaded, the workflow should be started and the UUIDs populated from the workflow page. Please note that the attached workflow is a pre-release version and might need some optimisations/further testing
Please rename the file to end with .zip after downloading.
Aws Workflow Selector-bar.zip (23.0 KB)
1
0
vaninithya_indla
38 Messages
2 years ago
Hi @spring-team.collibra.com
Thanks for sharing us the draft version of workflows.
After testing we have noticed some issues in the workflows. Please find below the feedback
Also in the assign LF-Tag sping boot code, the logic is written in a way that only column level tags are assigned, we cannot assign tags at database / table level alone.
Also , the next set of workflows on Grant and Revoke permissions, will there be any workflows released in next release ?
1
0
vaninithya_indla
38 Messages
2 years ago
Hi @spring-team.collibra.com
Thanks a lot for your update !
Can you please let us know if you have any tentative date of next release to plan our integration implementation activities?
Also, we have two questions on the overall Lake formation integration model
In the “users” domain , only IAM users are getting ingested , what about the IAM roles?
because LF-Tags can be assigned to a Role as well.
And whole Collibra to AWS integration is based on LF-Tag access control method. Is there any functionality available to provide access through named resource method?
Also can you share draft version of Grant Revoke workflows if available.
0
0
springboot_team
368 Messages
2 years ago
Hi @vani.nithya.indla ,
thanks for your questions and feedback. We will release the next version of the integration and the attached workflows in around two weeks. We will update you once the workflows have been released on the marketplace.
The two points you made are new features that the current integration version does not support. However, please feel free to post a feature request on the ideation portal or reach out to the Customer Success Manager and request these features.
Thanks
0
0
vaninithya_indla
38 Messages
2 years ago
Hi @spring-team.collibra.com ,
Thanks a lot for the update.
If you can share draft version of Grant revoke workflows, it would be great starting point for us to perform the testing before the release of main version.
We tried testing Grant-Revoke Tag Permissions manually but not sure what a “Aws-entry” is and also there are 4 more functionalities like
We would like to understand the difference between all these. Not able to identify what manual inputs we should be providing to perform this testing.
Thanks a lot.
Regards,
Vani
1
0
springboot_team
368 Messages
2 years ago
Hi @vani.nithya.indla, apologies for the late reply.
Regarding the draft version of the Grant to Tag/Resource workflows, currently, they are still under development (work in progress) and not ready to be released. However, we will keep you updated once these can be made available. Please note that an AWS Entry is a custom asset that is used to create the Grant to Tag permission. When the workflows are released, they will create the asset and assign the relations automatically so that the integration reads it.
Thanks
0
0
vaninithya_indla
38 Messages
2 years ago
Hi @spring-team.collibra.com
Thanks a lot for your constant support in attending our queries in testing the bidirectional integration.
We are still continuing our testing by manual creation of assets for Grant and Revoke and facing some issues.
We would like to know if there is any date the workflows are planned to be released in the next version release, so that we can plan our development activities based on that.
0
0
springboot_team
368 Messages
2 years ago
Hi @vani.nithya.indla,
Regarding the workflows and the next version, they are ready and are currently in the process of being released/made available on Collibra’s Marketplace.
Will update this post once they have been published.
0
0