V

Thursday, June 2nd, 2022 12:12 AM

Spring boot Integration to AWS Lake

Hi Team,

For AWS Lake to Collibra Bidirectional Integration
We have set up the Spring Boot Integration library with Collibra Integration Library and when trying to run the package we are getting the below error

com.collibra.marketplace.library.integration.exception.CollibraIntegrationLibraryException: {“message”:“There are one or more validation errors.”,“details”:“The Relation Key (:SOURCE) Asset ID ‘’ is not a valid UUID - it cannot be empty.”}

Can you please help us on this.

678 Messages

 • 

14.4K Points

3 years ago

@spring-team.collibra.com, can you help with this error?

368 Messages

3 years ago

Hello @vani.nithya.indla

Thanks for trying the AWS Lake Formation to Collibra integration.

Regarding the error encountered, can you please confirm that the asset/attribute/relation types defined com.collibra.aws.lakeformation.util.CustomConstants enum were updated according to your Collibra instance resource IDs?

Hi @spring-team.collibra.com
Thanks for your response
We have updated the constants file and data got ingested into Collibra.

But the assets in “Databases” domain got ingested with “Obsolete” status. Is there any root cause for this issue?

368 Messages

3 years ago

Hello @vani.nithya.indla,

Thanks for the update.

Regarding the assets in the Databases domain having obsolete status, the integration contains logic that marks assets that were not part of the last sync as obsolete.

Can you please confirm that the databases and their related information was included as part of the last sync?

Hi @spring-team.collibra.com

Yes, we enabled auto sync. We will check and get back on this.

2 years ago

Hi @spring-team.collibra.com

Thanks a lot for your assistance

We are trying to sync assets from Collibra to AWS and when trying to do that we mapped below properties to operational domains

We have created a tag under “Create-update -delete-Tag” Operational domain and mapped the UUID to “Collibra.domain,tag-to resource” configuration property

we are getting the below error when running the integration
“ERROR 13916 — [ scheduling-1] c.c.a.l.scheduler.ScheduledSyncProcess : Sync Scheduler Error Invalid UUID string:”

Can you kindly help us on this ?

Also there are three operational domains but only two properties are mapping to operational domains , Please advise us on this

368 Messages

2 years ago

Hello @vani.nithya.indla

Thanks for the update.

Regarding the error encountered, can you please confirm that the ID of the respective domain is set for the following properties?

  • collibra.domain.lf-tag-permissions
  • collibra.domain.tag-to-resource
  • collibra.domain.modify-tag-id

Meanwhile, regarding the documentation, just to let you know that will be updated accordingly, thanks.

Hi @spring-team.collibra.com ,

Thanks a lot for your response !

Please find the details below

  • collibra.domain.lf-tag-permissions = we created a domain as lf-tag-permissions and updated the UUID (as read only domain “LF-Tags” was updated to “collibra.domain.tag-id”
    )
  • collibra.domain.tag-to-resource - we updated with the ID of the “Assign-Remove-LF-Tag-To-Resource Physical”
  • collibra.domain.modify-tag-id - we updated with the ID of the domain " Create-Update-Delete-Tag"

For testing “Create Tag”, we created a aws-create-tag and aws-create-tag-value asset types under “Creat-Update-Delete-Tag” with some attributes like region, arn but still we are getting the same error.

368 Messages

2 years ago

Hello @vani.nithya.indla,

Thanks for the update.

Can you please provide the stack trace for the following error?

Additionally, can you please confirm that (1) all the domains IDs in the application.properties file and (2) the com.collibra.aws.lakeformation.util.CustomConstants IDs are set and exist on your Collibra Cloud instance?

Regarding the CustomConstants class, the ID of each asset type, attribute type and relation type should be defined. Example:

PERMISSIONS("Aws-Permission", "6fa4721c-8f5d-4c00-8aa9-4b4bd576ebbb"),

Also, just to note that a CMA file located in the src/main/resources/collibra_exports folder can be used to set up the required Collibra instance metamodel.

Moreover, can you please confirm that the AWS database is not empty? In case it is, you could also try disabling the AWS Data Catalog and AWS Glue services by setting the following property in the application.properties file:

trigger.api.glue-converter-enabled=false

Hi @spring-team.collibra.com

Thanks a lot for your response.

Yes, we have uploaded the .cma file in Collibra cloud instance and updated the constants file with UUID’s of all asset types, Attribute types, and Relation types and databases are not empty in AWS.
In the .cma file the below two assets were not provided

AWS-Revoke-Resource
AWS-Remove-Tag-From-Resource

After updating these two the ingestion worked fine.
Also, we noticed that in Databases domain , table to database relation is not being ingested from AWS Lake. Can you please advise us on this

368 Messages

2 years ago

Hi @vani.nithya.indla,
Thanks for your feedback, we will make sure to update the CMA file accordingly.

Also please note that the integration is creating a

  • Database to Schema
    and
  • A Schema to Table relation.

Could you please confirm whether these relations are visible from the diagram view of the Database asset page?

Thanks

Hi @spring-team.collibra.com

Thanks for your response.
Yes, Schema to Table relation is coming up.
We noticed an issue with “AssignLFTag” functionality

In order to test that, we created an" aws-assign-tag" asset and provided attributes Database and Tablename with Catalog id
The status is changed to “Resolved” but on AWS end LF-Tag is not getting assigned to the table.
But when we add column name attribute type, on AWS end LF-Tag is getting assigned at column level alone.

What is the logic behind it? We should be able to assign a LF-Tag on Database level/ Table Level/ Column Level but the integration is working only if we provide Column information. Integration is not throwing any error if column name is left empty but the assignment of LF-Tag to table is not happening.
Kindly help us on this issue.

Attached the Log file for your reference.log.txt (12.1 KB)

368 Messages

2 years ago

Hi @vani.nithya.indla,
Manually creating the AssignLFTag asset to perform an assign action can be a bit problematic since you have to replicate exactly the asset types/relations and attributes that the integration is expecting. That is why we are currently developing a set of workflows that help automate this process. I have attached the workflow with this message, please feel free to use it. They will be released with the next version of this integration.

To be able to use this workflow, import the whole zip file to the Collibra Cloud Settings page > Workflows > Definitions > Upload a file. Once uploaded, the workflow should be started and the UUIDs populated from the workflow page. Please note that the attached workflow is a pre-release version and might need some optimisations/further testing

Please rename the file to end with .zip after downloading.

Aws Workflow Selector-bar.zip (23.0 KB)

Hi @spring-team.collibra.com

Thanks a lot for your support.

We will upload the file and let you know our queries after testing :slight_smile:

2 years ago

Hi @spring-team.collibra.com

Thanks for sharing us the draft version of workflows.
After testing we have noticed some issues in the workflows. Please find below the feedback

  1. For Assign LF Tag functionality, after submitting the workflow, assets are created with asset type “Aws-create-tag” instead of “Aws-assign-Tag” and also tag values are not getting added as attributes.

Also in the assign LF-Tag sping boot code, the logic is written in a way that only column level tags are assigned, we cannot assign tags at database / table level alone.

  1. UpdateTag workflow is storing the input Values as UUID’s instead of labels

  1. Remove Assign Tag workflows user form is selecting LF-Tag keys from databases domain instead of LF-Tag domain , selection user form is not correct and LF-Tag values are not getting created in the domain

Also , the next set of workflows on Grant and Revoke permissions, will there be any workflows released in next release ?

368 Messages

Hi @vani.nithya.indla,
For issue 1, can you make sure that the ID is filled in correctly in the workflow setting for the AWS-Assign Tag property? In the released version of the workflows, the default properties will be set like the IDs in the CMA file.

Issue number 2 is a known issue and a change in the spring boot integration has been done to cater for this.

And both issue number 3 and the Grant and Revoke permissions (For LF-Tag and for Resource) workflows are being developed and going to be released.
Thanks.

2 years ago

Hi @spring-team.collibra.com

Thanks a lot for your update !

Can you please let us know if you have any tentative date of next release to plan our integration implementation activities?

Also, we have two questions on the overall Lake formation integration model

  1. In the “users” domain , only IAM users are getting ingested , what about the IAM roles?
    because LF-Tags can be assigned to a Role as well.

  2. And whole Collibra to AWS integration is based on LF-Tag access control method. Is there any functionality available to provide access through named resource method?

Also can you share draft version of Grant Revoke workflows if available.

368 Messages

2 years ago

Hi @vani.nithya.indla ,

thanks for your questions and feedback. We will release the next version of the integration and the attached workflows in around two weeks. We will update you once the workflows have been released on the marketplace.

The two points you made are new features that the current integration version does not support. However, please feel free to post a feature request on the ideation portal or reach out to the Customer Success Manager and request these features.

Thanks

2 years ago

Hi @spring-team.collibra.com ,

Thanks a lot for the update.

If you can share draft version of Grant revoke workflows, it would be great starting point for us to perform the testing before the release of main version.

We tried testing Grant-Revoke Tag Permissions manually but not sure what a “Aws-entry” is and also there are 4 more functionalities like

  1. Grant-Revoke Resource Batch Permissions
  2. Grant-Revoke Resource Permissions

We would like to understand the difference between all these. Not able to identify what manual inputs we should be providing to perform this testing.

Thanks a lot.

Regards,
Vani

Hi @spring-team.collibra.com

Please update us on the above query.

Thanks,
Vani

368 Messages

2 years ago

Hi @vani.nithya.indla, apologies for the late reply.

Regarding the draft version of the Grant to Tag/Resource workflows, currently, they are still under development (work in progress) and not ready to be released. However, we will keep you updated once these can be made available. Please note that an AWS Entry is a custom asset that is used to create the Grant to Tag permission. When the workflows are released, they will create the asset and assign the relations automatically so that the integration reads it.

Thanks

2 years ago

Hi @spring-team.collibra.com

Thanks a lot for your constant support in attending our queries in testing the bidirectional integration.
We are still continuing our testing by manual creation of assets for Grant and Revoke and facing some issues.

We would like to know if there is any date the workflows are planned to be released in the next version release, so that we can plan our development activities based on that.

368 Messages

2 years ago

Hi @vani.nithya.indla,

Regarding the workflows and the next version, they are ready and are currently in the process of being released/made available on Collibra’s Marketplace.

Will update this post once they have been published.

Loading...