Spring boot Integration to AWS Lake

@spring-team.collibra.com, can you help with this error?

Hello @vani.nithya.indla

Thanks for trying the AWS Lake Formation to Collibra integration.

Regarding the error encountered, can you please confirm that the asset/attribute/relation types defined com.collibra.aws.lakeformation.util.CustomConstants enum were updated according to your Collibra instance resource IDs?

Hello @vani.nithya.indla,

Thanks for the update.

Regarding the assets in the Databases domain having obsolete status, the integration contains logic that marks assets that were not part of the last sync as obsolete.

Can you please confirm that the databases and their related information was included as part of the last sync?

Hi @spring-team.collibra.com

Thanks a lot for your assistance

We are trying to sync assets from Collibra to AWS and when trying to do that we mapped below properties to operational domains

We have created a tag under “Create-update -delete-Tag” Operational domain and mapped the UUID to “Collibra.domain,tag-to resource” configuration property

we are getting the below error when running the integration
“ERROR 13916 — [ scheduling-1] c.c.a.l.scheduler.ScheduledSyncProcess : Sync Scheduler Error Invalid UUID string:”

Can you kindly help us on this ?

Also there are three operational domains but only two properties are mapping to operational domains , Please advise us on this

Hello @vani.nithya.indla

Thanks for the update.

Regarding the error encountered, can you please confirm that the ID of the respective domain is set for the following properties?

• collibra.domain.lf-tag-permissions
• collibra.domain.tag-to-resource
• collibra.domain.modify-tag-id

Meanwhile, regarding the documentation, just to let you know that will be updated accordingly, thanks.

Hello @vani.nithya.indla,

Thanks for the update.

Can you please provide the stack trace for the following error?

Additionally, can you please confirm that (1) all the domains IDs in the application.properties file and (2) the com.collibra.aws.lakeformation.util.CustomConstants IDs are set and exist on your Collibra Cloud instance?

Regarding the CustomConstants class, the ID of each asset type, attribute type and relation type should be defined. Example:

PERMISSIONS("Aws-Permission", "6fa4721c-8f5d-4c00-8aa9-4b4bd576ebbb"),

Also, just to note that a CMA file located in the src/main/resources/collibra_exports folder can be used to set up the required Collibra instance metamodel.

Moreover, can you please confirm that the AWS database is not empty? In case it is, you could also try disabling the AWS Data Catalog and AWS Glue services by setting the following property in the application.properties file:

trigger.api.glue-converter-enabled=false

Hi @vani.nithya.indla,
Thanks for your feedback, we will make sure to update the CMA file accordingly.

Also please note that the integration is creating a

• Database to Schema
  and
• A Schema to Table relation.

Could you please confirm whether these relations are visible from the diagram view of the Database asset page?

Thanks

Hi @vani.nithya.indla,
Manually creating the AssignLFTag asset to perform an assign action can be a bit problematic since you have to replicate exactly the asset types/relations and attributes that the integration is expecting. That is why we are currently developing a set of workflows that help automate this process. I have attached the workflow with this message, please feel free to use it. They will be released with the next version of this integration.

To be able to use this workflow, import the whole zip file to the Collibra Cloud Settings page > Workflows > Definitions > Upload a file. Once uploaded, the workflow should be started and the UUIDs populated from the workflow page. Please note that the attached workflow is a pre-release version and might need some optimisations/further testing

Please rename the file to end with .zip after downloading.

Aws Workflow Selector-bar.zip (23.0 KB)

Hi @spring-team.collibra.com

Thanks for sharing us the draft version of workflows.
After testing we have noticed some issues in the workflows. Please find below the feedback

1. For Assign LF Tag functionality, after submitting the workflow, assets are created with asset type “Aws-create-tag” instead of “Aws-assign-Tag” and also tag values are not getting added as attributes.
   
   
   
   

Also in the assign LF-Tag sping boot code, the logic is written in a way that only column level tags are assigned, we cannot assign tags at database / table level alone.

2. UpdateTag workflow is storing the input Values as UUID’s instead of labels

3. Remove Assign Tag workflows user form is selecting LF-Tag keys from databases domain instead of LF-Tag domain , selection user form is not correct and LF-Tag values are not getting created in the domain

Also , the next set of workflows on Grant and Revoke permissions, will there be any workflows released in next release ?

Hi @spring-team.collibra.com

Thanks a lot for your update !

Can you please let us know if you have any tentative date of next release to plan our integration implementation activities?

Also, we have two questions on the overall Lake formation integration model

1. In the “users” domain , only IAM users are getting ingested , what about the IAM roles?
   because LF-Tags can be assigned to a Role as well.

2. And whole Collibra to AWS integration is based on LF-Tag access control method. Is there any functionality available to provide access through named resource method?

Also can you share draft version of Grant Revoke workflows if available.

Hi @vani.nithya.indla ,

thanks for your questions and feedback. We will release the next version of the integration and the attached workflows in around two weeks. We will update you once the workflows have been released on the marketplace.

The two points you made are new features that the current integration version does not support. However, please feel free to post a feature request on the ideation portal or reach out to the Customer Success Manager and request these features.

Thanks

Hi @spring-team.collibra.com ,

Thanks a lot for the update.

If you can share draft version of Grant revoke workflows, it would be great starting point for us to perform the testing before the release of main version.

We tried testing Grant-Revoke Tag Permissions manually but not sure what a “Aws-entry” is and also there are 4 more functionalities like

1. Grant-Revoke Resource Batch Permissions
2. Grant-Revoke Resource Permissions

We would like to understand the difference between all these. Not able to identify what manual inputs we should be providing to perform this testing.

Thanks a lot.

Regards,
Vani

Hi @vani.nithya.indla, apologies for the late reply.

Regarding the draft version of the Grant to Tag/Resource workflows, currently, they are still under development (work in progress) and not ready to be released. However, we will keep you updated once these can be made available. Please note that an AWS Entry is a custom asset that is used to create the Grant to Tag permission. When the workflows are released, they will create the asset and assign the relations automatically so that the integration reads it.

Thanks

Hi @spring-team.collibra.com

Thanks a lot for your constant support in attending our queries in testing the bidirectional integration.
We are still continuing our testing by manual creation of assets for Grant and Revoke and facing some issues.

We would like to know if there is any date the workflows are planned to be released in the next version release, so that we can plan our development activities based on that.

Hi @vani.nithya.indla,

Regarding the workflows and the next version, they are ready and are currently in the process of being released/made available on Collibra’s Marketplace.

Will update this post once they have been published.