Scanning AWS S3 buckets and provisoning profiling results to Amazon RDS for consumption
If we run scanning from Collibra to AWS S3 and do profiling of data as well as classifications on the data. And if we would like then ‘export’ or provision the profiling results like the ‘mean’/’min’/’max’/’1st decile’ etc into an Amazon RDS layer.
What API’s from Collibra will we be dependent on? Or alternatively have you come across clients who have done something similar? (Amazon RDS is a managed relational database service for MySQL, PostgreSQL, MariaDB, Oracle BYOL, or SQL Server.)
Basically how do we get the data to the provisioned layer to make it consumable for other users.
Kind regards
manrichkotze
Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago
3 comments
benedictemarcelin
·4 years ago · EditedHi Manrich,
I’ll share the email response here as well as it may be useful to others as well.
The REST APIs may be used to retrieve the profiling attributes of the asset such as the Mean, Median, Mode, Minimum Value, Maximum Value and 1st Decile. You can find the list of APIs and documentation on the developer portal at https://developer.collibra.com and in your environment at https://<your_collibra_platform_url>/docs/index.html and there is a tutorial at https://developer.collibra.com/developer-tutorials/api-overview/
For example, using Find attributes: https://<your_collibra_platform_url>/docs/restv2/index.html#/Attributes/findAttributes
and testing using ‘Try it out’ with an asset to see the JSON response.
Best,
Bénédicte
manrichkotze
OP4 years ago · EditedThanks Benedicte, we managed to get this working via the REST API and exported the results. Thanks for the feedback.
Alexandra Jorgenson
Admin4 years ago · Edited@benedicte.marcelin, might you be able to help track down a response for Manrich?