Integrating ODPS (Open Data Product Specification) YAML via REST API for Automated Data Product Deployment
Summary: Siddharth Reddy is seeking advice on implementing a "Governance as Code" strategy using the Open Data Product Specification (ODPS v4.1) to automate data product deployment in Collibra. They plan to parse YAML files through a Python-based CI/CD pipeline and use the Collibra Core REST API for integration. They are looking for guidance on API mapping, resolving MIME type issues during YAML ingestion, and structuring CI/CD automation to manage data product versions without duplication. Suggestions on best practices, custom scripts, and reference architectures are requested.
Hi everyone,
I am looking to implement a "Governance as Code" approach for our data products using the Open Data Product Specification (ODPS v4.1). My goal is to use standardized ODPS YAML descriptors to automatically instantiate and update Data Products, assign stewardship, and link business glossaries directly within Collibra.
Since there is no native importer for ODPS, I am planning to parse these YAML files via a CI/CD pipeline (Python) and push the metadata into Collibra using the Core REST API.
Has anyone in the community successfully built a similar pipeline using ODPS or a declarative YAML approach? Specifically, I am looking for guidance on:
API Mapping: Best practices or custom scripts for mapping standard ODPS fields (e.g.,
productStrategy,dataAccess,metadata) to Collibra’s specific Asset and Relation Operating Models.YAML Ingestion: Have you encountered and resolved the
resource type (text/x-yaml) is invaliderror or similar MIME type issues when pushing declarative configurations into the platform?Automation Strategy: Recommendations for structuring the CI/CD triggers to handle version updates to the data product cleanly without creating duplicate assets.
Any examples, reference architectures, or specific REST API endpoint recommendations would be highly appreciated!
Regards,
Siddhartha
Balasubrahmanya MG
·1 week agoLooking the same details