E

4 Messages

 • 

200 Points

Tuesday, August 22nd, 2023 9:29 AM

data access and column masking doubts - any direct connection to Databases?

I understand the basics of Colibra, but got a basic foundation doubt regarding how much config work is manual vs ability to pull/push directly to/from databases.

  1. Can colibra connect to target data access Database (ex Snowflake or Oracle) and:
  • pull all initial and/or incremental metadata (schemas, tables, views, mviews, users/roles, …) ? I wonder if yes and whats name of this “Metadata tracking” component and where its being executed: Collibra cloud or agent or some other way?
  • Having ingested target Database initial metadata into Collibra “Data Catalog” -probably as a Collibra “data access” type of object- I suppose kicks the approvals workflow for changes on metadata. Does Collibra connects to remote Database abd execute SQL to apply ddl changes? or does it just notified Data Engineers of approval and manually Data Engineering team then kicks off whatever schema-change cicd pipeline they own to really modify database ?
  1. Additionally i suppose one can define row access policies and column masking policies in Collibra and I wonder if Collibra able to push these policies into target Database (ex Snowflake UDFs) ? or does this pushdown needs to be custom developed somehow?

  2. Is it possible to have a free trial of Collibra in order to test above points ?

Thanks in advance,
Emanuel O.

36 Messages

1 year ago

Hi @emanueol.gmail.com,
let my try to answer some of your questions.

  1. Collibra is able to pull metadata such as existing schemas, tables and columns including their data type, field length, primary key definition etc. from supported data sources and ingest them into the Collibra Data Catalog. To achieve this, customers need to install and operate an Edge Site in their own infrastructure and network. Please check the Edge Documentation for more information.
    If something changes in the data source (e.g. a column in a table is deleted), certain actions are triggered in Collibra. These actions are documented here.
    I’m not sure what you mean with “…execute SQL to apply ddl changes”. Collibra does not modify anything in the data source.

  2. Check out Collibra Protect which enables policy enforcement for supported data sources.

  3. There is a free trial available on the Collibra website. You can also request a demo tailored to your needs on the Collibra website.

4 Messages

 • 

200 Points

1 year ago

Great thanks @laurenz.hiller, regarding 1) i mean if Collibra able to push down ddl sql scripts changes on the target database as well, or if its only one direction ingestion of metadata ?
2) good, and thanks i see as i suspected Collibra push to Snowflake tag row access and column making policies using native snowflake features thats great :+1:
3) thanks.

36 Messages

Hi @emanueol.gmail.com,
the Collibra Data Catalog integrations are one-way. Collibra is not a Data Modeling tool like Erwin or PowerDesigner. Therefore, modeling data structures (e.g. in a ER-Diagram), creating DDLs out of it and push them to a target database for execution, is not a Collibra Use Case in my opinion. Collibra or Data Catalog solutions in general are designed to catalog existing data models, enrich them with business context and make them available to all relevant employees.

4 Messages

 • 

200 Points

Ok so Collibra only able to push down column masking/row access rules into Snowflake (pushing down native UDF), but not able to push down/modify tables/views etc.

So i guess one needs use Collibra rest api to pull metadata, detect changes and execute changes on target database… still bit of work seems. I mean its good to have all governance great features Collibra have but there’s then this need of somehow implement some custom process to push model changes downstream to target database.

Or probably whats best practice is to:

  • have collibra only pulls metadata, alert of changes etc…
  • use any other tool to modify target tables/views (aka models) on target database, like DBT, etc…

Thanks for your insights they were very useful.

36 Messages

Hi @emanueol.gmail.com,
I would recommend using Collibra only for its intended purposes (data cataloging, governance, etc.). For data modeling and DDL generation I’d use a dedicated tool.

Loading...