Need strategy to handle same data asset in all of systems environments

New to Collibra and we have a requirement from Data Governance Leader to list the Data assets that may be in a systems lower environments. Meaning if we have a table called MyTable in Dev, Test, Prod of System X we need to see the Data Asset MyTable in Collibra production as three different assets.
The vision is to give Compliance and Consumers the view of where all the data is, even in the lower environments of a system.

Has anyone architected their model to accommodate this in Collibra?

davidbotzenhart

Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago

5 comments

alvinuseree

·4 years ago · Edited

That does make a lot of sense. In lower environments where your controls might not be anywhere near as stringent as Production, you may not want highly sensitive data residing there (unless you’ve anonomised it). So that intelligence does have value.

You can also spin it another way. As an engineer developing in a lower environment, you would want a view of the data available in the lower environments.

I agree with Arthur, I highly discourage you from creating 3 instances of that metadata in Collibra as it becomes very difficult to manage and could affect the browsing experience of your wider user base.

In terms of how you flag it, it depends on what you want to capture.

If you want to capture additional metadata on the environments themselves (maybe different owners, functions or how those environments are used as a CI/CD pipeline) and perhaps link them to other assets (let’s say controls on each environment) you could do something like I’ve attached below. Through Collibra, you’d be able to flag that PII exists in the lower environments but perhaps thats fine given the controls you have on said environment (as an example).

If you just want to tag an asset as existing in Dev, Test, Prod then a characteristic or tag would work well.

davidbotzenhart

OP4 years ago · Edited

Thank you Robert and Alain for the reply. I understand questioning the value of having the three assets listed. The thinking is that data is “hidden” sometimes in the lower environments and Data Governance along with Compliance wants to have visibility into those environments to see ALL the data of the organization, not just what ends up in Prod. There could be exposed restricted data in the lower environment that presents a risk to the business. I hope that makes more sense.

alainmugwaneza

·4 years ago · Edited

I don’t also see the benefit to list the 3 environments.

It is feasible to document the 3 assets: MyTable-Dev, MyTable-Prod, MyTable-Test.

Create a DEV Physical Data Dictionary, a PROD Physical Data Dictionar, a TEST Physical Data Dictionary
You document

MyTable-DEV in DEV Physical Data Dictionary
MyTable-PROD in DEV Physical Data Dictionary
MyTable-PROD in DEV Physical Data Dictionary

What will happen is that your assets will be ingested and documented in the 3 Physical Data Elements, you can firstly focus on DEV and also … on specific use cases. You don’t have to work on ALL your “data assets”

Former User

·4 years ago · Edited

Hi, what would be the benefit to list also DEV and QA environments and content in Colllibra?

We limit ourselves to show only production content which is avaialble to the company. Also to limit the volume and hits that are generated on a search.

arthurburkhardt

·4 years ago · Edited

Whew… I’d say that Collibra is mostly geared towards business profiles. But the problem you are mentioning is very much an operational problem.
It might work if this is the focus of your governance and you don’t have too many assets, but otherwise, the sheer number of data assets you might end up with would make it impossible to govern.

I would suggest it might be easier to capture a characteristic to identify which data assets are available in which environment. Else, I would look for another type of solution to store such data.
Let’s say you want to capture all the data you have in all the databases in all environments, you might end up with hundreds of millions of elements if not billions. It would be a pain to manage that in an environment such as collibra and would make it very confusing for users to search information.

So not impossible, but be careful to not create too much noise with so many assets.