Hello fellow Data Citizens! We are looking at implementing AI Model Governance in Collibra and we would be very appreciative if any companies that have implemented this type of governance operating model would be open to having a discussion around best practices and possible asset types and relationships. The recent Blog post about AI Governance ( AI governance : the holy grail for all data scientists? | Collibra) is very intriguing to my Data Science team and we would like to accommodate their...

AI Model Governance | Best practices

laurenzhiller

·3 years ago · Edited

Collibra just announced that they are working on an OOTB Asset Model and workflows for AI Governance. Check out this recording for details.

Brian Lloyd

OP3 years ago · Edited

@laurenz.hiller Thank you for this! It makes perfect sense and is close to what we were putting together. I have not made progress lately because of being on vacation and then when I returned to work, I got sick so yeah me! I should be able to make progress on this topic for my internal customers/users in the next few weeks. I’ll let you know how we set things up. Thanks again for this conversation!!!

laurenzhiller

·3 years ago · Edited

Hi @blloyd.gannett.com,

here we go. This is how a first version of our asset model regarding AI Model Governance looks like - it’s heavily inspired by @alexandre.tkint.collibra.com, as we already had an exchange about this topic some months ago:

Some differences compared to @alexandre.tkint.collibra.com’s implementation:

We did not create a Feature Set Asset Type. Instead we just linked Tables to our Analytics Models.
Often our model results are stored in a database table again. So we created a second relation between Analytics Models and Tables.
In the moment we don’t store any evaluation metrics on the Analytics Models. But you could easily add Attribute Types for metrics like Accuracy, Precision, Recall, F1-Score etc.
We link the Analytics Models to the Applications, in which they are used.

Hops this gives some inspiration. AI Model Governance will certainly become more and more important as a result of the EU AI Act.

alexandretkint

·3 years ago · Edited

This is great Laurenz! Can we connect soon to discuss the results?

Interesting that you aren’t adding the features, which is just a question of how granular you want to go in mapping out the ai landscape.

laurenzhiller

·2 years ago · Edited

Hi @alexandre.tkint.collibra.com,
sent you a personal message.

michaelwilcox

·3 years ago · Edited

Hi @blloyd.gannett.com Our fearless leader Stijn Cristiaens recently wrote a blog post on "What it is, Why you need it… ". Our internal Data office at Collibra is also currently using the platform for AI / model governance - project led by @alexandre.tkint.collibra.com and @simla.sivanandan.collibra.com who may be able to provide additional insight as they are doing some very cool stuff in support of EU AI Act, etc.

AI governance: What it is, why you need it, and why it’s essential for your...

If your organization is leveraging (or planning to use) generative AI technologies, then it’s a good time to start thinking about AI Governance. Learn more here.

Est. reading time: 6 minutes

michaelwilcox

·3 years ago · Edited

@alexandre.tkint.collibra.com I think this is great. As the legislation and regulatory requirements are really just getting started for AI / Model governance, it was great to see just how fast you were able to connect them to the models that you had already built and surely also have the flexibility to revise & update as the Acts / policies are revised and develop over time. Hoping @blloyd.gannett.com & @laurenz.hiller can benefit from seeing the great work you have done with this to leverage as they build out this use case in their orgs.

michaelwilcox

·3 years ago · Edited

Hi @blloyd.gannett.com Our fearless leader Stijn Cristiaens recently wrote a blog post on "What it is, Why you need it… ". Our internal Data office at Collibra is also currently using the platform for AI / model governance - project led by @alexandre.tkint.collibra.com and @simla.sivanandan.collibra.com who may be able to provide additional insight as they are doing some very cool stuff in support of EU AI Act, etc.

AI governance: What it is, why you need it, and why it’s essential for your...

If your organization is leveraging (or planning to use) generative AI technologies, then it’s a good time to start thinking about AI Governance. Learn more here.

Est. reading time: 6 minutes

alexandretkint

·3 years ago · Edited

You are everywhere Michael ! Great to hear from you!

Here is an example, how I’m connecting a model with the upcoming policies. Nothing fixed / just a prototype. Thoughts?

Brian Lloyd

OP3 years ago · Edited

@laurenz.hiller - we are building a new Asset Type for AI Models. We have basically asked the Data Science team to provide us with a spreadsheet of what they want to track relating to their models. We are then creating the characteristics/attributes and will be defining any workflows that need to be followed. The intent for us starting out is to just record the models and what they do, who owns them then connect the models to the other assets in Collibra like Policies, Standards, Business Terms, Data Assets, etc…

This seems like a logical step for us and the Data Scientists can manage their catalog /inventory of models in Collibra with permissions and logging as opposed to using an Excel Spreadsheet to track.
I’ll share the final list of attributes we create soon.
-Brian

Brian Lloyd

OP3 years ago · Edited

@laurenz.hiller - we are building a new Asset Type for AI Models. We have basically asked the Data Science team to provide us with a spreadsheet of what they want to track relating to their models. We are then creating the characteristics/attributes and will be defining any workflows that need to be followed. The intent for us starting out is to just record the models and what they do, who owns them then connect the models to the other assets in Collibra like Policies, Standards, Business Terms, Data Assets, etc…

This seems like a logical step for us and the Data Scientists can manage their catalog /inventory of models in Collibra with permissions and logging as opposed to using an Excel Spreadsheet to track.
I’ll share the final list of attributes we create soon.
-Brian

alexandretkint

·3 years ago · Edited

blloyd.gannett.com:

nce/suggestions/best practices would

I love this conversation. thank you !
On the model side, what data assets would you link it to? (feel free to tell the story with your data asset types names)

laurenzhiller

·3 years ago · Edited

Hi @blloyd.gannett.com,
we are currently working on this use case. As soon as I have time, I’ll add some ideas on how an asset model for AI model governance can look like - give me a few days.