What is a Data Set? (ISO Metadata Registry Standards - ISO/IEC 11179-7:2019)
Introduction I have been working in many different industries (Insurance, Telecommunications, Retail...) until I joined UCB, a pharma company. What struck me most in the life sciences industry is the great diversity of data sources: financial, commercial, but also genomics data, IoT (industrial and medical), clinical data, assays, experiments, images, etc.TL;DR: Are data sets just relational data are can they be APIs, files, databases and more?
The ISO standard 11179-7 and other technology vendors seem to think data sets are more than just tabular data.
What do YOU think?
We talk more and more about Data Mesh and other approaches, with a focus on Data Products
All of this pointed me to question this question:
What is a data set?Collibra definition of a data set
A collection of related sets of Data Assets that are Data Elements or composed of Data Elements. Example: Customer contact information
| Example | Data Assets |
|---|---|
| SQL views | Columns in tables |
| Excel file | Columns in sheets |
| CSV file | Fields in file |
As such, some processes have been designed with that model in mind: Data Access enables to give access to database views (e.g. the upcoming integration with Snowflake or the Shop4Data).
But that seems a little narrow, doesn’t it?
What happens when the data is accessible through an API? Or through a storage bucket for a data lake (AWS S3, Azure Data Lake…)? Or an external system?
To make matter worse, the catalog experience displays a blank page for data sets with no documented data elements.
This seems a very restrictive view on governed data.
ISO Metadata Registry Standard
The International Standard Organization has been working on a Metadata Registry Standard for quite some years: ISO/IEC 11179 - Wikipedia
They released chapter 7 on Data Set metadata registration in 2019: ISO - ISO/IEC 11179-7:2019 - Information technology — Metadata registries (MDR) — Part 7: Metamodel for data set registration
This standard defines a full metamodel describing what a data set should be. The document is under intellectual property of the ISO, I cannot repost the model as is, but I’d like to point out two details in this simplified representation.
- The data set specification is the data set model/dictionary. It’s the logical documentation to explain what is in the data set.
- Data set distribution is the ACTUAL access to the data. The distribution can have different formats: CSV, XML, JSON, API, Database access, excel, etc. And you don’t have to choose just one, your data might be available in multiple formats!
This definition and use of data sets makes a lot of sense for me, and I would love to see more of that in Collibra!
Today, it seems the collibra definition mixes the “Data Set Specification” and the “Data Set Distribution”, which I believe leads to operational complexities.
Also, data models and documentation are a very important part of understanding data sets. At this time, there is no mechanism to visualize data models in collibra, but would be very useful
Examples of data set marketplaces
Here are examples of how other technologies have implemented the ISO 11179-7 standard to document data sets.
AWS MarketplaceAWS Marketplace: AWS Data Exchange for APIs (Test product) (amazon.com)
- “Data distribution” is listed as “Data sets”
- “Data specification” as “Data dictionaries and samples”
Open discussion
Have you met some issues with the existing collibra metamodel and the recent developments (catalog experience, shop4data, etc.)?
What are your company challenges in cataloging data outside the data warehouse, that might not be only relational data? Or are you ONLY focusing on the data IN the warehouse?
Would your data governance initiatives benefit from the vision laid out by the ISO Metadata Registry Standard?
maximiliencote
·4 years ago · EditedOh my, that is such a huge question!
In essence, we defined a data set as any collection of data that can be reused by a group of data consumers, with these main characteristics :
These characteristics are described into more details here :
How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh
There are problems with the centralized data lake. A future data mesh needs domains, self-service platforms, and product thinking.
https://towardsdatascience.com/data-as-a-product-vs-data-products-what-are-the-differences-b43ddbb0f123
Hence, as you point out, Collibra’s definition of a Data Set is too narrow for us, as it can encompass any data structures (relational or non-relational).
Regarding APIs, that is a topic on which we also have a lot of debates on. On my hand, I don’t think the API itself is a Data Set (as it looks more like a Database to me), and the Endpoints of the APIs would become the Data Sets (this is important, as API endpoints can have different levels of confidentiality for example, and are not all exposed to the same data consumers).
arthurburkhardt
OP4 years ago · EditedGood points! I fully agree with you, it’s nice to see other people aligning on what the definition of a Data Set.
Regarding your comments on APIs: Yes, the API itself is not the data set. The API is the “Distribution” mechanism of the data set. The model proposed by DCAT or ISO 11179:7 enables one data set to have multiple distributions: a CSV extract, a database access or an API access.
The problem with the current Collibra Data Set definition (and its related functionalities such as the catalog beta experience) is that it assumes there is only one distribution mechanism, and makes the “Data Elements” mandatory. If you have no Data Specification, the page is blank, therefore negatively impacting user experience.
Regarding your links on data mesh and data products: We’re exactly in that situation. For many of our users, their “data products” enabling them to reach their objectives are storage buckets in ADLS, scientific websites, databases, files, etc. which contain the data they need to support their analyses and decisions.
We ended up disabling the beta catalog experience in order to pursue this vision of documenting data sets with a broader definition, but I’m worried we might end up struggling against the limitations of the system (and the UI).
oliversteadman
·4 years ago · Edited@arthur.burkhardt My teams have spent collectively hundreds & hundreds of hours debating this
we always revert to definitions maintained by Data Catalog Vocabulary (DCAT) - Version 2 (w3.org) even though this is not a standard that we have officially adopted. I understand that no SaaS catalogues currently support DCAT, but given w3’s and DAMA’s rigorous approaches to debating & documenting these, I predict it won’t be many years before they do.
arthurburkhardt
OP4 years ago · EditedFantastic answer, thanks! Yes, DCAT is another great resource, which echoes the definition of ISO-11179-7.
In this case, there is no “Data set specification” to detail the logical data elements grouped by the data set, but there is the same data set “Distribution” concept, that allows to document different access mechanisms for a single data set.
I would hope that aligning on such standards would simplify the data access governance, by enabling automation on those data distributions.