Who Stole My Data Dictionary?
Hello Data Citizens!
My team is in the process of mapping other teams’ data for various critical data elements/personal information (PI) in compliance with data privacy law. One thing that has been incredibly painful is the absence of data dictionaries. More specifically, the lack of descriptions for each attribute.
I’d love to hear from others who may have experienced similar situations and/or any best practices for producing data that includes a data dictionary. I am not a developer but it would seem obvious to me that data dictionaries should always be included as part of development.
Thanks in advance!
Matt
levequematthew
Posted 6 years ago · Edited 1 year ago·Last reply 6 years ago
9 comments
Ronald Schaareman
·6 years ago · EditedHi,
Although it seems very obvious, from experience I know that descriptions / definitions are mostly not filled - usually out of an attitude of “that is obvious, right, so we don’t have to describe that”. Or sometimes because the field is so generic that the usage determines the definition, so it is impossible to have one at conception (but then at least say so!). And I have seen suppliers of large ERPs are guilty of the practice as well. My conclusion is that it is indeed still somewhat of a utopia to expect definitions for every object…
What we do is use Collibra DGC to add a business term to e.g. a table column and have a workflow (including a validation check) whether the definition for that business term is filled and the other asset is related to that. This way we can ensure that everything that is important at least have something of a definition (and hope for a savvy business steward who knows his or her definitions
) .
manrichkotze
·6 years ago · EditedOur approach is to first leverage off existing dictionaries lying around in business, upload and embed them into the Collibra tool and share the story that the Catalog now has descriptions and context to the data is provided. The next step was to embed uploading of dictionaries into existing data product life cycle development processes, and establishing it into the responsibilities of a data stewards.
manrichkotze
·6 years ago · EditedOur approach is to first leverage off existing dictionaries lying around in business, upload and embed them into the Collibra tool and share the story that the Catalog now has descriptions and context to the data is provided. The next step was to embed uploading of dictionaries into existing data product life cycle development processes, and establishing it into the responsibilities of a data stewards.
Former User
·6 years ago · EditedThis is also the direction which our company has gone in. Along with schema scanning where Data Dictionaries simply don’t exist.
What we have also done is create an Asset Type in Collibra which captures our most critical data in context (Business Process, LOB, Country, Regulation, Project). This then focuses the efforts of documentation to the niche 10% ish of an App which is the most critical so that we don’t have to boil the ocean with regards to documentation.
Former User
·6 years ago · EditedThat sounds like a very practical solution. Finding data dictionaries is very much searching for the proverbial needle in a haystack. They mostly exist in the mind of a developer somewhere - which is really quite scary, as the underlying data structure is the most important part - and should be controlled (governed) first, middle and last.
arthurburkhardt
·6 years ago · EditedOne good practice I noticed is: most technology systems have a built-in catalog. In databases, this is called “information_schema”, “system metadata”, etc. And all of those catalogs have a field called “Label” or “Description” or “Comment”.
Establishing a policy that all databases must document this field ensures that you can easily reuse it and integrate into the Collibra data catalog. Then, you can establish KPIs to make sure that tech teams do document this field properly, fix it at the source, and it is can be automatically retrieved in Collibra for continuous monitoring.
This way you can distribute the task to people who know best, not disrupt their workflow, and manage metadata closest to the data.
Brenda Boelter
·6 years ago · EditedThe most common complaint is that no one wants to write or maintain the data dictionary. My recommendation is to hire technical writers, librarians or analysts who love to write.
I was allowed to take on this task several years ago. My team worked with the subject matter experts and sample data to document our database columns. It was a huge effort but the rewards are huge and still paying off years later.
johnlieto
·6 years ago · EditedMatt - I like Karen’s approach, incorporating Business process is always an important aspect of supporting your business via Data Governance.
Our approach has been to align with our development teams on the work we are doing in the area of data education. This has helped to raise the awareness and the importance of defining tables, schemas and fields within the source Databases. When creating and managing Data Dictionaries, our approach is to to profile the data, understand the assets and create the Dictionaries leveraging existing system and field level meta data. If this is lacking, which is often the case, we leverage business resources to complete the definitions. This is definitely an on gong practice…
Former User
·6 years ago · EditedHi Matt
It’s crazy how often data dictionaries get missed and in fact a lot of the work our team is doing is in helping businesses include best practices for Information Management within existing BAU processes like development.
To kick start privacy work in the past I’ve focused on defining the business processes which use personal data attributes and then from there the systems that were impacted were analysed. Obviously not ideal as there is always a risk you miss a system that has personal data but it isn’t used by a process etc but it helped us identify the major risk areas and then the painful process of checking the other systems began.
The nice thing about this kind of work is it offers an excellent benefits case for doing Data Dictionaries going forward!