Data Ingestion method/best practices
I did an operating model audit and some questions came up.
- What should be ingested first – data elements or tables/columns?
- What is best practice, should or must I do an data element to column/table ingestion to start things?
- What is the recommended order by Collibra?
- Would there be a difference in the template/mapping required to do this?
The reason behind the questions is that we have been doing table comparisons and trying to figure out a process to match tables to existing documented elements.
fredhedrington
Posted 4 years ago · Edited 2 years ago·Last reply 4 years ago
5 comments
fredhedrington
OP4 years ago · EditedHey @tina.keller , I am thinking about what you stated and had additional questions. To your knowledge is there a way to map data elements to columns automatically?
fredhedrington
OP4 years ago · EditedHey @tina.keller , I am thinking about what you stated and had additional questions. To your knowledge is there a way to map data elements to columns automatically?
Tina Keller
·4 years ago · EditedI would imagine if you know the table and column then yes.
We ingest the data dictionary first. Then, we do the ingestions in the order I shared earlier. We include the relationships in the template so that Collibra knows what the relationships are (contains, is a source of, is a target of). We use the relationships so that the user community can see the relationships between attributes in the diagram views. I use the diagram views quite often when interacting with the business.
If you enter each attribute in manually you have to create the relationships.
fredhedrington
OP4 years ago · EditedHi Tina, thank your for the information. I appreciate your time and reply.
Tina Keller
·4 years ago · Edited@fred.hedrington We created our own excel ingestion templates. We have one for systems, one for integrations, one for data sets and one data elements. When we go to ingest, we do them in this order: 1) system, 2) data set 3) data element 4) integration