Best practices for ETL Mapping
We’re working on moving all of the existing documentation out of excel and wiki pages into Collibra and I’m not sure they best way to represent the mapping logic. For example, we have a case where separate columns in the raw data are added together to create the total column in the standardized data. As an added wrinkle, there are multiple flavors of raw data that might have different calculations:
- raw1.a + raw1.b + raw1.c = standard.total
- raw2.a+raw2.b=standard.total
What is the best place to represent this? On the raw or standardized asset? As you can probably guess, I’m pretty new to Collibra so any help is appreciated.
-Kathy
kathybarton
Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago
4 comments
ludovicjanssens
·4 years ago · EditedThe question is what you would like to depict. If you would like to showcase the ETL logic already developed, you’d better rely on technical data lineage. In case you would like a higher level mapping for more functional or business users, the mapping specification is the ideal instrument.
As such, I consider mapping specification more as an instrument for forward engineering. If you have your designers work with it, it could be used as a base for metadata driven integration schemes.
kathybarton
OP4 years ago · EditedThanks! Our goal is for this information to be developed by analysts in advance of onboarding a new data source. Also, technical lineage harvester is not available for our configuration so we are limited to manual entry. I’m hoping the mapping specification asset type and domain will help reduce the complexity of this information,
ludovicjanssens
·4 years ago · EditedHi Kathy,
Great! If that is your goal, the mapping specification will definitely be able to replace Excel-based mapping files many companies make.
Good luck!
junmiao
·4 years ago · EditedTake a look at Mapping Specification asset type to see if it fits your need. You can always extend it or create your own asset type too.