Connecting to a Data Warehouse that uses a Data Vault model design
I have an internal team that is interested in setting up a Data Warehouse using a Data Vault model design. They are considering using the Snowflake Data Vault feature to do this.
Does anyone have experience connecting their Collibra Metadata Management processes up with a data store using this sort of structure? Physically (well, actually it’s Virtually) the data in the system is just going to be stored in tables inside a SQL type database. But the internal structures of the tables follow a Hub and Satellite data model, and I am concerned about trying to get our Collibra tools to work with this sort of Data Warehouse.
Thanks!
markkuhnhenn
Posted 5 years ago · Edited 1 year ago·Last reply 4 years ago
6 comments
justinm_swenson
·4 years ago · EditedMark,
Although this is an older thread, I noticed this just looking for folks discussing Snowflake and dbt.
For us, dbt has been one of the best tool purchases in our Data Team’s history. We are very happy with it. I know it’s the number one tool adopted along with Snowflake for Data Engineers, Analytics Engineers, Data Analysts.
I’m still pushing for Collibra to have some kind of integration with it ASAP in 2022.
Anything in the “Modern Data Stack” should be a priority to gather metadata from for Collibra’s product teams. FiveTran, dtb, Snowflake, Census, Airflow, etc.
Justin
markkuhnhenn
OP5 years ago · EditedThanks Arthur.
Yes, I mistyped the tool name, it is the dbt tool. Evidently someone has had some success using the Colliba Accelerator process to develop a way to get data lineage information from JSON formatted output files from the dbt tool. I was hoping that this might be a process other people may have encountered. Hopefully as we make progress on this effort we’ll have some good information to share on our experiences with this.
Thanks again!
markkuhnhenn
OP5 years ago · Edited@arthur.burkhardt Any thoughts or experience working with project team using the dvt tools to build and populate Data Vault modeled data warehouses?
arthurburkhardt
·5 years ago · Edited@mkuhnhenn.geico.com I’m not sure I know what “dvt tools” stand for, would you be able to provide a link maybe?
Do you mean dbt maybe, as in data build tool ( https://www.getdbt.com/) ?
If so, this is one of the most exciting ELT frameworks I’ve seen in the past few years. It is a simple to use framework that is very developer friendly, great integration with all modern technologies and methodologies (git, dataops, python, etc.) and provides much of the needed governance features (lineage, in-line data quality, metrics, logging…).
I don’t have any experience on real projects, only on small PoCs, but I’m definitely looking forward to a project using this.
arthurburkhardt
·5 years ago · EditedYou don’t need to worry.
There are different data warehousing architectures: Kimball (aka dimensional modeling), Inmon (aka relational data warehouse), anchor modeling… Data vault is simply another such approach.
markkuhnhenn
OP5 years ago · EditedThank you very much, Arthur.
I was concerned that the data structures inside these Data Vault tables were grouped and arranged in logical patterns that did not correspond to the types of data patterns that the Collibra tools were accustomed to working with.
The Hub and Satellite pattern doesn’t seem that much different from the layout of Star Schemas and other data patterns, but the inclusion of source and time based data elements to account for change over time seemed to be new.
Thanks again!