Exploiting Hive Metastore of Databricks for lineage
Hi all
Wanted to check if anyone has made an attempt to exploit the Hive Metastore of Databricks for lineage?
Today, I loaded metadata of 2 databricks databases using the Collibra Marketplace provided Databricks driver. Here is the scenario -
Database 1 > Table_A
Database 2 > View_A based on Table_A
As the table, view relations are implicit, I expected the driver to show lineage/links between these 2 objects across databases but it did not.
So, I plan to get relationships between views & tables from Hive metastore, and load them in Collibra using Import APIs. The Marketplace drivers based connections can load the stand-alone metadata (like tables, columns, views, data types, descriptions)…
noor
Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago
2 comments
anieissac
·4 years ago · Editedhi @noor.shaik
could you please advise how the option of harvesting Hive metastore went through?
SheeriCabral
·4 years ago · EditedHi @noor.shaik ,
I’m the Senior Product Manager for Technical Lineage, ETL tools. I assume you’re talking about “Technical Lineage” (not “Data Lineage”, which is a business summary lineage), but if I’m assuming wrong, please let me know.
We do not have native support for Databricks at this time, with one exception: PowerBI support has a native databricks technical lineage integration - see here in the documentation.
We do have customers using Databricks (without PowerBI) that are successfully getting lineage using the custom technical lineage importer. So your plan is a good one - Hope that helps, and let me know if you meant data lineage and not technical lineage.