Using parquet files and the Reporting Data Layer
Has anyone used the Reporting Data Layer parquet files to track metadata quality and changes to metadata over time? Our team is looking into reporting solutions in Power BI to monitor the quality of our metadata in Collibra. Some examples of questions we want to answer using the report are:
- How many and which Business Terms are missing a System of Record relationship?
- How many and which columns are missing a relationship to their table/schema/database?
- Basic data about which assets are missing any required attributes as defined by our asset configuration
In addition to this, the team is interested in seeing if our data quality is improving over time (ex: assets missing required attributes is decreasing, etc). Has anyone ever used the parquet files from Collibra insights to develop these types of reports, or tracked changes to data over time using the parquet files for the purposes of reporting? Or if you didn’t use Collibra Insights, I would be curious to hear about how you implemented your reporting solution as well. I would greatly appreciate any insights someone from the community might have on this topic.
Thanks in advance!
arthurburkhardt
·4 years ago · EditedSo you can use the Collibra insights for this, or alternative solutions discussed here: Reporting Data Layer -
Product Q&A - The Data Citizens Community (collibra.com)
Overall, it seems you have two objectives:
Which assets are missing information?
You can simply use global views with a filter.
This will immediately give you a list your stewards can work on, instead of having to rely on an external BI system
How complete are my assets?
The articulation rules are exactly meant for that use case. They use attributes and statuses to calculate a data quality score, that can then be reported.
Articulation scores (collibra.com)