Data Profiling filter (Need active records not all the historical records)

Data profiling is very useful, as it can show the mean, median of say the gross principal balance on the account records. Trouble is, I only want the active records, that is the open/active accounts not the closed/inactive accounts

So for example, I have millions of historical records but only 50 thousand active. How do I profile only the active/open accounts?
Create a SQL view?

thanks for any assistance
Newbie Collibra Customer
Thomas

thomaswood

Posted 4 years ago · Edited 1 year ago·Last reply 3 years ago

4 comments

florentjuglair

·3 years ago · Edited

We’re also looking for the exact same capability, for the reasons mentioned by Thomas. Until it’s there we will not consider using Collibra instead of our current profiling solution.

Bineesh Babu

·3 years ago · Edited

Hello @thomas.wood ,

You can use Collibra Data Quality for this usecase where you can write your SQL Query with active accounts and build the dataset for Profiling. we also apply Automatic rules on the dataset so with Profiling you will get automatic rule without any manual effort

noor

·3 years ago · Edited

I am also looking for such capability, but for a different purpose - performance optimization.
I like to exclude some columns from each table in a schema that has got 3000+ tables.

Where do we apply such filters? Currently, we can only exclude a complete table, not columns.

I think from cataloging point of view, there are whole lot of improvements needed in the current state of Collibra DC.

Bineesh Babu

·3 years ago · Edited

Hello @noor.shaik,

Currently we haven’t seen a lot of requests for this feature in DGC to have exclude/include granularity on the column level for ingestion/profiling in DGC. Table level covers it well enough .

If your use-case is around data quality then we can do this in Collibra Data Quality . I will suggest to open a Ideation ticket for DGC so that we can see for the feedback and demand from other customers

Thanks
Bineesh