Data Profiling / Excluding tables
We have a very large databases. Some of the tables have over a billon rows. I need to understand the behavior of the exclude option.
When I initially run the job including all tables (with out data profiling), I will ingest all the metadata from all the tables.
If I then exclude a table, does it remove it from Collibra if it was already ingested previously?
I’m hoping I can load the entire database without data profiling, then run the job excluding large tables (with data profiling) without those excluded tables being removed.
Former User
Posted 5 years ago · Edited 1 year ago·Last reply 5 years ago
2 comments
arvindsingh
·5 years ago · Edited@cxlarsen.larsen
If your database supports push down sampling, then use that feature. It might speed up data profiling and data sampling process. More details here on push down sampling : https://productresources.collibra.com/docs/cloud-user/latest/#Catalog/DataProfiling/co_about-data-profiling.htm
If you exclude a table in your step 2, then schema refresh will create refresh conflicts for tables which were excluded in 2nd run.
You can clear those refresh conflicts and run schema refresh again. However, those tables will never be refreshed again until you remove those tables from exclusion in your connection properties, and you will keep getting refresh conflicts. You have to repeat step 1 and step 2 again if you want to refresh those initially excluded tables. It will be manual effort and it will not be my preference if you plan to automate schema refreshes.
If you still want to go ahead with your approach, you can create a workflow to clear out refresh conflicts. Thanks!
Former User
OP5 years ago · EditedIt doesn’t seem to behaving as you stated:
XCHANNELDATA is in the exclude list.
After the job ran it still shows in Collibra without anything populated in the refresh conflict. Does it just exclude from being refreshed and/or profiled?
I will be looking into push down sampling.