Classification and Profiling Via Jobserver or Edge - Are you using it? | Product Q&A

Classification and Profiling Via Jobserver or Edge - Are you using it?

We are working on classifying data in our organization a large scale utilizing Edge. In order to classify data, profiling must be completed. We maintain tons of data, and profiling something like 50k samples can often be below 1% (or even below 10%. When profiling data even at 10k sample rows it can take about 7-8 hours to complete the process for one sql database (we have hundreds) for edge or job server. I’m looking in to see what others are doing,
We are looking for some guidance in this area for what others are doing to improve our situation:
-Are you utilizing the connections for profiling and classifying data?
-How long is your profiling jobs taking?
-What sample size are you using?
-What percentage of data are you hitting w/ that sample size?
-Are you building out multiple servers to handle the scale of data?
-Do you use another tool to profile and classify and import into collibra?
-How are you building this data out?

joshallen

Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago

1 comment

noor

·4 years ago · Edited

Few thoughts -

We are currently using Jobserver for profiling & storing sample data (100 rows) for Snowflake. No classification.
A snowflake database has 2000+ tables. Profiling on a sample of 10k rows takes 24-48 hrs.
Jobserver crashed several times during this process. We raised a ticket with Collibra support. We tuned some settings related to Spark. However, later on, we realized the AWS EC2 machine on which Jobserver is installed is having the problem. The disk volumes are not able to burst as per the demands of data profiling, and hence have been advised by AWS to choose appropriate volumes.
Jobserver profiles and submits to DGC the profiling stats in the end all at one go. Seems this has been improved in EDGE. EDGE writes back to DGC the profiling stats table by table.
Also, I heard “true” parallelism in EDGE jobs can only be realized by next quarter. Then, you can run multiple jobs in parallel.
Also, EDGE makes sampled ata unreadable for now. I heard this will change by next quarter.
So, when we move to EDGE next quarter, we plan to multiple EDGE servers for multiple Snowflake databases (located in different regions > EU, APAC, US).

In general, irrespective of which software we use, profiling & classification is a resource intensive operation. I think it will take a while to stabilize things,