Jobserver Performance and Impact of PushDownSampling

Hi All,

PushdownSampling is a new feature added by Collibra in this release, see the link here: https://community.collibra.com/docs/cloud-user/2020.11/#Catalog/DataProfiling/co_about-data-profiling.htm

After testing, it does increase the performance of ingesting from a data source. But I have a few questions:

Is the pushDownSampling value entered applied to the Entire Data Source job, or per table? Does it has any correlation with the Number of Rows per table setting in the Data Profiling section in the DGC configuration in the console?

Also can we expect any boost in performance if we lower the table profile data size in Jobserver section in the DGC configuration in the console? Does it have any correlation with pushDownSampling value?

Any help or ideas in appreciated.

adityapawar

Posted 5 years ago · Edited 1 year ago·Last reply 4 years ago

4 comments

arvindsingh

·5 years ago · Edited

https://productresources.collibra.com/docs/cloud-user/latest/#Catalog/RegisterDataSource/JDBC/ref_db-connection-details-collibra-provided-driver.htm#

Select Oracle connector and go through instructions. You need to add property ‘pushDownSampling’ in driver configuration.

You can download driver from here : https://marketplace.collibra.com/listings/jdbc-driver-for-oracle/

Former User

·5 years ago · Edited

Having obtained further clarification:

pushDownSampling is a value that controls the maximum number of rows per table to analyse, implemented at the database level. It will limit the amount of samples retrieved to calculate the statistical values.
Number of Rows per table in the Data Profiling configuration section controls how many are shown visually in user interface

Former User

·5 years ago · Edited

Hi all,
We have jobserver on-prem using Oracle connector to get metadata information.
We need this information and their data samples without profiling information therefore we think that setting up this feature could be very usefull to accelerate the process but I can find documentation about how to set up this feature for Oracle connector.
Please, someone knows where can I find this instructions ?

thank you beforehand and best regards

noor

·4 years ago · Edited

Hi Chris,

so Jobserver calculates profiling statistics on the number of sampling rows returned by the database(example: 10000)? Or, it calculates on all the rows of the table by fetching them into cache?