Incremental Loads on Collibra DQ

Question

Hi Guys,

We have a use case where we need to schedule DQ jobs based only on the incremental loads on a table. For example, if 10 new rows have been added since the last DQ job, we only need to schedule/run the DQ job for those 10 rows. Is this use case possible via Collibra DQ?

kirkhaslbeck · Answer

@eric.gerstner.collibra.com 's answer is correct. The tool is designed to do this you simple pick the partition column or the column that would signal where old data ends and new data starts. Commonly a timestamp or numeric column. We use template variables like {rd} for this purpose so that your query is not hard coded. Works on files as well.

ericgerstner · Answer

@abhishek.chand.raja , Yes this is possible, please consider the below documents page on setting up partial scans. In setting up the query scope, you can leverage Collibra built variables {rd} which allow you to take advantage of a column like entry timestamp. The scope query is freeform SQL so you could build in whatever customizations work within that language (if you wanted to scan greater than a certain P-Key Value, etc.) https://productresources.collibra.com/docs/collibra/latest/Content/DataQuality/Explorer%20(no-code).htm

1	Grant Rollerson SUNCORP CORPORATE SERVICES PTY LTD
2	Tom Friesen Algonquin College
3	SB S Benson Plus3 IT
4	S stefan_busch
5	Brian Lloyd USA TODAY Co.
6	D daverichard