Incremental Data from Collibra Insights

Question

Hi

We are trying to extract only incremental load of insights data between two dates instead of extracting the complete insights data from the two dates. Is there a way we can do it or has this been done in any organization.

Thanks

Grant Rollerson · Answer

I've put this reply in more detail on other posts but pattern

extract insights to staging schema
apply SCD and insert into core schema (truncate staging)
transform core into reporting schema, incl. hot & cold facts
put tableau across reporting

Also look at Output Module

Hock Khoo · Answer

We dump the whole lot every day, and load up to the staging area.

We then Upsert any changes to a presentation schema any deletes / updates / inserts.

To understand what's changed, we use a hash on the columns and compare that

1	Grant Rollerson SUNCORP CORPORATE SERVICES PTY LTD
2	Tom Friesen Algonquin College
3	SB S Benson Plus3 IT
4	S stefan_busch
5	Brian Lloyd USA TODAY Co.
6	D daverichard