Cataloging Processes

Question

Does anyone have any best practices around cataloging jobs, pipelines, Processes, or the like? I’m unsure of which asset type to use.

ludovicjanssens · Answer

Personally, I would create a new data asset type “pipeline” or “data transformation” which could relate to:

different “data set” and “bi data set” assets that are involved (maybe using two relations like consuming and producing)
different “mapping specification” assets that logically document the transformation done in the pipeline (from the start node(s) to the end node(s)

The diagram would then allow to drill down to the physical data resources and to open technical lineage for those who need it.

Laura Beale · Answer

I have the same question. @mpclark.firstcommand.com did you find a solution for cataloging data pipelines? We have implemented technical lineage, but also need an asset type to capture functional and business attributes such as purpose, schedule, dependencies, etc.

michaelclark · Answer

Cataloging Processes ( S S I S , I nformatica, A D F)

1	Grant Rollerson SUNCORP CORPORATE SERVICES PTY LTD
2	Tom Friesen Algonquin College
3	SB S Benson Plus3 IT
4	S stefan_busch
5	Brian Lloyd USA TODAY Co.
6	D daverichard