How could CDQ interact with Azure Data Factory
i didnt see it in the supported connections but do we have any customers doing DQ on ADF?
ericgerstner
Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago
1 comment
i didnt see it in the supported connections but do we have any customers doing DQ on ADF?
ericgerstner
OP4 years ago · EditedDoing a quick check ADF reads like an ETL tool. Does it actually store information?
To help clarify - check out a recent interation of the CDQ Cloud. I think the layers of Control, Compute, Data really tell the best story.
People have asked us do you scan Tableau, and not wanting to offend a customer by retorting that Tableau is a BI tool, we can classifiy it in the control plane for manipulating data, thus not the best source.
I look at ADF as a control/compute plane for manipulating data.
https://docs.microsoft.com/en-us/azure/data-factory/#:~:text=Azure%20Data%20Factory%20is%20Azure’s,with%20full%20compatibility%20in%20ADF.
If you’re looking to leverage CDQ as part of that control, then think of it as a parallel invocation to the same data sources - leverage and API - we have three APIs: REST API, Scala API, and Py4J wrapper (around Scala API). We can use that to run DQ checks as part of their ETL process.
A graphic like this built with the ADF as the ETL arrow on their relative store might work
For general Azure support via CDQ, kindly check out https://datacitizens.collibra.com/forum/t/mircrosoft-azure-stack/1533