How does the Collibra DQ Agent work exactly?
Question: How does the Collibra DQ Agent work exactly?
This was answered in conversation with Vadim Vaks, Principal Enterprise Architect:
-
The Agent is responsible for working with a Connection.
a. Connection(s) must be associated with an Agent (see doc links at the bottom of this post). -
The Agent polls the DQ Metastore for a DQ Check job which has not been processed yet.
-
The Agent runs the DQ Check on an Apache Spark platform, and returns the results of the DQ Check to the DQ Metastore for display by the Web UI. (It is possible to use “pushdown” to avoid Apache Spark, that is a separate issue).
-
Lightweight: The agent is translator of the DQ Check to an actual Spark job, and it tracks the progress of that job.
- Agent is not activated before the Job is launched with [Run] from the Web UI Explorer.
- Could be run from REST API
- Could be run from Scala API
-
The “Agent” actually polls the DQ Metastore, looking for the OwlChecks that it needs to run, (roughly every 5 seconds):
-
Agent only activates when there is DQ Check request in the Metastore event queue.
-
DQ Checks gets created and pushed into this DQ Metastore, either from the DQ Web UI, the REST API.
-
The Scala API doesn’t use an agent. You’re already there, you have the
SparkContextalready. -
Agent logs get stored in the Agent Home: Stage 1 logs.
-
DQ Spark Submit JVM log: Stage 2 log:
- Job between compute space and the agent.
- Log directory of the Agent home.
- Spark standalone has some data in this log.
