Question: How can I configure Collibra DQ to run a large DQ Check Job (containing millions of rows of data)?Answer: We have some customer reports to us, sharing their configuration of Apache Spark, Kubernetes (K8s), Docker, and more: Collibra DQ ran on K8s Docker: 150 Million rows, with 31 columns, in 45 minutes (details :using 18 Spark Executors, 12 GB RAM and 4 CPUs per Executor, and Driver got 4 GB RAM). Success! 44 Million rows, with 495 columns, in 79 minutes (details: using 46 Spark Execut...

How can I configure Collibra DQ to run a large DQ Check Job (containing millions of rows of data)? | Data Quality & Observability

How can I configure Collibra DQ to run a large DQ Check Job (containing millions of rows of data)? | Data Quality & Observability | Collibra Community