How is Spark integrated with Collibra Data Quality and made available to customers? Are there any considerations or consequences when upgrading Spark that could impact Collibra Data Quality? Answer:Apache Spark is integrated at the compute-plane level. For example, in the Collibra DQ life-cycle we can see the steps and stages of the Apache Spark involvement. So at each Spark step and stage upgrading Spark could have an impact: [DQ Metastore - Postgres] DQ Check is requested using the DQ Web UI o...

DQ Life-cycle and Apache Spark | Data Quality & Observability

DQ Life-cycle and Apache Spark | Data Quality & Observability | Collibra Community