There is a warning about Big Query and the Spark version being prior to 3: However… The customer was on Spark 3.1 and yet still needed to set a Collibra DQ Connector property for Big Query using the flag: dq.bq.legacy=true Then it worked! Brian Mearns had this to say about that:"dq.bq.legacy=true will load entire table, only then apply predicates. It should not be required unless you’re on super old versions of Spark. It is a limitation with spark-bigquery-connector 2.11 — you need to use that f...

Spark 3.1 still needed Connection Properties for DQ.BQ.LEGACY | Data Quality & Observability

Spark 3.1 still needed Connection Properties for DQ.BQ.LEGACY | Data Quality & Observability | Collibra Community