TO BE REVIEWED: Spark 3.1 still needed Connection Properties for DQ.BQ.LEGACY

There is a warning about Big Query and the Spark version being prior to 3:

However…

The customer was on Spark 3.1 and yet still needed to set a Collibra DQ Connector property for Big Query using the flag: dq.bq.legacy=true

Then it worked!

Brian Mearns had this to say about that:
"dq.bq.legacy=true will load entire table, only then apply predicates. It should not be required unless you’re on super old versions of Spark. It is a limitation with spark-bigquery-connector 2.11 — you need to use that flag. If they’re on a compatible version of Spark, they should use the spark-bigquery-connector which matches their version. Their specific example it should not be needed, unless they’re still submitting to their yarn cluster that has legacy version of Spark.