28 Messages
Limitations of Spark-Submit jobs for CollibraDQ
Hello Fellow Data Citizens,
I hope you are well!
I have read that Spark-Submit is an available option to execute CollibraDQ jobs. However, It has some limitations. Can I please request more clarification/information. Thank you.
- Spark-Submit path is not supported in Production by Collibra support team
- CollibraDQ team is not supporting any bug fix
- Each Spark-Submit job requires a new Spark Cluster.
ashishsharma
5 Messages
2 years ago
@muhammad.salahuddin.asic.gov.au - The requirement of having a New SPARK cluster for each command line submitted spark-submit job is really a limitation at Databricks side. Databricks does not allow a direct spark submit on their cluster, they do not support it.
You could still submit DQ jobs from Databricks UI or APIs as alternative here, as illustrated in CollibraDQ docs:
https://productresources.collibra.com/docs/collibra/latest/Content/DataQuality/DQApis/DQ-Databricks%20Submit.htm
1
0
muhammadsalahuddin
28 Messages
2 years ago
Hi @ashish.sharma
Can I please request to share the location of DQ Web Run to find the complete list of parameters for JSON payload template for Spark-Submit job, I have checked the CollibraDQ documentation and the parameters for JSON Payload Template has been mentioned partially, Please refer the following screenshot. Thank you again.
2
0