How do I modify the settings for Spark to manage logs (and the disk space used)
[This is for standalone installations, but can be modified for K8s]
Customer noticed that disk space was filling up. The spark /tmp location can be redirected per this procedure:
https://dq-docs.collibra.com/installation/standalone/full-install#redirecting-spark-scratch
But that doesn’t solve the problem of these logs accumulating and taking up disk space.
The default is that cleanup is not enabled and they are retained for 7 days.
The number of seconds to retain application work directories on each worker. This is a Time To Live and should depend on the amount of available disk space you have. Application logs and jars are downloaded to each application work dir. Over time, the work dirs can quickly fill up disk space, especially if you run jobs very frequently.
This can be added into load-spark-env.shexport SPARK_WORKER_OPTS="${SPARK_WORKER_OPTS} -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1800 -Dspark.worker.cleanup.appDataTtl=3600"
See https://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts
(and scroll down a bit)
Thank you to Milind Pandit, Jessica Myers, and Kishore Gali for various pieces of this puzzle.
