Technical : Temp file (Run DQ Job) and Best practices
Question: When we run DQ job against the temp file and i can’t see “estimate job” button. that is valid scenario ?
Answer : yes. Thats valid scenario
Question: Are we doing any compute when upload the temp file ?
Answer : No. We cannot do any kind of compute at any sort of scale in Web itself (not even a count *) on any data of consequence. it will crash the web UI. Files at scale should be loaded on a distributed files system and session management (Livy) should be enabled. In that mode, DQ Web will offload processing of Estimate, GetDays with data, Filtergarms ect to the interactive Spark session.
Question: If file size is very large in that case do we need to estimate the job for executor , memory etc ? Or are we limiting the file size ?
Answer : Temp files are not really meant for any kind of serious use cases. There is not a way to distribute the compute as the file lives on a single nodes files system with no way for a distributed Spark session to access it.
Question : What is general best practice for Temp file ?
Answer: As a general best practice, Temp files should not be larger than 1GB. Also, Large datasets should be broken into multiple files and stored in a single folder that way, they can be processed in parallel. Single large file will crash a JVM. For example : Imagine a file that is 50GB in size. Would require a 70GB JVM to read and process… and that is not workable scenario.
This is the reason we rely on Spark so heavily. some file formats like CSV are splittable so Spark can go after a singe large file in pieces, but this is extremely suboptimal. But there is not way to split JSON files, for example : Each JVM has to read the entire file.
matthewtsui
·5 years ago · EditedThank you, @darshan.soni.collibra.com, this is really helpful!