Error estimating the execution of a job when the file is upload using S3
Hi everyone!
I’m having an issue trying to run a test with a S3 csv file, when I click on “Estimate job” I see the following error without information:
However, if I upload the same file in the Temp file section, I can execute everything correctly with results. In this option “Estimate job” doesn’t appear.
How can I see more information about the errors? Can we access the log files?
Why does it fail with S3 and not uploading the file?
Thanks!
anasalas
Posted 5 years ago · Edited 1 year ago·Last reply 5 years ago
7 comments
leonkim
·5 years ago · EditedI see. I’ll add it to bug list. Thanks.
The estimate job button is supposed to work only when Livy is enabled. Currently your “S3_ana_dq” does not have Livy enabled and causing estimate job to fail (on purpose). That is falsely disabling the Run button. We’ll have to fix it to let user override it / make this more obvious.
leonkim
·5 years ago · Editedif you can link me to the installation then I can take a look.
leonkim
·5 years ago · Editedif you can link me to the installation then I can take a look.
anasalas
OP5 years ago · EditedI’m using this instance -> http://35.196.107.21:9000/.
I have created a Remote file connection called “S3_ana_dq” and I have tried with the three uploaded files.
Thank you!
anasalas
OP5 years ago · EditedThank you Leon for your answer.
The version is 2021.07. About the Livy question, I’m not sure about it. I’m using a created installation recommended for Data Assessments.
I cannot click on “Estimate job” but if I click to “run” directly, It works fine.
Related to the error, can we access the log files to know more about it?
Thank you.
anasalas
OP5 years ago · EditedI was able to execute the job correctly if I don’t click on “Estimate Job”. However, If I click on “Estimate Job”, the “Run” button is disabled.
leonkim
·5 years ago · EditedTemp files are intended for small files that can be scanned without distributed compute. “Estimate Job” is expected to NOT appear in Temp file. See https://datacitizens.collibra.com/forum/t/technical-temp-file-run-dq-job-and-best-practices/890
Files at scale should be loaded on a distributed files system like S3, and session management (Livy) should be enabled. In that mode, DQ Web will offload processing of Estimate, GetDays with dataa, Filtergarms, etc to the interactive Spark session.
What version of DQ are you on? Was Livy session enabled for that S3 connection?