Associating $rd value to a DQ job source folder name to make it dynamic with respect to the run date
We have a DQ job with source as ADLS remote file folder. This is a monthly job. There is new ADLS folder created by the business every month with year and month in the name of the folder.
For example; SRC/2024/09 is the folder name for September run. We are manually editing the DQ job to pick up the latest folder before the scheduled monthly run and thus we are forced to keep the job in stage.
Is there any option of automating this by splitting the source name and replacing the month and year part of it with respective month and year part of $rd, so that the job will dynamically choose the latest folder name.
Any help is very much appreciated. Any other option of achieving the result would also be really helpful.
I tried dissecting the folder name and it throws error.
