12 Messages
JDBC Parquet Connector - Is there a way to exclude folders?
Turns out that in the Azure Data Lake that I need to run ingestions on there is a subfolder in the folder structure that should not be ingested. That folder is on the bottom of the folder structure, so I can’t just target the ingestion below it. Is there a way to exclude folders from a Data Lake parquet ingestion?
alvinuseree
157 Messages
3 years ago
Hmmm very much depends on how that folder is set up.
If you just have 1 sub-folder in the folder and all files in the sub-folder need to be omitted, you can use the property
IncludeSubdirectoriesand set to false.If not, then you might have to exclude the files one by one or use an asterisk wildcard if the files in the sub-folder use a naming convention.
If the Tables being created are prefixed with the sub-folder name i.e.
subfolder_file1, you could use the asterisk wildcard to just omit all tables that start withsubfolderCheck the link below or just read the screenshot:
https://productresources.collibra.com/docs/collibra/latest/Content/Catalog/RegisterDataSource/Jobserver/co_register-data-source-via-jobserver.htm
0
0