Workflows : High volumes

Hello,

I have a workflow that starts with a FindAssetsRequest.builder(), it retrieves up to 30k of assets that are sorted by name and filtered on the status of Published.

This workflow creates Data Issues and related Collibra Tasks. The issue that I am facing is, the workflow can take some time to execute. Note that the number of assets to retrieve will grow over time.

I would be obliged to get some feedback from others who have similar workflow issues and how they addressed the processing time for high volumes.

Best Regards

Bobby

robertinnes11

Posted 2 years ago · Edited 1 year ago·Last reply 1 year ago

2 comments

stevenwood

·1 year ago · Edited

Bobby

Performance is directly related to the number of assets being added/edited. It will also vary by environment since non-prod instances usually have a lower performance configuration. There are too many variables that could impact the execution time, so I don't think we can provide a direct answer. In general, I would suggest a coaching call to see if there are other ways to execute the WF. This might include using the output API and the import API. The Import API is designed for high volume and should perform better than creating assets using individual API calls. That's just a suggestion that a coach could review with you.

Steven

Unavailable member

·2 years ago · Edited

Their profile isn't available.

Hi Bobby,

This is Lauren from the Collibra Community team. Thank you for asking this question and providing details.

While we wait for someone with specific experience in this area to respond, we invite you to browse a few resources that might help:

Collibra Documentation
Community Resources
Community Leaders - we are all here to help each other. Feel free to contact a fellow Community member directly with your question.

In the meantime, we'll try to find an answer in the background at Collibra. We appreciate your patience and hope to see this resolved soon.

Best,