Long-Running Workflows Performance
Summary: Grant Gadomski inquires about the performance effects of running 150-200 multi-day workflows simultaneously in Collibra, comparing it to his experience with Appian, where workflows manage state changes. victormorisset explains that Collibra workflows have two states: Execution and Waiting. Execution consumes CPU and RAM, while Waiting, which occurs during user input, does not impact performance as it's database-stored. Performance is affected only when many workflows are in Execution. Grant Gadomski confirms his understanding and asks for documentation and whether workflows restore their state after an outage.
Hey there,
How bad would it be performance-wise (or otherwise) to have ~150-200 multi-day workflows running at one time?
For context, I'm considering whether we should build complex, multi-day approval workflows in Collibra vs. another tool. Coming from an Appian background, my approach has always been to store request state (created, approved, returned for revision, etc.) in a record/database entity, and use "workflows" only to move the request between states (kicking a new workflow off when someone takes action, and completing it when the action is finished & new request state is written to the database).
Is this a common practice with Collibra workflows as well? If not, have you seen/heard about any performance issues that may come from hundreds of workflows sitting in memory for days at a time?
Thank you!
Victor Morisset
·1 year ago · Edited@Grant Gadomski I'm not entirely sure to understand what you are trying to achieve. However, to clarify things. The Collibra workflow engine manages processes in two states:
Execution State: The workflow is actively running tasks, which consumes CPU and RAM.
Waiting State: The workflow is paused, typically for user input (like an approval). In this state, it doesn't impact performance as its status is simply stored in the database.
Performance problems occur when many workflows are in the Execution state simultaneously. Even a simple task, if started on hundreds of assets at once, can slow the system down because each workflow instance requires resources to start.
On the contrary, you could have thousands of workflow instances in a waiting state, it would not impact the system at all.
Grant Gadomski
OP1 year agoGot it, I think you answered my question, thanks Victor!
Basically I'm curious about whether workflow instances that are waiting (potentially days) for user input still sit in RAM. But it sounds like Collibra has the concept of a "waiting state", where these instances are moved to the "hard drive" until someone takes action. Is there any documentation on this that you know of?
Also, if Collibra goes down for any reason (cloud vendor outage, etc.), are workflows restored to their previous state when the system comes online again?
Thanks again!
Victor Morisset
·1 year agoYes, all workflow states are stored in the database, so there's no impact if the instance shuts down or is re-started