Extended profiling activities - system impact and scheduling?
Hello,
We have a large profiling task that we are testing, and for our Dev system the profiling took almost exactly 96 hours. The issue with scheduling a task that takes that long is if we run it on the weekends, it seems that almost every weekend lately has had an update. Yes, we can request deferral, but then it kind of breaks the whole process of scheduling the system.
I am curious what others are doing for system tasks like this, and how intensive is the profiling task against CData objects? I watched our Edge server using HTOP during the weekend of the 96 hour profiling, and the Edge server didn’t appear to be really using too much resources. Not much more than 5 or 6% per core, and only the occasional spike in hard drive use.
Has anyone run into system issues when the profiling task is running against that system?
uditgupta1
·3 years ago · Edited@drichard.beyondtrust.com Try following options if they can help :
ask Scheduling: Consider scheduling the profiling task during periods of lower system activity, such as non-business hours or times when system updates are less likely to occur. This may help minimize conflicts with other processes and reduce the need for deferrals.
Task Optimization: Explore if there are any optimization techniques available for the profiling task. Collibra may provide guidelines or recommendations to enhance the performance and reduce the execution time of the task. Optimizing the task can help minimize the impact on system operations and shorten the duration of the profiling process.
Testing and Staging Environments: Consider setting up separate testing and staging environments where you can perform lengthy or resource-intensive tasks without interfering with the production system. This allows you to validate the profiling task and its impact before running it on the live system.