Logging best practices
Summary: The author discusses best practices for log file management in Collibra. Collibra retains 30-day archives of log files by default, but extensions can be requested. The impact of these practices includes complying with industry log retention requirements and maintaining environment performance to avoid issues with restores or import jobs. Recommendations include setting up JDBC log parameters to match Jobserver logs, using the default logging level except when troubleshooting, and using external scripting to move logs to a non-Collibra storage if retention beyond 30 days is required. Validation can be performed using the Collibra Console.
Collibra maintains a 30-day archive of log files by default, but in special circumstances CollibraSupport can temporarily extend this period.
Impact
Comply with industry-specific log retention requirements.
Keep your environment performant, to avoid potential risk such as:
Issues with restores.
Issues with Import jobs.
Recommendations
When setting up JDBC logs, set parameters to follow the same folder structure as Jobserver logs to ensure they are written to easy to find locations with the appropriate permissions.
In general operation, the default logging level is usually sufficient. You should only set logging levels to a higher level of detail when you are troubleshooting issues with Collibra Support or Engineering teams. You should only maintain this logging change until the issue has been duplicated and captured in the logs. You should then return the logging levels to their default, because higher levels of logging can exhaust the available disk space faster than monitoring tools can detect.
By default, the system retains 30 days of logs in the archive. If there is a business or regulatory requirement to retain log files for longer than 30 days, we recommend external scripting to call the Collibra Support REST API (<base console url>/docs/rest/index.html#/support) to move archived files to a non-Collibra storage space, such as an S3 bucket, every 2 weeks.
Validation criteria
Use Collibra Console → Settings → Logs to monitor these recommendations.
Additional information
For more information, go the following resources:
Environment log settings for DGC services
