ERROR Logs Management
Hello all! How do you manage the logs from your Collibra instance?
- How do you monitor for issues to proactively help your datacitizen? (e.g. exception thrown in a workflow)
- What do you do with all of the ERROR messages ?(let’s not even talk about the WARN ones)
I implemented an automatic log collection to store all the log files and a log parser to analyze what is happening on our instance, and a log analyzer to monitor for issues.
And it just went through the roof the last 3 days!
Number of error messages per day
Most of them actually come from the beta edge release, so no big deal.
Number of error logs in April 2021
Anyways, very curious to see what you’re doing with your logs, and whether I should pay more attention to all the validation issues, outdated api v1 calls, etc.
arthurburkhardt
Posted 5 years ago · Edited 1 year ago·Last reply 3 years ago
7 comments
arthurburkhardt
OP3 years ago · EditedToday, one developer had an issue and could not upload a workflow. He looked at the logs trying to find an explanation for the error, but there were lots of errors and logs lines, nothing looked specifically related to his issues.
Here’s the breakdown of 16h of dev environment logs and where to find the error.
The error was in the 0.01% of loglines that actually meant something useful for us…
Luckily, we have developed good tools to easily parse logs and sift through the ocean of useless logs.
from collibra import Collibra, Console co = Console() df = co.get_log() df[~df['class'].isin(platform_classes)]victorabidoye
·3 years ago · EditedThank you for the response, Arthur. Can you provide the graphQL API endpoint url?
victorabidoye
·3 years ago · EditedThank you for the response, Arthur. Can you provide the graphQL API endpoint url?
arthurburkhardt
OP3 years ago · EditedOK, ok, here’s a code snippet:
victorabidoye
·3 years ago · EditedHi, how does the automated log collection work? is there an api to stream the log or download the log file?
arthurburkhardt
OP3 years ago · EditedI personally use the undocumented graphQL API to query log files and download them.
This has proven very useful to analyze trends overtime, report on issues, etc.
I also have some methods in my python library to directly query logs into a pandas dataframe for real time analysis.
Former User
·5 years ago · EditedVery impressive!
We focus our log parsing mainly on the dgc-jobs log file, key use case is to have a view of how often jobs such as imports fail (frequent fails generally give us a driver to revisit some of our training material) but also how often jobs run for.
We’ve really benefited from the improvements to import logging.
We briefly explored Single Sign On SAML log parsing as well as we have parties logging in from various areas of Europe and America and used to sit in different domain controllers. Admittedly the scope of that work was just a Proof of Concept but gave us some good insights into the behavior of SSO.