Checking logs in Console seems to be overly cumbersome. Only the last 200 entries can be checked via Console UI. To be able to see the rest, the log has to be downloaded, unpacked and opened in some reader. Has anyone found any workaround for this hassle?

Accessing Console Log | Developers

daniel

·3 years ago · Edited

If I am doing a lot of log reading, I use the API to get the log lines and then filter out all of the stack trace stuff to put into a local file, then I simply tail the new file (though you could just print to output too). I have this function as a part of a command line python script which then refreshes at whatever the given interval is (sometimes quite fast when debugging a workflow, or otherwise around 5 minutes for general log viewing).

def getLogFileLines():
    conn = HTTPSConnection(f"console-{client_env}.collibra.com")
    payload = ""
    headers = {
    'Authorization': f'Basic {creds}',
    'Content-Type': 'application/json',
      }
    time_now = round(time.time() * 1000)
    conn.request("GET", "/rest/log/service/85a195a8-b519-483e-9ba0-823d6f896d30/content?filename=dgc.log&lines=500&_=" + str(time_now), payload, headers)
    res = conn.getresponse()
    data = res.read()
    json_data  = json.loads(data)
    f =  io.open('C:\\Temp\\dgc.log', 'w', encoding='utf-8')
    for line in reversed(json_data):
        if not re.match(r'[ \t]', line):
            f.writelines(line + '\n')
    f.close()

arthurburkhardt

·3 years ago · Edited

@dcurran1.kpmg.com.au any reason you’re not using the userscript tip mentioned above? Clean up stack trace with Workflow Exceptions - Developers - the Data Citizens community (collibra.com)
For real-time log reading, that’s the most convenient way I’ve found.

Tomasz Tomiczek

·3 years ago · Edited

Hi Daniel,
I am interested in the part of your script. What is this id:
85a195a8-b519-483e-9ba0-823d6f896d30 which you are using?

conn.request("GET", "/rest/log/service/**85a195a8-b519-483e-9ba0-823d6f896d30**/content?filename=dgc.log&lines=500&_=" + str(time_now), payload, headers)

daniel

·3 years ago · Edited

Hey Tomasz,

That is a part of the URL when you go into the WebUI and click on the dgc log. It is just an internal ID so you may need to check if it is the same in your environment or not.

Roger Craig

·2 years ago · Edited

Where is the documentation for that REST API Endpoint? I can’t find the endpoint within the REST Core API.

daniel

·3 years ago · Edited

If I am doing a lot of log reading, I use the API to get the log lines and then filter out all of the stack trace stuff to put into a local file, then I simply tail the new file (though you could just print to output too). I have this function as a part of a command line python script which then refreshes at whatever the given interval is (sometimes quite fast when debugging a workflow, or otherwise around 5 minutes for general log viewing).

def getLogFileLines():
    conn = HTTPSConnection(f"console-{client_env}.collibra.com")
    payload = ""
    headers = {
    'Authorization': f'Basic {creds}',
    'Content-Type': 'application/json',
      }
    time_now = round(time.time() * 1000)
    conn.request("GET", "/rest/log/service/85a195a8-b519-483e-9ba0-823d6f896d30/content?filename=dgc.log&lines=500&_=" + str(time_now), payload, headers)
    res = conn.getresponse()
    data = res.read()
    json_data  = json.loads(data)
    f =  io.open('C:\\Temp\\dgc.log', 'w', encoding='utf-8')
    for line in reversed(json_data):
        if not re.match(r'[ \t]', line):
            f.writelines(line + '\n')
    f.close()

daniel

·3 years ago · Edited

Hey Arthur,

I made that script years ago when tinkering around, before coming across your method. My org also has tight restrictions on chrome plugins unfortunately, which means I continue with this script when need.
Only other benefit from this approach is to still write to a log file in the background locally that I can look at if needed when trying to track something complicated down.

Definitely would use the userscript though for those that can have the plugin

ravi

·3 years ago · Edited

Hi Pawel,

If you want to increase the entries, you can manually type the integer in the box where you see “Show the Last 200 entries” to 1000 or more.

Hope it helps.

Regards,
Ravi

arthurburkhardt

·3 years ago · Edited

ravi_collibra:

This is probably one of the most well hidden features of the console. I was so amazed when a collibra trainer showed it to me!

Also, another trick that I couldn’t live without: removing stack traces from the console logs.
Clean up stack trace with Workflow Exceptions - Developers - the Data Citizens community (collibra.com)

arthurburkhardt

·3 years ago · Edited

Yeah… the experience can be hit or miss… Collibra is difficult to master, and I guess it’s difficult to find and train coaches to perform at the highest level.
Actually, this is mostly what I do those days: download with a python script and parse into a pandas dataframe for easy manipulation.

The best solution would be to enable the integration with a proper log analytics engine (Elastic, Solr, or else), but there is no such functionality today.

ravi

·3 years ago · Edited

Hi Pawel,

If you want to increase the entries, you can manually type the integer in the box where you see “Show the Last 200 entries” to 1000 or more.

Hope it helps.

Regards,
Ravi

Pawel Musial

OP3 years ago · Edited

Thanks Arthur.
You’re reading my mind
Btw, my Collibra trainer insisted on downloading the logs if you need more than 200 lines