detailed DGC logging

Question

Hi All, we would like to explore detailed logging in the DGC, i.e.

.API calls - It seems there is an option for that in the console, but where can the actual logs be found?
Administrative functions logging: if somebody changes the scope of a domain, or removes an attribute from an asset type, etc. where are these actions logged?
Thanks!
Tom Kuppens

ruslanahmetzyanov · Answer

Hi all! We are also interested in getting detailed logging, thank you for insights!
However, the question to Collibra team - whether there is on roadmap detailed statistics:

search statistics - by userid/date/search pattern
view statistics - by userid/date/asset
?

arthurburkhardt · Answer

tom.kuppens.telenetgroup.be:

How do you get access to the repository?

The repo is a postgres database available in the backup zip (without password). You can restore it to a postgres server: https://datacitizens.collibra.com/forum/t/backup-recovery-issues-in-2022-08-2022-09/2333/4?u=arthur.burkhardt

tomkuppens1 · Answer

How do you get access to the repository? This looks mighty interesting! Thanks! Tom

tomkuppens1 · Answer

On the API log indeed the documentation is sparse. I was under the impression that the API call logging option under general settings would be more like a permanent logging feature. Under log settings, there is another “API call logging” setting (see below), which can only be enabled for a certain amount of time. Confusing. We raised a ticket for this, but no feedback yet.

I will dig a bit deeper in the OMRE, but it’s looks like an expensive solution for something that could be solved OOTB by simple event logging. It doesn’t have to be activity log stuff, I would settle for just low level dgc.log.
Yesterday a set of domains all of a sudden had its scope changed to a default scope. Many of our assets appeared empty. Everything is re-set now, and our users are happy again, but we have no clue what happened, no root cause analysis possible here.

Thanks for the insights, Arthur!

Tom

arthurburkhardt · Answer

No idea, I’m very confused by the documentation: Configure the logging of Collibra DGC API calls. It says logs are written to a database and overwritten (so I guess they don’t use an immutable log) and the documentation belongs to the “DiagnosticFiles” section, so I imagine the purpose is to create a diagnostic file only? It does not look like a feature that is meant to be activated permanently.
No such feature OOTB.

Collibra has developed a marketplace OMRE (Operating Model Reverse Engineering) that is pretty nice. You run it on a regular basis in DEV only (not prod), and it copies the metamodel into assets, relations, domains, etc. This gives you full visibility/audit on who created what, when, etc. It’s slow as hell though, since every single action is performed individually, rather than relying on mechanisms such as import/export. It takes about 1h30 only for the metamodel and I had to kill it after 12h when trying to also get metrics.
I have developed a small metadata tracking tool in python (150 lines of codes) that flattens the model into a set of relational tables, then perform SCD2 on top of it to keep track of everything that happens. This runs in about 2 minutes.

The OMRE has really nice features, such as prebuilt diagrams that are super useful for activities such as: please tell me all assignments that use this attributes/relations

or “how are glossary domain types being used?”

1	Grant Rollerson SUNCORP CORPORATE SERVICES PTY LTD
2	Tom Friesen Algonquin College
3	SB S Benson Plus3 IT
4	S stefan_busch
5	Brian Lloyd USA TODAY Co.
6	D daverichard