Userbehavior tracking costum build

Has anyone a way to track userstatistics?
A custom solution?

Collibra has Pen.io, but besides a monthly report we don’t have a overview.

jonasvan_reeth

Posted 5 years ago · Edited 1 year ago·Last reply 5 years ago

10 comments

Former User

·5 years ago · Edited

Some of the Pendo reports do provide decent insights but feel very one size fits all (because they are).

Still awaiting a day where the Raw Pendo data can regularly be shared with Citizens.

We asked our CAM for a one off dump of the Raw Pendo data for 3 months so we could get a flavour of the uplift in adoption (increased users, retention rates, hits on certain pages) as a result of a new workflow offering we deployed. The insights we gained from our own reporting on that data was very heplful.

You could ask your CAM for a view of the Raw Pendo Outputs, but unfortunately as there is no process around it (last I heard), it’ll likely be regarded as a one off request.

One day perhaps …

Edit:
Those insights I’m referring to used a mix of the activitesApi, log parsing and the Pendo Data itself stitched to our Op Model. From my personal experience, there was more than sufficient information from all of this to map out user journeys.

rajeshkailashnath

·5 years ago · Edited

We were in a similar position and when contacted the support team, we were given a query to extract the data from the backend postgres data base. We are currently on-prem and are looking for ways to get the same data when we moved to the cloud.(access to PG DB on the cloud is strictly prohibited)

Activities api is very limited in the number of records that are received. The query times out after 2000 records.
The date filters in the activities api available in the swagger, does not apply.
Navigation statistics does provide some valuable info, but the rich user details that are otherwise available in activities api is not available out of the box.
The hitch hiker’s guide to output module does not include steps to extract the activities data.

We have been able to setup google analytics and are able to get most of the details. But individual user related activity, type of activity etc cannot be obtained from google analytics.(There are ways to setup userids but that requires additions JS coding and needs to be done outside of collibra).

We have also been able to develop log crawlers to capture the below details but as noted above, if a user is just browsing the DGC, searching etc, those activities are not captured explicitly in the dgc.log.
UserName
SessionID
LicenseType
LoginTime
LogoutTime
Duration

The pendo reports need major enhancements since the data that is available is very rudimentary and does not help with identifying the patterns for usage across individual user groups/type of activity(Author vs Consumer).

We have been advised to look at the Insights API(seperate license?) when we move the cloud. That may be the best option as of now.

bartvanderlocht

·5 years ago · Edited

@rajesh.kailashnath

Can you explain how ‘Insights’ will help for this?

To my knowledge, it only contains the (full) DGC content (assets/domains/communities etc…), but user-activity is not included. The only information about users is their responsibilities

arthurburkhardt

·5 years ago · Edited

1. Activities api is very limited in the number of records that are received. The query times out after 2000 records.

It works great for me. The only problem is that the way they have implemented paging is incorrect., it shows {'total': 9223372036854775807, 'offset': 0, 'limit': 0}. So the only way to make it work properly is to iterate on pages and stop when there are no more new records. Clunky, but it works.

The date filters in the activities api available in the swagger, does not apply.

Works great for me, but remember those are java millis, not standard epoch, so you need to divide/multiply by 1000. e.g. in python

start_date = int(time.time()-600)*1000 # 1623331444000 => Activities for the last 10min(600 seconds)
r = c.get('activities', params={'startDate': start_date})
print(len(r.json()['results'])) # 58 => 58 activities

We have been advised to look at the Insights API(seperate license?) when we move the cloud. That may be the best option as of now.

I can confirm what @bart.vanderlocht said, there is no information there that will help you whatsoever… This the the types of data that are available in the insights reporting.

rajeshkailashnath

·5 years ago · Edited

We are currently on-prem and dont have access to the Insights reporting. This was something that was communicated by the CPS team, we are awaiting a demo to see if that meets our requirement. They also mentioned that Collibra is working on some major enhancements to the insights reporting to make this type of data available, so hoping it meets our needs.

rajeshkailashnath

·5 years ago · Edited

Thanks Arthur! We are exploring the option of iteratively extracting the data through Mule workflows. The date filters don’t seem to apply in the swagger even when the date is provided in UTC format. This may be happening due to the default limit.

arthurburkhardt

·5 years ago · Edited

The date format is not utc, but Java Millis.
Try with startDate = 1623331444000 instead.

rajeshkailashnath

·5 years ago · Edited

Activities api is very limited in the number of records that are received. The query times out after 2000 records.
The date filters in the activities api available in the swagger, does not apply.
Navigation statistics does provide some valuable info, but the rich user details that are otherwise available in activities api is not available out of the box.
The hitch hiker’s guide to output module does not include steps to extract the activities data.

We have been advised to look at the Insights API(seperate license?) when we move the cloud. That may be the best option as of now.

alvinuseree

·5 years ago · Edited

Fantastic news, sounds like that could address the pain point I mentioned - thanks for sharing mate

bartvanderlocht

·5 years ago · Edited

I depends on what you mean with ‘tracking userstatistics’. We are doing some, but we don’t have solution like Google Analytics in place

First, we are monitoring logins:
We have a script that downloads the log-files via the Console and parses the files (extracting username + timestap for each login). Finally the data is loaded in a database.
PowerBI connects to this database, and we have dashboards to monitor the evolution etc. (Of course, you can also write to a file, and then use excel for analysis)
We also load info from HR in this database, so we can also aggregate the logins per team/department etc.

On top of logins, we are also monitoring (or planning to monitor):

Searches by the users: also via parsing log-files
Workflow usage: we have standard logging at the beginning and end of each workflow, so again: we can extract this info easily from the log-files
Most view assets: you can extract via REST-api (/navigation/most_viewed)