Activities API
Hi
If I remember correctly, in the past when I retrieved data from the activity API endpoint (/rest/2.0/activities), it gave an indication of the total results (as you would expect) .
Now I notice it returns like the following -->
"{
“total”: 9223372036854776000,* ()
“offset”: 0,*
“limit”: 0,*
“results”: [*
{"*
so the value in the total-key became useless.
I also notice querying this endpoint became a lot slower then it was when we started with Collibra
and less metadata was in the platform. Even when defining multiple filters like user id and limits on amounts when trying to get data out of the endpoint. Often it also results in a status code 500
or keeps idle for hours.
I believe this feature is a really important one and should be stable (also better documented ; like probably together with 99% of all other Collibra data citizens I am still wondering how the ‘context id’ should be used correctly and the info boxes, which document the possible parameters, look like they have been written on a late evening like this one; not production worthy in my opinion and getting data out of the endpoint in acceptable duration without triggering 500 errors is a frustrating journey, or Is it just me? What are your experiences here?
Use-case:
*Let’s say for example you have a user which says 2 years ago he created a certain asset, *
*but this isn’t findable in the Data Governance Center anymore. Would this be a good fit in order to tackle *
*with the above endpoint; eg. check if the user actually created the asset he refers to (when he knows the name *
of the asset) ? If yes, Which filters should be taken into account to do this in an efficient way?
If no, which other solutions we have? Eg; the reporting data layer (but unfortunately this doesn’t go back in time
forever anymore like before,… so for data that was created 2 years ago and if you don’t have old snapshots this is not an option) . Back-ups are an option as well but can be time consuming to figure out if the user doesn’t know the exact date or year anymore he created the lost asset(s).
Interested in your experiences and tips! Thank you in advance
Edit --> I maybe seem a little sceptical in my previous writing, but still a big fan of the Collibra product and definitely a thumbs up towards The Collibra Engineering team regarding the scalability of the product in general;
I feel improvement for sure if we compare with the early days (when we killed the DGC multiple times a day).
Although it would be nice if impacting changes like reporting data layer snapshots with a limited storage time on Collibra end, while it was unlimited in the past, would be communicated beforehand / more transparent.
And important core components like the activity API would have more clean documentation,
also about it’s limitations.
Kind regards
Axel
Axel Schots
OP4 years ago · EditedHi @arthur.burkhardt
Thanks a lot for your feedback! : )
I agree it’s still usable, but in our case I really notice how the performance of the endpoints decreases as the amount of assets in the DGC increase and that is not a good thing (OK, it makes sense but then it should be scaled somehow accordingly) . I raised it towards The Collibra Engineering team and they are currently investigating.
Cool you implemented the stopping mechanism and thanks for sharing!
Although in my opinion this is like buying a brand new premium sports car and you have to
make an additional counter yourself in order to see how many fuell you have left.
Kind regards
Axel
arthurburkhardt
·4 years ago · EditedHey Axel,
Despite a few quirks, the actiivtyApi is still quite usable IMHO.
Erroneous total result count
The bogus total results has been there as far as I can remember. In my python integration library, I implemented a default result limit of 10000 and a mechanism to stop querying when the size of the result set does not increase anymore.if old_count == len_assets: self.logger.warning("Stopping fetching {resource} because no new results retrieved") breakContext ID
The ContextID is simply the UUID of a resource related to those activities. e.g. AssetID, DomainID, etc.For your query: “Tell me all asset created by user XXX between 3 years and 1 year ago”, you would get:
Do you get any results doing such a query?