Prevent synchronization API from deleting assets missing from the external system
Hi all,
As per the below screenshot from the documentation, it seems it is possible to finalize whether assets missing in the import input file need to be deleted from the Collibra catalog or need their status to be changed. I think the status change option is useful for performing delta loads into the catalog especially if the external system metadata is huge in volume & we get only the incremental metadata once the full/initial load is complete.
I may simply give the status as “candidate” (though I am yet to try). We are not interested at the moment to track the status of data assets. So, it doesn’t really matter for us what status the asset is in. But, I am not sure if this is an elegant way of solving the assets deletion problem at hand. Any other smart ideas are welcome please.
Former User
·4 years ago · EditedHi Noor,
Hmm ok, odd that it isn’t working for me. I might need to investigate further on why this is giving back an internal server error then.
Thanks for clarifying though, just wanted to make sure I wasn’t missing anything obvious
Cheers
Former User
·4 years ago · EditedHi Noor,
Do you have any examples of this in code by chance? I am trying to use the synchronization APIs too, however I can’t seem to get the Finalization one working.
Steps:
When I run it from the rest docs (the green screen you are on), or using Postman, I get a 500 error and nothing in the console logs. When I run it from my python code, I get a 415 Unsupported Media Type error.
Any pointers on what could be going wrong would be greatly appreciated
Another point on this, if I change it from CHANGE_STATUS to REMOVE_RESOURCES, it works fine, so it does seem like this finalizationParameters thing is the cause and I just need to get it right.
Cheers
noor
OP4 years ago · EditedHi Dan,
In the syntax which I shared in my earlier reply, use exactly the same one but replace the status id value that starts with 00000…with the resource id of the status you like to set. That you can find in “Settings —> Status” in the DGC UI.
{ \"STATUS_ID\": \"00000000-0000-0000-0000-000000005020\" };type=application/jsonthis is the only thing I wanted to emphasize upon. Else, the steps followed by you look same to what I did.
noor
OP4 years ago · Editedfor the benefit of all, those who want to try CHANGE_STATUS finalization strategy, there is a syntax error in the finalization parameters example given in the documentation. I have highlighted in orange color in the below screenshot. I got the correct syntax from support. You should use this:
{ \"STATUS_ID\": \"00000000-0000-0000-0000-000000005020\" };type=application/jsonNote: the status id is the resource id of an already existing status that you like to update to
arthurburkhardt
·4 years ago · EditedThe regular (not synchronization) batch does exactly that: load asset incrementally, do not remove missing assets.
noor
OP4 years ago · EditedYou are right Arthur. Thanks for the input.
This is what I tried.
I had 2 assets in the input file for FULL load.
In the next load, I deleted one asset, and I changed the attribute value of another asset.
The expected result is the deleted asset in the input file must not be deleted from the catalog —> passed
The expected result for attribute value change is that it is updated ----> passed
however, not sure why in the log it is written as if the attribute value is deleted first and created afresh (probably for performance improvement, Collibra does this way? )…
arthurburkhardt
·4 years ago · EditedCorrect, This is the standard behavior.