Import API Check on relations
Using the Import API v2 is great.
But what is a good approach for this situation?
When I’m trying to upload some assets with an relation there might be a risk.
When the Asset for the relation doesn’t exist yet ( for example another upload program failed )
the whole import fails and we get a vague error message with no specifiaction… (@collibra this really should be fixed )
To cover this I’ve handled 2 approaches concidering the sizes of the project/import.
-
get all the assets that could be made a relation to in a List before creating the import (ussually they are in 1 domain and i get all the assets from that domain)
and check if the asset exists, adding it if it exists. -
when building the import and getting to the relation part, I make an API call to check if the specific asset exists and adding it if it exists.
Are there other approaches to check the existance and preventing the fail of the import.
arthurburkhardt
·3 years ago · EditedAh ok. Then just use the mapping API endpoint, or even better: use the systemId-externalId as the asset fullname.
I personally use it in a few integrations (e.g. syncing the IT inventory management system), where I use the externalId as the full name. It works great, no issues whatsoever.
Just for reference, this whole topic is about import API, so maybe best to open a new topic if this is not relevant for managing relations in the import API.
victorabidoye
·3 years ago · EditedI created an idea to allow the creation of relation via the API using external system ID. This can partially solve the issue explained here. Pls vote for the idea: https://productresources.collibra.com/ideation-platform/?id=DCC-I-1170
arthurburkhardt
·3 years ago · EditedWait… Isn’t that already possible?
victorabidoye
·3 years ago · EditedI guess what you are showing is for the import api. My idea is asking for the externalSystemID to be used as an identifier for creating relations in the rest core api: /relations
pmahend3
·5 years ago · EditedHi Jonas,
With the Import V1 API, when the asset specified in the relationship section didn’t already exist, the API would create the said asset. However, this was changed with the newer version as part of making it a bit more strict. My understanding is that it was done to prevent idempotency issues.
Our Product Architect @adrian.hsieh.collibra.com could provide more insight on this.
jonasvan_reeth
OP5 years ago · EditedHi @pranay.mahendra,
Thanks for the reply.
I completely agree that the refered asset should not be created if not found, this is also not what I’m asking.
What I expected was more as I said before.
Ignore the relations not found
and in the api response return a list of missing assets/referals,
So we can respond on this.
The point being is that I started to use the import api to have a fast workflow.
With the extra check I have to do this is no longer the case and I try to find a fix for this issue.
arthurburkhardt
·5 years ago · EditedWild thought here: What if you uploaded the files through the import API, triggered a collibra workflow to open the temp files, perform the checks on the platform, then return the expected results?
If it passes, you just need to call the “finalize” endpoint. If it fails, you get the list of all unmatched relations.
jonasvan_reeth
OP5 years ago · EditedSounds like a good approach, but don’t we still have to rerun the import api & rebuild the import file?
arthurburkhardt
·5 years ago · EditedNo, that’s the beauty of it. You can load each file (increment of 50,000), then when you are done uploading file, there is a specific endpoint to trigger the import. So between loading files and triggering the import, you could check relationship constraint. I haven’t tried it, but it looks possible.
arthurburkhardt
·5 years ago · EditedYeah, I see what you mean. I had a quick look, and it seems like the “simulation” parameter could help achieve this, but the error message is way too high level to provide any specific insight.
{"conflicts":{"Line 1 contains the following error(s):":["No asset matches the specified criteria."]},"type":"CONFLICT","message":"The input processing did not complete successfully."}When using the migration feature, the simulation generates a nice excel reports of all the added/changed resources, but the import simulation does not provide any information whatsoever. Is that correct?
@pranay.mahendra Do you have any idea about that?
arthurburkhardt
·5 years ago · EditedYes, I encountered this issue while importing multiple types of assets and did not investigate further for lack of information (which errors, why, etc.)
Yes, checking existence before import is always a good strategy. This can be achieved through the core API as you seem to be doing (e.g. get assets in a domain), but also through the mappings or output module. The output module is super efficient because it allows to only retrieve the IDs, rather than returning the whole asset definition.
Here’s an hybrid example to show a few interesting behaviors of the output module:
There’s of course always a tradeoff: if you have 100,000 to import, and 100,000 assets to check against, it will be an efficient solution. If you have 10 assets to import and 1,000,000 assets to check against, your solution 2 will be better suited.
jonasvan_reeth
OP5 years ago · EditedHey Arthur,
I was playing with the same thought to use the outputmodel.
Indeed it might speed it up, but still it takes away the speed and efficiency of using the import model.
Something that takes normally 5 minutes to import, now takes an hour ( depending on how many relations there are ).
But as always thanks for the great suggestion,
I’m gonna keep this open in the hopes that someone of Collibra takes this up.
I’m hoping that they give an awnser in the trend of:
“In the future there will be a response: Succeeded but with following exceptions ( list of relations not mapped )” -> in this case we would be able to respond and still have a speedy workflow.