Issue with Marketplace API Importer (Read JSON payload and import into Collibra)

@spring-team.collibra.com, is this something you are able to help @joseph.nudi with or is there a different group internally I can reach out to?

@kristen.freer I appreciate the @. Note that we got through our biggest problem late last night (apologies for not updating the post). We knew the issue was tied to the custom “is of type” relation requested by the documentation. On the code side we missed parameter for the assignment of this custom relation to the local UUID specifically for folks not using the CMA file.

/ TODO: Change ID if you did not make use of the CMA file

•    DATA_ASSET_TYPE_OF_DATA_ASSET("385fd1fc-cd53-4974-a6d1-28649342d702");*
  

This gets us now to the point of the ingestion of an API being functional. We have an issue with the core API that was the focus of the effort, but this is relatively minor so far.

I would bring to the attention of the @spring-team.collibra.com that we did find another error which seems like it should not be related to the issue noted above. When running one sample API such as “Simple” after booting the maven/springboot instance, the first ingestion worked. If you run the SAME API again (repeatedly or otherwise) it works. However, running any other API, even a simple API like “Callback”, led to artifacts (I believe in the form of schema: aka data structure and data element assets) from the previous API (“Simple” in this case) being injected into their respective domains again WITHOUT linkages to the API and endpoint, etc.

That’s all for now and if anyone else has issues with getting this Marketplace tool up and running please feel free to reach out.

Joe

Hello @joseph.nudi

Thanks for trying the Read JSON payload an import into Collibra integration and for the information provided.

Regarding when using the sample “Callback” API, to confirm the issue is that there are relations between assets having type Code Value and Data Structure missing please?

This is a known issue that is currently being worked on. Will update this ticket once the new/updated version is made available on Collibra’s Marketplace.

@spring-team.collibra.com

Got it and thank you. That is likely where a breakdown is in another API we’re ingesting and it could be related to this issue. Note that this issue is CONSISTENT within an API. Running the same API repeatedly results in the same Data Structures being isolated.

However, I believe there is another issue going on unless the following is related to my instance only. Try this please:

Close any open integration services and reopen a new mvn springboot instance
Run the SIMPLE API and let it process as usual. No problems should occur.
Optionally, run SIMPLE API again. Still no problems.
Immediately (without closing mvn) run CALLBACK. You should see injected artifacts from the SIMPLE API suddenly show up (as unconnected data structures). The CALLBACK run still works, but you’ve also injected some garbage data structures almost as if the mvn/springboot integration has artifacts stuck in a buffer/memory somewhere and tries to process them.
If you delete all of your assets (or not) and rerun with a FRESH mvn kicked off, this duplication of SIMPLE data structures during the CALLBACK import does NOT occur.

Finally, in case it is related, note that the Activities in Collibra settings seems to show multiple .csv firing during an import. One before and one after the import (possibly). This may be correct/as desired though.

Hope this helps and please reach out with questions/concerns. Thank you,

Joe

Hello Joe,

Thanks for the update.

Regarding the Collibra Activities exports, these are used to be able to determine which assets were already on the Collibra instance and which ones were created. It is also used for the setting the old assets (not included in the last sync) status to Obsolete.

Hello @joseph.nudi,

Thanks for the information provided.

Regarding the API specification, request body, it is not currently supported by the integration. This has been added to the backlog for which we currently do not have an ETA available.

Meanwhile, for the previous issues, the updated integration will be made available on Collibra’s Marketplace soon. Will update this post once it has been released.

Hello @joseph.nudi ,

Please note that the latest version of the Open API integration has been published which includes a fix for the unnecessary Data Structure Relations issue. Please note that this fix contains an updated version of the CMA file for better convenience. Before uploading the CMA file please make sure to remove the custom relation Data Asset is of type Data Asset since the CMA file won’t update the id if the relation already exists.

Thanks