Import API Delete/Update behavior on relations

Hi all,

I have a scenario like explained below -

There is a column that is “contained in” 2 queries. Sample data is given here. Somehow, the Import API creates the first relation of the column, then deletes the first one, and finally creates the second one (you can imagine this as the last one). Any guesses on why this could be happening. Should I define the JSON structure the other way - as on, use the query name as the starting point to make the relationship. But, again, a query can “contain” several columns. So, not sure, even if I switch the relationship, the Import API would retain all the relationships or just one (i.e. the last one)

JSON template that tells second column from the input data above is column name, and first column is query name.

{
“resourceType”: “Asset”,
“displayName”: “${2}”,
“type”: “Column”,
“identifier”: “${2}”,
“domain”: “ABC”,
“community”: “DEF”,
“relations”: {
“71749003-72d3-4b92-a234-1550a79796c1:SOURCE”: {
“relation_name”: “${1}”,
“domain_name”: “ABC”,
“community_name”: “DEF”
}
}

noor

Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago

4 comments

arthurburkhardt

·4 years ago · Edited

I think this is getting too confusing.

When you import relations, you use the source or target asset as the identifier of the command.
For a given command, Collibra set/merge (overwrite) the relations for the given relation type.

So, you can define your 1:M relation in one go (col_1 is related to query_1 and query_2)
or you can take it the other way around and define from the source (query_1 is related to col_1, query_2 is related to query 2)
or you can turn it into a 1:1 by making each column only owned by one query. (a query is a data transformation mapping whereby multiple queries use the same source columns, but the outputed columns only belong to a specific query).

Import API allows to define 1:1, 1:M and M:M relations, but you need to define M in your import command. You cannot do M x 1 and hope it will work.

arthurburkhardt

·4 years ago · Edited

As you can see in Import API Documentation - Collibra - Developer, the operation for relations is SET/REPLACE

So if you send a command that says “here’s col_1, it’s related to Query_1” then another statement “here’s col_1, it’s related to Query_2”, the second statement will override the first one.

Either you need to declare two distinct assets (query_1 > col_1 and query_2 > col_1)
Or you need to declare the column with two relations ("here’s col_1, it’s related to query_1 and query_2)

arthurburkhardt

·4 years ago · Edited

As you can see in Import API Documentation - Collibra - Developer, the operation for relations is SET/REPLACE

Either you need to declare two distinct assets (query_1 > col_1 and query_2 > col_1)
Or you need to declare the column with two relations ("here’s col_1, it’s related to query_1 and query_2)

noor

OP4 years ago · Edited

Hi Arthur,

Trying to understand option 1 your suggested -

You want query name to be the starting point while creating relationships? If so, we will have data like this even:

query_1 > col_1
query_2 > col_2
query_2 > col_3 (----------> I hope this relation does not get replaced with the above one then)

Also, this behavior is observed where the input file row count is greater than 1000.

For a input file with row count around 300, all relationships are retained. Example:

SAP_Cube1 > Col_1
SAP_Cube2 > Col_1

In this case, Col_1 is part of several cubes in SAP. Collibra creates both relationships even though Col_1 is used as the starting point during creating relationships. I suspect there is some internal batching algorithm in Import API that is exhibiting the SET/REPLACE behavior.

Based on the above screenshot, then we can never create 1:M (example: Column is contained in Multiple Queries) or M:M relationships.