Does Collibra support complex data types

I am using Collibra DGC 5.7.6, our data dictionary has the columns with complex data types. They are Map, Struct, or Array. A Map represents a group of attributes. For example, Party is a Map. Party contains PartyID, data type, string and Party Name, data type, string.

How do I represent complex data type in Collibra physical data dictionary?

Thanks

Former User

Posted 5 years ago · Edited 1 year ago·Last reply 3 years ago

10 comments

laurenzhiller

·3 years ago · Edited

Hi @arthur.burkhardt and @warren.whale ,

I just came across this old post. In the meantime several of the mentioned integrations (e.g. Kafka) are available.

Unfortunately Collibra assumes that the world consists only of relational databases with tables and columns.
For example, also in Apache Kafka there are AVRO schemas that contain complex data structures (nested object arrays with fields etc.). Using the Collibra JDBC driver for Kafka leads to a flattened table with columns per Kafka Topic. The columns names ingested into Collibra are just qualified (Object.Object.Column). In my opinion this is way too simplified, as the objects do not exist as assets and the representation of the Kafka Topic is very difficult to understand for an end user, because the hierarchy of the elements is not visible in an easy way. Furthermore Collibra has no solution for handling multiple versions of the same topic. How do you deal with such data structures - do you use your own parsers instead of Collibra drivers and import the metadata via the Import API? Especially, do you have a solution regarding Kafka?

Best regards,
Laurenz

warrenwhale

·3 years ago · Edited

We did develop our own integration for ingesting and versioning xml & json structures which retained their original hierarchy but it was implemented via mulesoft so we can no longer use it. We have not migrated it to another technology yet. I agree that flattening everything to simplistic tables and columns completely misrepresents the many non-relational technologies currently in the market. Collibra needs to try harder to represent the real world.

laurenzhiller

·3 years ago · Edited

I created a corresponding idea in the Ideation Platform: https://ideas.collibra.com/ideas/DCC-I-1157
Let’s see if something happens on Collibra side.

warrenwhale

·5 years ago · Edited

We’re in the process developing an integration to capture complex json or XML structures typically used in an API solution but it is a non-trivial problem to solve. Once we figure that out we’ll likely extend that to deal with the complex tables. Fortunately we don’t have too many complex relational tables to deal with yet. In a coaching session today the coach didn’t see any initiative within Collibra to support complex structures. Hopefully they’ll see the demand and do something.

Former User

OP5 years ago · Edited

Thank you, Arthur. This approach makes sense.

arthurburkhardt

·5 years ago · Edited

I would create specific subtypes for those, just as you have “column” asset type. Does that answer your question?

arthurburkhardt

·5 years ago · Edited

@stijn.waegemans: Good question. Depends on the datatype. I think modelling a json schema in Collibra would be the archetype of that pattern. 5 basic values (scalar), array and map: It seems quite logical to model the 5 basic scalar values as data elements and the array and map as data structures.
Buuuut… array is one of those tricky ones. Are we talking about an array containing similar values (e.g. tags) or can it be a mixed bag? To err on the safe side, data structure it is.

I hear Collibra customers interested to catalog schemas and APIs, I would be very interested to see if Collibra comes out with an OOTB solution for describing semi-structured data.
From the roadmap, Kafka and HBase metadata connectors are coming, and I wonder whether we’re going to get a proper solution representing avro schemas and multi-dimensional maps, or if we’re just going to get a dumbed-down relational model, ala Apache Phoenix.
Any thoughts?

arthurburkhardt

·3 years ago · Edited

Yeah… it turned out to be the dumbed-down version.
I saw very inspiring examples of customers such as Adeo or CMA-CGM, they presented their use case, they managed the technical data outside of Collibra, keeping only the business level data in Collibra.
https://datacitizens.collibra.com/forum/t/data-citizens-france/430/4?u=arthur.burkhardt

I think that Collibra is not a good solution for technical metadata today, and especially not for anything non-relational. It’s great for the business metadata.
In the last product vision webinar presented by Laura Sellers, they introduced a 3-tiered data ecosystem:

The “Data Awareness” tier is essentially the space where deep technical metadata lies, and the positioned solution was the “Metadata lake”. Unfortunately it’s still in the distant roadmap (2023 or later)

Versioning is an interesting beast on it’s own and many fear there is no good solution to manage it.
This has been one of the most highly voted features of the ideation board for over 5 years now, and it has recently picked up a little traction. Ideation Platform – Collibra | Product Resource Center

Mind map of versioning use cases

Collibra still lacks capabilities to support data modelling (not only for creation, but even for publication and governance of models) beyond the very basic use cases. I don’t think I ever saw a convincing use case leveraging json, avro, xml or any such modeling languages. The recent development makes me think there might be better support of technical relational platforms, but I have not seen any signs that things might improve for non-relational usages.

arthurburkhardt

·5 years ago · Edited

I would create specific subtypes for those, just as you have “column” asset type. Does that answer your question?

Stijn Waegemans

·5 years ago · Edited

I would create specific subtypes for those, just as you have “column” asset type. Does that answer your question?

@arthur.burkhardt Would you create this as a subtype of data structure or data element or column?

@wendy.wang.rbc.com might be worth reaching out to your customer success manager, to see if they could point you to someone that can help with this specific problem. Probably a bit more context is needed.