JDBC Driver for Apache Kafka

Hi Jamie,
Yes, this page is for seeking help from the community. However, this JDBC driver is Supported, to if your issue requires it, you can log in a support ticket.
Best regards,
Paulo

I am looking also the Kafka asset model with an explanatory example. Is it available somewhere?

Hi @peep.kungas ,
the Asset Model the Kafka JDBC Driver creates is quite easy:
Schema > Table (= Kafka Topic) > Column (=Field inside of a Kafka Topic)

So basically, the driver treats Kafka as a relational database with tables and columns. Furthermore, the driver will ingest only one schema version of the Kafka topic (the most recent) in Collibra.

@laurenz.hiller, could you provide also naming convention examples for these assets, e.g. names and full names?

Hi @peep.kungas,

AVRO Schemas in Kafka typically contain complex data structures (nested object arrays with fields etc.). The Collibra JDBC driver for Kafka is not able to ingest these complex data structures in a natural way.
Instead, ingesting a Kafka topic leads to a flattened table with columns per Kafka Topic. The column names ingested into Collibra are just qualified (Object.Object.Column).

Example:
You have a Kafka topic called “customer” with three fields “customerId”, “firstName” and “lastName”. In addition, the topic contains an “address” object that contains “street”, “city” and “country” fields.

The following assets are created in Collibra:
Table: “customer”
Column: “customerId” (name) , “customer > customerId” (full name)
Column: “firstName” (name) , “customer > firstName” (full name)
Column: “lastName” (name) , “customer > lastName” (full name)
Column: “address.street” (name) , “customer > address.street” (full name)
Column: “address.city” (name) , “customer > address.city”" (full name)
Column: “address.country”" (name) , “customer > address.country” (full name)