262 Messages
Databricks Metadata through OOTB JDBC driver
Dear all,
The Spark JDBC driver (SparkJDBC42.jar) is unable to capture certain information from the below table structure:
-
table level comment
-
the TBLPROPERTIES key-value pair information
-
PARTITION BY information
However, it captures the column level comment (eg. the comment against employee_number column), all columns of employee table, their technical data types.
Please advise if I need to configure any additional properties to be ale to read/extract the information that the driver could not extract at the moment.
create table default.employee(
employee_number INT COMMENT ‘Unique identifier for an employee’,
employee_name VARCHAR(50),
employee_age INT
PARTITIONED BY (employee_age)
COMMENT ‘this is a table level comment’
TBLPROPERTIES (‘created.by.user’ = ‘Noor’, ‘created.date’ = ‘10-08-2021’);
noor
262 Messages
3 years ago
For the benefit for all -
We raised a ticket with Collibra support on this. Investigation is in progress.
We realized these things:
the driver is issuing “describe database_name.table_name” command on databricks. It should issue “describe table extended database_name.table_name” command to get the extended metadata like - table level comments, partition by information and so on so forth.
On a side note, even if the driver could get this information, where does Collibra store this information in the catalog - against what attribute at table level, is still an open question.
I will update here what the support says.
0
0