Kafka Job set up in Collibra DQ
Hi Team,
I was looking for information about running checks on kafka topics and found this guide : https://dq-docs.collibra.com/dq-job-examples/job-kafka.
I do have one question : do we have any guide around all available parameters? Also is there any description around the meaning of those parameters? I’m thinking about authentification parameters as we might be running those commands on the same server as the kafka server, hence there might be some authentification concerns. Please enlighten me if I’m missing something as I’m not really a kafka specialist.
Thank you,
borisahodikpe
Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago
3 comments
matthewtsui
·4 years ago · EditedTo manage customer and internal expectations: Kafka is available for use as tech preview. The UI is missing helpful wizards to assist in configuration. That said, we are not planning to extend support until April 2022 at the earliest.
Tip: Often, users can connect Collibra DQ to the DB that Kafka dumps into, such that you wouldn’t require Kafka
kirkhaslbeck
·4 years ago · Editedzkp: String,
zkh: String,
ds: String,
topic: String,
lb: Int,
format: String,
delim: String,
kh: String,
kp: String,
kcg: String,
interval: Int,
//Kafka Params
k_bootstrap: String,
k_group_id: String,
k_client_id: String,
k_keydeserializer: String,
k_valuedeserializer: String,
k_ssl_key_pwd: String,
k_ssl_keystore_loc: String,
k_ssl_keystore_pwd: String,
k_ssl_trststore_loc: String,
k_ssl_trststore_pwd: String,
k_auto_offset_rst: String,
k_auto_cmit: String,
k_sasl_krb_svc_nm: String,
k_sasl_mech: String,
k_secproto: String,
k_ssl_enabled_protos: String,
k_ssl_kstore_type: String,
k_ssl_proto: String,
k_ssl_prvdr: String,
k_ssl_tstore_type: String,
k_req_tmout_ms: String,
k_acmit_ms: String,
k_reconn_bckoff_ms: String,
k_retry_bckoff_ms: String,
k_ftch_min_bytes: String,
k_hrtbt_intvl_ms: String,
k_max_part_ftch_bytes: String,
k_sess_tmout_ms: String,
k_conn_max_idle_ms: String,
k_excl_int_topics: String,
k_max_poll_recs: String,
k_part_asgn_stgy: String,
k_rcvbuf_bytes: String,
k_sndbuf_bytes: String,
k_chk_crcs: String,
k_ftch_max_wait_ms: String,
k_intcpr_classes: String,
k_meta_max_age_ms: String,
k_metric_rpts: String,
k_metric_samps: String,
k_metric_sampwinms: String,
k_sasl_krb_kinit: String,
k_sasl_krb_min_relogin: String,
k_sasl_krb_tkt_rnw_jttr: String,
k_sasl_krb_tkt_rnw_wndw_factor: String,
k_ssl_ciphers: String,
k_ssl_epoint_id_algo: String,
k_ssl_kmgr_algo: String,
k_ssl_tmgr_algo: String,
sparkArgMap: Map[String, String],
cmdLine: String,
var header: String,
master: String,
timeSeries: String,
dateFormat: String,
dateColumn: String,
tsk: String,
tsv: String,
key1: String,
key2: String,
key3: String,
val1: String,
val2: String,
val3: String,
tfill: String,
vfill: String,
var schema: String,
max: String,
by: String,
dllb: String,
dlfunc: String,
dlcols: String,
var upperlim: String,
var lowerlim: String,
var skipfirstrow: Boolean,
var iot: Boolean,
var path: String,
val maxlull: Long
borisahodikpe
OP4 years ago · EditedHi @kirk.haslbeck.collibra.com, would you be able to help here? Thank you