Competitive question - portability of rules and 'streaming' rule maintenance
We are working with a customer very interested in Data Quality. We’ve done a few demos and they liked what they saw so far.
Ran into an obstacle with one team there as that team is using Oracle EDQ and apparently really likes it. Their objection or concern was stated this way:
“Is streaming technology for rule building utilized or how can the solution enable efficient rule maintenance over time and source to target mapping. What is Collibra’s approach to Data Quality and how it relates to the streaming (point-to-point) function used by other tools such as Infogix or Oracle EDQ?”
I couldn’t find much on Infogix beyond them saying they can do point to point or source to target reconciliation. I know Oracle allows ‘triggers’ to call DQ jobs, but I also know that’s well supported via Owl so no gap there.
We’ve talked to them about the ability to configure and manage source to target checks via Owl, as well as talked about the ability to run owlchecks via API / CLI.
They also talked about the ‘portability’ of rule sets, as in, the ability to take a set of rules running against dataset A and port those in one operation to run the same set of rules against dataset B which is another hop in the pipeline. They saw the rule repo and the ability to share rules across datasets that way, but they seemed concerned about that being a one-rule-at-a-time approach.
I’m not sure what they would be referring to beyond this. Has anyone heard this as a question or concern before?
We need to get them a response on this topic this week.
BigDataBear
·5 years ago · EditedHi Adam,
I have these notes on Collibra DQ with Streaming:
Kafka (hidden, but works, not in GA)
A. Timestamp is available.
B. Data:
C. Schema registry is currently not supported.
Dev team planning to expand to AWS Kinesis, Google Cloud Dataflow, and Azure Event Hub as well. Vadim V. said about this point, “These items are not on the plan of record yet. We likely won’t have a timeframe for this until we release Kafka support.”
adamblalock
OP5 years ago · EditedThanks for this… I think the heart of their question though is not whether or not we can run DQ checks against streaming data (such as Kafka), but more about how checks can be ported intelligently between layers in a data pipeline.
Like they mentioned, they perceive that in these other tools, they can take a set of rules running against dataset A (table in source 1) for example, and in a matter of a few clicks or one call, port those rules to all run against dataset B (table in source 2) which is the target, and so on for each layer down the pipeline. I showed them the ability to set up and configure source to target checks, but that is manual (or appears to be in the UI) and is one layer at a time, and doesn’t port the full set of rules to run against each object or layer in the pipeline. THAT I think is the functionality they’re looking for and seeing as a gap if we don’t have.
Hopefully this makes sense.