Functional: DQ Rules
Rules / General: Popular Questions
Q: Can You Select Rules Altogether And Apply To Dataset?
A: Yes, Select One By One For Now; Currently We Are Doing Custom But Productizing That Feature Is In Our Backlog
Q: Is there a library of OOTB rules?
A: Yes, please refer to the Rule Library
Q: Can You Weight Importance Of Rules?
A: Yes, You Can Tune And Assign Points
Q: How To Delete Rules?
A: Rule Page -> Bottom Panel -> Second Tab Rules Definition -> Far Right Of Table Contains Delete Icon. Please Note: Scroll Bar May Be Tucked On Far Right.
Rules Preview / Testing
Q: Can Data Producers Test DQ Rules Prior To Scheduling Execution?
A: Yes, users can validate their syntax with our validation tool. This runs on a subset of the data, not entire population.
Q: Can I test rules in real-time?
A: Roadmap item, ETA 1H 2022
Rules / Import Export
Q: Can One Export / Import Rules?
A: Yes, SQL-Based Rules; Very Common; Some Uploaded Existing Spreadsheets And Could Immediately Thousands Of Rules
Rules / Autogenerated
Q: For Autogenerated Rules, Are You Using ISO Currency Codes / Industry Standard Values? Are You Cross-Checking With Zip Codes?
A: You have option to build RegEx but right now, we are not integrating with external APIs and data sources. Users would need to onboard the data set and act as cross reference.
Q: Do Rules Change Over Time?
A: Yes, They Continuously / Incrementally Adapt; Users Can Also Intervene / Give Feedback; Baseline Gets Re-Calibrated With Every Scan
Q: Is there a suppress option to disable an autogenerated rule completely?
A: Yes.
Dimensions
Q: Can we see rules by dimension?
A: Yes, users can tag dimensions. Currently we have 6 dimensions: Accuracy, Completeness, Consistency, Timeliness, Uniqueness, Validity.
A: Additional dimensions ETA 1H 2022
Integration
Q: Can I synchronize DQ Rules from Collibra DQ in Collibra DIC?
A: Yes! https://dq-docs.collibra.com/integration/dq-connector
Q: Can You Propose A Rule In Collibra?
A: Future roadmap item.
Machine Learning
Q: How Are The Rules Automatically Built?
A: We Have Some Proprietary ML Models, Some Off The Shelf Libraries To Autogenerate The Rules. However, We Always Recommend Staying Focused On Our Features And How They Can Solve Use Cases / Problems And Spend Less Time Down The ML Rabbit Hole.
System Load
Q: Can You Minimize CPU / Memory Load Of Rule Execution On Hardware?
A: Yes, Rules Can Be Run As Push Down Or Spark Options. User Can Elect Where Runtime Occurs To Minimize Load On Hardware
Rules / Execution
Q: Rules Must Be Executed Near Real-Time To Data Arrival?
A: Yes, DQ Scans Can Be Part Of Orchestration. DQ Scan Can Be Invoked From REST Or Cmd Line As Part Of Trigger Or Job Control System)
Auditing: How to check whether a rule was created / updated / by who / when?
Q: Can One Audit Rule Creation / Modification To A User Level? Can One Receive An Alert When Rule Is Modified? A: Yes, Rules -> Rules -> Click ‘+’ Sign, See Audit Trail Below
RegEx: Guidance on creating RegEx rules
Q: Any advice with regards to creating complex RegEx rules for customers?
A: https://regex-generator.olafneumann.org/ In Conjunction With Collibra DQ’s RegEx Builder
Q: What scripting language is used to write RegEx in Collibra DQ?
A: Collibra DQ uses SparkSQL regex which is
Roughly identical to Java RegEx
kathybarton
·4 years ago · Edited@matthew.tsui.collibra.com Is there a way to create a rule that checks the % populated for a specific column? We have a requirement that some columns have “some” data in them but they can have some NULL/EMPTY values. How can I check if a population % threshold has been missed?