DQ Rules
Top Links
Rules / General: Popular Questions
Q: Can You Select Rules Altogether And Apply To Dataset?
A: Yes, Select One By One For Now; Currently We Are Doing Custom But Productizing That Feature Is In Our Backlog
Q: Is there a library of OOTB rules?
A: Yes, please refer to the Rule Repo.
Q: Can You Weight Importance Of Rules?
A: Yes, You Can Tune And Assign Points
Q: How To Delete Rules?
A: Rule Page -> Bottom Panel -> Second Tab Rules Definition -> Far Right Of Table Contains Delete Icon. Please Note: Scroll Bar May Be Tucked On Far Right.
Rules Preview / Testing
Q: Can Data Producers Test DQ Rules Prior To Scheduling Execution?
A: Yes, users can validate their syntax with our validation tool. This runs on a subset of the data, not entire population.
Q: Can I test rules in real-time?
A: Roadmap item, coming soon! Q4 2021
Rules / Import Export
Q: Can One Export / Import Rules?
A: Yes, SQL-Based Rules; Very Common; Some Uploaded Existing Spreadsheets And Could Immediately Thousands Of Rules
Rules / Autogenerated
Q: For Autogenerated Rules, Are You Using ISO Currency Codes / Industry Standard Values? Are You Cross-Checking With Zip Codes?
A: You have option to build RegEx but right now, we are not integrating with external APIs and data sources. Users would need to onboard the data set and act as cross reference.
Q: Do Rules Change Over Time?
A: Yes, They Continuously / Incrementally Adapt; Users Can Also Intervene / Give Feedback; Baseline Gets Re-Calibrated With Every Scan
Q: Is there a suppress option to disable an autogenerated rule completely?
A: Yes.
Dimensions
Q: Can we see rules by dimension?
A: Yes, users can tag dimensions. Currently we have 6 dimensions: Accuracy, Completeness, Consistency, Timeliness, Uniqueness, Validity.
Integration
Q: Can You Propose A Rule In Collibra?
A: Future roadmap item.
Machine Learning
Q: How Are The Rules Automatically Built?
A: We Have Some Proprietary ML Models, Some Off The Shelf Libraries To Autogenerate The Rules. However, We Always Recommend Staying Focused On Our Features And How They Can Solve Use Cases / Problems And Spend Less Time Down The ML Rabbit Hole.
System Load
Q: Can You Minimize CPU / Memory Load Of Rule Execution On Hardware?
A: Yes, Rules Can Be Run As Push Down Or Spark Options. User Can Elect Where Runtime Occurs To Minimize Load On Hardware
Rules / Execution
Q: Rules Must Be Executed Near Real-Time TO Data Arrival?
A: Yes, DQ Scans Can Be Part Of Orchestration. DQ Scan Can Be Invoked From REST Or Cmd Line As Part Of Trigger Or Job Control System)
Auditing: How to check whether a rule was created / updated / by who / when?
Q: Can One Audit Rule Creation / Modification To A User Level? Can One Receive An Alert When Rule Is Modified? A: Yes, Rules -> Rules -> Click ‘+’ Sign, See Audit Trail Below
RegEx: Guidance on creating RegEx rules
Q: Any advice with regards to creating complex RegEx rules for customers?
A: https://regex-generator.olafneumann.org/ In Conjunction With Collibra DQ’s RegEx Builder
kirkhaslbeck
·5 years ago · EditedYes. you can use as many DB /SQL alias columns as you like and they will have DQ. meaning you can concat or sum or simply copy a name twice. As long as it looks like a column it will flow through like a column.
for example if you have loan_data with cols
loan_amount, house_value, credit_score
and you wanted to create a column ltv loan_to_value_ratio. You could say loan_amount/house_value as ltv and then have data quality on loan to value ratios.
Lexanne Painter
·5 years ago · EditedA question I received from a customer and wanted to get some thoughts:
Can I make a target table column alias and expect to see that alias name column in the metadata result of the DQ check?
TIA!
BigDataBear
·5 years ago · EditedIf you RENAME a Collibra DQ Dataset, the rules associated with that dataset are dropped from the dataset, because the rules are associated with a dataset. This can be a painful realization so be careful with renaming a dataset.
BigDataBear
·5 years ago · EditedI wrote this rule to find non-latin characters in a name string, for a customer. I tested it and it works, but it may not work for all edge cases: Create Template WHERE = $colNm not rlike ‘[a-zA-Z\u00C0-\u017F\s]+’