Dataset size or configuration tips for detecting Categorical Outliers

Hi all,

I am trying to detect categorical outliers given the below use case.
As a construction company, I typically deal with vendors in construction type.
I want to catch a non-construction vendor.

Unable to catch the outlier, any tips on how best to configure?

ericgerstner

Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago

4 comments

ericgerstner

OP4 years ago · Edited

Further, to support the discussion on outliers consider the below use case

As noted above, outliers “keys” on a bucket, but organizations may try to understand what happens if the bucket is duplicated in rows. Eg.) I want to look for outliers of a personal sales history, but that sales rep may appear multiple times.

Note that the outlier is looking at individual records. While we can aggregate to say a weeks/days worth of rows, there is not a summation to applied to the daily records - the array is expanded to the timeframe, but the function applied is what we expect a cell to look like.

This is a good oppurtunity to differentiate CDQ from a more common BI Tool. BI Tools aggregate for business insight, in the case below John met his daily average for sales (~15000) - so not a business issue, if accepting the potentially bad data.

Something seems amiss in the data given John has a 0.00 entry. CDQ looks for outliers to check for data quality, and while that will lead to business insight for a DQ operator, one does not insist that the tool will be BI for a sales manager, simply that it will help ensure the data is reliable.

Example, given the use-case thus far, what if upon fixing the data, we see that John had 4,000 sales at 0900AM and 4,000 sales at 1130am which means he only sold 13,000 that day (4000 + 5000 + 4000) – that value would be an issue for the Sales Manager who would see it in his/her BI tool.

So when constructing the DQ checks (outliers patterns etc.), the mindset is DQ and thus outliers will be tailored to consider the cell first before we look to aggregate.

anasalas

·4 years ago · Edited

Hi Eric!

I really like this post. Thank you!

Related to the second and third examples, maybe we can combine the detection of outliers with duplicates or patterns or even rules to avoid that behavior because it is true that the outliers feature doesn’t detect the replication.

What do you think?

Thank you!
Ana.

ericgerstner

OP4 years ago · Edited

Absolutely, in the event the duplicate line items are incorrect, we’d hope patterns call them out.

ericgerstner

OP4 years ago · Edited

After working with the team - key in the Patterns menu does not refer to a PRIMARY KEY rather where to key in for the bucket- so key should NOT be unique, rather the applicable grouping for the pattern.

Also worth noting that a sufficient enough dataset row count change will result in an altered dataset and may influence the outlier.

Please note that outlier configuration ties to RECORDS https://datacitizens.collibra.com/forum/t/dataset-size-or-configuration-tips-for-detecting-records/1438/2