Outlier with Date
Use Case (Problem Statement): Client is having a problem with a five day lookback and missing days of data (on weekends and holidays). For example they want to run a five day lookback on Monday, but the missing data from Saturday/Sunday, which is causing the problem. There are only three days of data available for the outliers formula to use in that case and if the minimum history is 4 or 5, this will not work.
Use Case Solution (ideas):
- Extend the number of days of look back to seven or eight (7 | 8). Let’s call this variable “F” -dllb (this originally meant: “deep learning lookback”):
-dllb 7
- Reduce the minimum history to three or four (3 | 4). -dlminhist (“deep learning minimum history”, eg. the minimum, let’s call this variable “H”) observations are in the history, internal model:
-dlminhist 4
Historical:
There are two ways Outliers can be triggered 1. with a date, or 2. without a date.
Outlier without a date:
[x7=45], [x6=50], [x5=64], [x4=63], [x3=45], [x2=99], [x1=54]
Outlier == [x2] (99)
Outlier uses the IQR formula : Wikipedia for IQR
Outliers with a date:
Most recent value (today) is then compared to history of the values, and the question becomes is the MOST RECENT value the outlier in the set. The answer above is if x1=54 is processed today, then the answer is “no,” it is not an outlier in the set.
At customer, because there are not enough data points for the look back of 5 and the minhist of 3 (to satisfy the -dlminhist setting) they are not seeing any outliers.
Proposed Solution:
- Make sure the date is set in Outliers.
- Make the -dllb value is set to greater than 5.
- Make the -dlminhist value set to 3. (minimum is 2)
noor
·5 years ago · EditedOk. Thanks for sharing. Good initiative of sharing tips & tricks for some real world problems.
The workaround makes sense as some customers do not have their nightly batch cycles processed during weekends & holidays. So the business date in a fact table or dimension table will be the working day date only.
noor
·5 years ago · EditedOk. Thanks for sharing. Good initiative of sharing tips & tricks for some real world problems.
The workaround makes sense as some customers do not have their nightly batch cycles processed during weekends & holidays. So the business date in a fact table or dimension table will be the working day date only.
BigDataBear
OP5 years ago · EditedWelcome. Glad you find it useful.