DutchMLSchool 2022 - Anomaly Detection at Scale

July 4 - 6, 2022
2 n d E d i t i o n

BigML, Inc #DutchMLSchool
Anomaly Detection at Scale
Lessons Learned deploying thousands of Anomaly Detectors
Alvaro Clemente

Machine Learning Engineer, BigML
2

Agenda
3

BigML, Inc #DutchMLSchool 4
Agenda
Anomaly Detection Primer
1
Lessons
2
Conclusion
3

Anomaly Detection in a Nutshell
5

Identify the anomalies

1
2
3
4
6
5 7
8
9

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
YES

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
YES *

Identify examples that are different
from the rest of the dataset

Identify examples that are different from the rest of the dataset
• Dataset cleaning: Remove instances that are not
representative of your dataset
• Exploring Data: Identifying outlier situations in your
dataset

• Classification: working with very unbalanced data
or uncertainty

Anomaly Detectors for Classification
• Fraud Detection: Detecting money laundering
and fraud in bank transactions

• Intrusion Detection: Detecting unexpected
events in network traffic

• Quality Control: Detecting failures in
manufacturing processes

Lessons
13

Design
Domain
Training
Operation
• You don’t need Anomaly Detection

• Divide and Conquer
• Data Cleaning for free

• Automatic Experts

• Missing the forest for the trees
• Features, Features, Features

• Setup a Feedback Loop

• Customize Thresholds
• You can’t evaluate it!

• Adapt to new times

• Size matters

Lesson 1: You don’t need
 
Anomaly Detection
Design

Lesson 1: You don’t need Anomaly Detection
• You are interested in the unusual case

• You have very few examples of the
interesting class

• You can’t have fully labeled datasets

• The class shows unexpected
behaviors
When possible, other methods will give better control over the performance

Lesson 2: Divide and Conquer
Design

Lesson 2: Divide and Conquer
• Identify the different tasks in your problem domain

• Build a model trained with data from that specific
task

• Even with different features
• Each model will be easier to track and reason about
Prefer multiple simpler models over a single complex one

Lesson 3: Data cleaning for free
Domain

Lesson 3: Data cleaning for free
• Find issues in your data pipelines

• Have a fast feedback loop for reporting issues

• Have an off-switch when those issues are detected
Anomaly Detectors will find data issues

Lesson 4: Automatic Experts
Domain

Lesson 4: Automatic Experts
Use expert rules to check predictions
• A combination of anomalies and rules
will yield the best results

• You can automate your rules with a
model

• Train a model to detect False Positives

• This requires even more data

Lesson 5: Don’t miss the forest
 
for the trees
Domain

Lesson 5: Don’t miss the forest for the trees
Anomaly patterns contain very interesting information!
• Looking how the anomalies happen over time can reveal
very useful information

• Random failures → Random anomalies

• Significant events → Groups of anomalies

• Macro Alerts

Lesson 6: Features, Features,
Features
Training

Lesson 6: Features, Features, Features
Use the features to tune model behavior
• Feature engineering and selection is one of the
2 main ways to tune the model behavior

• Keep the number of features to a minimum

• Keep it explainable

Lesson 7: Setup a Feedback Loop
Training

Lesson 7: Setup a Feedback Loop
Setup a Feedback Loop for tuning model behavior
• Keep a database of predictions and outcomes

• Usually requires human inspection

• Monitor performance of the models for tuning

• With BigML*, you can update your models with
new data
* currently only available in private deployments

Lesson 8: Customize Thresholds
Training

Lesson 8: Customize Thresholds
Customize the thresholds to your requirements and data
• Anomaly can be a fuzzy and subjective concept

• Use multiple thresholds
• Low, medium and High
• Use dynamic thresholds
• Data driven

Lesson 9: You can’t evaluate it
Operation

Lesson 9: You can’t evaluate it!
Evaluating these models is complicated
• Evaluation will not be that simple

• Lack of information

• Macro events affect the individual performance

• Precision and Recall don’t translate so well to these kinds of problems

• Find some useful and realistic evaluation metrics

• Indirect metrics, business metrics (i.e: recall rates on cars)

• Manual exploration of random samples of data

Lesson 10: Adapt to new times
Operation

Lesson 10: Adapt to new times
Anomaly Detectors are very sensitive to changes in the working conditions
• Anomaly Detectors are very sensitive to
changes in the working conditions

• Model quality will deteriorate over time faster than
with other models

• Monitor the model performance

• Retrain often

• Find change indicators and disable proactively

Lesson 11: Size matters
Operation

Lesson 11: Size matters
Storage and caching for Anomaly Detectors is key for fast predictions
• You will be deploying a lot of these models

• Anomaly Detectors can be heavy

• Efficient storage, transport and loading will be key
• Near real time scenarios and distributed prediction

• Use efficient representations of the models

• Minomaly

Conclusion
37

Solve problems that are impractical
with traditional methods

Discover unexpected facts about
your systems

Q & A
40

MLToolbox Context
42

BODY SHOP PAINT SHOP ASSEMBLY SHOP
Cost of
fi
xing a
welding failure: $
Cost of
fi
xing a
welding failure: $$$
From Data to Real Time Alerts
Using ML to improve Quality Control

Detect as many failures as possible, while keeping a manageable number of car
extractions

A case for Anomaly Detectors
• Large amounts of data processed in real time

• 300,000 welds / day

• Extremely few examples of failures

• 1 / 15,000 welds fails (0.0067%!)

• Failures can have unexpected shapes

• Hundreds of machines doing slightly different tasks

• Unreliable labels

DutchMLSchool 2022 - Anomaly Detection at Scale

More Related Content

Similar to DutchMLSchool 2022 - Anomaly Detection at Scale

More from BigML, Inc

Recently uploaded

DutchMLSchool 2022 - Anomaly Detection at Scale