© MapR Technologies, confidential
®
© MapR Technologies, confidential
®
Anomaly Detection
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Agenda
•  What is anomaly detection?
•  Some example...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Who I am
•  Ted Dunning, Chief Application Architect...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Who we are
•  MapR makes the technology leading dist...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
What is Anomaly Detection?
•  What just happened tha...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
®
Looks pretty
anomalous
to me
© MapR Technologies, confidential
®
Will the real anomaly
please stand up?
© MapR Technologies, confidential
®
© MapR Technologies, confidential
What Are We Really Doing
•  We want action when some...
© MapR Technologies, confidential
®
A Second Look
© MapR Technologies, confidential
®
A Second Look
99.9%-ile
© MapR Technologies, confidential
®
With Spikes
© MapR Technologies, confidential
®
With Spikes
99.9%-ile including spikes
© MapR Technologies, confidential
®
How Hard Can it Be?
Online
Summarizer 99.9%-ile
t
x > t ? Alarm !
x
© MapR Technologies, confidential
®
© MapR Technologies, confidential
On-line Percentile Estimates
•  Apache Mahout has on...
© MapR Technologies, confidential
®
Spot the Anomaly
Anomaly?
© MapR Technologies, confidential
®
Maybe not!
© MapR Technologies, confidential
®
Where’s Waldo?
This is the real
anomaly
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Normal Isn’t Just Normal
•  What we want is a model ...
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
We Do Windows
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Windows on the World
•  The set of windowed signals ...
© MapR Technologies, confidential
®
Most Common Shapes (for EKG)
© MapR Technologies, confidential
®
Reconstructed signal
Original
signal
Reconstructed
signal
Reconstruction
error
© MapR Technologies, confidential
®
An Anomaly
Original technique for
finding 1-d anomaly works
against reconstruction err...
© MapR Technologies, confidential
®
Close-up of anomaly
Not what you want your
heart to do.
And not what the model
expects...
© MapR Technologies, confidential
®
A Different Kind of Anomaly
© MapR Technologies, confidential
®
Model Delta Anomaly Detection
Online
Summarizer
δ > t ?
99.9%-ile
t
Alarm !
Model
-
+ δ
© MapR Technologies, confidential
®
© MapR Technologies, confidential
The Real Inside Scoop
•  The model-delta anomaly det...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Example: Event Stream (timing)
•  Events of various ...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Poisson Distribution
•  Time between events is expon...
© MapR Technologies, confidential
®
Converting Event Times to Anomaly
99.9%-ile
99.99%-ile
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Example: Event Stream (sequence)
•  The scenario:
– ...
© MapR Technologies, confidential
®
Normal Event Flow
Image-1
Image-2
Image-3
Image-4
login
© MapR Technologies, confidential
®
Phishing Flow
Image-
1
Image-
1
Image-
1
Image-
1
login
Image-
1
Image-
1
Image-
1
Ima...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Key Observations
•  Regardless of exact details, the...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Recap (out of order)
•  Anomaly detection is best do...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Recap
•  Different systems require different models
...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
How Do I Build Such a System
•  The key is to combin...
© MapR Technologies, confidential
®
t
now
Hadoop is Not Very Real-time
Unprocessed
Data
Fully
processed
Latest
full
period...
© MapR Technologies, confidential
®
t
now
Hadoop works
great back here
Storm
works
here
Real-time and Long-time together
B...
© MapR Technologies, confidential
®
© MapR Technologies, confidential
Who I am
•  Ted Dunning, Chief Application Architect...
© MapR Technologies, confidential
®
© MapR Technologies, confidential ®
Upcoming SlideShare
Loading in...5
×

Strata 2014-tdunning-anomaly-detection-140211162923-phpapp01

484

Published on

This talk describes the general architecture common to anomaly detections systems that are based on probabilistic models. By examining several realistic use cases, I illustrate the common themes and practical implementation methods.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
484
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Strata 2014-tdunning-anomaly-detection-140211162923-phpapp01

  1. 1. © MapR Technologies, confidential ® © MapR Technologies, confidential ® Anomaly Detection
  2. 2. © MapR Technologies, confidential ® © MapR Technologies, confidential Agenda •  What is anomaly detection? •  Some examples •  Some generalization •  More interesting examples •  Sample implementation methods
  3. 3. © MapR Technologies, confidential ® © MapR Technologies, confidential Who I am •  Ted Dunning, Chief Application Architect, MapR tdunning@mapr.com tdunning@apache.org @ted_dunning •  Committer, mentor, champion, PMC member on several Apache projects •  Mahout, Drill, Zookeeper others
  4. 4. © MapR Technologies, confidential ® © MapR Technologies, confidential Who we are •  MapR makes the technology leading distribution including Hadoop •  MapR integrates real-time data semantics directly into a system that also runs Hadoop programs seamlessly •  The biggest and best choose MapR –  Google, Amazon –  Largest credit card, retailer, health insurance, telco –  Ping me for info
  5. 5. © MapR Technologies, confidential ® © MapR Technologies, confidential What is Anomaly Detection? •  What just happened that shouldn’t? –  but I don’t know what failure looks like (yet) •  Find the problem before other people see it –  especially customers and CEO’s •  But don’t wake me up if it isn’t really broken
  6. 6. © MapR Technologies, confidential ®
  7. 7. © MapR Technologies, confidential ® Looks pretty anomalous to me
  8. 8. © MapR Technologies, confidential ® Will the real anomaly please stand up?
  9. 9. © MapR Technologies, confidential ® © MapR Technologies, confidential What Are We Really Doing •  We want action when something breaks (dies/falls over/otherwise gets in trouble) •  But action is expensive •  So we don’t want false alarms •  And we don’t want false negatives •  We need to trade off costs
  10. 10. © MapR Technologies, confidential ® A Second Look
  11. 11. © MapR Technologies, confidential ® A Second Look 99.9%-ile
  12. 12. © MapR Technologies, confidential ® With Spikes
  13. 13. © MapR Technologies, confidential ® With Spikes 99.9%-ile including spikes
  14. 14. © MapR Technologies, confidential ® How Hard Can it Be? Online Summarizer 99.9%-ile t x > t ? Alarm ! x
  15. 15. © MapR Technologies, confidential ® © MapR Technologies, confidential On-line Percentile Estimates •  Apache Mahout has on-line percentile estimator –  very high accuracy for extreme tails –  new in version 0.9 !! •  What’s the big deal with anomaly detection? •  This looks like a solved problem
  16. 16. © MapR Technologies, confidential ® Spot the Anomaly Anomaly?
  17. 17. © MapR Technologies, confidential ® Maybe not!
  18. 18. © MapR Technologies, confidential ® Where’s Waldo? This is the real anomaly
  19. 19. © MapR Technologies, confidential ® © MapR Technologies, confidential Normal Isn’t Just Normal •  What we want is a model of what is normal •  What doesn’t fit the model is the anomaly •  For simple signals, the model can be simple … •  The real world is rarely so accommodating x ~ N(0,ε)
  20. 20. © MapR Technologies, confidential ® We Do Windows
  21. 21. © MapR Technologies, confidential ® We Do Windows
  22. 22. © MapR Technologies, confidential ® We Do Windows
  23. 23. © MapR Technologies, confidential ® We Do Windows
  24. 24. © MapR Technologies, confidential ® We Do Windows
  25. 25. © MapR Technologies, confidential ® We Do Windows
  26. 26. © MapR Technologies, confidential ® We Do Windows
  27. 27. © MapR Technologies, confidential ® We Do Windows
  28. 28. © MapR Technologies, confidential ® We Do Windows
  29. 29. © MapR Technologies, confidential ® We Do Windows
  30. 30. © MapR Technologies, confidential ® We Do Windows
  31. 31. © MapR Technologies, confidential ® We Do Windows
  32. 32. © MapR Technologies, confidential ® We Do Windows
  33. 33. © MapR Technologies, confidential ® We Do Windows
  34. 34. © MapR Technologies, confidential ® We Do Windows
  35. 35. © MapR Technologies, confidential ® © MapR Technologies, confidential Windows on the World •  The set of windowed signals is a nice model of our original signal •  Clustering can find the prototypes –  Fancier techniques available using sparse coding •  The result is a dictionary of shapes •  New signals can be encoded by shifting, scaling and adding shapes from the dictionary
  36. 36. © MapR Technologies, confidential ® Most Common Shapes (for EKG)
  37. 37. © MapR Technologies, confidential ® Reconstructed signal Original signal Reconstructed signal Reconstruction error
  38. 38. © MapR Technologies, confidential ® An Anomaly Original technique for finding 1-d anomaly works against reconstruction error
  39. 39. © MapR Technologies, confidential ® Close-up of anomaly Not what you want your heart to do. And not what the model expects it to do.
  40. 40. © MapR Technologies, confidential ® A Different Kind of Anomaly
  41. 41. © MapR Technologies, confidential ® Model Delta Anomaly Detection Online Summarizer δ > t ? 99.9%-ile t Alarm ! Model - + δ
  42. 42. © MapR Technologies, confidential ® © MapR Technologies, confidential The Real Inside Scoop •  The model-delta anomaly detector is really just a mixture distribution –  the model we know about already –  and a normally distributed error •  The output (delta) is (roughly) the log probability of the mixture distribution •  Thinking about probability distributions is good
  43. 43. © MapR Technologies, confidential ® © MapR Technologies, confidential Example: Event Stream (timing) •  Events of various types arrive at irregular intervals –  we can assume Poisson distribution •  The key question is whether frequency has changed relative to expected values •  Want alert as soon as possible
  44. 44. © MapR Technologies, confidential ® © MapR Technologies, confidential Poisson Distribution •  Time between events is exponentially distributed •  This means that long delays are exponentially rare •  If we know λ we can select a good threshold –  or we can pick a threshold empirically Δt ~ λe−λt P(Δt > T) = e−λT −logP(Δt > T) = λT
  45. 45. © MapR Technologies, confidential ® Converting Event Times to Anomaly 99.9%-ile 99.99%-ile
  46. 46. © MapR Technologies, confidential ® © MapR Technologies, confidential Example: Event Stream (sequence) •  The scenario: –  Web visitors are being subjected to phishing attacks –  The hook directs the visitors to a mocked login –  When they enter their credentials, the attacker uses them to log in –  We assume captcha’s or similar are part of the authentication so the attacker has to show the images on the login page to the visitor •  The problem: –  We don’t really know how the attack works
  47. 47. © MapR Technologies, confidential ® Normal Event Flow Image-1 Image-2 Image-3 Image-4 login
  48. 48. © MapR Technologies, confidential ® Phishing Flow Image- 1 Image- 1 Image- 1 Image- 1 login Image- 1 Image- 1 Image- 1 Image- 1 login Fake login Real login
  49. 49. © MapR Technologies, confidential ® © MapR Technologies, confidential Key Observations •  Regardless of exact details, there are patterns •  Event stream per user shows these patterns •  Phishing will have different patterns at much lower rate •  Measuring statistical surprise gives a good anomaly (fraud or malfunction) indicator
  50. 50. © MapR Technologies, confidential ® © MapR Technologies, confidential Recap (out of order) •  Anomaly detection is best done with a probability model •  Deep learning is a neat way to build this model –  converting to symbolic dynamics simplifies life •  -log p is a good way to convert to anomaly measure •  Adaptive quantile estimation works for auto- setting thresholds
  51. 51. © MapR Technologies, confidential ® © MapR Technologies, confidential Recap •  Different systems require different models •  Continuous time-series –  sparse coding or deep learning to build signal model •  Events in time –  rate model base on variable rate Poisson –  segregated rate model •  Events with labels –  language modeling –  hidden Markov models
  52. 52. © MapR Technologies, confidential ® © MapR Technologies, confidential How Do I Build Such a System •  The key is to combine real-time and long-time –  real-time evaluates data stream against model –  long-time is how we build the model •  Extended Lambda architecture is my favorite •  See my other talks on slideshare.net for info •  Ping me directly
  53. 53. © MapR Technologies, confidential ® t now Hadoop is Not Very Real-time Unprocessed Data Fully processed Latest full period Hadoop job takes this long for this data
  54. 54. © MapR Technologies, confidential ® t now Hadoop works great back here Storm works here Real-time and Long-time together Blended view Blended view Blended View
  55. 55. © MapR Technologies, confidential ® © MapR Technologies, confidential Who I am •  Ted Dunning, Chief Application Architect, MapR tdunning@mapr.com tdunning@apache.org @ted_dunning •  Committer, mentor, champion, PMC member on several Apache projects •  Mahout, Drill, Zookeeper others
  56. 56. © MapR Technologies, confidential ® © MapR Technologies, confidential ®
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×