Strata 2014-tdunning-anomaly-detection-140211162923-phpapp01

  • 346 views
Uploaded on

This talk describes the general architecture common to anomaly detections systems that are based on probabilistic models. By examining several realistic use cases, I illustrate the common themes and …

This talk describes the general architecture common to anomaly detections systems that are based on probabilistic models. By examining several realistic use cases, I illustrate the common themes and practical implementation methods.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
346
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
10
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. © MapR Technologies, confidential ® © MapR Technologies, confidential ® Anomaly Detection
  • 2. © MapR Technologies, confidential ® © MapR Technologies, confidential Agenda •  What is anomaly detection? •  Some examples •  Some generalization •  More interesting examples •  Sample implementation methods
  • 3. © MapR Technologies, confidential ® © MapR Technologies, confidential Who I am •  Ted Dunning, Chief Application Architect, MapR tdunning@mapr.com tdunning@apache.org @ted_dunning •  Committer, mentor, champion, PMC member on several Apache projects •  Mahout, Drill, Zookeeper others
  • 4. © MapR Technologies, confidential ® © MapR Technologies, confidential Who we are •  MapR makes the technology leading distribution including Hadoop •  MapR integrates real-time data semantics directly into a system that also runs Hadoop programs seamlessly •  The biggest and best choose MapR –  Google, Amazon –  Largest credit card, retailer, health insurance, telco –  Ping me for info
  • 5. © MapR Technologies, confidential ® © MapR Technologies, confidential What is Anomaly Detection? •  What just happened that shouldn’t? –  but I don’t know what failure looks like (yet) •  Find the problem before other people see it –  especially customers and CEO’s •  But don’t wake me up if it isn’t really broken
  • 6. © MapR Technologies, confidential ®
  • 7. © MapR Technologies, confidential ® Looks pretty anomalous to me
  • 8. © MapR Technologies, confidential ® Will the real anomaly please stand up?
  • 9. © MapR Technologies, confidential ® © MapR Technologies, confidential What Are We Really Doing •  We want action when something breaks (dies/falls over/otherwise gets in trouble) •  But action is expensive •  So we don’t want false alarms •  And we don’t want false negatives •  We need to trade off costs
  • 10. © MapR Technologies, confidential ® A Second Look
  • 11. © MapR Technologies, confidential ® A Second Look 99.9%-ile
  • 12. © MapR Technologies, confidential ® With Spikes
  • 13. © MapR Technologies, confidential ® With Spikes 99.9%-ile including spikes
  • 14. © MapR Technologies, confidential ® How Hard Can it Be? Online Summarizer 99.9%-ile t x > t ? Alarm ! x
  • 15. © MapR Technologies, confidential ® © MapR Technologies, confidential On-line Percentile Estimates •  Apache Mahout has on-line percentile estimator –  very high accuracy for extreme tails –  new in version 0.9 !! •  What’s the big deal with anomaly detection? •  This looks like a solved problem
  • 16. © MapR Technologies, confidential ® Spot the Anomaly Anomaly?
  • 17. © MapR Technologies, confidential ® Maybe not!
  • 18. © MapR Technologies, confidential ® Where’s Waldo? This is the real anomaly
  • 19. © MapR Technologies, confidential ® © MapR Technologies, confidential Normal Isn’t Just Normal •  What we want is a model of what is normal •  What doesn’t fit the model is the anomaly •  For simple signals, the model can be simple … •  The real world is rarely so accommodating x ~ N(0,ε)
  • 20. © MapR Technologies, confidential ® We Do Windows
  • 21. © MapR Technologies, confidential ® We Do Windows
  • 22. © MapR Technologies, confidential ® We Do Windows
  • 23. © MapR Technologies, confidential ® We Do Windows
  • 24. © MapR Technologies, confidential ® We Do Windows
  • 25. © MapR Technologies, confidential ® We Do Windows
  • 26. © MapR Technologies, confidential ® We Do Windows
  • 27. © MapR Technologies, confidential ® We Do Windows
  • 28. © MapR Technologies, confidential ® We Do Windows
  • 29. © MapR Technologies, confidential ® We Do Windows
  • 30. © MapR Technologies, confidential ® We Do Windows
  • 31. © MapR Technologies, confidential ® We Do Windows
  • 32. © MapR Technologies, confidential ® We Do Windows
  • 33. © MapR Technologies, confidential ® We Do Windows
  • 34. © MapR Technologies, confidential ® We Do Windows
  • 35. © MapR Technologies, confidential ® © MapR Technologies, confidential Windows on the World •  The set of windowed signals is a nice model of our original signal •  Clustering can find the prototypes –  Fancier techniques available using sparse coding •  The result is a dictionary of shapes •  New signals can be encoded by shifting, scaling and adding shapes from the dictionary
  • 36. © MapR Technologies, confidential ® Most Common Shapes (for EKG)
  • 37. © MapR Technologies, confidential ® Reconstructed signal Original signal Reconstructed signal Reconstruction error
  • 38. © MapR Technologies, confidential ® An Anomaly Original technique for finding 1-d anomaly works against reconstruction error
  • 39. © MapR Technologies, confidential ® Close-up of anomaly Not what you want your heart to do. And not what the model expects it to do.
  • 40. © MapR Technologies, confidential ® A Different Kind of Anomaly
  • 41. © MapR Technologies, confidential ® Model Delta Anomaly Detection Online Summarizer δ > t ? 99.9%-ile t Alarm ! Model - + δ
  • 42. © MapR Technologies, confidential ® © MapR Technologies, confidential The Real Inside Scoop •  The model-delta anomaly detector is really just a mixture distribution –  the model we know about already –  and a normally distributed error •  The output (delta) is (roughly) the log probability of the mixture distribution •  Thinking about probability distributions is good
  • 43. © MapR Technologies, confidential ® © MapR Technologies, confidential Example: Event Stream (timing) •  Events of various types arrive at irregular intervals –  we can assume Poisson distribution •  The key question is whether frequency has changed relative to expected values •  Want alert as soon as possible
  • 44. © MapR Technologies, confidential ® © MapR Technologies, confidential Poisson Distribution •  Time between events is exponentially distributed •  This means that long delays are exponentially rare •  If we know λ we can select a good threshold –  or we can pick a threshold empirically Δt ~ λe−λt P(Δt > T) = e−λT −logP(Δt > T) = λT
  • 45. © MapR Technologies, confidential ® Converting Event Times to Anomaly 99.9%-ile 99.99%-ile
  • 46. © MapR Technologies, confidential ® © MapR Technologies, confidential Example: Event Stream (sequence) •  The scenario: –  Web visitors are being subjected to phishing attacks –  The hook directs the visitors to a mocked login –  When they enter their credentials, the attacker uses them to log in –  We assume captcha’s or similar are part of the authentication so the attacker has to show the images on the login page to the visitor •  The problem: –  We don’t really know how the attack works
  • 47. © MapR Technologies, confidential ® Normal Event Flow Image-1 Image-2 Image-3 Image-4 login
  • 48. © MapR Technologies, confidential ® Phishing Flow Image- 1 Image- 1 Image- 1 Image- 1 login Image- 1 Image- 1 Image- 1 Image- 1 login Fake login Real login
  • 49. © MapR Technologies, confidential ® © MapR Technologies, confidential Key Observations •  Regardless of exact details, there are patterns •  Event stream per user shows these patterns •  Phishing will have different patterns at much lower rate •  Measuring statistical surprise gives a good anomaly (fraud or malfunction) indicator
  • 50. © MapR Technologies, confidential ® © MapR Technologies, confidential Recap (out of order) •  Anomaly detection is best done with a probability model •  Deep learning is a neat way to build this model –  converting to symbolic dynamics simplifies life •  -log p is a good way to convert to anomaly measure •  Adaptive quantile estimation works for auto- setting thresholds
  • 51. © MapR Technologies, confidential ® © MapR Technologies, confidential Recap •  Different systems require different models •  Continuous time-series –  sparse coding or deep learning to build signal model •  Events in time –  rate model base on variable rate Poisson –  segregated rate model •  Events with labels –  language modeling –  hidden Markov models
  • 52. © MapR Technologies, confidential ® © MapR Technologies, confidential How Do I Build Such a System •  The key is to combine real-time and long-time –  real-time evaluates data stream against model –  long-time is how we build the model •  Extended Lambda architecture is my favorite •  See my other talks on slideshare.net for info •  Ping me directly
  • 53. © MapR Technologies, confidential ® t now Hadoop is Not Very Real-time Unprocessed Data Fully processed Latest full period Hadoop job takes this long for this data
  • 54. © MapR Technologies, confidential ® t now Hadoop works great back here Storm works here Real-time and Long-time together Blended view Blended view Blended View
  • 55. © MapR Technologies, confidential ® © MapR Technologies, confidential Who I am •  Ted Dunning, Chief Application Architect, MapR tdunning@mapr.com tdunning@apache.org @ted_dunning •  Committer, mentor, champion, PMC member on several Apache projects •  Mahout, Drill, Zookeeper others
  • 56. © MapR Technologies, confidential ® © MapR Technologies, confidential ®