Detecting Anomalies in Streaming Data
•
2 likes•675 views
This document discusses anomaly detection in streaming data using Hierarchical Temporal Memory (HTM). It describes how HTM can be used to build a real-time anomaly detection system that continuously learns and predicts patterns in streaming data. It also introduces the Numenta Anomaly Benchmark (NAB), an open benchmark for evaluating streaming anomaly detection algorithms that contains labeled real-world data streams and a scoring system that rewards early detection of anomalies. Several state-of-the-art algorithms are shown to perform well on NAB, including HTM, and a competition is announced to expand NAB with more data and algorithms.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
Similar to Detecting Anomalies in Streaming Data
Similar to Detecting Anomalies in Streaming Data (20)
Recently uploaded
Recently uploaded (20)
Detecting Anomalies in Streaming Data
- 1. DETECTING ANOMALIES IN STREAMING DATA Data By The Bay May 19, 2016 Subutai Ahmad @SubutaiAhmad sahmad@numenta.com
- 2. OUTLINE • Real-time streaming analytics • Anomaly detection with Hierarchical Temporal Memory • Benchmarking real-time anomaly detection • Summary
- 3. Monitoring IT infrastructure Uncovering fraudulent transactions Tracking vehicles Real-time health monitoring Monitoring energy consumption Detection is necessary, but prevention is often the goal REAL-TIME ANOMALY DETECTION • Exponential growth in IoT, sensors and real-time data collection is driving an explosion of streaming data • The biggest application for machine learning is anomaly detection
- 5. EXAMPLE: PREVENTIVE MAINTENANCE Planned shutdown Behavioral change preceding failure Catastrophic failure
- 6. THE STREAMING ANALYTICS PROBLEM Given all past input and current input, decide whether the system behavior is anomalous right now. Must report decision, perform any retraining, bookkeeping, etc. before next input arrives. No look-ahead No training/test set split – everything must be done online System must be automated, and customized to each stream
- 7. HIERARCHICAL TEMPORAL MEMORY (HTM) • Powerful sequence memory derived from recent findings in experimental neuroscience • High capacity memory based system • Models temporal sequences in data • Inherently streaming • Continuously learning and predicting • No need to tune hyper-parameters • Open source: github.com/numenta
- 8. HTM PREDICTS FUTURE INPUT • Input to the system is a stream of data • Encoded into a sparse high dimensional vector • Learns temporal sequences in input stream and makes a prediction in the form of a sparse vector • represents a prediction for upcoming input HTM
- 9. ANOMALY DETECTION WITH HTM HTM Raw anomaly score Anomaly likelihood is an instantaneous measure of prediction error • 0 if input was perfectly prediction • 1 if it was completely unpredicted • Could threshold it directly to report anomalies, but in very noisy environments we can do better
- 10. ANOMALY LIKELIHOOD • Second order measure: did the predictability of the metric change? 1. Estimate historical distribution of anomaly scores 2. Check if recent scores are very different
- 11. ANOMALY LIKELIHOOD • Second order measure: did the predictability of the metric change? 1. Estimate historical distribution of anomaly scores 2. Check if recent scores are very different
- 12. ANOMALY DETECTION WITH HTM HTM Raw anomaly score Anomaly likelihood Learns temporal sequences Continuously makes predictions Continuously learning Was current input predicted? Has level of predictability changed significantly?
- 13. ANOMALIES IN IT INFRASTRUCTURE • Grok • Commercial server based product detects anomalies in IT infrastructure • Runs thousands of HTM anomaly detectors in real time • 10 milliseconds per input per metric, including continuous learning • No parameter tuning required • http://grokstream.com
- 14. ANOMALIES IN FINANCIAL DATA • HTM for Stocks • Real-time free demo application • Continuously monitors top 200 stocks • Available on iOS App Store or Google Play Store • Open source application: github.com/numenta/numenta-apps
- 15. OUTLINE • Real-time streaming analytics • Anomaly detection with Hierarchical Temporal Memory • Benchmarking real-time anomaly detection • Summary
- 16. EVALUATING STREAMING ANOMALY DETECTION • Most existing benchmarks are designed for batch data, not streaming data • Hard to find benchmarks containing real world data labeled with anomalies • There is a need for an open benchmark designed to test real-time anomaly detection • A standard community benchmark could spur innovation in streaming anomaly detection algorithms
- 17. NUMENTA ANOMALY BENCHMARK (NAB) • NAB: a rigorous benchmark for anomaly detection in streaming applications
- 18. NUMENTA ANOMALY BENCHMARK (NAB) • NAB: a rigorous benchmark for anomaly detection in streaming applications • Real-world benchmark data set • 58 labeled data streams (47 real-world, 11 artificial streams) • Total of 365,551 data points
- 19. NUMENTA ANOMALY BENCHMARK (NAB) • NAB: a rigorous benchmark for anomaly detection in streaming applications • Real-world benchmark data set • 58 labeled data streams (47 real-world, 11 artificial streams) • Total of 365,551 data points • Scoring mechanism • Rewards early detection • Different “application profiles”
- 20. NUMENTA ANOMALY BENCHMARK (NAB) • NAB: a rigorous benchmark for anomaly detection in streaming applications • Real-world benchmark data set • 58 labeled data streams (47 real-world, 11 artificial streams) • Total of 365,551 data points • Scoring mechanism • Rewards early detection • Different “application profiles” • Open resource • AGPL repository contains data, source code, and documentation • github.com/numenta/NAB • Ongoing competition to expand NAB
- 21. EXAMPLE: HOURLY SERVICE DEMAND Spike in demand Unusually low demand
- 22. EXAMPLE: PRODUCTION SERVER CPU Spiking behavior becomes the new norm Spike anomaly
- 23. HOW SHOULD WE SCORE ANOMALIES? • The perfect detector • Detects anomalies as soon as possible • Provides detections in real time • Triggers no false alarms • Requires no parameter tuning • Automatically adapts to changing statistics • Scoring methods in traditional benchmarks are insufficient • Precision/recall does not incorporate importance of early detection • Artificial separation into training and test sets does not handle continuous learning • Batch data files allow look ahead and multiple passes through the data
- 24. WHERE IS THE ANOMALY?
- 25. NAB DEFINES ANOMALY WINDOWS NAB scoring function gives higher score to earlier detections in window
- 26. OTHER DETAILS • Application profiles • Three application profiles assign different weightings based on the tradeoff between false positives and false negatives. • EKG data on a cardiac patient favors False Positives. • IT / DevOps professionals hate False Positives. • Three application profiles: standard, favor low false positives, favor low false negatives. • NAB emulates practical real-time scenarios • Look ahead not allowed for algorithms. Detections must be made on the fly. • No separation between training and test files. Invoke model, start streaming, and go. • No batch parameter tuning. Must be fully automated with single set of parameters across data streams. Any further parameter tuning must be done on the fly.
- 27. TESTING ALGORITHMS WITH NAB • NAB is designed to easily plug in and test new algorithms • Results with several algorithms: • Hierarchical Temporal Memory • Etsy Skyline • Popular open source anomaly detection technique • Mixture of statistical experts, continuously learning • Twitter ADVec • Open source anomaly detection released last year • Robust outlier statistics + piecewise approximation • Bayesian Online Change Point Detection • Formal Bayesian method for detecting anomalies in time series
- 28. NAB V1.0 RESULTS (58 FILES)
- 29. DETECTION RESULTS: CPU USAGE ON PRODUCTION SERVER Simple spike, all 3 algorithms detect Shift in usage Etsy Skyline Numenta HTM Twitter ADVec Red denotes False Positive Key
- 30. DETECTION RESULTS: MACHINE TEMPERATURE READINGS HTM detects purely temporal anomaly Etsy Skyline Numenta HTM Twitter ADVec Red denotes False Positive Key All 3 detect catastrophic failure
- 31. DETECTION RESULTS: TEMPORAL CHANGES IN BEHAVIOR OFTEN PRECEDE A LARGER SHIFT HTM detects anomaly 3 hours earlier Etsy Skyline Numenta HTM Twitter ADVec Red denotes False Positive Key
- 32. NAB COMPETITION!! • NAB is a resource for the streaming analytics community • Need additional real-world data files and more algorithms tested • NAB Competition offers cash prizes for: • Additional anomaly detection algorithms tested on NAB • Submission of real-world data files with labeled real anomalies • Cash prizes of $2,500 each for algorithms and data • Easy to enter, high likelihood of winning! • Go to http://numenta.org/nab for details
- 33. SUMMARY • Anomaly detection for streaming data imposes unique challenges • Stringent real-time constraints and automation requirements • Typical batch methodologies do not work well • HTM learning algorithms • Can be used to create a streaming anomaly detection system • Performs very well across a wide range of datasets • Open source, commercially deployable • NAB is an open source benchmark for streaming anomaly detection • Includes a labeled dataset with real world data • Scoring methodology designed for practical real-time applications • NAB competition!
- 34. RESOURCES Grok (anomalies in IT infrastructure): http://grokstream.com HTM Studio (desktop app for easy experimentation): contact me Open Source Repositories: Algorithm code: https://github.com/numenta/nupic HTM Stocks demo: https://github.com/numenta/numenta-apps NAB code + paper: https://github.com/numenta/nab Apache Flink: https://github.com/nupic-community/flink-htm Contact info: Subutai Ahmad sahmad@numenta.com, @SubutaiAhmad Alex Lavin alavin@numenta.com, @theAlexLavin