Downloaded 318 times
![33
Example: Simple exponential forgetting (Normal distribution model)
Define alpha – forgetting factor
Compute initial average, sumOfSquares
using initial samples
For each new sample, x[t]
If |x[t]-average[t-1]|> 3* Stddev[t-1]
Flag x[t] as an anomalous sample
average[t] = alpha*x[t] + (1-alpha)*average[t-1]
sumOfSquares[t] = alpha*x^2 + (1-alpha)*sumOfSquares[t-1]
Stddev[t] = sqrt(sumOfSquares[t] – average[t]^2)](https://image.slidesharecdn.com/anomalydetectionanodotdraft-160707064125/75/Analytics-for-large-scale-time-series-and-event-data-33-2048.jpg)
Uploaded byAnodot
Analytics for large-scale time series and event data
The document discusses the significance of anomaly detection in large-scale analytics, particularly for industrial IoT and business incident detection. It outlines design principles, methods, and the Anodot system for automatic anomaly detection, which involves learning normal and abnormal behaviors from metrics. Key considerations for effective anomaly detection include real-time processing, model adaptability, and the distinction between supervised and unsupervised learning methods.
More Related Content
Similar to Analytics for large-scale time series and event data
![[DevFest Strasbourg 2025] - NodeJs Can do that !!](https://cdn.slidesharecdn.com/ss_thumbnails/devfeststrasbourg2025-nodejscandothat-251127142731-da65b6fd-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...](https://cdn.slidesharecdn.com/ss_thumbnails/md-mobileengineerandsoftwareengineerarewestillrelevantsidiqpermana-251127010650-55224ef1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-digitalaccessibilitywhydevelopersneedtoknowandcarein2025-251127011019-0674441d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-agenticaiarchitectureredefiningsystemcommunication-251124030838-e6c70cc2-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-phpinaiagethelaravelway-251125012602-ef9d330e-thumbnail.jpg?width=640&height=640&fit=bounds)
Analytics for large-scale time series and event data
- 1. 1 Building Anomaly Detection ForLarge Scale Analytics Ira Cohen, Chief Data Scientist 16th May, 2016
- 2. 2 Outline Anomaly detection? Whydo I need it? Design principals for Anomaly Detection What is anomaly detection? Anomaly Detection Methods The Anodot System
- 3.
- 4. 4 Detecting the Unknowns Saves Time + Money Industrial IoT Proactive Maintenance Detecting issues in factories/machines Web Services Detecting business incidents + unknown business opportunities Machine Learning Closing the “Machine Learning” loop Tracking and detecting ”unknowns” not modeled during training Security Detection of unknown breach/attack patterns
- 5. 5 Business Incidents -More go undetected as the business grows $$$$ $$ $ $$
- 6. 6 Detecting Business Incidents:Metric Driven Detection Business Business Generation: Leads, visitors, usage, engagements App: Performance, errors, usability Infra utilization/state: Middleware, network, System e.g., Purchases per product, Conversions per campaign… Per Geo, user segment, page, browser, device… Per class, method, feature… Per host, database, switch…
- 7. 7 Detecting Business Incidents:Metric Driven Detection Drop in # of visitors Decrease in ad conversion on Android Price glitch – increase in purchases / decrease in revenue
- 8. 8 Setting alerts withthresholdsDashboards Manual Detection of Business incidents
- 9. 9 Manual Solutions: Drowningin a “Sea of Data” MISSED INCIDENTS FALSE ALARMS GENUINE ALERTS Too many parameters to set thresholds Too much data to analyze in real time
- 10. 10 What is AnomalyDetection?
- 11.
- 12. 12 Anomaly Detection 12 • Illposed problem • What is an anomaly?
- 13. 13 Anomaly Detection inTime Series Signals Unexpected change of temporal pattern of one or more time series signals.
- 14. 14 Anomaly detection: DesignPrincipals
- 15. 15 Anomaly Detection: DesignConsiderations Timeliness Real time vs. Retroactive Detection Scale 100’s vs. Millions of metrics Rate of change Adaptive vs. Offline learning Conciseness Univariate vs. Multivariate methods Well defined incidents? Supervised vs. Unsupervised methods
- 16. 16 Timeliness: Real timevs. Retroactive Detection Real time decision making Non-real time decision making Reduction in visitors/revenues Check for bugs Increase in product purchase Increase inventory Increase in ad conversion w/o increase in impressions check for fraud Capacity Planning Marketing budget allocations Data Cleaning Scheduled Maintenance
- 17. 17 Timeliness: Real timevs. Retroactive Detection Real time decision making Non-real time decision making Online learning: Cannot iterate over the data More prone to False Positives Scales more easily Batch learning: can iterate over the data Easier to remove False Positives Poor scaling
- 18. 18 Rate of change Constantchange Very slow change • Most common case • ”Closed” systems – e.g., airplanes, large machinery • Requires adaptive algorithms • Learn once and apply the model for a long time
- 19. 19 Conciseness of Anomalies UnivariateAnomaly Detection Multivariate Anomaly Detection • Learn normal model for each metric • Anomaly detection at the metric level • Easier to scale • Causes anomaly storms: Can’t see the forest from the trees • Easier to model many types of behaviors • Learn single model for all metrics • Anomaly detection of complete incident • Hard to scale • Hard to interpret the anomaly • Often requires metric behaviour to be homogeneous Hybrid approach • Learn normal model for each metric • Combine anomalies to single incidents if metrics are related • Scalable • Can combine multiple types of metric behaviours
- 20. 20 Well defined incidents? Yes- Supervised methods No - Unsupervised methods • Requires a well defined set of incidents to identify • Learning a model to classify samples as normal or abnormal • Requires labeled examples of anomalies • Cannot detect new types of incidents • Learning a normal model only • Statistical test to detect anomalies • Can detect any type of anomaly known or unknown Semi-Supervised methods • Use few labelled examples to improve detection of unsupervised methods. • Or – use unsupervised detection for unknown cases, supervised detection to classify already known cases.
- 21.
- 22. 22 Unsupervised Anomaly Detection Generalscheme Step 1 Step 2 Step 3 Model the normal behavior of the metric(s) using a statistical model Devise a statistical test to determine if samples are explained by the model. Apply the test for each sample. Flag as anomaly if it does not pass the test
- 23. 23 Very Simple Model 1σ1σ 2σ2σ 3σ3σ μ 99.7% 95.4% 68% Assumenormal behavior is the Normal distribution Estimate the average, standard deviation over all samples Test: any sample |x-average|> 3*standard deviation is abnormal
- 24. 24 A single modeldoes not fit them all! Smooth (stationary) Irregular sampling Multi Modal Sparse Discrete “Step”
- 25. 25 Metric types distribution Basedon 50,000,000 metrics sampled from dozens of companies Nearly constant, 2% Discrete, 15% Sparse, 3% Multi Modal, 5% Smooth, 38% Irregular sampling, 37% All Industries
- 26. 26 Example: The importanceof modeling seasonality Single seasonal pattern
- 27. 27 Example: The importanceof modeling seasonality Multiple seasonal patterns (“Amplitude modulation”)
- 28. 28 Example: The importanceof modeling seasonality Multiple seasons – Additive signals
- 29. 29 Seasonality Distribution Season: 3hours, 2% Season: 12 hours, 1% Season: 2 hours, 1% Season: 1 hours, 1% Season: 6 hours, 0.5% Season: 4 hours, 0.2% Season: 5 hours, 0.1% Season: 24 hours, 69% Season: Weekly, 26% Note: Only 14% of the metrics have season
- 30. 30 Example Methods todetect seasonality Finding maximums in Auto- correlation of signal Computationally expensive More robust to gaps Finding maximum(s) in Fourier transform of signal Challenging to detect low frequency seasons Challenging to discover multiple seasons Sensitive to missing data Exhaustive search based on cost function Computationally expensive Robust to gaps Challenging to discover multiple seasons
- 31. 31 Real time detection@ scale = Online learning algorithms 1 2 3 Initialize model For each new sample test if anomaly Update model parameters with each new sample
- 32. 32 Example Online Models/Algorithms 4 2 1 3 SimpleMoving Average Double/Triple exponential (Holt- Winters) Kalman Filters + ARIMA and variations Single exponential forgetting
- 33. 33 Example: Simple exponentialforgetting (Normal distribution model) Define alpha – forgetting factor Compute initial average, sumOfSquares using initial samples For each new sample, x[t] If |x[t]-average[t-1]|> 3* Stddev[t-1] Flag x[t] as an anomalous sample average[t] = alpha*x[t] + (1-alpha)*average[t-1] sumOfSquares[t] = alpha*x^2 + (1-alpha)*sumOfSquares[t-1] Stddev[t] = sqrt(sumOfSquares[t] – average[t]^2)
- 34. 34 Update rate withonline models: Avoiding pitfalls What should be the learning rate? Too Slow Too Fast
- 35. 35 Update rate withonline models: Avoiding pitfalls What should be the learning rate? “Al Dente” Auto tuning required!
- 36. 36 Update rate withonline models: Avoiding pitfalls How to update a model when there is an anomaly? Strategy A: Update as usual Most of the anomaly is missed
- 37. 37 Update rate withonline models: Avoiding pitfalls Full anomaly captured How to update a model when there is an anomaly? Strategy B: Adapt the learning rate
- 38. 38 Batch models 1 23 4 Collect historical samples Segment samples to similarly behaving segments Cluster segments according to some similarity measure Mark as anomalies segments that are in small or no clusters
- 39. 39 Example Batch AnomalyDetection Methods Multi-model distributions: • Gaussian models • Generalized mixture models One sided SVM PCA Clustering methods (K-Means, DBScan, Mean- Shift) MOST COMMON IN USE Hidden Markov Models
- 40. 40 Anomaly detection methods- examples NAME ADAPTIVE? REALTIME? SCALABLE? UNI-MULTI VARIATE Holt-Winters Yes Yes Yes Univariate ARIMA + Kalman Yes Yes Yes Both HMM No Yes No Multivariate GMM No No No Both DBScan No No No Multivariate K-Means No No No Multivariate
- 41. 41 Large scale anomalydetection – the Anodot system
- 42. 42 Automatic Anomaly Detectionin five Steps: The Anodot Way Metrics Collection – Universal, scale to millions Normal behavior learning Abnormal behavior learning Behavioral Topology Learning Feedback Based Learning 1 2 3 4 5
- 43. 43 Large Scale AnomalyDetection System Architecture Kafka Events Queue Anomaly Grouping Signals Correlation Map Real-Time Rollups Store Cassandra Anodotd REST WebApp Online Base Line Learning Aggregator Elasticsearch DWH S3 HADOOP HIVE Offline Learning Management & Portal Anodot-Web User Mgmt RDBMS Customer DS Agent
- 44.