Uploaded byArun Kejariwal
Finding bad apples early: Minimizing performance impact
The document discusses advanced techniques for anomaly detection in high-volume data environments, specifically focusing on real-time information consumption and performance monitoring across microservices and clusters. It outlines a scalable, algorithmic approach for identifying performance regressions and discrepancies in system behavior, leveraging historical time series analysis. The presentation also details applications in production environments, challenges in data consistency, and plans for future enhancements in data granularity and classification.
More Related Content
Similar to Finding bad apples early: Minimizing performance impact
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Finding bad apples early: Minimizing performance impact
- 1. Arun Kejariwal (@arun_kejariwal) 1 Finding BadApples Early: Minimizing Performance Impact
- 2. 2 Real-time information consumption >500M Tweets per day >300 hrs video uploaded to YouTube every minute 69% Y/Y Growth in Data Traffic Mobile E-commerce Mobile Trading
- 3. 3 Monitoring ! Availability, Performance, Reliability Largenumber of micro services Large number of clusters Multiple data centers Millions of time series Data fidelity Anomaly Detection
- 4. 4 Anomaly Detection: Whybother? Performance regression Potentially user impacting Sources Software Inefficiency Hardware Slow Failure Keynote Template
- 5.
- 6. 6 Anomaly Detection Algorithms > 50yrs of Research ✒ image No model fits all Time series analysis Univariate Multivariate Frequency domain analysis Fourier transform Wavelet transform Clustering K-Means
- 7. 📈 7 Flavors of AnomalyDetection Overview Univariate Time Series # Clicks Multi-variate Time Series CPU, Memory Set of Time Series # Tasks Completed
- 8.
- 9. 9 Host level (contd.) KeynoteTemplate Contextual Application Topology Map Hierarchical Datacenter ! Applications ! Services ! Hosts • Automatically discover Developer / Architect’s view of the application - for the Operations team - Framework for system config and context • Real-time, streaming architecture - Keeps up with today’s elastic infrastructure • Scale to 1000s of hosts, 100s of (micro) services • Present evolution of system state over time - DVR-like replay of health, system changes, failures Evolving Needs of Modern Operations
- 10. 10 Large clusters “Slow” nodes FailedNodes Keynote Template Visual analysis?
- 11. 11 ➢ Our Approach Algorithmic:Simple, fast and effective Automated Scalable to millions of time series Keynote Template
- 12. 12 The Approach Bird’s EyeView Input Set of time series Nodes of the same cluster Same metric, e.g., CPU utilization Output Anomalous time series Classification
- 13. 13 📊The Approach (contd.) Metricsof Interest Requests Tasks completed Latency p99 p999 CPU Bytes out (Kafka) rx_bytes TCP retransmits iowait, iowait_max Examples
- 14. 14 ✓The Approach (contd.) Robustagainst anomalies Average Median Examples
- 15. 15 ✓The Approach (contd.) Robustagainst irregular patterns Examples
- 16. 16 The Approach (contd.) Parameters 6hr 1 day Historical window width
- 17. 17 The Approach (contd.) Parameters Large Highprecision Low recall Small Low precision High recall Classification threshold
- 18. 18 The Approach (contd.) Computerolling median time series Compute pairwise distance matrix between time series Distance measure Euclidean Minkowski Compute vector of sum of pairwise distances Classify the time series which satisfy the following as anomalous: Sum(Pairwise Distances)/Median(Pairwise Distances) > threshold Outline
- 19. 19 The Approach (contd.) Inconsistentdata Time series of unequal length Data collection issue Cold start issue How to address? Truncate at right Omit inconsistent timestamps Interpolate Large number of anomalies Non-uniform underlying trend across the time series Edge cases
- 20. 20 Applications In production Hadoop clusters #Tasks completed Datastores Slow writes Micro-services CPU utilization '
- 21. 21 Applications (contd.) ' Hadoop Big elephantin the room Multiple clusters Thousands of nodes Slow nodes -> Long job completion times Potential business impact Finding laggards in a Hadoop cluster
- 22. 22 Applications (contd.) ' Rank nodesin decreasing order of “slowness” Long history window Compute heavy ad-hoc jobs may skew analysis Burst in new jobs Report, via e-mail report, the slowest nodes Subject to an input sensitivity parameter Specified by the SRE Finding laggards in a Hadoop cluster
- 23. 23 Applications (contd.) ' Similar workloadacross nodes Which band should be classified as anomalous? Load imbalance Finding multiple bands
- 24. 24 Applications (contd.) ' Disparate hardware Loadimbalance Finding heterogeneity
- 25. 25 Limitations New nodes comingonline Nodes dying Open problems
- 26. 26 Next steps Support forfiner granularity data Minutely Sub-minutely
- 27. 27 Next steps (contd.) Multi-classclassification Presence of multiple bands
- 28. 28 Next steps (contd.) Cloudand Containers
- 29. 29 For the curious Similaritymeasures Match count based sequence similarity Normalized length of Longest Common Subsequence (LCS) Clustering k-Means, Phased k-means, k-Medoids EM Single-linkage clustering One-class SVM
- 30.