Netflix strives to provide an amazing experience to each member. To accomplish this, Netflix needs to maintain very high availability across our systems. However, at a certain scale, humans can no longer scale their ability to monitor the status of all systems, making it critical for Netflix to build tools and platforms that can automatically monitor their production environments and make intelligent real-time operational decisions to remedy the problems they identify. In this session, we discuss how Netflix uses data mining and machine learning techniques to automate decisions in real-time with the goal of supporting operational availability, reliability, and consistency. We review how we got to the current states, the lessons we learned, and the future of real-time analytics at Netflix. While Netflix's scale is larger than most other companies, we believe the approaches and technologies we discuss are highly relevant to other production environments, and audience members should come away with actionable ideas that are implementable in, and benefit, most other environments.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chris Sanden, Netflix
Roy Rapoport, Netflix
October 2015
BDT207
Real-Time Analytics In Service of
Self-Healing Ecosystems

2. @chris_sanden
Chris & Roy
@royrapoport

3. Prerequisites

4. Expectations
(Reasonable)
Telemetry
System
Real-Time
Analytics
System(s)
Data
Orchestration
Systems
Decision
Observation

5. Bad News: An Evolution

6. Bad News: An Evolution

7. Bad News: An Evolution

8. Not Bad
• Absolutely necessary
• Pretty useful
• Insufficient

9. Scale At Scale
We’ve Got
1,982,562,395
Problems
And Boredom Ain’t One

10. Scale At Scale
Complexity in a
Few* Dimensions
* For sufficiently large values of “few”

11. Scale At Scale
421,010

12. Scale At Scale
Telemetry Volume
is silly

13. Scale At Scale
2,000,000,000
is silly

14. Scale At Scale
14”

15. Scale At Scale

16. Scale At Scale

17. Scale At Scale

18. MMO: Most Memorable Outage
• One device (out of ~103)
• One test cell (out of ~101)
• One test (out of ~104)
• Couldn’t view House of Cards S3E1
• For a week

19. Scale At Scale
We have weird, device-specific problems all
the time, and interactions with A/B tests only
make them more complicated, so I'm not
sure we have a pat moral of the story except
that we really like alerting and fast
responses.
- Matt McCarthy

20. Bad News About the Cloud
• Infrastructure no longer the bottleneck
• Before: Weeks to change infrastructure
• After: API call
• TTD expectations vastly higher
• AWS makes us the lameness bottleneck

21. Good News About the Cloud
• Infrastructure no longer the bottleneck
• Before: Weeks to change infrastructure
• After: API call
• Rapid recovery, automated response
possible
• AWS: Enabling productive laziness for 9
years and counting

22. Don’t Forget to Bring a Towel!
Monitoring Capabilities You’ll Find Useful
• Time series
• Event Streaming
• Dependency Discovery and Inspection

23. Real-Time Analytics
1. Prediction
2. Detection
3. Correlation

24. 1. Prediction

25. 1.1 Predictive Scaling

26. Predictive Scaling
Auto Scaling is reactive.
• SCALE UP by 10%
• WHEN Requests Per Second > 120
• FOR 10 consecutive minutes
• FOLLOWED-BY a cool-down of 15 minutes

27. Predictive Scaling
Advanced Use Cases
• Rapid, reoccurring, spike in demand
• Variable traffic patterns
• Outages

28. Predictive Scaling
Concept
• Anticipate change in traffic and workload.
• Predict the resources needed a head of time.
• Proactively scale up or down.

29. Predictive Scaling
Metric Selection
• Clear, relatively stable, and recurring pattern.
• Independent of cluster performance.

30. Predictive Scaling
Requests Per Second (RPS)

31. Predictive Scaling
Fast Fourier Transformation (FFT)

32. Predictive Scaling
FFT-based Prediction

33. Prediction
Predictive Scaling
Action Plan

34. Predictive Scaling
Metric
FFT
Prediction
Action
Plan
Scale
Prediction Workflow

35. Predictive Scaling
Predictive-reactive Auto Scaling
• A hybrid approach
• Predict the workload of a cluster in advance and proactively scale.
• Use auto scaling to handle unexpected surges in workload.

36. 2. Detection

37. 2.1 Anomaly Detection

38. a·nom·a·ly de·tec·tion
[uh-nom-uh-lee] [dih-tek-shuh n]
1. identification of observations which do not conform to an expected pattern.
2. a task that keeps data scientists up at night.

39. “Blips”
Anomaly Detection
Anomaly Types
“Bloops”

40. Anomaly Detection
Static Threshold

41. Anomaly Detection
Static Threshold

42. Anomaly Detection
Prediction Algorithms
• FFT-based Prediction
• Double Exponential Smoothing (DES)
• Holt-Winters
• ARIMA
• Etc.

43. Anomaly Detection
Metric Prediction Residual Threshold
Detection Workflow

44. Anomaly Detection
Double Exponential Smoothing

45. Anomaly Detection
Double Exponential Smoothing

46. Anomaly Detection
Double Exponential Smoothing

47. Anomaly Detection
Statistical Techniques
• Three-sigma (3-sigma)
• Kolmogorov-Smirnov (KS)
• Interquartile Range (IQR)
• Grubbs Test
• Least Squares
• Etc.

48. Anomaly Detection
Metric Prediction Residual Threshold
Detection Workflow

49. Anomaly Detection
Metric Prediction Residual 3-sigma
Detection Workflow

50. Anomaly Detection
Metric Prediction Residual IQR
Combine
Votes
3-sigma
KS
Detection Workflow - Ensemble Approach

51. Anomaly Detection
Advanced Detection Techniques
• Robust Anomaly Detection (RAD) - Netflix
• Seasonal Hybrid ESD - Twitter
• Extendible Generic Anomaly Detection System (EGADS) - Yahoo
• Kale - Etsy

52. 2.2 Outlier Detection

53. out·li·er de·tec·tion
[out-lahy-er] [dih-tek-shuh n]
1. identification of unusual members from a set of generating mechanisms.
2. not be confused with anomaly detection.

54. Time
Population Outlier Detection
Anomaly
Detection

55. Server Outlier Detection
Netflix runs on thousands of servers
• A small percentage of servers become unhealthy.
• Customer experience may be degraded.
• Time wasted looking for evidence.

56. Server Outlier Detection

57. Server Outlier Detection

58. Server Outlier Detection
Cluster Analysis
• Unsupervised machine learning.
• If a server belongs to a group it should be near lots of other points as
measured by some distance function.
Assumption
• Servers running the same hardware and software should behave similar.

59. Server Outlier Detection
DBSCAN - Density-Based Spatial Clustering of Applications with Noise

60. Server Outlier Detection
Metric DBSCAN Filter Action
Detection Workflow

61. Server Outlier Detection
Actions / Remediation
• Send e-mail
• Page service owner
• Terminate instance
• Remove from service
• Detach from a load balancer

62. 2.3 Automated Canary Analysis

63. Automated Canary Analysis
Canary Release Process
• A change is gradually rolled out to production.
• Checkpoints are performed along the way.
• A decision is made at each checkpoint.

64. Automated Canary Analysis
Advantages
• Better degree of trust and safety in deployments.
• Faster deployment cadence.
• Lower investment in simulation engineering.

65. Automated Canary Analysis
Canary Process
Current Version
(v1.0)
New Version
(v1.1)
Load
Balancer
Traffic
100 Servers
5 Servers
95%
5%
Metrics

66. Automated Canary Analysis
Canary Process
Current Version
(v1.0)
New Version
(v1.1)
Load
Balancer
Traffic
0 Servers
100 Servers
100%
Metrics

67. Automated Canary Analysis

68. Automated Canary Analysis
Automated Analysis
• Identify a set of metrics to compare.
• Use a statistical test to identify the difference between v1.0 and v1.1
• Mann–Whitney
• Kolmogorov-Smirnov
• Generate a score that indicates overall similarity.
• Percentage of metrics that match in performance.

69. Automated Canary Analysis
Metrics
Statistical
Test
Calculate
Score
Decision
Analysis Workflow

70. Automated Canary Analysis
Augmented Canary Process
Previous Version
(v1.0)
New Version
(Canary - v1.1)
Load
Balancer
Traffic
88 Servers
6 Servers
Previous Version
(Control - v1.0)
6 Servers
AnalysisMetrics

71. Automated Canary Analysis

72. 3. Correlation

73. Correlation Analysis
Automated Finger-Pointing for Fun and Profit
You Want Service-Oriented Architecture?
We’ve got Service-Oriented Architecture

74. Correlation Analysis
Automated Finger-Pointing for Fun and Profit
A
B C
D
E
F
G
H
I
J

75. Correlation Analysis
Automated Finger-Pointing for Fun and Profit
A
B C
D
E
F
G
H
I
J

76. Correlation Analysis
Something Else Is Also Weird!
CPU up
Alert triggered
HTTP 400
Correlated spike
HTTP requests
Correlated drop

77. Correlation Analysis
If you care about this, you don’t care about that …
I Care About
This Metric!
I Also Care About
This Metric!
Maybe not?

78. Conclusion

79. Magic!
In Conclusion
Not Magic!
DES
IQR
FFT
DBSCAN
RAD/RPCA

80. In Conclusion
(Reasonable)
Telemetry
System
Real-Time
Analytics
System(s)
Data
Orchestration
Systems
Decision
Observation

81. In Conclusion
Wanna play?

82. Useful Links
• Prediction
• Predictive Auto Scaling: http://techblog.netflix.com/2013/12/scryer-netflixs-predictive-auto-scaling.htm
• FFT: https://en.wikipedia.org/wiki/Fast_Fourier_transform
• Detection
• Double Exponential Smoothing: https://en.wikipedia.org/wiki/Exponential_smoothing
• Interquartile Range (IQR): https://en.wikipedia.org/wiki/Interquartile_range
• Ensemble Learning: http://www.scholarpedia.org/article/Ensemble_learning
• Robust Anomaly Detection (RAD): http://techblog.netflix.com/2015/02/rad-outlier-detection-on-big-data.html
• DBSCAN: https://en.wikipedia.org/wiki/DBSCAN
• Server Outlier Detection: http://techblog.netflix.com/2015/07/tracking-down-villains-outlier.html
• Canary Release Process: http://martinfowler.com/bliki/CanaryRelease.html
• Automated Canary Analysis: http://www.infoq.com/presentations/canary-analysis-deployment-pattern
• Nonparametric tests: https://en.wikipedia.org/wiki/Nonparametric_statistics
• Correlation
• Pearson Correlation: https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

83. Attributions
• http://aggronaut.com
• http://designsold.com/pictures-of-kittens/
• http://slate.com, Illustration by Phil Plait
• http://www-rohan.sdsu.edu/
• http://scikit-learn.org/stable/documentation.html

84. Remember to complete
your evaluations!

85. Thank you!
@chris_sanden
@royrapoport