A brief overview of Real-Time Analytics at Netflix and the challenges we've faced in designing and deploying production ready products based on real-time data.

1. Real-Time Analytics @ Netflix
Cody Rioux - @codyrioux
Real-Time Analytics - Insight Engineering

2. Overview.
● Real-Time Analytics
○ Anomaly / Outlier Detection
○ Canary Analysis
● Architecture
● Challenges
○ Cold Start
○ Concept Drift
○ Configuration
○ Change Deployment
○ User Acceptance

3. We are drowning in information but starved for knowledge.
- John Naisbitt
Real-Time Analytics

4. Real-Time Analytics
● Part of Insight Engineering.
● Build systems that make intelligent decisions about our operational environment.
○ Make decisions in near real-time.
○ Automate actions in the production environment.
● Support operational availability and reliability.

5. One of these things is not like the others.
Anomaly and Outlier Detection

6. Unexpected
value for a
given
generating
mechanism.

7. Terminology
Outlier Anomaly

9. Good builds gone bad!
Automated Canary Analysis

10. Old Version (v1.0)
New Version
(Canary - v1.1)
Load
Balancer
Customers
88 Servers
6 Servers
Metrics
Netflix Canary Release Process.
Old Version
(Control - v1.0)
6 Servers
Analysis

11. A Data Scientist’s capability to extract value from data is largely coupled with the maturity of the
data platform of its company. - Robert Chang
Analytic Architecture

12. CSI (REST)Customers
First Generation Architecture

13. OpenPy
Load BalancerCustomers
Models
Second Generation Architecture
OpenPy
...
RTA Data PollerTelemetry
Data

14. Heating things up beginning at absolute zero
Challenge Zero: Cold Start

15. Tagged and curated
data is incredibly
expensive.
Photo credit Damian Gadal under Creative Commons Attribution 2.0

17. OpenPy
Load BalancerCustomers
Models
New Architecture
OpenPy
...
RTA Data PollerTelemetry
Data
Data Tagger
Data
Store

18. Change in data stream over time.
Challenge One: Concept Drift

19. Photo credit SuperCar-RoadTrip.frunder Creative Commons Attribution
Change in the
data over time
in unforeseen
ways causes
degradation in
performance.

20. Solutions: Concept Drift
Monitoring the behavior of
analytics and soliciting user
feedback.

21. OpenPy
Load Balancer
Customers
Models
New Architecture
OpenPy
...
RTA Data PollerTelemetry
Data
Data Tagger
Data
Store
Feedback

22. Removing the burden of configuration complexity for the user.
Challenge Two: Configuration

23. Configurations
are complex.
Assumptions
and meta-
analytics
eliminate user
burden.

24. General is inherently more
complex than specific.

25. OpenPy
Load Balancer
Customers
Models
New Architecture
OpenPy
...
RTA Data PollerTelemetry
Data
Data Tagger
Data
Store
Feedback

26. Progress is impossible without change, and those who cannot change their minds cannot
change anything
- George Bernard Shaw
Challenge Three: Change
Deployment

27. OpenPy
Load Balancer
Customers
Models
New Architecture
OpenPy
...
RTA Data PollerTelemetry
Data
Data Tagger
Data
Store
Feedback

28. Rest API
/REST/v1/anomaly
/REST/v2/anomaly
/REST/v3/anomaly
...

29. Mantis (Stream
Processor)
Customers
Models
New Architecture
Spark
Telemetry
Data
Data Tagger Data
Store
Feedback
Fact
Table
Versioned JAR
Files

30. “Machine learning is really good at partially solving just about
any problem.” - cdixon
Challenge Four: User
Acceptance

31. User Acceptance
● Understandable analytics.
● Favor probabilities for inputs and outputs.
● Conceptual documentation.

32. User Acceptance
?

33. Our platform is less bad than it used to be. :)
Recap

34. Mantis (Stream
Processor)
Customers
Models
New Architecture
Spark
Telemetry
Data
Data Tagger Data
Store
Feedback
Fact
Table
Versioned JAR
Files

35. Recap
● Cold Start: Data Tagging
● Concept Drift: Feedback Loop
● Configuration: Assumptions and Meta Analytics
● Change Deployment: Versioned Analytics
● User Acceptance: Docs, probabilities, understandable analytics.

36. Literature
Machine Learning: The High
Interest Credit Card of Technical
Debt (Sculley et al., 2014)

37. Literature
● Practical Machine Learning: A New Look at Anomaly Detection (Dunning, 2014)
● Distinguishing cause from effect using observational data: methods and
benchmarks (Mooij et al., 2014)
● Enhancing Performance Prediction Robustness by Combining Analytical Modeling
and Machine Learning (Didona et al., 2015)

38. Questions?
crioux@netflix.com
@codyrioux
linkedin.com/in/codyrioux