1. Netflix’s Global Cloud
Edge Architecture
Mikey Cohen mikey@netflix.com
Edge Engineering Platform
Netflix

2. Over 44 million subscribers
in over 40 countries

3. Netflix accounts for over 30% of peak internet traffic
in North America

4. One billion hours ~
100,000 years
per month...

5. Netflix supports over 1000 device types

6. Edge Services
● Front door to Netflix
● Edge Routing - Zuul
● API - Edge Server
● Playback services

7. How does Netflix Streaming work?*
* A simplified view

8. How does Netflix Streaming work?
Netflix
Services
in Amazon Cloud
Your CE Device
CDN

9. Device Under the Hood
Netflix
Services
in Amazon Cloud
Your CE Device
CDN
User Interface
Netflix Streaming Platform
DRM encodingCE integration

10. User Interface loaded, data retrieved from
Netflix Edge Service
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services

11. User Interface loaded, data retrieved from
Netflix Edge Service
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services

12. User Interface Loaded
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services

13. Movie Authorization
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services
Authorize

14. Movie Authorization
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services
Authorize

15. Obtaining License
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services
License

16. Movie starts streaming
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services
PlayData

17. Movie starts streaming
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services

18. Periodic “bookmark” calls note place in
movie
User Interface
Netflix Streaming Platform
DRM
Netflix
Services
in Amazon Cloud
encoding
Your CE Device
CDN
CE integration
Edge
Services
bookmark

19. Edge Services - What we are talking about
today
User Interface
Netflix Streaming Platform
DRM encoding
Your CE Device
CDN
CE integration
bookmarkNetflix
Services
in Amazon Cloud Edge Services

20. Edge’s lofty mission
● High Availability
● Good performance
● Data broker between many services and
devices in a global, high volume, rapidly
innovating, highly dynamic service
● Clients and services are constantly changing

21. Edge stats
● Billions of incoming requests per day
○ Over 10X outgoing service calls per request
● About 10 device changes per day
● Daily service pushes
● Daily routing changes

22. Architecture Goals
● Infrastructure
○ Availability
○ Resiliency
○ Scalability
● Application
○ Platform diversity
○ Rapid innovation
○ A/B Testing
● Delivery
○ Automation
○ Insights

23. Netflix’s Global Cloud Architecture

24. High Level Regional Edge Architecture
ELB
Edge Service
Netflix Services
ELB
Playback Service
ELB
Zuul
Website Service

25. Zuul
ELB
Edge Service
Netflix Services
ELB
Playback Service
ELB
Zuul
Website Service

26. What is Zuul?
● Open source framework for dynamically
reading, writing, and executing filters that act on
incoming HTTP requests
● Dynamically compiled filters written in Groovy
○ Any JVM language supported
● Filters share state through a request scoped
context

27. How we use Zuul
● Authentication
● Insights
● Stress Testing
● Canary Testing
● Dynamic Routing
● Service Migration
● Load Shedding
● Security
● Static Response handling
● Active/Active traffic management

28. Zuul Filter Characteristics
● Type
● Execution Order
● Criteria
● Action

29. Zuul Filter Lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

30. Zuul Filter Lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

31. zuul filter lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

32. zuul filter lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

33. zuul filter lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

34. zuul filter lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

35. zuul filter lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

36. zuul filter lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

37. zuul filter lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

38. Example Filter
File: DeviceDelayFilter.groovy
1 class DeviceDelayFilter extends ZuulFilter {
2
3 def static Random rand = new Random()
4 @Override
5 String filterType() {
6 return 'pre'
7 }
8
9 @Override
10 int filterOrder() {
11 return 5
12 }
13
14 @Override
15 boolean shouldFilter() {
16 return RequestContext.getRequest().
17 getParameter("deviceType")?equals("BrokenDevice"):false
18 }
19
20 @Override
21 Object run() {
22 sleep(rand.nextInt(20000)) //Sleep for a random number of seconds between [0-20]
23 }
24 }

39. Filter deployment

40. Active/Active

41. Multiple Active Regions
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
Services

42. Multiple Active Regions - NM vs GE
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

43. Multiple Active Regions- Cassandra Replication across regions
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

44. DNS Misrouting
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

45. DNS Misrouting
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

46. Geo lookup resolves IP in west
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS
GEO

47. Zuul east routes to Zuul west
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS
GEO

48. Response is from west
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS
GEO

49. Regional Failure
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

50. Catastrophe in US-East
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

51. East Coast is Down
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

52. Switch DNS to point to US-West
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

53. East traffic flows to West
ZUUL
API
Cassandra
Services
ZUUL
API
Cassandra
ServicesDNS
DNS

54. Edge Server (API)

55. The Edge Service - Netflix’s API Tier
ELB
Edge Service
Netflix Services
ELB
Playback Service
ELB
Zuul
Website Service

56. What’s wrong with REST for Netflix?

57. REST
● One Size Fits all
● One Data Format Fits All
● REST tends to be atomic
● Average 25 REST requests to build up a
page.

58. Netflix’s Groovy Scripting Layer

59. Edge Scripting Tier
● Device teams write scripts for their device
○ control content, format, endpoints
● Code injected directly into Edge Service at
runtime
○ Scripts are in production in about 30 seconds

60. Endpoint
Code
(Groovy)
Endpoint Controller
RxJava
Async Service Layer API
Hystrix (Fault tolerance)Endpoint
Manager
JVM
Netflix Services
Edge Server Architecture

61. Endpoint
Code
(Groovy)
Endpoint Controller
RxJava
Async Service Layer API
Hystrix (Fault tolerance)Endpoint
Manager
JVM
Netflix Services
Pushing a Script
UI Engineer
/ps3/home
script

62. Endpoint
Code
(Groovy)
Endpoint Controller
RxJava
Async Service Layer API
Hystrix (Fault tolerance)Endpoint
Manager
JVM
Netflix Services
Pushing a Script
UI Engineer
/ps3/home
script

63. Endpoint
Code
(Groovy)
Endpoint Controller
RxJava
Async Service Layer API
Hystrix (Fault tolerance)Endpoint
Manager
JVM
Netflix Services
Controller pulls new script / compiles
UI Engineer
/ps3/home
script

64. Endpoint
Code
(Groovy)
Endpoint Controller
RxJava
Async Service Layer API
Hystrix (Fault tolerance)Endpoint
Manager
JVM
Netflix Services
Script Activated
UI Engineer
Activate

65. Service Layer

66. Endpoint
Code
(Groovy)
Endpoint Controller
RxJava
Async Service Layer API
Hystrix (Fault tolerance)Endpoint
Manager
JVM
Netflix Services
Service Layer

67. Purpose of the Service Layer
● Interface to business logic (our API)
● Shield data consumers from service
changes
● Combine and expose business data in a
logical and consistent manner
● All Service Layer methods are async using
RxJava
○ Hides concurrency and underlying implementation

68. RxJava

69. Endpoint
Code
(Groovy)
Endpoint Controller
RxJava
Async Service Layer API
Hystrix (Fault tolerance)Endpoint
Manager
JVM
Netflix Services
RxJava

70. RxJava
● Why?
○ How do you expose an async service as an API?
○ Solution to compose async flows and sequences of
data
○ Rich set of operators to filter and interact with data

71. How RxJava Helps
● Need to hide concurrency from script writers
○ Minimize the “bad things” consumers of our API on
box can do.
○ Hide the internal implementation
■ Change concurrency of any given call
■ Switch to non-blocking IO

72. Hystrix
Service Resiliency

73. Endpoint
Code
(Groovy)
Endpoint Controller
RxJava
Async Service Layer API
Hystrix (Fault tolerance)Endpoint
Manager
JVM
Netflix Services
Hystrix

81. How Hystrix helps
● Latency and Fault Tolerance
○ Stop cascading failures. Fallbacks and graceful degradation. Fail fast and rapid recovery.
○ Thread and semaphore isolation with circuit breakers.
● Realtime Operations
○ Realtime monitoring and configuration changes. Watch service and property changes take effect
immediately as they spread across a fleet.
○ Be alerted, make decisions, affect change and see results in seconds.
● Concurrency
○ Parallel execution. Concurrency aware request caching. Automated batching through request collapsing.

82. Hystrix Dashboard Example

83. DELIVERY

84. Edge Delivery
● Continuous deployment
● Automated system integrity analysis
● Tools for facilitating delivery

85. Automated Deployment Pipeline

86. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Edge Cluster Organization

87. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Most Requests to Main Origin

88. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Some requests to Canary

89. Canary Analysis

90. Canary Analysis Detail

91. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Response Validation

92. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Fork response to Main and Canary

93. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Validate response
Validate
response
integrity

94. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Targeted Debugging

95. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Targeted Debugging

96. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Targeted Debugging

97. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Squeezing the Origin

98. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
SQUEEZE
ORIGIN
ELB
Squeezing the Origin

99. ZUUL ZUUL-CANARY
ZUUL-DEBUG ZUUL-SQUEEZE
MAIN
ORIGIN
CANARY
ORIGIN
DEBUG
ORIGIN
ELB
Finding service Capacity
SQUEEZE
ORIGIN

100. Scryer - Predictive auto-scaling
● Why?
○ Reactive doesn’t work in all cases
○ Reacting is sometimes too late
■ Sunday morning cartoons
○ Reactive overreacts
■ Superbowl, World Cup, Outages
■ Fixed size scaling
○ All in All - more reliable and saves money

101. Daily Traffic Patterns

102. Scryer Predictions

103. How does Scryer work?
● Traffic shape analysis
○ Monday vs Monday
○ Sunday vs Sunday, etc
○ FFT based smoothing

104. Filtering out Noise

105. Ignoring outages

106. Accounting for regular spikey traffic

107. Iteratively apply FFT

108. Other Scryer Factors
● Traffic volume analysis
○ At least 4 weeks of data
○ Linear regression based on time of day
○ Correct the prediction based on today’s trend.
● Instance factors
○ Instance startup time
○ Instance capacity (obtained by squeeze testing)
● Scale (up/down) actions scheduled based on prediction

109. The Future

110. Future - Large Projects on Edge
● Async, non-blocking servers
● Service layer redesign
● Internal Insights
● Global Insights

111. Edge Architecture Today
ELB
API Service
Netflix Services
ELB
Streaming
Service
ELB
Zuul
Website Service
Zuul Zuul

112. Future Edge Architecture
ELB
API/ Edge Service
Netflix Services
Playback Services
ELB
Zuul
Website

113. Future Edge Architecture
ELB
API/ Edge Service
Netflix Services
Playback Services
ELB
Zuul
Website

114. Future Edge Architecture
ELB
API/ Edge Service
Netflix Services
Playback Services
ELB
Zuul
Website

115. Future Edge Architecture
ELB
API/ Edge Service
Netflix Services
Playback Services
ELB
Zuul
Website

116. Future Edge Architecture
ELB
API/ Edge Service
Netflix Services
Playback Services
ELB
Zuul
Website

117. Global Insights
API/ Edge
Service
Netflix
Services
Playback Services
Zuul
User
Interface
Insight
EngineEvent Stream
Client
Data

118. User Interface Designs

121. Netflix in the Cloud -
5 years later
Lessons learned

122. What Did We Learn?

123. Failure is Assured!

124. ● Code failure - Continuous delivery
● Service failure - fallbacks and redundancy
● Instances and Zone failure - redundancy
● Cloud infrastructure failure - Multiple active regions
● Human failure - Automation
Building for Failure

125. Drawbacks of the cloud
● Some failures are difficult to detect the cause
○ Huge variability in instance performance that are
almost impossible to explain.
○ Network barriers
○ Multi tenancy
○ Firewalls
● Very limited access to information/ ability to fix issues

126. Software focus: Cloud’s greatest strength
● Scale our business
● Automate processes
● Radically experiment
● Remain resilient
● Move quickly

127. Netflix Culture - Our secret sauce
● Freedom and responsibility
● Highly aligned teams
● Aversion to process
● Design for necessity
● Design for failure
● Engineering teams operating their services

128. Netflix OSS
● Zuul - Smart edge router
● RxJava - Functional reactive libraries
● Hystrix - SOA resiliency
● + a lot more!

129. For more Info on Netflix Cloud Technology:
Read our Technology Blog : http://techblog.netflix.com/
Check out our Open Source Cloud Projects : http://netflix.github.io