Traditionally, a tug of war has existed between service reliability and engineering velocity. Increasing speed to fuel product innovation has meant making tradeoffs in reliability. Netflix standardizes common functionality, like service discovery, configuration, metrics, logging, and RPC across services. This frees teams to focus on the unique business value of their service. It also enables us to evolve and maintain platform components independently from individual services. Even with a standard set of components, service owners still need to combine these disparate elements into a coherent platform. We reduce this friction by providing a preassembled platform where teams only need to provide their business logic, and not worry about assembling the service from scratch. We can further streamline the service lifecycle by providing automation and tooling for development, testing, deployment and operations. We provide "one click" solutions to automatically generate the associated pipelines, machinery, and infrastructure that's required to run their service reliably in production. These patterns, while described in a Netflix context, can be broadly applicable to increase both reliability and velocity of your microservices architecture.

1. The Paved PaaS to
Microservices
December, 3, 2017
Yunong Xiao,
Principal Software Engineer, Netflix
yunong@netflix.com,
@yunongx,
yunongx
http://yunong.io
linkedin.com/in/yunongxiao

4. 纸牌屋

5. What is ?
Great Content
Incredibly Scale

6. Watch:
Any time
Any where
Any device
Great Content

7. 110 000 000 paying customers
in over 190 countries
streaming 125 000 000 hours/day
Incredible Scale: Numbers

8. Incredible Scale: Subscriber Growth
20M
38M
56M
74M
92M
110M
2011 2012 2013 2014 2015 2016 2017

9. Incredible Scale: Bandwidth
1/3 of ALL internet traffic in North America

10. How?

11. Microservices Architecture
Member Experience
Streaming
Playback
Telemetry
Sign up
Recommendations
…

12. Naive Solution: Every client talks to every service
30 clients * 800 services = 24000 connections
Clients Services

13. Too Complex!

14. Netflix Edge API
http://api.netflix.com
Clients Services

15. What is the Netflix Edge API?
Bridges the gap between frontend and backend

16. What is the Netflix Edge API?
Provides 1 consistent API over 100s of
disparate services

17. What is the Netflix Edge API?
Central integration point for frontend and
backend

18. Requirements?

19. 1000s of A/B tests per year
A/B

20. Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Frequent code pushes

21. Customization

22. Netflix Edge
API
Backend
Service A
Backend
Service B
Backend
Service C
Backend
Service N
…
PROBLEMS:
Not customizable: 1 service for all devices
Slow rate of change: centralized service, 1 code push a month

23. 100s of services
Customized for each
client
Pushing multiple
times/week
TV
iOS
Android
Windows
Browsers
Discovery
Playback
Non-
member
…
Backend
Service A
Backend
Service B
Backend
Service C
Backend
Service N
…
Clients Standalone Services Edge API Backend Services

24. Goals
Velocity Reliability
100s of services
1000s of A/B tests
Pushing multiple times/week/service
99.99% Availability

25. Who’s the owner?
Client Engineers Server Engineers

26. Q: Can we achieve velocity &
reliability with client engineers?
A: Try Platform as a Service

27. –Wikipedia
“Platform as a service (PaaS)… allows customers to
develop, run, and manage applications without the
complexity of
building and maintaining the infrastructure and
platform…”
What is a Platform as a Service (Paas), Anyway?

28. Our PaaS to Achieving Reliability & Velocity
1. Standardized components
2. Preassembled platform
3. Automation and tooling

29. 1. Standardized Components

30. What’s Inside a Microservice?
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

31. Netflix Open Source Libraries
http://netflix.github.io

32. Why Standards?

33. Mélange of RPC

34. Microservice Interactions
µServiceClient
uService
µService
µService
µService
…
µService
…
µService

35. Failure: When not If
Netflix 不不⼯工作了了， 我得当⽗父⺟母了了，#不不公平
Netflix 不不⼯工作了了， 就像我的好朋友死了了

36. – Managers everywhere
“Is it fixed yet?”
Mean Time to Detection (MTTD)
Mean Time to Repair (MTTR)

37. N Flavors of RPC
µServiceClient
uService
µService
µService
µService
…
µService
…
µService
How do you debug the system?

38. One Standard RPC
µServiceClient
uService
µService
µService
µService
…
µService
…
µService
Reduces MTTR

39. Benefits of Standardizing
Consistency Leverage Interoperability Quality Support
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

40. You want to be different?

41. http://jobs.netflix.com/culture
Netflix Culture

42. • Use non-standard
components/architecture
• Fosters innovation
• Reintegrate changes back
into platform
Freedom Responsibility&
• Use new technologies
judiciously
• Don’t develop alone in a
vacuum
• Increases burden to the
rest of the company

43. 2. Preassembled Platform

44. Assembly Required
RPC
Discovery
Registration
Runtime
OS
Configuration
Metrics
Logging
Tracing
Dashboards
Alerts
Stream
Processing

45. Let’s use an Analogy

48. Most engineers aren’t experts
on how a microservice is built,
they just want a pre-built
microservice that works

49. Microservice Assembly Instructions
Read documentation
Copy/paste (wrong) source code
Which versions of dependencies?
How to initialize all components?
How to configure all components?
Missing components
…

50. Velocity Reliability
Not a single line of business logic!
New feature
Slow: Can Take Weeks

51. Preassembled Platform
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

52. Easy Access to All Platform Components

53. Out of the Box

54. Component Management

55. Logs Metrics Alerts

56. Insights

57. Application, system, and runtime metrics & logs
Different for each service!

58. Consistent application, system, and runtime
insight
Reduces MTTD & MTTR

59. Launching a microservice
safely

60. Configures and Initializes Correctly
BOEING 747-400 NORMAL PROCEDURES CHECKLIST
POWER UP / SAFETY CHECK
First Officer Captain
CIRCUITBREAKERS...………………...CHECKED
BATTERY……………...………………….………ON
STANDBY POWER………………...………..AUTO
HYDRAULIC DEMAND PUMPS….………..…..OFF
WINDSHIELD WIPERS....……………….……..OFF
ALTERNATE FLAPS AND GEAR…………….OFF
GEAR LEVER…….………….....…………..DOWN
FLAPS…..……………………………....CHECKED
APU….………………………………...….RUNNING
ELECTRICAL SYSTEM....……SET/APU AVAIL ON
APU BLEED AIR…...……………………………ON
ISOLATION VALVES…………………………OPEN
PACKS………………...……………………NORMAL
PREFLIGHT
First Officer Captain
EMERGENCY EQUIPMENT…….…….CHECKED
FIRE PROTECTION…………...………CHECKED
INTERRUPT SWITCHES………………………ON
PASSENGER OXYGEN……………..……NORMAL
STAB TRIM CUTOUT SWITCHES………….AUTO
TRANSPONDER…………………………………SET
SOURCE SELECTORS…………………...…….SET
CLOCKS…………………………………………..SET
CRT SELECTORS…………………………NORMAL
PFD………………………………………CHECKED
ND…………………………………………CHECKED
AUTOBRAKES…………………………………RTO
EIU SELECTOR………………………………AUTO
HDG REFERENCE SWITCH….…………NORMAL
FMC MASTER SELECTOR…………………LEFT
GROUND PROX SYSTEM……………CHECKED
BEFORE STARTING
First Officer Captain
HYDRAULIC DEMAND PUMPS…………………….
.............................................AUTO, AUX (1 AND 4)
BRAKE PRESSURE……..……………….NORMAL
FUEL QUANTITY………………..…………. ____KG
FUEL SYSTEM………………………………….SET
X-FEEDS…………………OPEN (1 & 4 CLOSED)
SEAT BELTS SIGN………………………………ON
NOTOC………………………………….CHECKED
SHIPS PAPERS…………………………ON BOARD
PERFORMANCE DATA......…CHECKED AND SET

62. Versions Updates Compatibility

63. Stay on paved path!

64. Adjust as Needed
Config overrides
Startup & shutdown hooks
Access to 3rd party libs
Swap, disable, or configure
components
Raw component access

65. 3. Automation and Tooling

66. Ship a Feature

67. Development
Testing
Deployment Operations
Steps

68. Development

69. Goal: Achieve higher velocity
through better developer
ergonomics

70. Q: What’s the first thing you have to
do when developing a
microservice?

72. Environment bootstrap Integrate tooling & services Run local & cloud
CLI for common dev experience

73. Local Development
Live reload
Attach debugger
localhost
PROD

74. Testing

75. Catch Bugs in Test, not Prod

76. Testing
Preassembled Platform
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing
Owned by
service team

77. Preassembled Platform
Unstable
Test Environment
Availability:
<<<<<99.99%
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

78. Solution: Provide First Class Mocks
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing
Preassembled Platform

79. Platform Testing API
• Just like a runtime API, need a testing API
• Provide mocks interface for components
• Gets platform out of the loop for providing mocks

80. Deployment

81. “Production is war!”

82. Experience Differences
Veteran Recruit

83. Deploy and Manage Services
• Pre-configured pipelines for publish, canary, deployment and
cleanup
• Single command deploy to any stack
• Integration for automated canary analysis
• Pre-configured auto scaling

84. Deploy and Manage Services
• Pre-configured pipelines for publish, canary, deployment and
cleanup
• Single command deploy to any stack
• Integration for automated canary analysis
• Pre-configured auto scaling

85. Netflix runs 100% in the cloud

86. How does Netflix Manage
Cloud Infrastructure?

88. http://www.spinnaker.io

89. What is a “Pipeline”?

90. Safe reproducible pipelines

91. Deploy and Manage Services
• Pre-configured pipelines for publish, canary, deployment and
cleanup
• Single command deploy to any stack
• Integration for automated canary analysis
• Pre-configured auto scaling

92. Single Command
Reduces user error
Abstraction over tooling
Increases developer velocity

93. Deploy and Manage Services
• Pre-configured pipelines for publish, canary, deployment and
cleanup
• Single command deploy to any stack
• Integration for automated canary analysis
• Pre-configured auto scaling

94. Production
Clients
99.9%
Canary: New Build
0.1%
What’s a Canary?
Relatively “safe” way to test new builds in production
100%

96. Provide a pre-configured pipeline for canaries

97. Deploy and Manage Services
• Pre-configured pipelines for publish, canary, deployment and
cleanup
• Single command deploy to any stack
• Integration for automated canary analysis
• Pre-configured auto scaling

101. Operations

102. Consolidated View

103. RPS
Latency
Error rates
CPU
Memory
Generated Dashboards & Alerts

104. CPU profiling Core dump analysis
Automated Analytics & Tooling

105. Our PaaS to Velocity & Reliability
1. Standardized components
2. Preassembled platform
3. Automation and tooling

106. Questions? 问题？
We’re hiring! 欢迎加⼊入我们的团队！
December, 3, 2017
Yunong Xiao,
Principal Software Engineer, Netflix
yunong@netflix.com,
@yunongx,
yunongx
http://yunong.io
linkedin.com/in/yunongxiao