The Netflix API team's journey towards Continuous Delivery.

1. How we learned to stop worrying and start deploying
The Netflix API service
Sangeeta Narayanan
@sangeetan
http://www.linkedin.com/in/sangeetanarayanan
http://bit.ly/1wq2kkN

4. Global Streaming for Movies and TV Shows

5. High Quality Original Content

7. Over 50 Million Subscribers
Over 40 Countries

8. > 34% of Peak Downstream Traffic in North
America
Over 2 billion streaming hours a month

10. Personalized User Experience

12. http://goo.gl/VhokZV
Role of API
Enable rapid innovation
Conduit for metadata between Devices and
Services
Implements business logic
Scale with business growth
Maintain resiliency

13. Going back in time…
http://bit.ly/1yOWEjr

14. PM: When can I get my feature?

15. PM: When can I get my feature?
Us: 2 -4 weeks

16. PM: When can I get my feature?
Us: 2 -4 weeks - ish…

17. PM: When can I get my feature?
Us: 2 -4 weeks - ish…
IF all goes well…

18. 2 week release cycle

19. Not Quite!

21. !
Stop being the bottleneck!
http://bit.ly/1zmYbAy

22. What’s not
working?

23. Heavy weight Code Management

24. Slow, non-repeatable builds

25. Constantly Changing Dependencies

26. Slow, unreliable tests
Low coverage
Manual on-device testing

27. Manual
deployments

28. Push Lead!

29. Requirements for new system
On-Demand, Rapid Feature Delivery
Intuitive and painless
Easy recovery from errors
Insight and Communication
Balance between Agility & Stability

30. Release
Release
Release
Patch
Patch
Patch
2 week Releases + Ad-Hoc Patches
http://bit.ly/1E6a9yn

31. MR
MR
IR
IR
IR
IR
3 week Major Releases + Weekly Incremental Releases

32. Automate SCM Tasks

33. Automated Dependency Validation

34. Test Strategy
Increasing
confidence

35. Test Runtimes 60%
No False Positives

36. Improved Result Reporting

37. Automated Deployments
Internal Environments
Using Asgard API
Connected to builds
Driven from CI Server

38. Pipelines

39. • Multiple
deployments/day
!
• Multiple internal
environments
!
• Multiple AWS
regions
http://bit.ly/13qrIfw

40. Team Cohesion
!
• Shared ownership - no silos
• Increased partner satisfaction
• Greater productivity

41. http://bit.ly/1xJQqjD
Aiming Higher

42. Faster, Better, All the way!
Shorter Feedback Loop
Increased Confidence
Richer Insight & Communication

43. Build
Bake
Test
Deploy
Build
Test Bake
Deploy

44. Branching Strategy
!
Modeled after github-flow
!
Automated Pull Request Processing
!
Automated Patch Branching

45. Single long-lived branch
Always deploy-able
Feature branches

46. More, Better, Faster & to Prod
Shorter Feedback Loop
Increased Confidence
Richer Insight & Communication

47. Automated Canary
Analysis
Aggregate Health Score
>1500 metrics
Configurable
Multiple regions
~1%$Traffic$
Old$Code$(Baseline)$ New$Code$(Canary)$

50. Developer Canaries
(dynamically provisioned)

51. Dependency Validation Canary

52. Deployed
Not intended for deployment
Not deployable; failed tests

53. Hands Free Production Deployments
http://bit.ly/1wQ8fPQ

54. Red/Black Deployments

55. Production Traffic
Old Code

56. Production Traffic
Old Code New Code

57. Production Traffic
Old Code New Code

60. Feature Rollback

61. Full Rollback

62. Old Code New Code
Re-enable
traffic
Production Traffic

63. Production Traffic
Old Code New Code

64. More, Better, Faster & to Prod
Shorter Feedback Loop
Increased Confidence
Richer Insight & Communication

69. API Server Deployment Velocity
#Deployments
#Rollbacks
Deployments & Rollbacks/week
since Oct 2012

70. Take-aways
Build Agility into Architecture
Embrace Change; Don’t Fight it!
Failure is inevitable
Insight is key

71. What’s Next
“Tiered” Canary Analysis
Failure Injection Testing
Throughput Trending
http://goo.gl/zjiV6W

72. Culture
http://goo.gl/7l7xkM

73. Freedom and Responsibility

75. http://netflix.github.io/
http://techblog.netflix.com

76. How we learned to stop worrying and start deploying
The Netflix API service
Sangeeta Narayanan
@sangeetan
http://www.linkedin.com/in/sangeetanarayanan
http://bit.ly/1wq2kkN
Thank You!