Infrastructure & tools that support operations of microservices at scale at Netflix

1. The machinery
around microservices
Tools & practices for operating
microservices @
Sangeeta Narayanan (@sangeetan)
Director, Edge Developer Experience
Dec 2016

3. #NetflixOffline

4. >83M subscribers
>1000 device types
>42B streaming hours in 2015
600 hours of original content
1000s of playbacks per second
3 AWS regions; 3 AZs each

6. Subscribers/year (millions)
20
40
60
80
2007 2009 2011 2013 2015

7. Subscribers/year (millions)
20
40
60
80
2007 2009 2011 2013 2015 2016
??

8. http://bit.ly/2gFd6mq
Many things have to come together for a successful launch of
this complexity

9. http://www.martinfowler.com/articles/microservices.html

10. Quality
Velocity
Availability
http://bit.ly/2fWUNd9

11. Increasing scale
http://bit.ly/2gVNWQT

12. Making
microservices work
Tools, Practices, Culture
http://bit.ly/2h7Q2NY

13. CDN data plane not shown

14. Architectural benefits
❖ Scalability
❖ Resiliency
❖ Velocity
❖ Autonomy

15. ❖ Dynamic Routing
❖ Traffic shaping
❖ Insights, debugging
❖ Traffic failover
Custom cloud gateway

16. Philosophies
❖ Own your service - operate
what you build
❖ Move fast - optimize for
velocity
❖ Stay safe - prepare for failure
❖ Focus on automation
http://bit.ly/2fydxy6

17. Develop Deploy
Operate Support
App Development &
Delivery
Insights
Netflix engineers own the entire lifecycle of their services. They are provided a rich set of tools
and libraries to simplify their tasks.

18. Development
❖ Standardized platform
❖ IPC
❖ Service discovery
❖ Logging, Monitoring
❖ Configuration Management
❖ and more
❖ DevEx tooling
❖ API discovery and documentation
❖ Usage tracking and lifecycle management
❖ Error simulations and behavior overrides

19. http://netflix.github.io/

21. Deployment

22. ❖ Automated canary analysis
❖ Self-service canaries for developers
❖ Automated regional deployments
❖ Choice of deployment strategy
❖ Pipelines for config changes

26. http://bit.ly/2g0H5RO http://bit.ly/2fXA0GA

27. Support
❖ Diverse audience
❖ Internal partners: service consumers, dependencies,
customer services
❖ External partners
❖ Production and lower environments
❖ Micro level insights

28. Fine grained insights via real time event stream queries

29. Request call graph

30. Request profiling

31. Operational Insights
❖ Aggregate health dashboards
❖ Real time alerting
❖ Anomaly detection
❖ Automated correlation analysis

36. Anomaly Detection

39. Failure testing
Drills and “game-day” exercises
Latency and error simulations

40. Automated traffic failover

41. Production Readiness Checklist

42. Culture

43. Context; not Control

44. http://bit.ly/2h93a1f
Highly aligned
Loosely coupled

45. Freedom
and
Responsibility

47. http://www.martinfowler.com/articles/microservices.html
People
Practices
Technology

49. Thank You