![@atseitlin
About Netflix
Netflix is the world’s
leading Internet
television network with
more than 36 million
members in 40
countries enjoying more
than one billion hours
of TV shows and movies
per month, including
original series[1]
[1] http://ir.netflix.com/](https://image.slidesharecdn.com/qconnewyork-resiliencythroughfailure-130615071714-phpapp01/85/Resiliency-through-failure-QConNY-2013-2-320.jpg)
Uploaded byAriel Tseitlin
Resiliency through failure @ QConNY 2013
The document outlines Netflix's approach to achieving high availability in its cloud infrastructure by designing systems that can gracefully handle failures. It emphasizes the importance of proactively inducing failure through methods such as the Simian Army tools (e.g., Chaos Monkey) to validate system resiliency and improve operational robustness. The document also highlights a blameless culture that fosters learning from failures and the significance of deep system visibility to understand and resolve issues.
More Related Content
Similar to Resiliency through failure @ QConNY 2013
Resiliency through failure @ QConNY 2013
- 1. @atseitlin Resiliency through failure Netflix'sApproach to Extreme Availability in the Cloud Ariel Tseitlin http://www.linkedin.com/in/atseitlin @atseitlin
- 2. @atseitlin About Netflix Netflix isthe world’s leading Internet television network with more than 36 million members in 40 countries enjoying more than one billion hours of TV shows and movies per month, including original series[1] [1] http://ir.netflix.com/
- 3.
- 4. @atseitlin How Netflix StreamingWorks Customer Device (PC, PS3, TV…) Web Site or Discovery API User Data Personalization Streaming API DRM QoS Logging OpenConnect CDN Boxes CDN Management and Steering Content Encoding Consumer Electronics AWS Cloud Services CDN Edge Locations Browse Play Watch
- 5.
- 6.
- 7. @atseitlin Our goal isavailability • Members can stream Netflix whenever they want • New users can explore and sign up for the service • New members can activate their service and add new devices
- 8. @atseitlin Failure is allaround us • Disks fail • Power goes out. And your generator fails. • Software bugs introduced • People make mistakes Failure is unavoidable
- 9. @atseitlin We design aroundfailure • Exception handling • Clusters • Redundancy • Fault tolerance • Fall-back or degraded experience (Hystrix) • All to insulate our users from failure Is that enough?
- 10. @atseitlin It’s not enough •How do we know if we’ve succeeded? • Does the system work as designed? • Is it as resilient as we believe? • How do we prevent drifting into failure? The typical answer is…
- 11. @atseitlin More testing! • Unittesting • Integration testing • Stress testing • Exhaustive test suites to simulate and test all failure mode Can we effectively simulate a large- scale distributed system?
- 12. @atseitlin Building distributed systemsis hard Testing them exhaustively is even harder • Massive data sets and changing shape • Internet-scale traffic • Complex interaction and information flow • Asynchronous nature • 3rd party services • All while innovating and building features Prohibitively expensive, if not impossible, for most large-scale systems
- 13. @atseitlin What if wecould reduce variability of failures?
- 14. @atseitlin There is anotherway • Cause failure to validate resiliency • Test design assumption by stressing them • Don’t wait for random failure. Remove its uncertainty by forcing it periodically
- 15. @atseitlin And that’s exactlywhat we did
- 16.
- 17.
- 18. @atseitlin Chaos Monkey taughtus… • State is bad • Clusters are good • Surviving single instance failure is not enough
- 19.
- 20.
- 21. @atseitlin Chaos Gorilla taughtus… • Hidden assumptions on deployment topology • Infrastructure control plane can be a bottleneck • Large scale events are hard to simulate • Rapidly shifting traffic is error prone • Smooth recovery is a challenge • Cassandra works as expected
- 22. @atseitlin What about largercatastrophes? Anyone remember Sandy?
- 23. @atseitlin Chaos Kong (*someday soon*)
- 24. @atseitlin The Sick andWounded
- 25.
- 26.
- 27.
- 28. @atseitlin Latency Monkey taughtus • Startup resiliency is often missed • An ongoing unified approach to runtime dependency management is important (visibility & transparency gets missed otherwise) • Know thy neighbor (unknown dependencies) • Fall backs can fail too
- 29.
- 30. @atseitlin Clutter accumulates • Complexity •Cruft • Vulnerabilities • Cost
- 31.
- 32. @atseitlin Janitor Monkey taughtus… • Label everything • Clutter builds up
- 33. @atseitlin Ranks of theSimian Army • Chaos Monkey • Chaos Gorilla • Latency Monkey • Janitor Monkey • Conformity Monkey • Circus Monkey • Doctor Monkey • Howler Monkey • Security Monkey • Chaos Kong • Efficiency Monkey
- 34. @atseitlin Observability is key •Don’t exacerbate real customer issues with failure exercises • Deep system visibility is key to root-cause failures and understand the system
- 35. @atseitlin Organizational elements • Everyengineer is an operator of the service • Each failure is an opportunity to learn • Blameless culture Goal is to create a learning organization
- 36.
- 37. @atseitlin Open Source Projects Github/ Techblog Apache Contributions Techblog Post Coming Soon Priam Cassandra as a Service Astyanax Cassandra client for Java CassJMeter Cassandra test suite Cassandra Multi-region EC2 datastore support Aegisthus Hadoop ETL for Cassandra AWS Usage Spend analytics Governator Library lifecycle and dependency injection Odin Cloud orchestration Blitz4j Async logging Exhibitor Zookeeper as a Service Curator Zookeeper Patterns EVCache Memcached as a Service Eureka / Discovery Service Directory Archaius Dynamics Properties Service Edda Config state with history Denominator Ribbon REST Client + mid-tier LB Karyon Instrumented REST Base Serve Servo and Autoscaling Scripts Genie Hadoop PaaS Hystrix Robust service pattern RxJava Reactive Patterns Asgard AutoScaleGroup based AWS console Chaos Monkey Robustness verification Latency Monkey Janitor Monkey Bakeries / Aminotor Legend
- 38. @atseitlin How does itall fit together?
- 39.
- 40. @atseitlin Our Current Catalogof Releases Free code available at http://netflix.github.com
- 41. @atseitlin Takeaways Regularly inducing failurein your production environment validates resiliency and increases availability Use the NetflixOSS platform to handle the heavy lifting for building large-scale distributed cloud- native applications
- 42. @atseitlin Thank you! Any questions? ArielTseitlin http://www.linkedin.com/in/atseitlin @atseitlin
Editor's Notes
- #41 The genre box shots were chosen because we have rights to use them, we are starting to make specific logos for each project going forward.