Uploaded byAdrian Cockcroft
AWS Re:Invent - High Availability Architecture at Netflix
The document discusses Netflix's cloud architecture and its operational strategies, specifically focusing on fault tolerance and high availability. It outlines various components such as load balancers, data replication across availability zones, and real-world examples of challenges and mitigation strategies in a cloud environment. The information emphasizes Netflix's approach to maintaining service reliability through automated responses, monitoring, and learned lessons from past outages.
In this document
Powered by AI
Intro to Netflix's cloud infrastructure, emphasizing their reliance on AWS cloud services.
Highlights years of production at Netflix from 2009 to 2011, indicating significant growth phases.
Discusses Netflix architecture applicable to any cloud, demonstrating components like user data, CDN, and device interactions.
Details on deployment in balanced availability zones with load balancers and data replication using Cassandra.
Examines various failure probabilities and mitigation strategies for applications, AWS regions, and data storage.
Links to key tech blogs discussing monitoring dynamic properties and fault tolerance in high-volume environments.
Covers classification of potential platform failures and outages in zones, focusing on service continuity.
Outlines regional network outages, cascading capacity overload, and impacts on load management.
Summarizes hardening the cloud through experiences and solutions developed from AWS outages.
Describes scalable and durable deployment patterns for managing Cassandra clusters and monitoring services.
Emphasizes Netflix's OSS contributions and resources for developers, including links to tech blogs and Github.
Encourages audience feedback on the presentation and overall experience at the re:Invent event.
More Related Content
![[Oracle Cloud Days Tokyo 2015] Oracle Database 12c最新情報 ～Maximum Availability ...](https://cdn.slidesharecdn.com/ss_thumbnails/d2-1bmaafordl-200702093455-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to AWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at Netflix
- 2.
- 3. (I’m skipping allthe cloud intro etc. Netflix runs in the cloud, if you hadn’t figured that out already you aren’t paying attention and should go to the other Netflix talks at AWS Re:Invent or read slideshare.net/netflix)
- 5.
- 9. Architecture applies toany cloud or datacenter Illustrated today using real world examples
- 10. Consumer User Data Electronics Web Site or Browse Discovery API AWS Cloud Services Personalization CDN Edge Locations DRM Customer Play Device (PC, Streaming API PS3, TV…) QoS Logging CDN Management and Steering Watch OpenConnect CDN Boxes Content Encoding
- 11. Each icon isthree to a few hundred instances across Cassandra three AWS zones memcached Web service Start Here S3 bucket Personalization movie group chooser
- 13. Deployed in ThreeBalanced Availability Zones Load Balancers Zone A Zone B Zone C Cassandra and Evcache Cassandra and Evcache Cassandra and Evcache Replicas Replicas Replicas
- 14. Triple Replicated Persistence Load Balancers Zone A Zone B Zone C Cassandra and Evcache Cassandra and Evcache Cassandra and Evcache Replicas Replicas Replicas
- 15. Isolated Regions US-East Load Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas
- 16. Failure Mode Probability Mitigation Plan Application Failure High Automatic degraded response AWS Region Failure Low Wait for region to recover AWS Zone Failure Medium Continue to run on 2 out of 3 zones Datacenter Failure Medium Migrate more functions to cloud Data store failure Low Restore from S3 backups S3 failure Low Restore from remote archive
- 17. Run what youwrote Rapid detection Rapid Response
- 18.
- 19.
- 20.
- 21. http://techblog.netflix.com/2012/11/edda-learn-stories-of-your-cloud.html Eureka Services metadata AWS Instances, ASGs, AppDynamics Request etc. flow Edda Monkeys
- 25.
- 26. Classify and namethe types of things that might go wrong in the platform or infrastructure
- 27. Zone Network Outage US-East Load Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Zone Dependent Zone Power Outage Service Outage Dependent Service could be @NetflixOSS platform or underlying infrastructure
- 29. Regional Network Outage US-East Load Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Control Plane Overload
- 31. Cascading Capacity Overload US-East Load Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Capacity demand migrates to services Platform and Infrastructure Migrating demand across regions may in another zone that don’t scale up fast Software Bugs and Global just spread the problem further… enough to take the load Configuration Errors “Oops…”
- 33. Hardening the cloud Lessons Learned at Scale Why Netflix Stays Up (Mostly)
- 35.
- 36.
- 37.
- 38. @NetflixOSS Eureka servicedirectory failed to mark down dead instances due to a configuration error US-East Load Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Effect: higher latency and errors Zone Power Outage Mitigation: Fixed configuration, and made Applications not using Zone- zone aware routing the default aware routing kept trying to talk to dead instances and timing out
- 40. Zone Enable DNS CommandQueue Per-Zone Control Plane Command Queues US-East Load Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas
- 41. A highly scalable,available and durable deployment pattern
- 42. Single function CassandraCluster Many Different Single-Function REST Clients Managed by Priam Between 6 and 72 nodes Stateless Data Access REST Service Astyanax Cassandra Client Optional Each icon represents a horizontally scaled service of three to hundreds of Datacenter instances deployed over three availability zones Update Flow Appdynamics Service Flow Visualization
- 43. Linux Base AMI(CentOS or Ubuntu) Optional Apache frontend, Java (JDK 6 or 7) memcached, non- java apps AppDynamics appagent monitoring Tomcat Monitoring Log rotation to S3 Application war file, base servlet, Healthcheck, status servlets, JMX AppDynamics GC and thread platform, client interface jars, interface, Servo autoscale machineagent dump logging Astyanax Epic/Atlas
- 44.
- 46. Linux Base AMI(CentOS or Ubuntu) Tomcat and Priam on JDK Java (JDK 7) Healthcheck, Status AppDynamics appagent monitoring Cassandra Server Monitoring AppDynamics Local Ephemeral Disk Space – 2TB of SSD or 1.6TB disk holding Commit log and GC and thread dump SSTables machineagent logging Epic/Atlas
- 47.
- 48. Cassandra Cassandra Cassandra Cassandra Cassandra S3 Backup Cassandra Cassandra Cassandra Cassandra Cassandra Cassandra Archive
- 49.
- 50.
- 51. Legend Github /Techblog Priam Exhibitor Servo and Autoscaling Scripts Cassandra as a Service Zookeeper as a Service Apache Contributions Astyanax Curator Honu Techblog Post Only Cassandra client for Java Zookeeper Patterns Log4j streaming to Hadoop Coming Soon CassJMeter EVCache Circuit Breaker - Hystrix Cassandra test suite Memcached as a Service Robust service pattern Cassandra Multi-region EC2 Eureka / Discovery Asgard - AutoScaleGroup based AWS datastore support Service Directory console Aegisthus Archaius Chaos Monkey Hadoop ETL for Cassandra Dynamics Properties Service Robustness verification Edda Explorers Latency Monkey Queryable config history Governator - Library lifecycle and Server-side latency/error injection Janitor Monkey dependency injection Odin REST Client + mid-tier LB Bakeries and AMI Workflow orchestration Blitz4j - Async logging Configuration REST endpoints Build dynaslaves
- 53. http://github.com/Netflix http://techblog.netflix.com http://slideshare.net/Netflix http://www.linkedin.com/in/adriancockcroft
- 54. We are sincerelyeager to hear your FEEDBACK on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance.