Source: http://ir.netflix.com

(I’m skipping all the cloud
intro etc. Netflix runs in thecloud, if you hadn’t figured that out already you aren’t paying attention and should go to the other Netflixtalks at AWS Re:Invent or read slideshare.net/netflix)

In production atNetflix20092009201020102010201020102011

Architecture applies to any cloud
or datacenter Illustrated today using real world examples

Consumer User DataElectronics Web Site
or Browse Discovery APIAWS Cloud Services PersonalizationCDN EdgeLocations DRM Customer Play Device (PC, Streaming API PS3, TV…) QoS Logging CDN Management and Steering Watch OpenConnect CDN Boxes Content Encoding

Each icon is three to
a few hundred instances across Cassandra three AWS zones memcached Web service Start Here S3 bucketPersonalization moviegroup chooser

Deployed in Three Balanced Availability
Zones Load Balancers Zone A Zone B Zone CCassandra and Evcache Cassandra and Evcache Cassandra and Evcache Replicas Replicas Replicas

Triple Replicated Persistence Load Balancers
Zone A Zone B Zone CCassandra and Evcache Cassandra and Evcache Cassandra and Evcache Replicas Replicas Replicas

Isolated Regions US-East Load Balancers
EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone CCassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas

Failure Mode Probability Mitigation PlanApplication
Failure High Automatic degraded responseAWS Region Failure Low Wait for region to recoverAWS Zone Failure Medium Continue to run on 2 out of 3 zonesDatacenter Failure Medium Migrate more functions to cloudData store failure Low Restore from S3 backupsS3 failure Low Restore from remote archive

Run what you wrote Rapid
detection Rapid Response

http://techblog.netflix.com/2012/06/annoucing-archaius-dynamic-properties.html

http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html

http://techblog.netflix.com/2012/11/edda-learn-stories-of-your-cloud.html Eureka Services metadata AWS
Instances, ASGs, AppDynamics Request etc. flow Edda Monkeys

http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html

Classify and name the types
of things thatmight go wrong in the platform or infrastructure

Zone Network Outage US-East Load
Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Zone DependentZone Power Outage Service Outage Dependent Service could be @NetflixOSS platform or underlying infrastructure

Regional Network Outage US-East Load
Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone CCassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Control Plane Overload

Cascading Capacity Overload US-East Load
Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra ReplicasCapacity demand migrates to services Platform and Infrastructure Migrating demand across regions mayin another zone that don’t scale up fast Software Bugs and Global just spread the problem further…enough to take the load Configuration Errors “Oops…”

Hardening the cloud Lessons Learned
at ScaleWhy Netflix Stays Up (Mostly)

http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html

http://googleappengine.blogspot.com/2012/10/about-todays-app-engine-outage.html

http://aws.amazon.com/message/67457/http://techblog.netflix.com/2012/07/lessons-netflix-learned-from-aws-storm.html

@NetflixOSS Eureka service directory failed
to mark down dead instances due to a configuration error US-East Load Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Effect: higher latency and errorsZone Power Outage Mitigation: Fixed configuration, and made Applications not using Zone- zone aware routing the default aware routing kept trying to talk to dead instances and timing out

Zone Enable DNSCommand Queue Per-Zone
Control Plane Command Queues US-East Load Balancers EU-West Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas

A highly scalable, available and
durable deployment pattern

Single function Cassandra Cluster Many
Different Single-Function REST Clients Managed by Priam Between 6 and 72 nodes Stateless Data Access REST Service Astyanax Cassandra Client OptionalEach icon represents a horizontally scaled service of three to hundreds of Datacenterinstances deployed over three availability zones Update Flow Appdynamics Service Flow Visualization

Linux Base AMI (CentOS or
Ubuntu)Optional Apache frontend, Java (JDK 6 or 7)memcached, non- java apps AppDynamics appagent monitoring Tomcat MonitoringLog rotation to S3 Application war file, base servlet, Healthcheck, status servlets, JMX AppDynamics GC and thread platform, client interface jars, interface, Servo autoscale machineagent dump logging Astyanax Epic/Atlas

http://github.com/netflix

Linux Base AMI (CentOS or
Ubuntu)Tomcat and Priam on JDK Java (JDK 7) Healthcheck, Status AppDynamics appagent monitoring Cassandra Server Monitoring AppDynamics Local Ephemeral Disk Space – 2TB of SSD or 1.6TB disk holding Commit log and GC and thread dump SSTables machineagent logging Epic/Atlas

http://github.com/netflix

Cassandra Cassandra Cassandra Cassandra Cassandra
S3 BackupCassandra Cassandra Cassandra Cassandra Cassandra Cassandra Archive

@NetflixOSS

http://techblog.netflix.com

Legend Github / Techblog Priam
Exhibitor Servo and Autoscaling Scripts Cassandra as a Service Zookeeper as a ServiceApache Contributions Astyanax Curator HonuTechblog Post Only Cassandra client for Java Zookeeper Patterns Log4j streaming to Hadoop Coming Soon CassJMeter EVCache Circuit Breaker - Hystrix Cassandra test suite Memcached as a Service Robust service pattern Cassandra Multi-region EC2 Eureka / Discovery Asgard - AutoScaleGroup based AWS datastore support Service Directory console Aegisthus Archaius Chaos Monkey Hadoop ETL for Cassandra Dynamics Properties Service Robustness verification Edda Explorers Latency Monkey Queryable config history Governator - Library lifecycle and Server-side latency/error injection Janitor Monkey dependency injection Odin REST Client + mid-tier LB Bakeries and AMI Workflow orchestration Blitz4j - Async logging Configuration REST endpoints Build dynaslaves

http://github.com/Netflix http://techblog.netflix.com http://slideshare.net/Netflixhttp://www.linkedin.com/in/adriancockcroft

We are sincerely eager tohear
your FEEDBACK on thispresentation and on re:Invent. Please fill out an evaluation form when you have a chance.