• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Devops at Netflix (re:Invent)
 

Devops at Netflix (re:Invent)

on

  • 7,942 views

How Netflix operates for maximum freedom and agility.

How Netflix operates for maximum freedom and agility.

Video here: https://www.youtube.com/watch?v=s0rCGFetdtM

Statistics

Views

Total Views
7,942
Views on SlideShare
4,204
Embed Views
3,738

Actions

Likes
10
Downloads
101
Comments
0

12 Embeds 3,738

http://www.buggymind.com 3663
http://mapleware-textcube.blogspot.kr 21
http://buggymind.com 20
https://twitter.com 13
http://www.twylah.com 7
http://localhost 4
http://www.google.co.kr 2
http://mapleware-textcube.blogspot.in 2
http://egile.tistory.com 2
http://mapleware-textcube.blogspot.com 2
http://www.onlydoo.com 1
http://moderation.local 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • My friends Joe and Carl already told you about Nac and our build system. This allows the devs to take control of their deployment. Each team is responsible for their own deployments and uptime. When something breaks, we have a system that lets us page a team who then gets on and fixes their stuff. Each team is responsible for their own destiny. So how do we stay reliable when we have no control? Information.
  • Automate as much as you can
  • The more automated things are, the easier it is to be a sysadmin. Application startup Configuration Code deployment Full system deployment The more automated things are, the easier it is to scale especially in a virtualized environment with auto-scaling And virtualized computing added the last bit, the ability to automate system deployment. (Ok, that ’ s not entirely true, but watch me wave my hands and say it is)
  • In most places, you have this. Standard image with tools to manage the systems and the deployment.
  • By building for three, you can reasonably lose one of your instances and still be stable.
  • replication factor quorum reads / writes
  • In most systems, you worry about the software and installing it on an OS. At Netflix, the smallest thing we worry about is the instance image, which lives in a cluster. We ’ ve essentially built a platform for doing automated deployment of Java code (and some Python too!)
  • So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines? We like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because: • More Reliable: less systems that can fail at deploy time right when we need them most. • Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise. • Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster
  • OK, back to the build pipeline again. We have a vague “ app bundles ” output in this diagram. Let ’ s delve into how we manage the application bundle artifacts in more detail.
  • The first step of the baking process is to create the “ base ” image that we will use for baking all app images. This is done once every week or two. We start with a standard Linux distro as a foundation (CentOS now, Ubuntu on the way), and add in our favorite, our custom and customized packages: • Apache, Java (JDK 6 and 7), Tomcat, Perl, Python, provisioning and startup scripts, log management tools, monitoring agents, etc. The end result is a beefed-up OS image that is ready to go, and just needs an app added.
  • The first step of the baking process is to create the “ base ” image that we will use for baking all app images. This is done once every week or two. We start with a standard Linux distro as a foundation (CentOS now, Ubuntu on the way), and add in our favorite, our custom and customized packages: • Apache, Java (JDK 6 and 7), Tomcat, Perl, Python, provisioning and startup scripts, log management tools, monitoring agents, etc. The end result is a beefed-up OS image that is ready to go, and just needs an app added.
  • Gateway classifies and routes events based on severity and the systems involved. The gateway currently processes around 48K events a day
  • (step through) here are some best practices we ’ ve learned over the last year with EC2. Some of these we follow well, and some we need to follow better.
  • At Netflix we use autoscaling the help manage reliability and cost. Here is one of our clusters scaling up and down. We are tuning for the holidays, so you can see parts where we are doing squeeze tests and adjusting the scaling speed and values.
  • Amazon will help you as well. One way they do this is by providing zones. Each zone is like an island that is loosely connected to the other zones, but mostly distinct.
  • So how do you get better than 99.95% uptime? Multiple zones! By spreading your systems out across multiple zones, you should be able to withstand the failure of one zone. In a little bit, I ’ ll go over how reddit and Netflix used a multizone strategy to survive outages.
  • Amazon, as well as other providers, offer multiple regions as well. Regions are essentially like separate providers with the same featureset. Your data does not get shared across regions
  • You can contact me in one of these ways, or ask your question now. thank you.

Devops at Netflix (re:Invent) Devops at Netflix (re:Invent) Presentation Transcript

  • RainmakersHow Netflix Operates Clouds for Maximum Freedom and Agility Jeremy Edberg Reliability Architect, Netflix
  • Do you have... • A release Engineer? • A QA department? • Chef or Puppet to manage your systems?Tweet @jedberg with feedback!
  • Do you have... • Upwards of 100 releases a day?Tweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • With more than 30 million streaming members in the United States, Canada, Latin America, theUnited Kingdom, Ireland and the Nordics, Netflix isthe worlds leading internet subscription service for enjoying movies and TV programs streamed over the internet to PCs, Macs and TV. Source: http://ir.netflix.com Tweet @jedberg with feedback!
  • The Netflix Way • Everything is “built for three” • Fully automated build tools to test and make packages • Fully automated machine image bakery • Fully automated image deployment • Independent teams responsible for both Dev and OpsTweet @jedberg with feedback!
  • PhilosophyTweet @jedberg with feedback!
  • Automate all the things!Tweet @jedberg with feedback!
  • Automate all the things! • Application startup • Configuration • Code deployment • System deploymentTweet @jedberg with feedback!
  • Automation • Standard base image • Tools to manage all the systems • Automated code deploymentTweet @jedberg with feedback!
  • Shared state should be stored in a shared service Data on an instance should be replicated to other instancesTweet @jedberg with feedback!
  • “Build for Three” We hold a boot camp for new engineers to teach them how to build for a highly distributed environment.Tweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • Netflix on AWS 2012 2012 2012 IPv6 IPv6 IPv6 Open ConnectTweet @jedberg with feedback!
  • Highly aligned, loosely coupled • Services are built by different teams who work together to figure out what each service will provide. • The service owner publishes an API that anyone can use.Tweet @jedberg with feedback!
  • Advantages to a Service Oriented Architecture • Easier auto-scaling • Easier capacity planning • Identify problematic code-paths more easily • Narrow in the effects of a change • More efficient local cachingTweet @jedberg with feedback!
  • Freedom and Responsibility • Developers deploy when they want • They also manage their own capacity and autoscaling • And fix anything that breaks at 4am!Tweet @jedberg with feedback!
  • All systems choices assume some part will fail at some point.Tweet @jedberg with feedback!
  • The Monkey Theory • Simulate things that go wrong • Find things that are differentTweet @jedberg with feedback!
  • Execution Photo from I, Robot, copyright 20th Century FoxTweet @jedberg with feedback!
  • Netflix built a global PaaS • Service Oriented Architecture • HTTP/Rest interfaces between servicesTweet @jedberg with feedback!
  • Netflix PaaS features • Supports all regions and zones • Multiple accounts • Cross region/account replication • Internationalized, localized and GeoIP routed • Advanced key management • Autoscaling with 1000s of instances • Monitoring and alerting on millions of metricsTweet @jedberg with feedback!
  • What AWS Provides • Instances • Machine Images • Elastic IPs • Load Balancers • Security groups / Autoscaling groups • Availability zones and regionsTweet @jedberg with feedback!
  • Linux Base AMI (CentOS or Ubuntu) Optional Java (JDK 6 or 7) Apache Appdynamics App Agent Monitoring monitoring Tomcat Log Rotation to S3 Application war file, base Healthcheck, status GC and servlet, platform, interface servelets, JMX interface, Appdynamics thread dump jars for dependent services Servo autoscale Machine Agent loggingTweet @jedberg with feedback!
  • The Netflix Platform Discovery (Eureka)Entrypoints Circut Breakers (Hystrix) (Edda)Configuration Cassandra (Priam & (Archaius) Astyanax & CassJMeter) Zookeeper (Exhibitor) Cryptex logging (Blitz4j & Honu) AKMSEvCache NIWS Proxiesi18n Geo L10n Base Open SourceTweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • N ov C D r u ra e to c 20 12 A x sty Fe an b STweet @jedberg with feedback! o er a M Pr v ar m ia C A e r as sJ pr Ex M M r hi b et a y ito Ju n A s rch Ju A a l d sg iu ar C A Open Source at Netflix M ha Edda Blitz4j ug Hystrix on os keGovernator Se y p Eu a re O k ct
  • Finding things • Discovery (Eureka) • Application to instance mapping • Heartbeat to keep track of health • Entrypoints (Edda) • Local database of AWS resources • NIWS (Netflix Internal Web Service) • On instance software load balancer • Handles retry logic • Geo (Geolocation library) • Provides IP to Lat/Lon mapping for any service that needs it.Tweet @jedberg with feedback!
  • Entrypoints (Edda) • REST API • GET /REST/v2/instance/$id • Keeps track of all resources • Autoscaling groups, EIPs, Instances, Applications, Clusters, HistoryTweet @jedberg with feedback!
  • Entrypoints Exploration Find all active instances GET /REST/v2/view/instances Find all instances in a GET /REST/v2/group/clusters cluster Show only ASG name, /v2/aws/autoScalingGroups/edda-v123;_pp: (autoScalingGroupName,instances: instance ID and health (instanceId,lifecycleState)) Which ASG contains a /v2/aws/autoScalingGroups;instances.instanceId=i- 96f3ca3a particular instance?Tweet @jedberg with feedback!
  • Keeping it all Straight • Configuration (Archaius) • Global variables (Fast properties) • Base • Base system. Prod vs. Test, etc • Zookeeper (Curator) • Locks, other similar coordination • Logging (Blitz4j and Honu) • Keep track of what happened and store it for post analysis.Tweet @jedberg with feedback!
  • Keeping it Secure • Cryptex • Service for key management • High, medium and low value keys • AKMS (Amazon Key Management System) • Hands out keys to instances (and dev boxes) so they don’t have to store the key on the instanceTweet @jedberg with feedback! For more info, see SEC201: Security Panel
  • Storing it• Cassandra (Priam, astyanax) • Configure and access Cassandra • Provide OO abstractions handle connection pooling, discovery of hosts• EVCache (Eccentric Volatile Cache) • Wrapper for memcached to handle zone awareness and replication• Proxies • Get data out of the datacenter and into the cloud.Tweet @jedberg with feedback!
  • Data What do we do with it all?Tweet @jedberg with feedback!
  • We store it! • Cache (memcached) • Cassandra • RDS (MySql)Tweet @jedberg with feedback!
  • CassandraTweet @jedberg with feedback!
  • Why Cassandra? • Availability over consistency • Writes over reads • We know Java • Open source + supportTweet @jedberg with feedback!
  • Using Cassandra at Netflix • Priam • Zero touch auto-config • State management • Token assignment • Node replacement • Backup/restore to/from S3 • Astyanax • OO abstraction to Cassandra • Multi-region supportTweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • Cassandra ArchitectureTweet @jedberg with feedback!
  • Cassandra ArchitectureTweet @jedberg with feedback! For more info, see DAT202: Optimizing your Cassandra Database on AWS
  • Tools • Asgard • AWS usage • Atlas • Chronos • Build system • Explorers (Cassandra and SimpleDB)Tweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon MachineTweet @jedberg with feedback! Image
  • api-frontend api-usprod-v007 api-usprod-v008Tweet @jedberg with feedback!
  • api-frontend api-usprod-v007 api-usprod-v008Tweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • Netflix has moved the granularity from the instance to the clusterTweet @jedberg with feedback!
  • Why Bake? Traditional: •launch OS Generic AMI •install packages Instance •install app Netflix: •launch OS+app App AMI InstanceTweet @jedberg with feedback!
  • Getting Baked Artifactory Artifactory app bundles Ivy snapshot / release libraries libraries / appsJenkins Jenkins resolve resolve test test publish publish sync sync compile compile build build report report source Perforce / /Git Perforce Git Ant targets Groovy all over Tweet @jedberg with feedback!
  • Base Image Baking S3 / EBS foundation foundation AMI AMI Linux: CentOS, Fedora, Ubuntu base base AMI AMI mount snapshot Ready for Yum // Apt Yum Apt app install Bakery Bakery bake AWS RPMs: Apache, Java... ec2 slave instancesTweet @jedberg with feedback!
  • App Image Baking S3 / EBS base AMI base AMI Linux, Apache, Java, Tomcat app app AMI AMI mount snapshot Jenkins // Yum // Jenkins Yum Ready Artifactory Artifactory to launch! install Bakery Bakery AWS app bundle ec2 slave instancesTweet @jedberg with feedback!
  • Linux Base AMI (CentOS or Ubuntu) Optional Java (JDK 6 or 7) Apache Appdynamics App Agent Monitoring monitoring Tomcat Log Rotation to S3 Application war file, base Healthcheck, status GC and servlet, platform, interface servelets, JMX interface, Appdynamics thread dump jars for dependent services Servo autoscale Machine Agent loggingTweet @jedberg with feedback!
  • Linux Base AMI (CentOS or Ubuntu) Optional Java (JDK 6 or 7) Apache Appdynamics App Agent Monitoring monitoring JBoss Log Rotation to S3 Application war file, base Healthcheck, status GC and servlet, platform, interface servelets, JMX interface, Appdynamics thread dump jars for dependent services Servo autoscale Machine Agent loggingTweet @jedberg with feedback!
  • Linux Base AMI (CentOS or Ubuntu) Optional Python Apache monitoring Monitoring Django Log Rotation to S3 Application file, base server, platform, interface Appdynamics logging libs for dependent services Machine AgentTweet @jedberg with feedback!
  • The Monkey Theory • Simulate things that go wrong • Find things that are differentTweet @jedberg with feedback!
  • The simian army • Chaos -- Kills random instances • Chaos Gorilla -- Kills zones • Chaos Kong -- Kills regions • Latency -- Degrades network and injects faults • Conformity -- Looks for outliers • Circus -- Kills and launches instances to maintain zone balance • Doctor -- Fixes unhealthy resources • Janitor -- Cleans up unused resources • Howler -- Yells about bad things like Amazon limit violations • Security -- Finds security issues and expiring certificatesTweet @jedberg with feedback! For more info, see ARC301: Intro to Chaos Monkey & the Simian Army
  • What’s going on?!Tweet @jedberg with feedback!
  • AtlasTweet @jedberg with feedback!
  • {  "clusters": [    "epic_aggregator",    "epic_aggregator-dev"  ], {  "alerts": [       "metricName": "EpicPlugin_MetricCount",    // you can use javascript style comments in the config       "applyTo": "instance",    {       "description": "${instanceId} is reporting too many metrics",      "metricName": "EpicPlugin_NumDropped",       "condition": {      "applyTo": "cluster",         "type": "NumOccurrences",      "condition": {         "num": 4,        "type": "StaticThreshold",         "condition": {        "max": 0.0           "type": "StaticThreshold",      },           "max": 0.0      "severity": "major",         }      "description": "plugin is dropping metrics"       },    },       "additionalDetails": {    {         "statusUrl": "http://${publicDnsName}:7001/Status",      "metricName": "EpicPlugin_NumDropped_Instance",         "nacClusterUrl": "nac${env}/${region}/cluster/show/${cluster}"      "applyTo": "instance",       }      "condition": {       "overrides": {        "type": "NumOccurrences",         "subject": "${instanceId} is reporting too many metrics",        "num": 4,         "incident_key": "${metricName}:${instanceId}",        "condition": {         "service_key_override": "12345",          "type": "StaticThreshold",         "email_override": "devnull@netflix.com"          "max": 0.0       },        }       "severity": "minor"      },     }      "overrides": {   ]        "service_key_override": "12345", }        "require_instance_status_not_in: ["DOWN", "OUT_OF_SERVICE"],        "email_override": "devnull@netflix.com"      },      "severity": "minor"    },    Example Alert Config Tweet @jedberg with feedback!
  • Alert TuningTweet @jedberg with feedback!
  • Alert Systems CORE CORE Atlas Event Event Paging Paging Service Gateway Service alerting Gateway alerting CORE CORE Appdynamics Agent Amazon Amazon Agent SES api SES api CORE CORE Agent Agent api api Other Other Team’ss Team’ Agent AgentTweet @jedberg with feedback!
  • Tweet @jedberg with feedback!
  • ChronosTweet @jedberg with feedback!
  • Data Collection Pipeline Data Processing Pipeline TextTweet @jedberg with feedback! For more info, see BDT303: Data Science with Elastic MapReduce
  • Chuckwa/Honu messages / min 63 billion messages a dayTweet @jedberg with feedback!
  • Best PracticesTweet @jedberg with feedback!
  • Incident Reviews Ask the key questions: • What went wrong? • How could we have detected it sooner? • How could we have prevented it? • How can we prevent this class of problem in the future? • How can we improve our behavior for next time?Tweet @jedberg with feedback!
  • Best Practices for Data • Have multiple copies of all data • Keep those copies in multiple AZs • Avoid keeping state on a single instance • Take frequent snapshots of EBS disks • No secret keys on the instanceTweet @jedberg with feedback!
  • Netflix autoscaling 2 Deployment Text 1 Traffic PeakTweet @jedberg with feedback!
  • AWS Usage Dollar amounts have been carefully removedTweet @jedberg with feedback!
  • Going multi-zoneTweet @jedberg with feedback!
  • Benefits of Amazon’s Zones • Loosely connected • Low latency between zones • 99.95% uptime guarantee per regionTweet @jedberg with feedback!
  • Going Multi-regionTweet @jedberg with feedback!
  • Leveraging Multi-region • 100% uptime is theoretically possible. • You have to replicate your data • This will cost moneyTweet @jedberg with feedback!
  • Circuit Breakers (Hystrix) Be liberal in what you accept, strict in what you sendTweet @jedberg with feedback!
  • Just a quick reminder... • (Some of) Netflix is open source: • https://github.com/netflixTweet @jedberg with feedback!
  • We are sincerely eager to hear your feedback on thispresentation and on re:Invent. Please fill out an evaluation form when you have a chance.
  • Questions?Tweet @jedberg with feedback!
  • Getting in touch Email: jedberg@{gmail,netflix}.com Twitter: @jedberg Web: www.jedberg.net Facebook: facebook.com/jedberg Linkedin: www.linkedin.com/in/jedbergTweet @jedberg with feedback!