Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
RainmakersHow Netflix Operates Clouds for Maximum Freedom and Agility             Jeremy Edberg      Reliability Architect...
Do you have...                           •    A release Engineer?                           •    A QA department?         ...
Do you have...                      •     Upwards of 100 releases a day?Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
With more than 30 million streaming members in  the United States, Canada, Latin America, theUnited Kingdom, Ireland and t...
The Netflix Way                 •      Everything is “built for three”                 •      Fully automated build tools ...
PhilosophyTweet @jedberg with feedback!
Automate all the things!Tweet @jedberg with feedback!
Automate all the things!                     •     Application startup                     •     Configuration            ...
Automation                      •     Standard base image                      •     Tools to manage all the systems      ...
Shared state should be       stored in a shared service     Data on an instance should       be replicated to other       ...
“Build for Three”  We hold a boot camp for new engineers to teach them how        to build for a highly distributed enviro...
Tweet @jedberg with feedback!
Netflix on AWS                                2012   2012   2012                                IPv6   IPv6   IPv6        ...
Highly aligned, loosely coupled                  •     Services are built by different teams                        who wo...
Advantages to a Service           Oriented Architecture             •     Easier auto-scaling             •     Easier cap...
Freedom and Responsibility                  •     Developers deploy when they want                  •     They also manage...
All systems choices assume     some part will fail at some                point.Tweet @jedberg with feedback!
The Monkey Theory                           • Simulate things                                that go wrong                ...
Execution                                       Photo from I, Robot, copyright 20th Century FoxTweet @jedberg with feedback!
Netflix built a global PaaS               • Service Oriented                      Architecture               • HTTP/Rest i...
Netflix PaaS features           •     Supports all regions and zones           •     Multiple accounts           •     Cro...
What AWS Provides     •     Instances     •     Machine Images     •     Elastic IPs     •     Load Balancers     •     Se...
Linux Base AMI (CentOS or Ubuntu)    Optional                       Java (JDK 6 or 7)    Apache                       Appd...
The Netflix Platform           Discovery      (Eureka)Entrypoints       Circut Breakers (Hystrix)      (Edda)Configuration...
Tweet @jedberg with feedback!
N                                   ov      C                                   D     r u ra                              ...
Finding things    •      Discovery (Eureka)         •    Application to instance mapping         •    Heartbeat to keep tr...
Entrypoints (Edda)               •     REST API                   •      GET /REST/v2/instance/$id               •     Kee...
Entrypoints Exploration         Find all active instances    GET /REST/v2/view/instances          Find all instances in a ...
Keeping it all Straight        •     Configuration (Archaius)             •  Global variables (Fast properties)        •  ...
Keeping it Secure     •     Cryptex          •     Service for key management          •     High, medium and low value ke...
Storing it•    Cassandra (Priam, astyanax)    •     Configure and access Cassandra    •     Provide OO abstractions handle...
Data                            What do we do with it all?Tweet @jedberg with feedback!
We store it!                    • Cache (memcached)                    • Cassandra                    • RDS (MySql)Tweet @...
CassandraTweet @jedberg with feedback!
Why Cassandra?               • Availability over consistency               • Writes over reads               • We know Jav...
Using Cassandra at Netflix        •     Priam             •   Zero touch auto-config             •   State management     ...
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Cassandra ArchitectureTweet @jedberg with feedback!
Cassandra ArchitectureTweet @jedberg with feedback!   For more info, see DAT202: Optimizing your Cassandra Database on AWS
Tools             •     Asgard             •     AWS usage             •     Atlas             •     Chronos             •...
Tweet @jedberg with feedback!
Elastic Load                                   Balancer     Auto Scaling        Group                                Secur...
api-frontend   api-usprod-v007                             api-usprod-v008Tweet @jedberg with feedback!
api-frontend   api-usprod-v007                             api-usprod-v008Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Tweet @jedberg with feedback!
Netflix has moved the               granularity from the             instance to the clusterTweet @jedberg with feedback!
Why Bake? Traditional: •launch OS               Generic AMI •install packages                            Instance •install...
Getting Baked                                                    Artifactory                                              ...
Base Image  Baking                                                S3 / EBS                                                ...
App Image Baking                                                    S3 / EBS                                              ...
Linux Base AMI (CentOS or Ubuntu)    Optional                       Java (JDK 6 or 7)    Apache                       Appd...
Linux Base AMI (CentOS or Ubuntu)    Optional                       Java (JDK 6 or 7)    Apache                       Appd...
Linux Base AMI (CentOS or Ubuntu)    Optional                       Python    Apache                        monitoring   M...
The Monkey Theory                           • Simulate things                                that go wrong                ...
The simian army                •     Chaos -- Kills random instances                •     Chaos Gorilla -- Kills zones    ...
What’s going on?!Tweet @jedberg with feedback!
AtlasTweet @jedberg with feedback!
{  "clusters": [    "epic_aggregator",    "epic_aggregator-dev"  ],                                                       ...
Alert TuningTweet @jedberg with feedback!
Alert Systems                                           CORE                                            CORE   Atlas      ...
Tweet @jedberg with feedback!
ChronosTweet @jedberg with feedback!
Data Collection Pipeline                                Data Processing Pipeline                                        Te...
Chuckwa/Honu messages / min                                63 billion                                messages a           ...
Best PracticesTweet @jedberg with feedback!
Incident Reviews                                Ask the key questions:                •      What went wrong?             ...
Best Practices for Data        •     Have multiple copies of all data        •     Keep those copies in multiple AZs      ...
Netflix autoscaling      2                                Deployment                                     Text      1      ...
AWS Usage                                Dollar amounts have been carefully removedTweet @jedberg with feedback!
Going multi-zoneTweet @jedberg with feedback!
Benefits of Amazon’s Zones           •     Loosely connected           •     Low latency between zones           •     99....
Going Multi-regionTweet @jedberg with feedback!
Leveraging Multi-region         •     100% uptime is theoretically possible.         •     You have to replicate your data...
Circuit Breakers (Hystrix)        Be liberal in what you accept, strict in what you sendTweet @jedberg with feedback!
Just a quick reminder...                      •     (Some of) Netflix is open source:                            •   https...
We are sincerely eager to  hear your feedback on thispresentation and on re:Invent. Please fill out an evaluation    form ...
Questions?Tweet @jedberg with feedback!
Getting in touch         Email: jedberg@{gmail,netflix}.com         Twitter: @jedberg         Web: www.jedberg.net        ...
Upcoming SlideShare
Loading in …5
×

Do you have... • Upwards Devops at Netflix (re:Invent)

0 views

Published on

Do you have... • Upwards of 100 releases a day?Tweet @jedberg with feedback!

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download Full doc Ebook here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download PDF EBOOK here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/toudkt5 } ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download Full doc Ebook here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download PDF EBOOK here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/toudkt5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/toudkt5 } ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hey ver nice article
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

×