Downloaded 363 times
Uploaded byAdrian Cockcroft
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is doing Continuous Delivery
The document discusses Netflix's cloud architecture and operational strategies, emphasizing their transition to a microservices model utilizing AWS and DevOps practices. It highlights the evolution of IT cycles from mainframe to cloud-native approaches, detailing the challenges and solutions Netflix faced during its scaling process. Key elements include automation, continuous deployment, and customer-centric metrics for optimizing performance and service delivery.
More Related Content
Similar to Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is doing Continuous Delivery
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is doing Continuous Delivery
- 1. Now Playing onNetflix: Adventures in a Cloudy Future CMG November 2013 Adrian Cockcroft @adrianco @NetflixOSS http://www.linkedin.com/in/adriancockcroft
- 2. Netflix Member WebSite Home Page Personalization Driven – How Does It Work?
- 3. How Netflix Usedto Work Consumer Electronics Oracle Monolithic Web App AWS Cloud Services MySQL CDN Edge Locations Oracle Datacenter Customer Device (PC, PS3, TV…) Monolithic Streaming App MySQL Content Management Limelight/Level 3 Akamai CDNs Content Encoding
- 4. How Netflix StreamingWorks Today Consumer Electronics User Data Web Site or Discovery API AWS Cloud Services Personalization CDN Edge Locations DRM Datacenter Customer Device (PC, PS3, TV…) Streaming API QoS Logging OpenConnect CDN Boxes CDN Management and Steering Content Encoding
- 5.
- 6. Netflix Scale • Tensof thousands of instances on AWS – Typically 4 core, 30GByte, Java business logic – Thousands created/removed every day • Thousands of Cassandra NoSQL storage nodes – Mostly 8 core, 60Gbyte, 2TByte of SSD – 65 different clusters, over 300TB data, triple zone – Over 40 are multi-region clusters (6, 9 or 12 zone) – Biggest 288 nodes, 300K rps, 1.3M wps
- 7. Reactions over time 2009“You guys are crazy! Can’t believe it” 2010 “What Netflix is doing won’t work” 2011 “It only works for ‘Unicorns’ like Netflix” 2012 “We’d like to do that but can’t” 2013 “We’re on our way using Netflix OSS code”
- 8. "This is theIT swamp draining manual for anyone who is neck deep in alligators." Adrian Cockcroft, Cloud Architect at Netflix
- 10.
- 11. Goal of TraditionalIT: Reliable hardware running stable software
- 13.
- 14.
- 15.
- 17. Incidents – Impactand Mitigation Public Relations Media Impact PR Y incidents mitigated by Active Active, game day practicing X Incidents High Customer Service Calls CS YY incidents mitigated by better tools and practices XX Incidents Affects AB Test Results Metrics impact – Feature disable XXX Incidents No Impact – fast retry or automated failover XXXX Incidents YYY incidents mitigated by better data tagging
- 18. Web Scale Architecture AWS Route53 DynECT DNS UltraDNS DNS Automation RegionalLoad Balancers Regional Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas
- 19. CIO Says SpeedIT Up!
- 20. “Get inside youradversaries' OODA loop to disorient them” Colonel Boyd, USAF
- 21. Land grab opportunity Engage customers Deliver Measure Customers Act Competitive Move Observe Colonel Boyd, USAF “Getinside your adversaries' OODA loop to disorient them” Customer Pain Point Analysis Orient Model Hypotheses Implement Decide Commit Resources Plan Response Get Buy-in
- 22. Territory Expansion Print Ad Campaign Upgrade Mainframe Measure Revenue Act Foreign Competition Observe Mainframe Era -1 year cycle Customer Pain Point Systems Analysis Orient Capacity Model Customize Vendor SW Decide Vendor Evaluation 5 year Plan Board Level Buyin
- 23. 80’s Mainframe InnovationCycle • • • • • Cost $1M to $100M Duration 1 to 5 years Bet the whole company Cost of failure – bankrupt or bought Cobol and DB2 on MVS
- 24. Territory Expansion TV Advert Campaign Install Servers Measure Revenue Act Foreign Competition Observe Client/Server Era –3 month cycle Customer Pain Point Data Warehouse Orient Capacity Estimate Customize Vendor SW Decide Vendor Evaluation 1 year Plan CIO Level Buy-in
- 25. 90’s Client ServerInnovation Cycle • • • • • Cost $100K to $10M Duration 3 – 12 months Bet a product line or division Cost of failure – revenue hit, CIO’s job C++ and Oracle on Solaris
- 26. Territory Expansion Web Display Ads Measure Sales Install Capacity Act Competitive Moves Observe Commodity Era –2 week agile train Customer Pain Point Data Warehouse Orient Capacity Estimate Code Feature Decide Feature Priority 2 Week Plan Business Buy-in
- 27. 00’s Commodity AgileInnovation Cycle • • • • • Cost $10K to $1M Duration 2 – 12 weeks Bet a product feature Cost of failure – product mgr reputation Java and MySQL on RedHat Linux
- 28. Train Model ProcessHand-Off Steps Product Manager Developer QA Integration Team Operations Deploy Team BI Analytics Team
- 29. What Happened? Rate ofchange increased Cost and size and risk of change reduced
- 30. Cloud Native Construct ahighly agile and highly available service from ephemeral and assumed broken components
- 31. Real Web ServerDependencies Flow (Netflix Home page business transaction as seen by AppDynamics) Each icon is three to a few hundred instances across three AWS zones Cassandra memcached Start Here Personalization movie group choosers (for US, Canada and Latam) Web service S3 bucket
- 32. Continuous Deployment No timefor handoff to IT
- 33. Developer Self Service Freedomand Responsibility
- 34. Developers run what theywrote Root access and pagerduty
- 35. IT is aCloud API DEVops automation
- 36. Github all thethings! Leverage social coding
- 38. Putting it alltogether…
- 39. Land grab opportunity Launch AB Test Automatic Deploy Measure Customers Act Competitive Move Observe Continuous Deliveryon Cloud Customer Pain Point Analysis Orient Model Hypotheses Increment Implement Decide Plan Response Share Plans JFDI
- 40. Continuous Innovation Cycle • • • • • Costnear zero, variable expense Duration hours to days Bet a decoupled microservice code push Cost of failure – near zero, instant rollback Clojure/Scala/Python on NoSQL on Cloud
- 41. Continuous Deploy Hand-OffSteps Product Manager A/B test setup and enable Self service hypothesis test results Developer Automated test Self service deploy, on call Self service analytics
- 42. Continuous Deploy Automation Checkin code, Jenkins build Bake AMI, launch in test env Functional and performance test Production canary test Production red/black push
- 43.
- 44.
- 45. Global Deploy Automation Afternoonin California Night-time in Europe If passes test suite, canary then deploy West Coast Load Balancers East Coast Load Balancers Europe Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Canary then deploy Next day on West Coast Canary then deploy Next day on East Coast After peak in Europe
- 46. Ephemeral Instances • Largestservices are autoscaled • Average lifetime of an instance is 36 hours Autoscale Up Autoscale Down P u s h
- 47. (New Today!) PredictiveAutoscaling 24 Hours predicted traffic vs. actual More morning load Sat/Sun high traffic Lower load on Weds Prediction driving AWS Autoscaler to plan capacity
- 48.
- 49. Takeaway Speed Wins Assume Broken CloudNative Automation Github is your “app store” and resumé @adrianco @NetflixOSS http://netflix.github.com