Download as PDF, PPTX
Uploaded byAdrian Cockcroft
Netflix on Cloud - combined slides for Dev and Ops
This document contains slides from a presentation given by Adrian Cockcroft on Netflix's use of cloud computing on Amazon Web Services (AWS). The summary includes: 1) Netflix moved most of its infrastructure to AWS to leverage AWS's scale and features rather than building its own datacenters, as capacity growth was unpredictable and datacenters were inflexible. 2) Netflix uses many AWS services including EC2, S3, EBS, EMR and more. It deployed a large movie encoding farm on EC2, stores content on S3, uses EMR/Hadoop for log analysis, and a CDN for content delivery. 3) Netflix has learned that cloud tools don't always scale for large
More Related Content
![[AWSマイスターシリーズ] AWS CloudFormation](https://cdn.slidesharecdn.com/ss_thumbnails/20131127aws-meister-regenerate-cloudformation-131202184424-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Netflix on Cloud - combined slides for Dev and Ops
Netflix on Cloud - combined slides for Dev and Ops
- 1. Ne#lix in the Cloud Nov 6, 2010 Adrian Cockcro: @adrianco #ne#lixcloud h=p://www.linkedin.com/in/adriancockcro:
- 2. Combined Slides For both Developer and OperaJons Audiences (2-‐3 hours of content) Teaser intro slides also at slideshare.net/adrianco Oct 14th Beta #devops subset -‐ Cloud CompuJng Meetup Nov 3rd GA – QConSF Developer Oriented subset Nov 6th, 2010 – Combined slides at slideshare.net/adrianco
- 3. With more than 16 million subscribers in the United States and Canada, Ne9lix, Inc. is the world’s leading Internet subscripAon service for enjoying movies and TV shows. Source: h=p://ir.ne#lix.com
- 4. Why Give This Talk?
- 5. Ne#lix is Path-‐finding The Cloud ecosystem is evolving very fast Share with and learn from the cloud community
- 6. We want to use clouds, not build them Cloud technology should be a commodity Public cloud and open source for agility and scale
- 7. We are looking for talent Ne#lix wants to connect with the very best engineers
- 8.
- 9. We stopped building our own datacenters Capacity growth rate is acceleraJng, unpredictable Product launch spikes -‐ iPhone, Wii, PS3, XBox Datacenter is large inflexible capital commitment
- 10. Customers Q3 year/year +52% Total and +145% Streaming 18 16 14 12 10 8 6 4 2 0 2009Q2 2009Q3 2009Q4 2010Q1 2010Q2 2010Q3 Source: h=p://ir.ne#lix.com
- 11. Leverage AWS Scale “the biggest public cloud” AWS investment in tooling and automaJon AWS zones for high availability, scalability AWS skills are common on resumes…
- 12. Leverage AWS Feature Set “two years ahead of the others” EC2, S3, SDB, SQS, EBS, EMR, ELB, ASG, IAM, RDB…
- 13. “The cloud lets its users focus on delivering differenAaAng business value instead of wasAng valuable resources on the undifferen)ated heavy li0ing that makes up most of IT infrastructure.” Werner Vogels Amazon CTO
- 14. Ne#lix Deployed on AWS Content Logs Play WWW API Video S3 DRM Search Metadata Masters EMR CDN Movie Device EC2 Hadoop rouJng Choosing Config TV Movie S3 Hive Bookmarks RaJngs Choosing Business Mobile CDN Logging Similars Intelligence iPhone
- 15. Movie Encoding farm (2009) • Tens of thousands of videos Content • Thousands of EC2 instances Video • Encoding apps on MS Windows Masters • ~100 speed/format permutaJons • Petabytes of S3 EC2 • Content Delivery Networks S3 “Ne9lix is one of the largest customers of the biggest CDNs Akamai and Limelight” CDN
- 16. Hadoop -‐ ElasJc Map-‐Reduce (2009) • Web Access Logs Logs • Streaming Service Logs S3 • Terabyte per day scale • Easy Hadoop via Amazon EMR EMR • Hive SQL “Data Mart” Hadoop • Gateway to Datacenter BI Hive Slideshare.net talks evamtse “Ne#lix: Hive User Group” h=p://slidesha.re/aqJLAC adrianco “Crunch Your Data In The Cloud” h=p://slidesha.re/dx4oCK Business Intelligence
- 17. Streaming Service Back-‐end (early 2010) • PC/Mac Silverlight Player Support Play • Highly available “play bu=on” DRM • DRM Key Management CDN • Generate route to stream on CDN rouJng • Lookup bookmark for user/movie Bookmarks • Update bookmark for user/movie • Log quality of service Logging
- 18. Web site, a page at a Jme (through 2010) • Clean presentaJon layer rewrite WWW • Search auto-‐complete Search • Search backend and landing page • Movie and genre choosing Movie • Star raJngs and recommendaJons Choosing • Similar movies RaJngs • Page by page to 80% of views (leave account signup in Datacenter for now) Similars
- 19. API for TV devices and iPhone etc. (2010) • REST API: developer.ne#lix.com API • Interfaces to everything else Metadata • TV Device ConfiguraJon • Personalized movie choosing Device Config • iPhone Launch in the cloud only TV Movie Choosing “Ne9lix is an API for streaming to TVs Mobile (we also do DVD’s and a web site)” iPhone
- 20. Ne#lix EC2 Instances per Account Encoding Test and ProducJon Log Analysis
- 21.
- 22. Datacenter oriented tools don’t work Ephemeral instances High rate of change
- 23. Cloud Tools Don’t Scale for Enterprise Too many are “Startup” oriented Built our own tools Drove vendors hard
- 24. “fork-‐li:ed” apps don’t work well Fragile Too many datacenter oriented assumpJons
- 25. Faster to re-‐code from scratch • Opportunity to pay down technical debt • Re-‐architected and re-‐wrote most of the code • Fine grain web services • Leveraged many open source Java projects • SystemaJcally instrumented • “NoSQL” SimpleDB backend
- 26. “In the datacenter, robust code is best pracAce. In the cloud, it’s essenAal.”
- 27. Takeaway Ne9lix is path-‐finding the use of public AWS cloud to replace in-‐house IT for non-‐trivial applicaAons with hundreds of developers and thousands of systems. (Pause for quesJons before we dive into details)
- 28. What, Why and How? The details…
- 29. Synopsis • The Goals – Faster, Scalable, Available and ProducJve • AnJ-‐pa=erns and Cloud Architecture – The things we wanted to change and why • Cloud Bring-‐up Strategy – Developer TransiJons and Tools • Roadmap and Next Steps
- 30. Goals • Faster – Lower latency than the equivalent datacenter web pages and API calls – Measured as mean and 99th percenJle – For both first hit (e.g. home page) and in-‐session hits for the same user • Scalable – Avoid needing any more datacenter capacity as subscriber count increases – No central verJcally scaled databases – Leverage AWS elasJc capacity effecJvely • Available – SubstanJally higher robustness and availability than datacenter services – Leverage mulJple AWS availability zones – No scheduled down Jme, no central database schema to change • ProducJve – OpJmize agility of a large development team with automaJon and tools – Leave behind complex tangled datacenter code base (~8 year old architecture) – Enforce clean layered interfaces and re-‐usable components
- 31. Cloud Architecture Pa=erns Where do we start?
- 32. Datacenter AnJ-‐Pa=erns What do we currently do in the datacenter that prevents us from meeJng our goals?
- 33. Architecture • So:ware Architecture – The abstracJons and interfaces that developers build against • Systems Architecture – The service instances that define availability, scalability • Compose-‐ability – so:ware architecture that is independent of the systems architecture – decoupled flexible building block components
- 34. Rewrite from Scratch Not everything is cloud specific Pay down technical debt Robust pa=erns
- 35. Old Datacenter vs. New Cloud Arch Central SQL Database Distributed Key/Value NoSQL SJcky In-‐Memory Session Shared Memcached Session Cha=y Protocols Latency Tolerant Protocols Tangled Service Interfaces Layered Service Interfaces Instrumented Code Instrumented Service Pa=erns Fat Complex Objects Lightweight Serializable Objects Components as Jar Files Components as Services
- 36. The Central SQL Database • Datacenter has a central database – Everything in one place is convenient unJl it fails – Customers, movies, history, configuraJon • Schema changes require downJme AnA-‐paTern impacts scalability, availability
- 37. The Distributed Key-‐Value Store • Cloud has many key-‐value data stores – More complex to keep track of, do backups etc. – Each store is much simpler to administer DBA – Joins take place in java code • No schema to change, no scheduled downJme • Latency for Memcached vs. Oracle vs. SimpleDB – Memcached is dominated by network latency <1ms – Oracle for simple queries is a few milliseconds – SimpleDB has replicaJon and REST overheads >10ms
- 38. The SJcky Session • Datacenter SJcky Load Balancing – Efficient caching for low latency – Tricky session handling code – Middle Jer load balancer has issues in pracJce • Encourages concentrated funcJonality – one service that does everything AnA-‐paTern impacts producAvity, availability
- 39. The Shared Session • Cloud Uses Round-‐Robin Load Balancing – Simple request-‐based code – External shared caching with memcached • More flexible fine grain services – Works be=er with auto-‐scaled instance counts
- 40. Cha=y Opaque and Bri=le Protocols • Datacenter service protocols – Assumed low latency for many simple requests • Based on serializing exisJng java objects – Inefficient formats – IncompaJble when definiJons change AnA-‐paTern causes producAvity, latency and availability issues
- 41. Robust and Flexible Protocols • Cloud service protocols – JSR311/Jersey is used for REST/HTTP service calls – Custom client code includes service discovery – Support complex data types in a single request • Apache Avro – Evolved from Protocol Buffers and Thri: – Includes JSON header defining key/value protocol – Avro serializaJon is half the size and several Jmes faster than Java serializaJon, more work to code
- 42. Persisted Protocols • Persist Avro in Memcached – Save space/latency (zigzag encoding, half the size) – Less bri=le across versions – New keys are ignored – Missing keys are handled cleanly • Avro protocol definiJons – Can be wri=en in JSON or generated from POJOs – It’s hard, needs be=er tooling
- 43. Tangled Service Interfaces • Datacenter implementaJon is exposed – Oracle SQL queries mixed into business logic • Tangled code – Deep dependencies, false sharing • Data providers with sideways dependencies – Everything depends on everything else AnA-‐paTern affects producAvity, availability
- 44. Untangled Service Interfaces • New Cloud Code With Strict Layering – Compile against interface jar – Can use spring runJme binding to enforce • Service interface is the service – ImplementaJon is completely hidden – Can be implemented locally or remotely – ImplementaJon can evolve independently
- 45. Untangled Service Interfaces Two layers: • SAL -‐ Service Access Library – Basic serializaJon and error handling – REST or POJO’s defined by data provider • ESL -‐ Extended Service Library – Caching, conveniences – Can combine several SALs – Exposes faceted type system (described later) – Interface defined by data consumer in many cases
- 46. Service InteracJon Pa=ern Sample Swimlane Diagram
- 47. Service Architecture Pa=erns • Internal Interfaces Between Services – Common pa=erns as templates – Highly instrumented, observable, analyJcs – Service Level Agreements – SLAs • Library templates for generic features – Instrumented Ne#lix Base Servlet template – Instrumented generic client interface template – Instrumented S3, SimpleDB, Memcached clients
- 48. CLIENT Request Start Timestamp, Client Inbound Request End outbound deserialize end Timestamp serialize start Jmestamp Jmestamp Inbound Client deserialize outbound start serialize end Jmestamp Jmestamp Client network receive Jmestamp Service Request Client Network send Jmestamp Instruments Every Service network send Jmestamp Step in the call Service Network receive Jmestamp Service Service outbound inbound serialize end serialize start Jmestamp Jmestamp Service Service outbound inbound SERVICE execute serialize start serialize end Jmestamp request start Jmestamp Jmestamp, execute request end Jmestamp
- 49. Boundary Interfaces • Isolate teams from external dependencies – Fake SAL built by cloud team – Real SAL provided by data provider team later – ESL built by cloud team using faceted objects • Fake data sources allow development to start – e.g. Fake IdenJty SAL for a test set of customers – Development solidifies dependencies early – Helps external team provide the right interface
- 50. One Object That Does Everything • Datacenter uses a few big complex objects – Movie and Customer objects are the foundaJon – Good choice for a small team and one instance – ProblemaJc for large teams and many instances • False sharing causes tangled dependencies – UnproducJve re-‐integraJon work AnA-‐paTern impacAng producAvity and availability
- 51. An Interface For Each Component • Cloud uses faceted Video and Visitor – Basic types hold only the idenJfier – Facets scope the interface you actually need – Each component can define its own facets • No false-‐sharing and dependency chains – Type manager converts between facets as needed – video.asA(PresentaJonVideo) for www – video.asA(MerchableVideo) for middle Jer
- 52. So:ware Architecture Pa=erns • Object Models – Basic and derived types, facets, serializable – Pass by reference within a service – Pass by value between services • ComputaJon and I/O Models – Service ExecuJon using Best Effort – Common thread pool management
- 53.
- 54. API AWS EC2 Front End ELB Discovery Service API Proxy API etc. API ELB Component API SQS Services Oracl e Oracle Oracle memcached memcached ReplicaJon EBS Ne@lix S3 Data Center AWS Storage SimpleDB
- 55. Ne#lix UndifferenJated Li:ing • Middle Tier Load Balancing • Discovery (local DNS) • EncrypJon Services • Caching • Distributed App Management We want cloud vendors to do all this for us as well!
- 56. Load Balancing in AWS • Middle Jer currently not supported in AWS – ELB are public-‐facing only – Cannot apply security group sezngs • ELB verJcal scalability for concentrated clients – Too few proxy IP addresses leads to hot spots • ELB needs support for balancing heurisJcs – ProporJonal balance across Availability Zones – Weighted Least connecJons, Weighted Round Robin • Zone aware rouJng – Default to instances in the same Availability Zone – Falls back to cross-‐zone on failure
- 57. Discovery • Discovery Service (Redundant instances per zone) – Simple REST interface – Cloud apps register with Discovery • Apps send heartbeats every 30 sec to renew lease – App evicted a:er 3 missed heartbeats – Can re-‐register if the problem was transient • Apps can store custom metadata – Version number, AMI id, Availability Zone, etc. • So:ware Round-‐robin Load Balancer – Query Discovery for instances of specific applicaJon – Baked into Ne#lix REST client (JSR311/Jersey based) AWS Middle-‐)er ELB would eliminate most use cases
- 58. Database MigraJon • Why SimpleDB? – No DBA’s in the cloud, Amazon hosted service – Work started two years ago, fewer viable opJons – Worked with Amazon to speed up and scale SimpleDB • AlternaJves? – InvesJgaJng adding Cassandra and Membase to the mix – Need several opJons to match use cases well • Detailed SimpleDB Advice – Sid Anand -‐ QConSF Nov 5th – Ne#lix’ TransiJon to High Availability Storage Systems – Blog -‐ h=p://pracJcalcloudcompuJng.com/ – Download Paper PDF -‐ h=p://bit.ly/bhOTLu
- 59. Oracle to SimpleDB (See Sid’s paper for details) • SimpleDB Domains – De-‐normalize mulJple tables into a single domain – Work around size limits (10GB per domain, 1KB per key) – Shard data across domains to scale – Key – Use distributed sequence generator, GUID or natural unique key such as customer-‐id – Implement a schema validator to catch bad a=ributes • ApplicaJon layer support – Do GROUP BY and JOIN operaJons in the applicaJon – Compose relaJons in the applicaJon layer – Check constraints on read, and repair data as a side effect • Do without triggers, PL/SQL, clock operaJons
- 60. Tools and AutomaJon • Developer and Build Tools – Jira, Eclipse, Hudson, Ivy, ArJfactory – Builds, creates .war file, .rpm, bakes AMI and launches • Custom Ne#lix ApplicaJon Console – AWS Features at Enterprise Scale (hide the keys!) – Auto Scaler Group is unit of deployment to producJon • Open Source + Support – Apache, Tomcat, OpenJDK, CentOS • Monitoring Tools – Keynote – service monitoring and alerJng – AppDynamics – Developer focus for cloud – EpicNMS – flexible data collecJon and plots h=p://epicnms.com – Nimso: NMS – ITOps focus for Datacenter + Cloud alerJng
- 61.
- 62.
- 63.
- 64.
- 65. Monitoring Vision • Problem – Too many tools, each with a good reason to exist – Hard to get an integrated view of a problem – Too much manual work building dashboards – Tools are not discoverable, views are not filtered • SoluJon – Get vendors to add deep linking and embedding – IntegraJon “portal” Jes everything together – Dynamic portal generaJon, relevant data, all tools
- 66. Cloud Monitoring Mechanisms • Keynote – External URL monitoring • Amazon CloudWatch – Metrics for ELB and Instances • AppDynamics – End to end transacJon view showing resources used – Powerful real Jme debug tools for latency, CPU and Memory • Nimso: NMS – Scalable and reliable monitoring and alerJng, integraJon portal • Epic – Flexible and easy to use to extend and embed plots • Logs – High capacity logging and analysis framework – Hadoop (log4j -‐> chukwa -‐> EMR)
- 68. Snapshots for a Business TransacJon Sort Call Graphs to Top, pick a slow one
- 69. Drill in to Slow Call Slow Asynchronous S3 Write – no big deal…
- 70. Cloud Bring-‐Up Strategy Simplest and Soonest
- 71. Shadow Traffic RedirecJon • Early a=empt to send traffic to cloud – Real traffic stream to validate cloud back end – Uncovered lots of process and tools issues – Uncovered Service latency issues • TV Device calls Datacenter API – Returns Genre/movie list for a customer – Asynchronously duplicates request to cloud – Start with send-‐and-‐forget mode, ignore response
- 72. Shadow Redirect Instances Modified Datacenter Datacenter Service Instances Modified Cloud Cloud Service One request per Instances visit Data Sources queueservice videometadata
- 73. First Web Pages in the Cloud
- 74.
- 75. First Page • First full page – Starz Channel Genre – Simplest page, no sub-‐genres, minimal personalizaJon – Lots of investment in new Struts based page design – Uses idenJty cookie to lookup in member info svc • New “merchweb” front end instance – movies.ne#lix.com points to merchweb instance • Uncovered lots of latency issues – Used memcached to hide S3 and SimpleDB latency – Improved from slower to faster than Datacenter
- 76. Starz Page Cloud Instances Front End merchweb mulJple requests Middle Tier starz memcached per visit Data Sources queueservice rentalhistory videometadata
- 77. Controlled Cloud TransiJon • WWW calling code chooses who goes to cloud – Filter out corner cases, send percentage of users – The URL that customers see is h=p://movies.ne#lix.com/WiContentPage?csid=1 – If problem, redirect to old Datacenter page h=p://www.ne#lix.com/WiContentPage?csid=1 • Play Bu=on and Star RaJng AcJon redirect – Point URLs for acJons that create/modify data back to datacenter to start with
- 78.
- 79. Cloud Developer Setup • Cloud Boot Camp – Room full of engineers sharing the pain for 1-‐2 days – Built a very rough prototype working web site – Get everyone hands-‐on in the cloud with a new code base – Debug lots of tooling and conceptual issues very fast – Member info in SimpleDB with developer’s accounts only • Cloud Specific Key Setup – It’s a pain, need to configure your IDE’s JVM – Needed to integrate with AWS security model • Startup Guide Wiki Pages – What object facets already exist, how to make your own – What components already exist or are work in progress
- 80. Developer Instances Collision Sam and Rex both want to deploy web front end for development Sam Rex web in test account
- 81. Per-‐Service Namespace RouJng Developers choose what to share Sam Rex Mike web-‐sam web-‐rex web-‐dev backend-‐dev backend-‐dev backend-‐mike
- 82. Developer Service Namespaces • Developer specific service instances – Configured via Java properJes at runJme – RouJng implemented by REST client library • Server ConfiguraJon – Configure discovery service version string – Registers as <appname>-‐<namespace> • Client ConfiguraJon – Route traffic on per-‐service basis including namespace
- 83.
- 84. WWW Page by Page during Q2/Q3/Q4 • Simplest possible page first – Minimal dependencies • Add pages as dependent services come online • Home page – most complex and highest traffic • Leave low traffic pages for later cleanup gradual migraAon from Datacenter pages
- 85. Big-‐Bang TransiJon • iPhone Launch (August/Sept) – No capacity in the datacenter, cloud only – App Store gates release, not gradual, can’t back out – Market is huge (exisJng and new customers) – Has to work at large scale on day one • Datacenter Shadow Redirect Technique – Used to stress back-‐end and data sources • SOASTA Cloud Based Load GeneraJon – Used to stress test API and end-‐to-‐end funcJonality
- 86. Current Work for Cloud Pla#orm • Drive latency and availability goals – More Aggressive caching – Improving Fault and latency robustness • Logging and monitoring portal/dashboards – Working to integrate tools and data sources – Need be=er observability and automaJon • EvaluaJng a range of NoSQL choices – Broad set of use cases, no single winner – Good topic for another talk…
- 87.
- 88. Next Few Years… • “System of Record” moves to Cloud – Master copies of data live only in the cloud, with backups etc. – Cut the datacenter to cloud replicaJon link • InternaJonal Expansion – Global Clouds – Rapid deployments to new markets • GPU Clouds opJmized for video encoding • Cloud StandardizaJon – Cloud features and APIs should be a commodity not a differenJator – DifferenJate on scale and quality of service – CompeJJon also drives cost down – Higher resilience – Higher scalability We would prefer to be an insignificant customer in a giant cloud
- 89. Remember the Goals Faster Scalable Available ProducJve Track progress against these goals
- 90. Takeaway Ne9lix is path-‐finding the use of public AWS cloud to replace in-‐house IT for non-‐trivial applicaAons with hundreds of developers and thousands of systems. h=p://www.linkedin.com/in/adriancockcro: @adrianco #ne#lixcloud