Netflix in the Cloud at SV Forum

1. Cloud Architecture at Ne0lix How Ne0lix Built a Scalable Java oriented PaaS running on AWS SVForum March 27th, 2012 Adrian Cockcro9 @adrianco #ne=lixcloud h@p://www.linkedin.com/in/adriancockcro9

2. Adrian Cockcro9 •  Director, Architecture for Cloud Systems, Ne=lix Inc. –  Previously Director for PersonalizaOon Pla=orm •  DisOnguished Availability Engineer, eBay Inc. 2004-‐7 –  Founding member of eBay Research Labs •  DisOnguished Engineer, Sun Microsystems Inc. 1988-‐2004 –  2003-‐4 Chief Architect High Performance Technical CompuOng –  2001 Author: Capacity Planning for Web Services –  1999 Author: Resource Management –  1995 & 1998 Author: Sun Performance and Tuning –  1996 Japanese EdiOon of Sun Performance and Tuning •  SPARC & Solarisパフォーマンスチューニング (サンソフトプレスシリーズ) •  More –  Twi@er @adrianco – Blog h@p://perfcap.blogspot.com –  PresentaOons at h@p://www.slideshare.net/adrianco

3. Why Ne=lix, Why Cloud, Why AWS Part 1 of 3

4. What kind of Cloud? •  So9ware as a Service – SaaS –  Replaces in house applicaOons –  Targets end users •  Pla=orm as a Service – PaaS –  Replaces in house operaOons funcOons –  Targets developers •  Infrastructure as a Service – IaaS –  Replaces in house datacenter capacity –  Targets developers and ITops

5. What Ne=lix Did •  Moved to SaaS –  Corporate IT – OneLogin, Workday, Box, Evernote… –  Tools – Pagerduty, AppDynamics, ElasOc MapReduce •  Built our own PaaS <-‐ today’s focus –  Customized to make our developers producOve –  When we started, we had li@le choice •  Moved incremental capacity to IaaS –  No new datacenter space since 2008 as we grew –  Moved our streaming apps to the cloud

6. Why Use Public Cloud?

7. Things We Don’t Do

8. Be@er Business Agility

9. Data Center Ne=lix could not build new datacenters fast enough Capacity growth is acceleraOng, unpredictable Product launch spikes -‐ iPhone, Wii, PS3, Xbox InternaOonal – Canada, LaOn America, UK/Ireland

10. Ne=lix.com is now ~100% Cloud A few small back end data sources sOll in progress All internaOonal product is cloud based USA speciﬁc logisOcs remains in the Datacenter Working on SOX, PCI as scope starts to include AWS

11. Ne=lix Choice was AWS with our own pla=orm and tools Unique pla=orm requirements and extreme scale, agility and ﬂexibility

12. Leverage AWS Scale “the biggest public cloud” AWS investment in features and automaOon Use AWS zones and regions for high availability, scalability and global deployment

13. But isn’t Amazon a compeOtor? Many products that compete with Amazon run on AWS We are a “poster child” for the AWS Architecture Ne=lix is one of the biggest AWS customers Co-‐opeOOon -‐ compeOtors are also partners

14. Could Ne=lix use another cloud? Would be nice, we use three interchangeable CDN Vendors But no-‐one else has the scale and features of AWS You have to be this tall to ride this ride… Maybe in 2-‐3 years?

15. We want to use clouds, we don’t have Ome to build them Public cloud for agility and scale We use electricity too, but don’t want to build our own power staOon… AWS because they are big enough to allocate thousands of instances per hour when we need to

16. What about other PaaS? •  CloudFoundry – Open Source by VMWare –  Developer-‐friendly, easy to get started –  Missing scale and some enterprise features •  Rightscale –  Widely used to abstract away from AWS –  Creates it’s own lock-‐in problem… •  AWS is growing into this space –  We didn’t want a vendor between us and AWS –  We wanted to build a thin PaaS, that gets thinner

17. Ne=lix Deployed on AWS 2009 2009 2010 2010 2010 2011 Content Logs Play WWW API CS Video InternaOonal Masters S3 DRM Sign-‐Up Metadata CS lookup Device DiagnosOcs EC2 EMR Hadoop CDN rouOng Search Conﬁg & AcOons Movie TV Movie Customer S3 Hive Bookmarks Choosing Choosing Call Log Business Social CDNs Logging RaOngs Facebook CS AnalyOcs Intelligence

18. Cloud Architecture Pa@erns Where do we start?

19. Goals •  Faster –  Lower latency than the equivalent datacenter web pages and API calls –  Measured as mean and 99th percenOle –  For both ﬁrst hit (e.g. home page) and in-‐session hits for the same user •  Scalable –  Avoid needing any more datacenter capacity as subscriber count increases –  No central verOcally scaled databases –  Leverage AWS elasOc capacity eﬀecOvely •  Available –  SubstanOally higher robustness and availability than datacenter services –  Leverage mulOple AWS availability zones –  No scheduled down Ome, no central database schema to change •  ProducOve –  OpOmize agility of a large development team with automaOon and tools –  Leave behind complex tangled datacenter code base (~8 year old architecture) –  Enforce clean layered interfaces and re-‐usable components

20. Datacenter AnO-‐Pa@erns What do we currently do in the datacenter that prevents us from meeOng our goals?

21. Rewrite from Scratch Not everything is cloud speciﬁc Pay down technical debt Robust pa@erns

22. Ne=lix Datacenter vs. Cloud Arch Central SQL Database Distributed Key/Value NoSQL SOcky In-‐Memory Session Shared Memcached Session Cha@y Protocols Latency Tolerant Protocols Tangled Service Interfaces Layered Service Interfaces Instrumented Code Instrumented Service Pa@erns Fat Complex Objects Lightweight Serializable Objects Components as Jar Files Components as Services

23. So9ware Architecture Pa@erns •  Object Models –  Basic and derived types, facets, serializable –  Pass by reference within a service –  Pass by value between services •  ComputaOon and I/O Models –  Service ExecuOon using Best Eﬀort / Futures –  Common thread pool management –  Circuit breakers to manage and contain failures

24. Model Driven Architecture •  TradiOonal Datacenter PracOces –  Lots of unique hand-‐tweaked systems –  Hard to enforce pa@erns –  Some use of Puppet to automate changes •  Model Driven Cloud Architecture –  Perforce/Ivy/Jenkins based builds for everything –  Every producOon instance is a pre-‐baked AMI –  Every applicaOon is managed by an Autoscaler Every change is a new AMI

25. Ne=lix PaaS Principles •  Maximum FuncOonality –  Developer producOvity and agility •  Leverage as much of AWS as possible –  AWS is making huge investments in features/scale •  Interfaces that isolate Apps from AWS –  Avoid lock-‐in to speciﬁc AWS API details •  Portability is a long term goal –  Gets easier as other vendors catch up with AWS

26. Ne=lix Global PaaS •  Architecture Features and Overview •  Portals and Explorers •  Pla=orm Services •  Pla=orm APIs •  Pla=orm Frameworks •  Persistence •  Scalability Benchmark

27. Global PaaS? Toys are nice, but this is the real thing… •  Supports all AWS Availability Zones and Regions •  Supports mulOple AWS accounts {test, prod, etc.} •  Cross Region/Acct Data ReplicaOon and Archiving •  InternaOonalized, Localized and GeoIP rouOng •  Security is ﬁne grain, dynamic AWS keys •  Autoscaling to thousands of instances •  Monitoring for millions of metrics •  ProducOve for 100s of developers on one product •  23M+ users USA, Canada, LaOn America, UK, Eire

28. Basic PaaS EnOOes •  AWS Based EnOOes –  Instances and Machine Images, ElasOc IP Addresses –  Security Groups, Load Balancers, Autoscale Groups –  Availability Zones and Geographic Regions •  Ne=lix PaaS EnOOes –  ApplicaOons (registered services) –  Clusters (versioned Autoscale Groups for an App) –  ProperOes (dynamic hierarchical conﬁguraOon)

29. Core PaaS Services •  AWS Based Services –  S3 storage, to 5TB ﬁles, parallel mulOpart writes –  SQS – Simple Queue Service. Messaging layer. •  Ne=lix Based Services –  EVCache – memcached based ephemeral cache –  Cassandra – distributed data store •  External Services –  GeoIP Lookup interfaced to a vendor –  Keystore HSM in Ne=lix Datacenter

30. Instance Architecture Linux Base AMI (CentOS or Ubuntu) OpOonal Apache frontend, Java (JDK 6 or 7) memcached, non-‐java apps Tomcat AppDynamics appagent Monitoring Log rotaOon ApplicaOon servlet, base Healthcheck, status to S3 GC and thread server, pla=orm, interface servlets, JMX interface, AppDynamics dump logging jars for dependent services Servo autoscale machineagent Epic

31. Security Architecture •  Instance Level Security baked into base AMI –  Login: ssh only allowed via portal (not between instances) –  Each app type runs as its own userid app{test|prod} •  AWS Security, IdenOty and Access Management –  Each app has its own security group (ﬁrewall ports) –  Fine grain user roles and resource ACLs •  Key Management –  AWS Keys dynamically provisioned, easy updates –  High grade app speciﬁc key management support

32. Portals and Explorers •  Ne=lix ApplicaOon Console (NAC) –  Primary AWS provisioning/conﬁg interface •  AWS Usage Analyzer –  Breaks down costs by applicaOon and resource •  Cassandra Explorer –  Browse clusters, keyspaces, column families •  Base Server Explorer –  Browse service endpoints conﬁguraOon, perf

35. Pla=orm Services •  Discovery – service registry for “ApplicaOons” •  IntrospecOon – Entrypoints •  Cryptex – Dynamic security key management •  Geo – Geographic IP lookup •  Pla=ormservice – Dynamic property conﬁguraOon •  LocalizaOon – manage and lookup local translaOons •  Evcache – ephemeral volaOle cache •  Cassandra – Cross zone/region distributed data store •  Zookeeper – Distributed CoordinaOon (Curator) •  Various proxies – access to old datacenter stuﬀ

36. Metrics Framework •  System and ApplicaOon –  CollecOon, AggregaOon, Querying and ReporOng –  Non-‐blocking logging, avoids log4j lock contenOon –  Honu-‐Streaming -‐> S3 -‐> EMR -‐> Hive •  Performance, Robustness, Monitoring, Analysis –  Tracers, Counters – explicit code instrumentaOon log –  Real Time Tracers/Counters –  SLA – service level response Ome percenOles –  Servo annotated JMX extract to Cloudwatch •  Latency Monkey Infrastructure –  Inject random delays into service responses

37. Ne0lix Pla0orm Persistence •  Ephemeral VolaOle Cache – evcache –  Discovery-‐aware memcached based backend –  Client abstracOons for zone aware replicaOon –  OpOon to write to all zones, fast read from local •  Cassandra –  Highly available and scalable (more later…) •  MongoDB –  Complex object/query model for small scale use •  MySQL –  Hard to scale, legacy and small relaOonal models

38. Priam – Cassandra AutomaOon Available at h@p://github.com/ne=lix •  Ne=lix Pla=orm Tomcat Code •  Zero touch auto-‐conﬁguraOon •  State management for Cassandra JVM •  Token allocaOon and assignment •  Broken node auto-‐replacement •  Full and incremental backup to S3 •  Restore sequencing from S3 •  Grow/Shrink Cassandra “ring”

39. Astyanax Available at h@p://github.com/ne=lix •  Cassandra java client •  API abstracOon on top of Thri9 protocol •  “Fixed” ConnecOon Pool abstracOon (vs. Hector) –  Round robin with Failover –  Retry-‐able operaOons not Oed to a connecOon –  Ne=lix PaaS Discovery service integraOon –  Host reconnect (fixed interval or exponenOal backoff) –  Token aware to save a network hop – lower latency –  Latency aware to avoid compacOng/repairing nodes – lower variance •  Batch mutaOon: set, put, delete, increment •  Simplified use of serializers via method overloading (vs. Hector) •  ConnecOonPoolMonitor interface for counters and tracers •  Composite Column Names replacing deprecated SuperColumns

40. Astyanax Query Example Paginate through all columns in a row ColumnList<String> columns; int pageize = 10; try { RowQuery<String, String> query = keyspace .prepareQuery(CF_STANDARD1) .getKey("A") .setIsPaginaOng() .withColumnRange(new RangeBuilder().setMaxSize(pageize).build()); while (!(columns = query.execute().getResult()).isEmpty()) { for (Column<String> c : columns) { } } } catch (ConnecOonExcepOon e) { }

41. High Availability •  Cassandra stores 3 local copies, 1 per zone –  Synchronous access, durable, highly available –  Read/Write One fastest, least consistent -‐ ~1ms –  Read/Write Quorum 2 of 3, consistent -‐ ~3ms •  AWS Availability Zones –  Separate buildings –  Separate power etc. –  Fairly close together

42. “TradiOonal” Cassandra Write Data Flows Single Region, MulOple Availability Zone, Not Token Aware Cassandra • Disks • Zone A 2 2 4 2 1.  Client Writes to any Cassandra 3 3 Cassandra If a node goes oﬄine, Cassandra Node • Disks 5 • Disks 5 hinted handoﬀ 2.  Coordinator Node • Zone C 1 • Zone A completes the write replicates to nodes when the node comes and Zones Non Token back up. 3.  Nodes return ack to Aware coordinator Clients Requests can choose to 4.  Coordinator returns 3 wait for one node, a Cassandra Cassandra ack to client • Disks • Disks 5 quorum, or all nodes to 5.  Data wri@en to • Zone C • Zone B ack the write internal commit log disk (no more than Cassandra SSTable disk writes and • Disks 10 seconds later) • Zone B compacOons occur asynchronously

43. Astyanax -‐ Cassandra Write Data Flows Single Region, MulOple Availability Zone, Token Aware Cassandra • Disks • Zone A 1.  Client Writes to Cassandra 2 2 Cassandra If a node goes oﬄine, nodes and Zones • Disks 3 • Disks 3 hinted handoﬀ 2.  Nodes return ack to • Zone C 1 • Zone A completes the write client 3.  Data wri@en to Token when the node comes back up. internal commit log Aware disks (no more than Clients 2 Requests can choose to 10 seconds later) Cassandra Cassandra wait for one node, a • Disks • Disks 3 quorum, or all nodes to • Zone C • Zone B ack the write Cassandra SSTable disk writes and • Disks • Zone B compacOons occur asynchronously

44. Data Flows for MulO-‐Region Writes Token Aware, Consistency Level = Local Quorum 1.  Client writes to local replicas If a node or region goes offline, hinted handoff 2.  Local write acks returned to completes the write when the node comes back up. Client which conOnues when Nightly global compare and repair jobs ensure 2 of 3 local nodes are everything stays consistent. commi@ed 3.  Local coordinator writes to remote coordinator. Cassandra 100+ms latency 4.  When data arrives, remote Cassandra •  Disks •  Disks •  Zone A •  Zone A coordinator node acks and Cassandra 2 2 Cassandra Cassandra 4 Cassandra 6 6 3 5 Disks 6 copies to other remote zones 6 •  Disks •  Disks •  Zone C •  Zone A •  •  Zone C 4 Disks A •  •  Zone 1 4 5.  Remote nodes ack to local US EU coordinator Clients Clients Cassandra 2 Cassandra Cassandra 5 Cassandra 6.  Data flushed to internal •  Disks •  Zone C •  Disks 6 •  Zone B •  Disks •  Zone C •  Disks 6 •  Zone B commit log disks (no more Cassandra Cassandra than 10 seconds later) •  Disks •  Disks •  Zone B •  Zone B

45. Cassandra Backup •  Full Backup Cassandra Cassandra Cassandra –  Time based snapshot –  SSTable compress -‐> S3 Cassandra Cassandra •  Incremental S3 Backup Cassandra Cassandra –  SSTable write triggers compressed copy to S3 Cassandra Cassandra •  Archive Cassandra Cassandra –  Copy cross region A

46. ETL for Cassandra •  Data is de-‐normalized over many clusters! •  Too many to restore from backups for ETL •  SoluOon – read backup ﬁles using Hadoop •  Aegisthus –  h@p://techblog.ne=lix.com/2012/02/aegisthus-‐bulk-‐data-‐pipeline-‐out-‐of.html –  High throughput raw SSTable processing –  Re-‐normalizes many clusters to a consistent view –  Extract, Transform, then Load into Teradata

47. Cassandra Archive A Appropriate level of paranoia needed… •  Archive could be un-‐readable –  Restore S3 backups weekly from prod to test, and daily ETL •  Archive could be stolen –  PGP Encrypt archive •  AWS East Region could have a problem –  Copy data to AWS West •  ProducOon AWS Account could have an issue –  Separate Archive account with no-‐delete S3 ACL •  AWS S3 could have a global problem –  Create an extra copy on a diﬀerent cloud vendor….

48. Tools and AutomaOon •  Developer and Build Tools –  Jira, Perforce, Eclipse, Jenkins, Ivy, ArOfactory –  Builds, creates .war ﬁle, .rpm, bakes AMI and launches •  Custom Ne=lix ApplicaOon Console –  AWS Features at Enterprise Scale (hide the AWS security keys!) –  Auto Scaler Group is unit of deployment to producOon •  Open Source + Support –  Apache, Tomcat, Cassandra, Hadoop –  Datastax support for Cassandra, AWS support for Hadoop via EMR •  Monitoring Tools –  Alert processing gateway into Pagerduty –  AppDynamics – Developer focus for cloud h@p://appdynamics.com

49. Open Source Strategy •  Release PaaS Components git-‐by-‐git –  Source at github.com/ne=lix –  Intros and techniques at techblog.ne=lix.com –  Blog post or new code every week or so •  MoOvaOons –  Give back to Apache licensed OSS community –  MoOvate, retain, hire top engineers –  Create a community that adds features and ﬁxes

50. Current OSS Projects and Posts Github / Techblog Priam Exhibitor Servo Apache Project Techblog Post Astyanax Curator Autoscaling scripts CassJMeter Zookeeper Honu Cassandra EVCache Circuit Breaker Aegisthus

51. Scalability TesOng •  Cloud Based TesOng – fricOonless, elasOc –  Create/destroy any sized cluster in minutes –  Many test scenarios run in parallel •  Test Scenarios –  Internal app speciﬁc tests –  Simple “stress” tool provided with Cassandra •  Scale test, keep making the cluster bigger –  Check that tooling and automaOon works… –  How many ten column row writes/sec can we do?

52. <DrEvil>ONE MILLION</DrEvil>

53. Scale-‐Up Linearity h@p://techblog.ne=lix.com/2011/11/benchmarking-‐cassandra-‐scalability-‐on.html Client Writes/s by node count – ReplicaJon Factor = 3 1200000 1099837 1000000 800000 600000 537172 400000 366828 200000 174373 0 0 50 100 150 200 250 300 350

54. Availability and Resilience

55. Chaos Monkey •  Computers (Datacenter or AWS) randomly die –  Fact of life, but too infrequent to test resiliency •  Test to make sure systems are resilient –  Allow any instance to fail without customer impact •  Chaos Monkey hours –  Monday-‐Thursday 9am-‐3pm random instance kill •  ApplicaOon conﬁguraOon opOon –  Apps now have to opt-‐out from Chaos Monkey

56. Responsibility and Experience •  Make developers responsible for failures –  Then they learn and write code that doesn’t fail •  Use Incident Reviews to find gaps to fix –  Make sure its not about finding “who to blame” •  Keep Omeouts short, fail fast –  Don’t let cascading Omeouts stack up •  Make configuraOon opOons dynamic –  You don’t want to push code to tweak an opOon

57. Resilient Design – Circuit Breakers h@p://techblog.ne=lix.com/2012/02/fault-‐tolerance-‐in-‐high-‐volume.html

58. PaaS OperaOonal Model •  Developers –  Provision and run their own code in producOon –  Take turns to be on call if it breaks (pagerduty) –  Conﬁgure autoscalers to handle capacity needs •  DevOps and PaaS (aka NoOps) –  DevOps is used to build and run the PaaS –  PaaS constrains Dev to use automaOon instead –  PaaS puts more responsibility on Dev, with tools

59. What’s Le9 for Corp IT? •  Corporate Security and Network Management –  Billing and remnants of streaming service back-‐ends in DC •  Running Ne=lix’ DVD Business –  Tens of Oracle instances Corp WiFi Performance –  Hundreds of MySQL instances –  Thousands of VMWare VMs –  Zabbix, CacO, Splunk, Puppet •  Employee ProducOvity –  Building networks and WiFi –  SaaS OneLogin SSO Portal –  Evernote Premium, Safari Online Bookshelf, Dropbox for Teams –  Google Enterprise Apps, Workday HCM/Expense, Box.com –  Many more SaaS migraOons coming…

60. ImplicaOons for IT OperaOons •  Cloud is run by developer organizaOon –  Product group’s “IT department” is the AWS API and PaaS –  CorpIT handles billing and some security funcOons Cloud capacity is 10x bigger than Datacenter –  Datacenter oriented IT didn’t scale up as we grew –  We moved a few people out of IT to do DevOps for our PaaS •  TradiOonal IT Roles and Silos are going away –  We don’t have SA, DBA, Storage, Network admins for cloud –  Developers deploy and “run what they wrote” in producOon

61. Ne=lix PaaS OrganizaOon Developer Org ReporOng into Product Development, not ITops Ne=lix Cloud Pla=orm Team Cloud Ops Build Tools Pla=orm and Cloud Cloud Reliability Architecture and Database Performance SoluOons Engineering AutomaOon Engineering Perforce Jenkins Pla=orm jars Cassandra Future planning ArOfactory JIRA Benchmarking Monitoring Alert RouOng Key store Security Arch Monkeys Incident Lifecycle Base AMI, Bakery Zookeeper JVM GC Tuning Eﬃciency Ne=lix App Console Wiresharking Entrypoints Cassandra AWS VPC PagerDuty Hyperguard AWS API AWS Instances AWS Instances AWS Instances Powerpoint J

62. Roadmap for 2012 •  Readiness for global Ne=lix launches •  More resiliency and improved availability •  More automaOon, orchestraOon •  “Hardening” the pla=orm •  Lower latency for web services and devices •  Working towards IPv6 support •  More open sourced components

63. Wrap Up Answer your remaining quesOons… What was missing that you wanted to cover? Next up – Jason Chan on Security Architecture

64. Takeaway Ne>lix has built and deployed a scalable global Pla>orm as a Service. Key components of the Ne>lix PaaS are being released as Open Source projects so you can build your own custom PaaS. h@p://github.com/Ne=lix h@p://techblog.ne=lix.com h@p://slideshare.net/Ne=lix h@p://www.linkedin.com/in/adriancockcro9 @adrianco #ne=lixcloud

Netflix in the Cloud at SV Forum

Netflix in the Cloud at SV Forum

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(20)

Similar to Netflix in the Cloud at SV Forum

Similar to Netflix in the Cloud at SV Forum(20)

More from Adrian Cockcroft

More from Adrian Cockcroft(9)

Recently uploaded

Recently uploaded(20)

Netflix in the Cloud at SV Forum