Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Ne#lix	  Cloud	  Architecture	   Velocity	  Conference	  June	  14,	  2011	               Adrian	  Cockcro=	    @adrianco	...
Who,	  Why,	  What	             Ne#lix	  in	  the	  Cloud	     Cloud	  Challenges	  and	  Learnings	                     (...
Ne#lix	  Inc.	       With	  more	  than	  23	  million	  subscribers	  in	  the	  United	       States	  and	  Canada,	  N...
Unlimited	  streaming	  for	  $7.99/month,	  large	  and	  growing	  catalog	  of	  movies	  and	  TV	  
Adrian	  Cockcro=	  •  Director,	  Architecture	  for	  Cloud	  Systems,	  Ne#lix	  Inc.	        –  Previously	  Director	...
Why	  is	  Ne#lix	  Talking	  about	                 Cloud?	  
Ne#lix	  is	  Path-­‐finding	     The	  Cloud	  ecosystem	  is	  evolving	  very	  fast	  Share	  with	  and	  learn	  from...
We	  want	  to	  use	  clouds,	               not	  build	  them	     Cloud	  technology	  should	  be	  a	  commodity	  P...
Why	  Use	  Cloud?	                           	          For	  Be@er	  Business	  Agility	  For	  Unpredictable	  Business...
Data	  Center	                  Ne#lix	  could	  not	                                       build	  new	                  ...
23	  Million	  Customers	                          2011-­‐Q1	  year/year	  customers	  +69%	  	            25	            ...
Out-­‐Growing	  Data	  Center	               h@p://­‐ne#lix-­‐api.html   	        ...	  is	  now	  ~100%	  Cloud	     Account	  sign-­‐up	  is	  currently	  being	  moved	  to	  cloud	       All	  ...
Leverage	  AWS	  Scale	     “the	  biggest	  public	  cloud”	         AWS	  investment	  in	  tooling	  and	  automaOon	  ...
Leverage	  AWS	  Feature	  Set	        “the	  market	  leader”	  EC2,	  S3,	  SDB,	  SQS,	  EBS,	  EMR,	  ELB,	  ASG,	  IA...
“The	  cloud	  lets	  its	  users	  focus	           on	  delivering	  differenAaAng	           business	  value	  instead	...
We	  want	  to	  use	  clouds,	  we	  don’t	  have	  Ome	  to	  build	  them	                    Public	  cloud	  for	  ag...
Ne#lix	  EC2	  Instances	  per	  Account	            (summer	  2010,	  producOon	  is	  much	  higher	  now…)	  “Many	  Th...
Ne#lix	  Deployed	  on	  AWS	  Content	            Logs	             Play	          WWW	             API	      Video	     ...
Cloud	  Encoding	  Pipeline	                                                                     Encode	       S3	      En...
Cloud	  Architecture	           Ignite!	  
Product	  Trade-­‐off	  User	  Experience	     ImplementaOon	    Consistent	           Development	    Experience	         ...
Ne#lix	  Cloud	  Goals	  •  Faster	       –  Lower	  latency	  than	  the	  equivalent	  datacenter	  web	  pages	  and	  ...
Old	  Datacenter	  vs.	  New	  Cloud	  Arch	      Central	  SQL	  Database	          Distributed	  Key/Value	  NoSQL	   SO...
The	  Central	  SQL	  Database	  •  Datacenter	  has	  a	  central	  database	     –  Everything	  in	  one	  place	  is	 ...
The	  Distributed	  Key-­‐Value	  Store	  •  Cloud	  has	  many	  key-­‐value	  data	  stores	      –  More	  complex	  to...
The	  SOcky	  Session	  •  Datacenter	  SOcky	  Load	  Balancing	     –  Efficient	  caching	  for	  low	  latency	     –  T...
The	  Shared	  Session	  •  Cloud	  Uses	  Round-­‐Robin	  Load	  Balancing	      –  Simple	  request-­‐based	  code	     ...
Cha@y	  Opaque	  and	  Bri@le	  Protocols	  •  Datacenter	  service	  protocols	      –  Assumed	  low	  latency	  for	  m...
Robust	  and	  Flexible	  Protocols	  •  Cloud	  service	  protocols	     –  JSR311/Jersey	  is	  used	  for	  REST/HTTP	 ...
Persisted	  Protocols	  •  Persist	  Avro	  in	  Memcached	     –  Save	  space/latency	  (zigzag	  encoding,	  half	  the...
Tangled	  Service	  Interfaces	  •  Datacenter	  implementaOon	  is	  exposed	     –  Oracle	  SQL	  queries	  mixed	  int...
Untangled	  Service	  Interfaces	  •  New	  Cloud	  Code	  With	  Strict	  Layering	      –  Compile	  against	  interface...
Untangled	  Service	  Interfaces	  Two	  layers:	  •  SAL	  -­‐	  Service	  Access	  Library	      –  Basic	  serializaOon...
Service	  InteracOon	  Pa@ern	      Sample	  Swimlane	  Diagram	  
Service	  Architecture	  Pa@erns	  •  Internal	  Interfaces	  Between	  Services	     –  Common	  pa@erns	  as	  templates...
CLIENT	                                                                    Request	  Start	                               ...
Boundary	  Interfaces	  •  Isolate	  teams	  from	  external	  dependencies	     –  Fake	  SAL	  built	  by	  cloud	  team...
One	  Object	  That	  Does	  Everything	  •  Datacenter	  uses	  a	  few	  big	  complex	  objects	      –  Movie	  and	  ...
An	  Interface	  For	  Each	  Component	  •  Cloud	  uses	  faceted	  Video	  and	  Visitor	      –  Basic	  types	  hold	...
So=ware	  Architecture	  Pa@erns	  •  Object	  Models	     –  Basic	  and	  derived	  types,	  facets,	  serializable	    ...
Ne#lix	  Systems	  Architecture	  
API	   AWS	  EC2	                                           Front	  End	  Load	  Balancer	               Discovery	       ...
Database	  MigraOon	  •  Why	  SimpleDB?	      –  No	  DBA’s	  in	  the	  cloud,	  Amazon	  hosted	  service	      –  Work...
Cloud	  OperaOons	    Model	  Driven	  Architecture	  Capacity	  Planning	  &	  Monitoring	  
Tools	  and	  AutomaOon	  •  Developer	  and	  Build	  Tools	       –  Jira,	  Perforce,	  Eclipse,	  Jenkins,	  Ivy,	  Ar...
Model	  Driven	  Architecture	  •  Datacenter	  PracOces	     –  Lots	  of	  unique	  hand-­‐tweaked	  systems	     –  Har...
High	  Availability	  Zones	  •  Each	  zone	  is	  a	  separate	  datacenter	      –  Private	  power,	  cooling,	  netwo...
Cross	  Region	  Backups	  •  Data	  is	  backed	  up	  into	  a	  different	  cloud	  region	      –  Different	  AWS	  S3	...
Model	  Driven	  ImplicaOons	  •  Automated	  “Least	  Privilege”	  Security	     –  Tightly	  specified	  security	  group...
Ne#lix	  App	  Console	  
Auto	  Scale	  Group	  ConfiguraOon	  
Learnings	  •  Datacenter	  oriented	  tools	  don’t	  work	       –  Ephemeral	  instances	       –  High	  rate	  of	  c...
Capacity	  Planning	  &	  Monitoring	  
Capacity	  Planning	  in	  Clouds	                       (a	  few	  things	  have	  changed…)	  •    Capacity	  is	  expen...
Monitoring	  Issues	  •  Problem	     –  Too	  many	  tools,	  each	  with	  a	  good	  reason	  to	  exist	     –  Hard	 ...
Data	  Sources	                                        • External	  URL	  availability	  and	  latency	  alerts	  and	  re...
AppDynamics	          How	  to	  look	  deep	  inside	  your	  cloud	  applicaOons	  •  AutomaOc	  Monitoring	     –  Base...
Using	  AppDynamics	  (simple	  example	  from	  early	  2010)	  
Point	  Finger	  and	  Assess	  Impact	   (an	  async	  S3	  write	  was	  slow,	  no	  big	  deal)	  
Monitoring	  Summary	  •  Broken	  datacenter	  oriented	  tools	  is	  a	  big	  problem	  •  IntegraOng	  many	  differen...
Wrap	  Up	  
ImplicaOons	  for	  IT	  OperaOons	  •  Cloud	  is	  run	  by	  developer	  organizaOon	      –  Our	  IT	  department	  i...
Next	  Few	  Years…	  •  “System	  of	  Record”	  moves	  to	  Cloud	  (now)	        –  Master	  copies	  of	  data	  live...
Takeaway	                                  	  Ne9lix	  is	  path-­‐finding	  the	  use	  of	  public	  AWS	   cloud	  to	  ...
Amazon Cloud Terminology                                   See for Japanese                      ...
Upcoming SlideShare
Loading in …5



Netflix Velocity Conference 2011

Slide deck from June 14th 2011 Velocity Conference workshop presentation

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Netflix Velocity Conference 2011

  1. Ne#lix  Cloud  Architecture   Velocity  Conference  June  14,  2011   Adrian  Cockcro=   @adrianco  #ne#lixcloud  h@p://  
  2. Who,  Why,  What   Ne#lix  in  the  Cloud   Cloud  Challenges  and  Learnings   (Ignite)  Systems  and  OperaOons  Architecture    
  3. Ne#lix  Inc.   With  more  than  23  million  subscribers  in  the  United   States  and  Canada,  Ne9lix,  Inc.  is  the  world’s  leading   Internet  subscripAon  service  for  enjoying  movies  and   TV  shows.     InternaAonal  Expansion   We  plan  to  expand  into  an  addiAonal  market  in  the   second  half  of  2011…  If  the  second  market  meets  our   expectaAons…  we  will  conAnue  to  invest  and  expand   aggressively  in  2012.  Source:  h@p://  
  4. Unlimited  streaming  for  $7.99/month,  large  and  growing  catalog  of  movies  and  TV  
  5. Adrian  Cockcro=  •  Director,  Architecture  for  Cloud  Systems,  Ne#lix  Inc.   –  Previously  Director  for  PersonalizaOon  Pla#orm  •  DisOnguished  Availability  Engineer,  eBay  Inc.  2004-­‐7   –  Founding  member  of  eBay  Research  Labs  •  DisOnguished  Engineer,  Sun  Microsystems  Inc.  1988-­‐2004   –  2003-­‐4  Chief  Architect  High  Performance  Technical  CompuOng   –  2001  Author:  Capacity  Planning  for  Web  Services   –  1999  Author:  Resource  Management   –  1995  &  1998  Author:  Sun  Performance  and  Tuning   –  1996  Japanese  EdiOon  of  Sun  Performance  and  Tuning   •   SPARC  &  Solaris ( )  
  6. Why  is  Ne#lix  Talking  about   Cloud?  
  7. Ne#lix  is  Path-­‐finding   The  Cloud  ecosystem  is  evolving  very  fast  Share  with  and  learn  from  the  cloud  community  
  8. We  want  to  use  clouds,   not  build  them   Cloud  technology  should  be  a  commodity  Public  cloud  and  open  source  for  agility  and  scale  
  9. Why  Use  Cloud?     For  Be@er  Business  Agility  For  Unpredictable  Business  Growth  
  10. Data  Center   Ne#lix  could  not   build  new   datacenters  fast   enough   Capacity  growth  is  acceleraOng,  unpredictable   Product  launch  spikes  -­‐  iPhone,  Wii,  PS3,  XBox  
  11. 23  Million  Customers   2011-­‐Q1  year/year  customers  +69%     25   20   15   10   5   0  Source:  h@p://  
  12. Out-­‐Growing  Data  Center   h@p://­‐ne#lix-­‐api.html   37x  Growth  Jan   2010-­‐Jan  2011  Datacenter  Capacity  
  13.  is  now  ~100%  Cloud   Account  sign-­‐up  is  currently  being  moved  to  cloud   All  internaOonal  product  will  be  cloud  based   USA  specific  logisOcs  remains  in  the  Datacenter    
  14. Leverage  AWS  Scale   “the  biggest  public  cloud”   AWS  investment  in  tooling  and  automaOon  Use  many  AWS  zones  for  high  availability,  scalability   AWS  skills  are  most  common  on  resumes…  
  15. Leverage  AWS  Feature  Set   “the  market  leader”  EC2,  S3,  SDB,  SQS,  EBS,  EMR,  ELB,  ASG,  IAM,  RDB,  VPC…   h@p://  
  16. “The  cloud  lets  its  users  focus   on  delivering  differenAaAng   business  value  instead  of   wasAng  valuable  resources   on  the  undifferen)ated   heavy  li0ing  that  makes   up  most  of  IT   infrastructure.”      Werner  Vogels    Amazon  CTO    
  17. We  want  to  use  clouds,  we  don’t  have  Ome  to  build  them   Public  cloud  for  agility  and  scale   AWS  because  they  are  big  enough  to  allocate  thousands   of  instances  per  hour  when  we  need  to  
  18. Ne#lix  EC2  Instances  per  Account   (summer  2010,  producOon  is  much  higher  now…)  “Many  Thousands”   Content  Encoding   Test  and  ProducOon   Log  Analysis   “Several  Months”  
  19. Ne#lix  Deployed  on  AWS  Content   Logs   Play   WWW   API   Video   S3   DRM   Sign-­‐Up   Metadata   Masters   EMR   CDN   Device   EC2   Search   Hadoop   rouOng   Config   Movie   TV  Movie   S3   Hive   Bookmarks   Choosing   Choosing   Business   Mobile   CDN   Logging   RaOngs   Intelligence   iPhone  
  20. Cloud  Encoding  Pipeline   Encode   S3   Encode   S3  Movie   Master   Network   S3   Copy  to   CDN   Stream  Studios   Ne#lix   Master   Mezza-­‐ Mezza-­‐ to    50+   Origin   Origin   Tapes   Upload   nine   files   CDN   to  TV   nine   files   Licensed  content  is  provided  to  Ne#lix  as  high  quality  master  tapes   Many  formats  are  reduced  to  a  single  high  quality  mezzanine  format  on  S3   Individual  formats  and  speeds  are  encoded  in  over  50  combinaOons    Many  formats  for  older  and  newer  hardware  and  various  game  consoles    Many  speeds  from  mobile  through  standard  and  high  definiOon   StaOc  files  are  copied  to  each  Content  Delivery  Network’s  “origin  server”   CDNs  migrate  files  to  “edge  servers”  near  the  end  user   Files  stream  to  PC/Mac/iPad  or  TV  over  HTTP  using  “range  get”  to  move  chunks  
  21. Cloud  Architecture   Ignite!  
  22. Product  Trade-­‐off  User  Experience   ImplementaOon   Consistent   Development   Experience   complexity   OperaOonal   Low  Latency   complexity  
  23. Ne#lix  Cloud  Goals  •  Faster   –  Lower  latency  than  the  equivalent  datacenter  web  pages  and  API  calls   –  Measured  as  mean  and  99th  percenOle   –  For  both  first  hit  (e.g.  home  page)  and  in-­‐session  hits  for  the  same  user  •  Scalable   –  Avoid  needing  any  more  datacenter  capacity  as  subscriber  count  increases   –  No  central  verOcally  scaled  databases   –  Leverage  AWS  elasOc  capacity  effecOvely  •  Available   –  SubstanOally  higher  robustness  and  availability  than  datacenter  services   –  Leverage  mulOple  AWS  availability  zones   –  No  scheduled  down  Ome,  no  central  database  schema  to  change  •  ProducOve   –  OpOmize  agility  of  a  large  development  team  with  automaOon  and  tools   –  Leave  behind  complex  tangled  datacenter  code  base  (~8  year  old  architecture)   –  Enforce  clean  layered  interfaces  and  re-­‐usable  components  
  24. Old  Datacenter  vs.  New  Cloud  Arch   Central  SQL  Database   Distributed  Key/Value  NoSQL   SOcky  In-­‐Memory  Session   Shared  Memcached  Session   Cha@y  Protocols   Latency  Tolerant  Protocols   Tangled  Service  Interfaces   Layered  Service  Interfaces   Instrumented  Code   Instrumented  Service  Pa@erns   Fat  Complex  Objects   Lightweight  Serializable  Objects   Components  as  Jar  Files   Components  as  Services  
  25. The  Central  SQL  Database  •  Datacenter  has  a  central  database   –  Everything  in  one  place  is  convenient  unOl  it  fails   –  Customers,  movies,  history,  configuraOon  •  Schema  changes  require  downOme     AnA-­‐paUern  impacts  scalability,  availability  
  26. The  Distributed  Key-­‐Value  Store  •  Cloud  has  many  key-­‐value  data  stores   –  More  complex  to  keep  track  of,  do  backups  etc.   –  Each  store  is  much  simpler  to  administer   DBA   –  Joins  take  place  in  java  code  •  No  schema  to  change,  no  scheduled  downOme  •  Latency  for  Memcached  vs.  Oracle  vs.  SimpleDB   –  Memcached  is  dominated  by  network  latency  <1ms   –  Oracle  for  simple  queries  is  a  few  milliseconds   –  SimpleDB  has  replicaOon  and  REST  overheads  >10ms  
  27. The  SOcky  Session  •  Datacenter  SOcky  Load  Balancing   –  Efficient  caching  for  low  latency   –  Tricky  session  handling  code   –  Middle  Oer  load  balancer  has  issues  in  pracOce  •  Encourages  concentrated  funcOonality   –  one  service  that  does  everything     AnA-­‐paUern  impacts  producAvity,  availability  
  28. The  Shared  Session  •  Cloud  Uses  Round-­‐Robin  Load  Balancing   –  Simple  request-­‐based  code   –  External  shared  caching  with  memcached  •  More  flexible  fine  grain  services   –  Works  be@er  with  auto-­‐scaled  instance  counts  
  29. Cha@y  Opaque  and  Bri@le  Protocols  •  Datacenter  service  protocols   –  Assumed  low  latency  for  many  simple  requests  •  Based  on  serializing  exisOng  java  objects   –  Inefficient  formats   –  IncompaOble  when  definiOons  change     AnA-­‐paUern  causes  producAvity,  latency  and   availability  issues  
  30. Robust  and  Flexible  Protocols  •  Cloud  service  protocols   –  JSR311/Jersey  is  used  for  REST/HTTP  service  calls   –  Custom  client  code  includes  service  discovery   –  Support  complex  data  types  in  a  single  request  •  Apache  Avro   –  Evolved  from  Protocol  Buffers  and  Thri=   –  Includes  JSON  header  defining  key/value  protocol   –  Avro  serializaOon  is  half  the  size  and  several  Omes   faster  than  Java  serializaOon,  more  work  to  code  
  31. Persisted  Protocols  •  Persist  Avro  in  Memcached   –  Save  space/latency  (zigzag  encoding,  half  the  size)   –  Less  bri@le  across  versions   –  New  keys  are  ignored   –  Missing  keys  are  handled  cleanly  •  Avro  protocol  definiOons   –  Can  be  wri@en  in  JSON  or  generated  from  POJOs   –  It’s  hard,  needs  be@er  tooling  
  32. Tangled  Service  Interfaces  •  Datacenter  implementaOon  is  exposed   –  Oracle  SQL  queries  mixed  into  business  logic  •  Tangled  code   –  Deep  dependencies,  false  sharing  •  Data  providers  with  sideways  dependencies   –  Everything  depends  on  everything  else   AnA-­‐paUern  affects  producAvity,  availability  
  33. Untangled  Service  Interfaces  •  New  Cloud  Code  With  Strict  Layering   –  Compile  against  interface  jar   –  Can  use  spring  runOme  binding  to  enforce  •  Service  interface  is  the  service   –  ImplementaOon  is  completely  hidden   –  Can  be  implemented  locally  or  remotely   –  ImplementaOon  can  evolve  independently  
  34. Untangled  Service  Interfaces  Two  layers:  •  SAL  -­‐  Service  Access  Library   –  Basic  serializaOon  and  error  handling   –  REST  or  POJO’s  defined  by  data  provider  •  ESL  -­‐  Extended  Service  Library   –  Caching,  conveniences   –  Can  combine  several  SALs   –  Exposes  faceted  type  system  (described  later)   –  Interface  defined  by  data  consumer  in  many  cases  
  35. Service  InteracOon  Pa@ern   Sample  Swimlane  Diagram  
  36. Service  Architecture  Pa@erns  •  Internal  Interfaces  Between  Services   –  Common  pa@erns  as  templates   –  Highly  instrumented,  observable,  analyOcs   –  Service  Level  Agreements  –  SLAs  •  Library  templates  for  generic  features   –  Instrumented  Ne#lix  Base  Servlet  template   –  Instrumented  generic  client  interface  template   –  Instrumented  S3,  SimpleDB,  Memcached  clients  
  37. CLIENT   Request  Start   Timestamp,   Client   Inbound   Request  End   outbound   deserialize  end   Timestamp   serialize  start   Omestamp   Omestamp   Inbound   Client   deserialize   outbound   start   serialize  end   Omestamp   Omestamp  Client  network   receive   Omestamp   Service  Request   Client  Network   send   Omestamp   Instruments  Every   Service  network  send   Omestamp   Step  in  the  call   Service   Network   receive   Omestamp   Service   Service   outbound   inbound   serialize  end   serialize  start   Omestamp   Omestamp   Service   Service   outbound   inbound   serialize  start   SERVICE  execute   serialize  end   request  start   Omestamp   Omestamp   Omestamp,   execute  request   end  Omestamp  
  38. Boundary  Interfaces  •  Isolate  teams  from  external  dependencies   –  Fake  SAL  built  by  cloud  team   –  Real  SAL  provided  by  data  provider  team  later   –  ESL  built  by  cloud  team  using  faceted  objects  •  Fake  data  sources  allow  development  to  start   –  e.g.  Fake  IdenOty  SAL  for  a  test  set  of  customers   –  Development  solidifies  dependencies  early   –  Helps  external  team  provide  the  right  interface  
  39. One  Object  That  Does  Everything  •  Datacenter  uses  a  few  big  complex  objects   –  Movie  and  Customer  objects  are  the  foundaOon   –  Good  choice  for  a  small  team  and  one  instance   –  ProblemaOc  for  large  teams  and  many  instances  •  False  sharing  causes  tangled  dependencies   –  UnproducOve  re-­‐integraOon  work     AnA-­‐paUern  impacAng  producAvity  and   availability  
  40. An  Interface  For  Each  Component  •  Cloud  uses  faceted  Video  and  Visitor   –  Basic  types  hold  only  the  idenOfier   –  Facets  scope  the  interface  you  actually  need   –  Each  component  can  define  its  own  facets  •  No  false-­‐sharing  and  dependency  chains   –  Type  manager  converts  between  facets  as  needed   –  video.asA(PresentaOonVideo)  for  www   –  video.asA(MerchableVideo)  for  middle  Oer  
  41. So=ware  Architecture  Pa@erns  •  Object  Models   –  Basic  and  derived  types,  facets,  serializable   –  Pass  by  reference  within  a  service   –  Pass  by  value  between  services  •  ComputaOon  and  I/O  Models   –  Service  ExecuOon  using  Best  Effort   –  Common  thread  pool  management  
  42. Ne#lix  Systems  Architecture  
  43. API   AWS  EC2   Front  End  Load  Balancer   Discovery   Service   API  Proxy   API  etc.   Load  Balancer   Component   API   SQS   Services   Oracl e   Oracle   Oracle   memcached   memcached   ReplicaOon   EBS   NeAlix   S3   Data  Center  AWS  Storage   SimpleDB  
  44. Database  MigraOon  •  Why  SimpleDB?   –  No  DBA’s  in  the  cloud,  Amazon  hosted  service   –  Work  started  two  years  ago,  fewer  viable  opOons   –  Worked  with  Amazon  to  speed  up  and  scale  SimpleDB  •  AlternaOves?   –  Rolling  out  Cassandra  as  “upgrade”  from  SimpleDB   –  Need  several  opOons  to  match  use  cases  well  •  Detailed  NoSQL  and  SimpleDB  Advice   –  Sid  Anand    -­‐  QConSF  Nov  5th  –  Ne#lix’  TransiOon  to  High   Availability  Storage  Systems   –  Blog  -­‐  h@p://   –  Download  Paper  PDF  -­‐  h@p://  
  45. Cloud  OperaOons   Model  Driven  Architecture  Capacity  Planning  &  Monitoring  
  46. Tools  and  AutomaOon  •  Developer  and  Build  Tools   –  Jira,  Perforce,  Eclipse,  Jenkins,  Ivy,  ArOfactory   –  Builds,  creates  .war  file,  .rpm,  bakes  AMI  and  launches  •  Custom  Ne#lix  ApplicaOon  Console   –  AWS  Features  at  Enterprise  Scale  (hide  the  AWS  security  keys!)   –  Auto  Scaler  Group  is  unit  of  deployment  to  producOon  •  Open  Source  +  Support   –  Apache,  Tomcat,  Cassandra,  Hadoop,  OpenJDK,  CentOS  •  Monitoring  Tools   –  Keynote  –  service  monitoring  and  alerOng   –  Custom  metric  collecOon  and  alerOng  under  development   –  Datastax  OpsCenter  –  Cassandra  Monitoring   –  AppDynamics  –  Developer  focus  for  cloud  h@p://  
  47. Model  Driven  Architecture  •  Datacenter  PracOces   –  Lots  of  unique  hand-­‐tweaked  systems   –  Hard  to  enforce  pa@erns  •  Model  Driven  Cloud  Architecture   –  Perforce/Ivy/Jenkins  based  builds  for  everything   –  Every  producOon  instance  is  a  pre-­‐baked  AMI   –  Every  applicaOon  is  managed  by  an  Autoscaler   Every  change  is  a  new  AMI  
  48. High  Availability  Zones  •  Each  zone  is  a  separate  datacenter   –  Private  power,  cooling,  network  connecOons   –  Located  close  together  for  low  latency  •  ASG  Instances  are  distributed  over  3  zones  •  Data  wri@en  to  one  zone  appears  in  all  zones  •  Ne#lix  survived  total  failure  of  one  zone  (!)   –  Increase  capacity  of  exisOng  zones  by  50%   –  Small  or  zero  downOme  
  49. Cross  Region  Backups  •  Data  is  backed  up  into  a  different  cloud  region   –  Different  AWS  S3  account,  encrypted  for  security   –  AddiOonal  archive’s  created  on  a  different  vendor  •  Restore  to  a  new  region   –  Create  model  driven  architecture   –  Send  traffic  to  new  region  
  50. Model  Driven  ImplicaOons  •  Automated  “Least  Privilege”  Security   –  Tightly  specified  security  groups   –  Fine  grain  IAM  keys  to  access  AWS  resources   –  Performance  tools  security  and  integraOon  •  Model  Driven  Performance  Monitoring   –  Hundreds  of  instances  appear  in  a  few  minutes…   –  Tools  have  to  “garbage  collect”  dead  instances    
  51. Ne#lix  App  Console  
  52. Auto  Scale  Group  ConfiguraOon  
  53. Learnings  •  Datacenter  oriented  tools  don’t  work   –  Ephemeral  instances   –  High  rate  of  change   –  Need  too  much  hand-­‐holding  and  manual  setup  •  Many  Cloud  Tools  Don’t  Scale  for  Enterprise   –  Too  many  tools  are  “Startup”  oriented   –  Built  our  own  tools  for  1000’s  of  instances   –  Drove  vendors  to  be  dynamic,  scale,  add  APIs  •  Un-­‐modified  Datacenter  Apps  are  Fragile   –  Too  many  datacenter  oriented  assumpOons   –  We  re-­‐wrote  our  code  base!   –  (We  re-­‐write  it  conOnuously  anyway)  
  54. Capacity  Planning  &  Monitoring  
  55. Capacity  Planning  in  Clouds   (a  few  things  have  changed…)  •  Capacity  is  expensive  •  Capacity  takes  Ome  to  buy  and  provision  •  Capacity  only  increases,  can’t  be  shrunk  easily  •  Capacity  comes  in  big  chunks,  paid  up  front  •  Planning  errors  can  cause  big  problems  •  Systems  are  clearly  defined  assets  •  Systems  can  be  instrumented  in  detail  •  Depreciate  assets  over  3  years  (reservaOons!)  
  56. Monitoring  Issues  •  Problem   –  Too  many  tools,  each  with  a  good  reason  to  exist   –  Hard  to  get  an  integrated  view  of  a  problem   –  Too  much  manual  work  building  dashboards   –  Tools  are  not  discoverable,  views  are  not  filtered  •  SoluOon   –  Get  vendors  to  add  deep  linking  URLs  and  APIs   –  IntegraOon  “portal”  Oes  everything  together   –  Underlying  dependency  database   –  Dynamic  portal  generaOon,  relevant  data,  all  tools  
  57. Data  Sources   • External  URL  availability  and  latency  alerts  and  reports  –  Keynote   External  TesOng   • Stress  tesOng  -­‐  SOASTA   • Ne#lix  REST  calls  –  Chukwa  to  DataOven  with  GUID  transacOon  idenOfier   Request  Trace  Logging   • Generic  HTTP  –  AppDynamics  service  Oer  aggregaOon,  end  to  end  tracking   • Tracers  and  counters  –  log4j,  tracer  central,  Chukwa  to  DataOven   ApplicaOon  logging   • Trackid  and  Audit/Debug  logging  –  DataOven,  Appdynamics    GUID  cross  reference   • ApplicaOon  specific  real  Ome  –  Nimso=,  Appdynamics,  Epic   JMX    Metrics   • Service  and  SLA  percenOles  –  Nimso=,  Appdynamics,  Epic,logged  to  DataOven   • Stdout  logs  –  S3  –  DataOven,  Nimso=  alerOng  Tomcat  and  Apache  logs   • Standard  format  Access  and  Error  logs  –  S3  –  DataOven,  Nimso=  AlerOng   • Garbage  CollecOon  –  Nimso=,  Appdynamics   JVM   • Memory  usage,  call  stacks,  resource/call  -­‐  AppDynamics   • system  CPU/Net/RAM/Disk  metrics  –  AppDynamics,  Epic,  Nimso=  AlerOng   Linux   • SNMP  metrics  –  Epic,  Network  flows  –   • Load  balancer  traffic  –  Amazon  Cloudwatch,  SimpleDB  usage  stats   AWS   • System  configuraOon    -­‐  CPU  count/speed  and  RAM  size,  overall  usage  -­‐  AWS  
  58. AppDynamics   How  to  look  deep  inside  your  cloud  applicaOons  •  AutomaOc  Monitoring   –  Base  AMI  bakes  in  all  monitoring  tools   –  Outbound  calls  only  –  no  discovery/polling  issues   –  InacOve  instances  removed  a=er  a  few  days    •  Incident  Alarms  (deviaOon  from  baseline)   –  Business  TransacOon  latency  and  error  rate   –  Alarm  thresholds  discover  their  own  baseline   –  Email  contains  URL  to  Incident  Workbench  UI  
  59. Using  AppDynamics  (simple  example  from  early  2010)  
  60. Point  Finger  and  Assess  Impact   (an  async  S3  write  was  slow,  no  big  deal)  
  61. Monitoring  Summary  •  Broken  datacenter  oriented  tools  is  a  big  problem  •  IntegraOng  many  different  tools   –  They  are  not  designed  to  be  integrated   –  We  have  “persuaded”  vendors  to  add  APIs  •  If  you  can’t  see  deep  inside  your  app,  you’re  L  
  62. Wrap  Up  
  63. ImplicaOons  for  IT  OperaOons  •  Cloud  is  run  by  developer  organizaOon   –  Our  IT  department  is  Amazon  Cloud   –  Forming  “Cloud  OperaOons  Reliability  Eng”  team    •  TradiOonal  IT  Roles  are  going  away   –  Don’t  need  SA,  DBA,  Storage,  Network  admins   –  Database  Engineering  Team  runs  SDB/Cassandra  
  64. Next  Few  Years…  •  “System  of  Record”  moves  to  Cloud  (now)   –  Master  copies  of  data  live  only  in  the  cloud,  with  backups   –  Cut  the  datacenter  to  cloud  replicaOon  link,  turn  off  Oracle  databases  •  InternaOonal  Expansion  –  Global  Clouds  (later  in  2011)   –  Rapid  deployments  to  new  markets  •  Cloud  StandardizaOon?   –  Cloud  features  and  APIs  should  be  a  commodity  not  a  differenOator   –  DifferenOate  on  scale  and  quality  of  service   –  CompeOOon  and  scale  drives  cost  down   –  Higher  resilience  and  scalability     We  would  prefer  to  be  an  insignificant  customer  in  a  giant  cloud  
  65. Takeaway    Ne9lix  is  path-­‐finding  the  use  of  public  AWS   cloud  to  replace  in-­‐house  IT  for  non-­‐trivial  applicaAons  with  hundreds  of  developers  and   thousands  of  systems.   h@p://   @adrianco  #ne#lixcloud  
  66. Amazon Cloud Terminology See for Japanese This is not a full list of Amazon Web Service features•  AWS  –  Amazon  Web  Services  (common  name  for  Amazon  cloud)  •  AMI  –  Amazon  Machine  Image  (archived  boot  disk,  Linux,  Windows  etc.  plus  applicaOon  code)  •  EC2  –  ElasOc  Compute  Cloud   –  Range  of  virtual  machine  types  m1,  m2,  c1,  cc,  cg.  Varying  memory,  CPU  and  disk  configuraOons.   –  Instance  –  a  running  computer  system.  Ephemeral,  when  it  is  de-­‐allocated  nothing  is  kept.   –  Reserved  Instances  –  pre-­‐paid  to  reduce  cost  for  long  term  usage   –  Availability  Zone  –  datacenter  with  own  power  and  cooling  hosOng  cloud  instances   –  Region  –  group  of  Availability  Zones  –  US-­‐East,  US-­‐West,  EU-­‐Eire,  Asia-­‐Singapore,  Asia-­‐Japan  •  ASG  –  Auto  Scaling  Group  (instances  booOng  from  the  same  AMI)  •  S3  –  Simple  Storage  Service  (h@p  access)  •  EBS  –  ElasOc  Block  Storage  (network  disk  filesystem  can  be  mounted  on  an  instance)  •  RDB  –  RelaOonal  Data  Base  (managed  MySQL  master  and  slaves)  •  SDB  –  Simple  Data  Base  (hosted  h@p  based  NoSQL  data  store)  •  SQS  –  Simple  Queue  Service  (h@p  based  message  queue)  •  SNS  –  Simple  NoOficaOon  Service  (h@p  and  email  based  topics  and  messages)  •  EMR  –  ElasOc  Map  Reduce  (automaOcally  managed  Hadoop  cluster)  •  ELB  –  ElasOc  Load  Balancer  •  EIP  –  ElasOc  IP  (stable  IP  address  mapping  assigned  to  instance  or  ELB)  •  VPC  –  Virtual  Private  Cloud  (extension  of  enterprise  datacenter  network  into  cloud)  •  IAM  –  IdenOty  and  Access  Management  (fine  grain  role  based  security  keys)  
  • GOTOSatoru

    Oct. 19, 2020
  • tungdang51

    Aug. 27, 2020
  • Gudimetla

    Jul. 30, 2020
  • ThomasFahey1

    Oct. 16, 2019
  • needleyang

    Sep. 8, 2018
  • sanghunbaek98

    Jan. 18, 2017
  • chulminchoi56

    Dec. 20, 2016
  • ruichaolin

    Nov. 14, 2016
  • ssuserc6b4c6

    Oct. 25, 2016
  • net79

    Oct. 6, 2016
  • ssuser020868

    Mar. 8, 2016
  • HyukKwon1

    Jan. 20, 2016
  • LahcenJerid

    Dec. 23, 2015
  • zihado

    Dec. 13, 2015

    Oct. 12, 2015
  • redbrick9

    Aug. 5, 2015
  • bigbluebus

    Jun. 5, 2015
  • supingsue

    May. 17, 2015
  • hrenam

    Apr. 10, 2015
  • yoonsunjeon1

    Mar. 6, 2015

Slide deck from June 14th 2011 Velocity Conference workshop presentation


Total views


On Slideshare


From embeds


Number of embeds