SlideShare a Scribd company logo
1 of 60
Download to read offline
Cloud	
  Architecture	
  Tutorial	
  
       Running	
  in	
  the	
  Cloud	
  

       Qcon	
  London	
  March	
  5th,	
  2012	
  
                Adrian	
  Cockcro6	
  
               @adrianco	
  #ne:lixcloud	
  
       h>p://www.linkedin.com/in/adriancockcro6	
  
                     Part	
  3	
  of	
  3	
  
Running	
  in	
  the	
  Cloud	
  
Bring-­‐up	
  Strategy	
  for	
  Developers	
  and	
  TesRng	
  
       Capacity	
  Planning	
  and	
  Workloads	
  
                  Running	
  Cassandra	
  
              Monitoring	
  and	
  Scalability	
  
              Availability	
  and	
  Resilience	
  
               OrganizaRonal	
  Structure	
  
                                 	
  
                                 	
  
                                 	
  
Cloud	
  Bring-­‐Up	
  Strategy	
  
    Simplest	
  and	
  Soonest	
  
Shadow	
  Traffic	
  RedirecRon	
  
•  Early	
  a>empt	
  to	
  send	
  traffic	
  to	
  cloud	
  
    –  Real	
  traffic	
  stream	
  to	
  validate	
  cloud	
  back	
  end	
  
    –  Uncovered	
  lots	
  of	
  process	
  and	
  tools	
  issues	
  
    –  Uncovered	
  Service	
  latency	
  issues	
  
•  TV	
  Device	
  calls	
  Datacenter	
  API	
  and	
  Cloud	
  API	
  
    –  Returns	
  Genre/movie	
  list	
  for	
  a	
  customer	
  
    –  Asynchronously	
  duplicate	
  request	
  to	
  cloud	
  
    –  Start	
  with	
  send-­‐and-­‐forget	
  mode,	
  ignore	
  response	
  
Shadow	
  Redirect	
  Instances	
  

   Modified	
  
                                            Datacenter	
  
  Datacenter	
                               Service	
  
   Instances	
  


Modified	
  Cloud	
                        Cloud	
  Service	
  
                                                                 One	
  request	
  per	
  
  Instances	
                                                            visit	
  




 Data	
  Sources	
     queueservice	
     videometadata	
  
Video	
  Metadata	
  Server	
  
•  VMS	
  instance	
  isolates	
  new	
  pla:orm	
  from	
  old	
  codebase	
  
    –  Isolate/unblock	
  cloud	
  team	
  from	
  metadata	
  team	
  schedule	
  
    –  Datacenter	
  code	
  supports	
  obsolete	
  movie	
  object	
  
    –  VMS	
  ESL	
  is	
  designed	
  to	
  support	
  new	
  video	
  facet	
  object	
  

•  VMS	
  subsets	
  and	
  pre-­‐processes	
  the	
  metadata	
  
    –  Only	
  load	
  data	
  used	
  by	
  cloud	
  services	
  
    –  Fast	
  bulk	
  loads	
  for	
  VMS	
  clients	
  speed	
  startup	
  Rmes	
  
    –  Explore	
  next	
  generaRon	
  metadata	
  cache	
  architecture	
  
                                                  	
  
        Pa$ern	
  –	
  Add	
  services	
  to	
  isolate	
  old	
  and	
  new	
  code	
  base	
  
First	
  Web	
  Pages	
  in	
  the	
  Cloud	
  
First	
  Page	
  
•  First	
  full	
  page	
  –	
  Basic	
  Genre	
  
    –  Simplest	
  page,	
  no	
  sub-­‐genres,	
  minimal	
  personalizaRon	
  
    –  Lots	
  of	
  investment	
  in	
  new	
  Struts	
  based	
  page	
  design	
  
    –  Uses	
  idenRty	
  cookie	
  to	
  lookup	
  in	
  member	
  info	
  svc	
  
•  New	
  “merchweb”	
  front	
  end	
  instance	
  
    –  movies.ne:lix.com	
  points	
  to	
  merchweb	
  instance	
  
•  Uncovered	
  lots	
  of	
  latency	
  issues	
  
    –  Used	
  memcached	
  to	
  hide	
  S3	
  and	
  SimpleDB	
  latency	
  
    –  Improved	
  from	
  slower	
  to	
  faster	
  than	
  Datacenter	
  
Genre	
  Page	
  Cloud	
  Instances	
  


  Front	
  End	
                                                     merchweb	
  




                                                                                        mulRple	
  requests	
  
 Middle	
  Tier	
                         genre	
                   	
  memcached	
         per	
  visit	
  




Data	
  Sources	
     queueservice	
  
                      rentalhistory	
  
                                                      videometadata	
  
Controlled	
  Cloud	
  TransiRon	
  
•  WWW	
  calling	
  code	
  chooses	
  who	
  goes	
  to	
  cloud	
  
   –  Filter	
  out	
  corner	
  cases,	
  send	
  percentage	
  of	
  users	
  
   –  The	
  URL	
  that	
  customers	
  see	
  is	
  
      h>p://movies.ne:lix.com/WiContentPage?csid=1	
  
   –  If	
  problem,	
  redirect	
  to	
  old	
  Datacenter	
  page	
  
      h>p://www.ne:lix.com/WiContentPage?csid=1	
  
•  Play	
  Bu>on	
  and	
  Star	
  RaRng	
  AcRon	
  redirect	
  
   –  Point	
  URLs	
  for	
  acRons	
  that	
  create/modify	
  data	
  
      back	
  to	
  datacenter	
  to	
  start	
  with	
  
Cloud	
  Development	
  and	
  TesRng	
  Issues	
  
Boot	
  Camp	
  
•  One	
  day	
  “Ne:lix	
  Cloud	
  Training”	
  class	
  
    –  Has	
  been	
  run	
  6	
  Rmes	
  for	
  20-­‐45	
  people	
  each	
  Rme	
  
•  Half	
  day	
  of	
  presentaRons	
  
•  Half	
  day	
  hands-­‐on	
  
    –  Create	
  your	
  own	
  hello	
  world	
  app	
  
    –  Launch	
  in	
  AWS	
  test	
  account	
  
    –  Login	
  to	
  your	
  cloud	
  instances	
  
    –  Find	
  monitoring	
  data	
  on	
  your	
  cloud	
  instances	
  
    –  Connect	
  to	
  Cassandra	
  and	
  read/write	
  data	
  
Very	
  First	
  Boot	
  Camp	
  
•  Pathfinder	
  Bootstrap	
  Mission	
  
    –  Room	
  full	
  of	
  engineers	
  sharing	
  the	
  pain	
  for	
  1-­‐2	
  days	
  
    –  Built	
  a	
  very	
  rough	
  prototype	
  working	
  web	
  site	
  
•  Get	
  everyone	
  hands-­‐on	
  with	
  a	
  new	
  code	
  base	
  
    –  Debug	
  lots	
  of	
  tooling	
  and	
  conceptual	
  issues	
  very	
  fast	
  
    –  Used	
  SimpleDB	
  to	
  create	
  mock	
  data	
  sources	
  
•  Cloud	
  Specific	
  Key	
  Setup	
  
    –  Needed	
  to	
  integrate	
  with	
  AWS	
  security	
  model	
  
    –  New	
  concepts	
  for	
  datacenter	
  developers	
  
Developer	
  Instances	
  Collision	
  
Sam	
  and	
  Rex	
  both	
  want	
  to	
  deploy	
  web	
  front	
  end	
  for	
  
                             development	
  



    Sam	
                                                       Rex	
  

                                 web	
  in	
  
                                  test	
  
                                account	
  
Per-­‐Service	
  Namespace	
  Stack	
  RouRng	
  
         Developers	
  choose	
  what	
  to	
  share	
  




     Sam	
               Rex	
               Mike	
  
  web-­‐sam	
         web-­‐rex	
          web-­‐dev	
  

backend-­‐dev	
     backend-­‐dev	
     backend-­‐mike	
  
Developer	
  Namespace	
  Stacks	
  
•  Developer	
  specific	
  service	
  instances	
  
   –  Configured	
  via	
  Java	
  properRes	
  at	
  runRme	
  
   –  RouRng	
  implemented	
  by	
  REST	
  client	
  library	
  
•  Server	
  ConfiguraRon	
  
   –  Configure	
  discovery	
  service	
  version	
  string	
  
   –  Registers	
  as	
  <appname>-­‐<namespace>	
  
•  Client	
  ConfiguraRon	
  
   –  Route	
  traffic	
  on	
  per-­‐service	
  basis	
  including	
  
      namespace	
  
Capacity	
  Planning	
  Metrics	
  and	
  
             Methods	
  
What	
  is	
  Capacity	
  Planning	
  
•  We	
  care	
  about	
  
     –  CPU,	
  Memory,	
  Network	
  and	
  Disk	
  resource	
  uRlizaRon	
  
     –  ApplicaRon	
  response	
  Rmes	
  and	
  throughput	
  

•  We	
  need	
  to	
  know	
  
     –  how	
  much	
  of	
  each	
  resource	
  we	
  are	
  using	
  now,	
  and	
  will	
  use	
  in	
  
        the	
  future	
  
     –  how	
  much	
  headroom	
  we	
  have	
  to	
  handle	
  higher	
  loads	
  

•  We	
  want	
  to	
  understand	
  
     –  how	
  headroom	
  varies	
  
     –  how	
  it	
  relates	
  to	
  applicaRon	
  response	
  Rmes	
  and	
  throughput	
  
Capacity	
  Planning	
  Norms	
  
•    Capacity	
  is	
  expensive	
  
•    Capacity	
  takes	
  Rme	
  to	
  buy	
  and	
  provision	
  
•    Capacity	
  only	
  increases,	
  can’t	
  be	
  shrunk	
  easily	
  
•    Capacity	
  comes	
  in	
  big	
  chunks,	
  paid	
  up	
  front	
  
•    Planning	
  errors	
  can	
  cause	
  big	
  problems	
  
•    Systems	
  are	
  clearly	
  defined	
  assets	
  
•    Systems	
  can	
  be	
  instrumented	
  in	
  detail	
  
•    Depreciate	
  assets	
  over	
  3	
  years	
  
Capacity	
  Planning	
  in	
  Clouds	
  
                     (a	
  few	
  things	
  have	
  changed…)	
  

•    Capacity	
  is	
  expensive	
  
•    Capacity	
  takes	
  Rme	
  to	
  buy	
  and	
  provision	
  
•    Capacity	
  only	
  increases,	
  can’t	
  be	
  shrunk	
  easily	
  
•    Capacity	
  comes	
  in	
  big	
  chunks,	
  paid	
  up	
  front	
  
•    Planning	
  errors	
  can	
  cause	
  big	
  problems	
  
•    Systems	
  are	
  clearly	
  defined	
  assets	
  
•    Systems	
  can	
  be	
  instrumented	
  in	
  detail	
  
•    Depreciate	
  assets	
  over	
  3	
  years	
  (reservaRons!)	
  
Capacity	
  is	
  expensive	
  
              h>p://aws.amazon.com/s3/	
  &	
  h>p://aws.amazon.com/ec2/                                                   	
  
•  Storage	
  (Amazon	
  S3)	
  	
  
      –  $0.125	
  per	
  GB	
  –	
  first	
  50	
  TB	
  /	
  month	
  of	
  storage	
  used	
  
      –  $0.055	
  per	
  GB	
  –	
  storage	
  used	
  /	
  month	
  over	
  5	
  PB	
  
•  Data	
  Transfer	
  (Amazon	
  S3)	
  	
  
      –  $0.000	
  per	
  GB	
  –	
  all	
  data	
  transfer	
  in	
  is	
  free,	
  first	
  GB	
  out	
  is	
  free	
  
      –  $0.120	
  per	
  GB	
  –	
  first	
  10	
  TB	
  /	
  month	
  data	
  transfer	
  out	
  
      –  $0.050	
  per	
  GB	
  –	
  data	
  transfer	
  out	
  /	
  month	
  over	
  350	
  TB	
  
•  Requests	
  (Amazon	
  S3	
  Storage	
  access	
  is	
  via	
  h>p)	
  
      –  $0.01	
  per	
  1,000	
  PUT,	
  COPY,	
  POST,	
  or	
  LIST	
  requests	
  
      –  $0.01	
  per	
  10,000	
  GET	
  and	
  all	
  other	
  requests	
  
      –  $0	
  per	
  DELETE	
  
•  CPU	
  (Amazon	
  EC2)	
  
      –  Small	
  (Default)	
  $0.085/hour,	
  Extra	
  Large	
  $0.68/hour,	
  Four	
  XL	
  $2.00/hour	
  
      –  Small	
  (Default)	
  $0.08/hour,	
  Extra	
  Large	
  $0.64/hour,	
  Four	
  XL	
  $1.80/hour	
  
•  Network	
  (Amazon	
  EC2)	
  
      –  Inbound/Outbound	
  around	
  $0.10	
  per	
  GB	
  
Capacity	
  comes	
  in	
  big	
  chunks,	
  paid	
  up	
  front	
  

•  Capacity	
  takes	
  Rme	
  to	
  buy	
  and	
  provision	
  
    –  No	
  minimum	
  price,	
  monthly	
  billing	
  
    –  “Amazon	
  EC2	
  enables	
  you	
  to	
  increase	
  or	
  decrease	
  
       capacity	
  within	
  minutes,	
  not	
  hours	
  or	
  days.	
  You	
  can	
  
       commission	
  one,	
  hundreds	
  or	
  even	
  thousands	
  of	
  
       server	
  instances	
  simultaneously”	
  
•  Capacity	
  only	
  increases,	
  can’t	
  be	
  shrunk	
  easily	
  
    –  Pay	
  for	
  what	
  is	
  actually	
  used	
  
•  Planning	
  errors	
  can	
  cause	
  big	
  problems	
  
    –  Size	
  only	
  for	
  what	
  you	
  need	
  now	
  
Systems	
  are	
  clearly	
  defined	
  assets	
  

•  You	
  are	
  running	
  in	
  a	
  “stateless”	
  mulR-­‐
   tenanted	
  virtual	
  image	
  that	
  can	
  die	
  or	
  be	
  
   taken	
  away	
  and	
  replaced	
  at	
  any	
  Rme	
  

•  You	
  don’t	
  know	
  exactly	
  where	
  it	
  is,	
  you	
  can	
  
   choose	
  to	
  locate	
  “US-­‐East”	
  or	
  “Europe”	
  etc.	
  

•  You	
  can	
  specify	
  zones	
  that	
  will	
  not	
  share	
  
   components	
  to	
  avoid	
  common	
  mode	
  failures	
  
Systems	
  can	
  be	
  instrumented	
  in	
  detail	
  

•  Each	
  cloud	
  node	
  allocaRon	
  is	
  unique	
  
    –  So	
  elasRc	
  usage	
  pa>erns	
  keep	
  creaRng	
  new	
  nodes	
  
    –  “garbage	
  collect”	
  nodes	
  that	
  won’t	
  be	
  seen	
  again	
  
    –  Need	
  to	
  map	
  EIP	
  and	
  Cassandra	
  tokens	
  to	
  instances	
  


•  Ne:lix	
  SoluRon	
  –	
  Entrypoints	
  Slots	
  
    –  Each	
  Autoscale	
  Group	
  has	
  a	
  size	
  
    –  Each	
  instance	
  is	
  given	
  a	
  slot	
  number	
  up	
  to	
  size	
  
    –  Replacements	
  pick	
  empty	
  slots	
  
Depreciate	
  assets	
  over	
  3	
  years	
  
                 (reservaRons!)	
  
•    Reduced	
  costs	
  in	
  return	
  for	
  commitment	
  
•    One	
  or	
  three	
  years,	
  upfront	
  payment	
  
•    Payment	
  can	
  be	
  depreciated	
  as	
  capital	
  asset	
  
•    Low,	
  medium	
  or	
  high	
  usage	
  reservaRons	
  
      –  Save	
  more	
  if	
  you	
  use	
  them	
  more	
  
•  Spot	
  market	
  instances	
  
      –  Unused	
  reservaRons	
  sold	
  to	
  other	
  users	
  cheap	
  
      –  Will	
  be	
  yanked	
  at	
  any	
  Rme	
  if	
  needed	
  
A	
  Discussion	
  of	
  Workloads	
  and	
  
         How	
  They	
  Behave	
  
Workload	
  CharacterisRcs	
  

    •  A	
  quick	
  tour	
  through	
  a	
  taxonomy	
  of	
  
       workload	
  types	
  

    •  Start	
  with	
  the	
  easy	
  ones	
  and	
  work	
  up	
  

    •  Why	
  personalized	
  workloads	
  are	
  different	
  
       and	
  hard	
  

    •  Some	
  examples	
  and	
  coping	
  strategies	
  

3/12/12	
                                                              Slide	
  176	
  
Simple	
  Random	
  Arrivals	
  


    •  Random	
  arrival	
  of	
  transacRons	
  with	
  fixed	
  mean	
  
       service	
  Rme	
  
              –  Li>le’s	
  Law:	
  QueueLength	
  =	
  Throughput	
  *	
  Response	
  
              –  URlizaRon	
  Law:	
  URlizaRon	
  =	
  Throughput	
  *	
  ServiceTime	
  

    •  Complex	
  models	
  are	
  o6en	
  reduced	
  to	
  this	
  model	
  
              –  By	
  averaging	
  over	
  longer	
  Rme	
  periods	
  since	
  the	
  formulas	
  
                 only	
  work	
  if	
  you	
  have	
  stable	
  averages	
  
              –  By	
  wishful	
  thinking	
  (i.e.	
  how	
  to	
  fool	
  yourself)	
  

3/12/12	
                                                                                    Slide	
  177	
  
Mixed	
  random	
  arrivals	
  of	
  transacRons	
  
                  with	
  stable	
  mean	
  service	
  Rmes	
  
    •  Think	
  of	
  the	
  grocery	
  store	
  checkout	
  analogy	
  
              –  Trolleys	
  full	
  of	
  shopping	
  vs.	
  baskets	
  full	
  of	
  shopping	
  
              –  Baskets	
  are	
  quick	
  to	
  service,	
  but	
  get	
  stuck	
  behind	
  carts	
  
              –  RelaRve	
  mixture	
  of	
  transacRon	
  types	
  starts	
  to	
  ma>er	
  

    •  Many	
  transacRonal	
  systems	
  handle	
  a	
  mixture	
  
              –  Databases,	
  web	
  services	
  

    •  Consider	
  separaRng	
  fast	
  and	
  slow	
  transacRons	
  
              –  So	
  that	
  we	
  have	
  a	
  “10	
  items	
  or	
  less”	
  line	
  just	
  for	
  baskets	
  
              –  Separate	
  pools	
  of	
  servers	
  for	
  different	
  services	
  
              –  The	
  old	
  rule	
  -­‐	
  don’t	
  mix	
  OLTP	
  with	
  DSS	
  queries	
  in	
  databases	
  

    •  Performance	
  is	
  o6en	
  thread-­‐limited	
  
              –  Thread	
  limit	
  and	
  slow	
  transacRons	
  constrains	
  maximum	
  throughput	
  

    •  Model	
  mix	
  using	
  analyRcal	
  solvers	
  (e.g.	
  PDQ	
  perfdynamics.com)	
  

3/12/12	
                                                                                                             Slide	
  178	
  
Load	
  dependent	
  servers	
  –	
  varying	
  
                  mean	
  service	
  Rmes	
  
•  Mean	
  service	
  Rme	
  may	
  increase	
  at	
  high	
  throughput	
  
       –  Due	
  to	
  non-­‐scalable	
  algorithms,	
  lock	
  contenRon	
  
       –  System	
  runs	
  out	
  of	
  memory	
  and	
  starts	
  paging	
  or	
  frequent	
  GC	
  

•  Mean	
  service	
  Rme	
  may	
  also	
  decrease	
  at	
  high	
  throughput	
  
       –  Elevator	
  seek	
  and	
  write	
  cancellaRon	
  opRmizaRons	
  in	
  storage	
  
       –  Load	
  shedding	
  and	
  simplified	
  fallback	
  modes	
  

•  Systems	
  have	
  “Rpping	
  points”	
  if	
  the	
  service	
  Rme	
  increases	
  
       –      Hysteresis	
  means	
  they	
  don’t	
  come	
  back	
  when	
  load	
  drops	
  
       –      This	
  is	
  why	
  you	
  have	
  to	
  kill	
  catatonic	
  systems	
  
       –      Best	
  designs	
  shed	
  load	
  to	
  be	
  stable	
  at	
  the	
  limit	
  –	
  circuit	
  breaker	
  pa>ern	
  
       –      PracRcal	
  opRon	
  is	
  to	
  try	
  to	
  avoid	
  Rpping	
  points	
  by	
  reducing	
  variance	
  
	
  
•  Model	
  using	
  discrete	
  event	
  simulaRon	
  tools	
  
       –  Behaviour	
  is	
  non-­‐linear	
  and	
  hard	
  to	
  model	
  

3/12/12	
                                                                                                                            Slide	
  179	
  
Self-­‐similar	
  /	
  fractal	
  workloads	
  
•  Bursty	
  rather	
  than	
  random	
  arrival	
  rates	
  

•  Self-­‐similar	
  
       –  Looks	
  “random”	
  at	
  close	
  up,	
  stays	
  “random”	
  as	
  you	
  zoom	
  out	
  
       –  Work	
  arrives	
  in	
  bursts,	
  transacRons	
  aren’t	
  independent	
  
       –  Bursts	
  cluster	
  together	
  in	
  super-­‐bursts,	
  etc.	
  

•  Network	
  packet	
  streams	
  tend	
  to	
  be	
  fractal	
  

•  Common	
  in	
  pracRce,	
  too	
  hard	
  to	
  model	
  
       –  Probably	
  the	
  most	
  common	
  reason	
  why	
  your	
  model	
  is	
  wrong!	
  


3/12/12	
                                                                                  Slide	
  180	
  
State	
  Dependent	
  Service	
  Workloads	
  

    •  Personalized	
  services	
  that	
  store	
  user	
  state/history	
  
              –  TransacRons	
  for	
  new	
  users	
  are	
  quick	
  
              –  TransacRons	
  for	
  users	
  with	
  lots	
  of	
  state/history	
  are	
  slower	
  
              –  As	
  user	
  base	
  builds	
  state	
  and	
  ages	
  you	
  get	
  into	
  trouble…	
  

    •  Social	
  Networks,	
  RecommendaRon	
  Services	
  
              –  Facebook,	
  Flickr,	
  Ne:lix,	
  Twi>er	
  etc.	
  

    •  “Abandon	
  hope	
  all	
  ye	
  who	
  enter	
  here”	
  
              –  Not	
  tractable	
  to	
  model,	
  repeatable	
  tests	
  are	
  tricky	
  
              –  Long	
  fat	
  tail	
  response	
  Rme	
  distribuRon	
  and	
  Rmeouts	
  

    •  Try	
  to	
  transform	
  workloads	
  to	
  more	
  tractable	
  forms	
  

3/12/12	
                                                                                                Slide	
  181	
  
Example	
  -­‐	
  Twi>er	
  Workload	
  
•  @adrianco	
  tweets	
  –	
  copy	
  to	
  3600	
  or	
  so	
  other	
  users	
  
•  @zoecello	
  tweets	
  many	
  Rmes	
  a	
  day	
  	
  –	
  to	
  over	
  1M	
  users	
  
•  @barackobama	
  tweets	
  every	
  few	
  days	
  –	
  to	
  over	
  12M	
  users	
  
•  It’s	
  the	
  same	
  transacRon,	
  but	
  the	
  service	
  Rme	
  varies	
  by	
  several	
  
   orders	
  of	
  magnitude	
  
•  The	
  best	
  (most	
  acRve	
  and	
  connected	
  =	
  most	
  valuable)	
  users	
  
   trigger	
  a	
  “denial	
  of	
  service	
  a>ack”	
  on	
  the	
  systems	
  when	
  they	
  
   tweet	
  
•  Cascading	
  effect	
  as	
  many	
  others	
  re-­‐tweet	
  

3/12/12	
                                                                                      Slide	
  182	
  
Example	
  -­‐	
  Ne:lix	
  Movie	
  Choosing	
  
                     •  “Pick	
  24	
  genres/subgenres	
  etc.	
  of	
  75	
  movies	
  each	
  for	
  me”	
  
                           –  used	
  by	
  TV	
  based	
  devices	
  like	
  Xbox360,	
  PS/3,	
  iPhone	
  app	
  
                     •  New	
  user	
  
                           –    No	
  history	
  of	
  what	
  they	
  have	
  rented	
  (DVD)	
  or	
  streamed	
  
                           –    No	
  star	
  raRngs	
  for	
  movies,	
  possibly	
  some	
  genre	
  raRngs	
  
                           –    Basic	
  demographic	
  info	
  
                           –    Fast	
  to	
  calculate,	
  easy	
  to	
  find	
  many	
  good	
  choices	
  to	
  return	
  
                     •  User	
  with	
  several	
  years	
  tenure	
  
                           –  Thousands	
  of	
  movies	
  rented	
  or	
  streamed,	
  “seen	
  it	
  already”	
  
                           –  Hundreds	
  to	
  thousands	
  of	
  star	
  raRngs,	
  lots	
  of	
  genre	
  raRngs	
  
                           –  Requests	
  may	
  Rme	
  out	
  and	
  return	
  fewer	
  or	
  worse	
  choices	
  




3/12/12	
                                                                                                            Slide	
  183	
  
Workload	
  Modelling	
  Survival	
  
                             Methods	
  
    •  Simplify	
  the	
  workload	
  algorithms	
  
              –  move	
  from	
  hard	
  or	
  impossible	
  to	
  simpler	
  models	
  
              –  decouple,	
  cache	
  and	
  pre-­‐compute	
  to	
  get	
  constant	
  service	
  Rmes	
  

    •  Stand	
  further	
  away	
  
              –  averaging	
  is	
  your	
  friend	
  –	
  gets	
  rid	
  of	
  complex	
  fluctuaRons	
  

    •  Minimalist	
  Models	
  
              –  most	
  models	
  are	
  far	
  too	
  complex	
  –	
  the	
  classic	
  beginners	
  error…	
  
              –  the	
  art	
  of	
  modelling	
  is	
  to	
  only	
  model	
  what	
  really	
  ma>ers	
  

    •  Don’t	
  model	
  details	
  you	
  don’t	
  use	
  
              –  model	
  peak	
  hour	
  of	
  the	
  week,	
  not	
  day	
  to	
  day	
  fluctuaRons	
  
              –  e.g.	
  “Will	
  the	
  web	
  site	
  survive	
  next	
  Sunday	
  night?”	
  


3/12/12	
                                                                                                       Slide	
  184	
  
Running	
  Cassandra	
  
Cassandra	
  Use	
  Cases	
  
•  Key	
  by	
  Customer	
  –	
  Cross-­‐region	
  clusters	
  
     –  Many	
  app	
  specific	
  Cassandra	
  clusters,	
  read-­‐intensive	
  
     –  Keys+Rows	
  in	
  memory	
  using	
  m2.4xl	
  Instances	
  

•  Key	
  by	
  Customer:Movie	
  –	
  e.g.	
  Viewing	
  History	
  
     –  Growing	
  fast,	
  write	
  intensive	
  –	
  m1.xl	
  instances	
  
     –  Keys	
  cached	
  in	
  memory,	
  one	
  cluster	
  per	
  region	
  

•  Large	
  scale	
  data	
  logging	
  –	
  lots	
  of	
  writes	
  
     –  Column	
  data	
  expires	
  a6er	
  Rme	
  period	
  
     –  Distributed	
  counters,	
  one	
  cluster	
  per	
  region	
  
Ne:lix	
  Pla:orm	
  Cassandra	
  AMI	
  
•  Tomcat	
  server	
  with	
  Priam	
  
   –  Always	
  running,	
  registers	
  with	
  pla:orm	
  
   –  Manages	
  Cassandra	
  state,	
  tokens,	
  backups	
  
•  Removed	
  Root	
  Disk	
  Dependency	
  on	
  EBS	
  
   –  Use	
  S3	
  backed	
  AMI	
  for	
  stateful	
  services	
  
   –  Normally	
  use	
  EBS	
  backed	
  AMI	
  for	
  fast	
  provisioning	
  
Ne:lix	
  ContribuRons	
  to	
  Cassandra	
  
•  Cassandra	
  as	
  a	
  mutable	
  toolkit	
  
     –  Cassandra	
  is	
  in	
  Java,	
  pluggable,	
  well	
  structured	
  
     –  Ne:lix	
  has	
  a	
  building	
  full	
  of	
  Java	
  engineers….	
  
     –  We	
  changed	
  Cassandra	
  to	
  make	
  it	
  run	
  much	
  be>er	
  on	
  AWS	
  

•  ContribuRons	
  delivered	
  to	
  Cassandra	
  
     –  0.8	
  Prototype	
  off-­‐heap	
  row	
  cache,	
  SSTable	
  write	
  callback	
  
     –  1.x	
  OpRmizaRons	
  reduced	
  impact	
  of	
  repair	
  &	
  compacRon	
  
     –  January	
  2012	
  –	
  Ne:lix	
  engineer	
  becomes	
  core	
  commi>er	
  

•  Cassandra	
  Based	
  Projects	
  on	
  github.com/Ne:lix	
  
     –  Priam	
  AWS	
  integraRon	
  and	
  backup	
  using	
  Tomcat	
  helper	
  
     –  Astyanax	
  	
  Java	
  client	
  library	
  
     –  CassJMeter	
  for	
  performance	
  and	
  regression	
  tesRng	
  
Monitoring	
  Tools	
  
Monitoring	
  Vision	
  
•  Problem	
  
   –  Too	
  many	
  tools,	
  each	
  with	
  a	
  good	
  reason	
  to	
  exist	
  
   –  Hard	
  to	
  get	
  an	
  integrated	
  view	
  of	
  a	
  problem	
  
   –  Too	
  much	
  manual	
  work	
  building	
  dashboards	
  
   –  Tools	
  are	
  not	
  discoverable,	
  views	
  are	
  not	
  filtered	
  
•  SoluRon	
  
   –  Get	
  vendors	
  to	
  add	
  deep	
  linking	
  and	
  embedding	
  
   –  IntegraRon	
  “portal”	
  Res	
  everything	
  together	
  
   –  Dynamic	
  portal	
  generaRon,	
  relevant	
  data,	
  all	
  tools	
  
Cloud	
  Monitoring	
  Mechanisms	
  
•  Keynote	
  or	
  Gomez	
  etc.	
  
     –  External	
  URL	
  monitoring	
  
•  Amazon	
  CloudWatch	
  
     –  Metrics	
  for	
  ELB	
  and	
  Instances	
  
•  AppDynamics	
  
     –  End	
  to	
  end	
  transacRon	
  view	
  showing	
  resources	
  used	
  
     –  Powerful	
  real	
  Rme	
  debug	
  tools	
  for	
  latency,	
  CPU	
  and	
  Memory	
  
•  Epic	
  (Ne:lix	
  in-­‐house	
  project)	
  
     –  Flexible	
  and	
  easy	
  to	
  use	
  to	
  extend	
  and	
  embed	
  plots	
  
•  Logs	
  
     –  High	
  capacity	
  logging	
  and	
  analysis	
  framework	
  
     –  Hadoop	
  (log4j	
  -­‐>	
  Honu	
  -­‐>	
  EMR)	
  
Using	
  AppDynamics	
  
(simple	
  example	
  from	
  early	
  2010)	
  
AppDynamics	
  Monitoring	
  of	
  Cassandra	
  –	
  AutomaRc	
  Discovery	
  
Scalability	
  TesRng	
  
•  Cloud	
  Based	
  TesRng	
  –	
  fricRonless,	
  elasRc	
  
    –  Create/destroy	
  any	
  sized	
  cluster	
  in	
  minutes	
  
    –  Many	
  test	
  scenarios	
  run	
  in	
  parallel	
  

•  Test	
  Scenarios	
  
    –  Internal	
  app	
  specific	
  tests	
  
    –  Simple	
  “stress”	
  tool	
  provided	
  with	
  Cassandra	
  

•  Scale	
  test,	
  keep	
  making	
  the	
  cluster	
  bigger	
  
    –  Check	
  that	
  tooling	
  and	
  automaRon	
  works…	
  
    –  How	
  many	
  ten	
  column	
  row	
  writes/sec	
  can	
  we	
  do?	
  
<DrEvil>ONE	
  MILLION</DrEvil>	
  
Scale-­‐Up	
  Linearity	
  
  h>p://techblog.ne:lix.com/2011/11/benchmarking-­‐cassandra-­‐scalability-­‐on.html	
  


                        Client	
  Writes/s	
  by	
  node	
  count	
  –	
  Replica:on	
  Factor	
  =	
  3	
  
1200000	
  
                                                                                                   1099837	
  
1000000	
  

 800000	
  

 600000	
  
                                                              537172	
  
 400000	
                                        366828	
  

 200000	
                           174373	
  

        0	
  
                0	
             50	
         100	
        150	
            200	
     250	
        300	
          350	
  
Stress	
  Client	
  Latency	
  
      Includes	
  ~10ms	
  Scheduling	
  Overhead	
  –	
  for	
  be>er	
  latency	
  data	
  see	
  
	
  h>p://techblog.ne:lix.com/2012/03/jmeter-­‐plugin-­‐for-­‐cassandra.html                           	
  
Measured	
  at	
  the	
  Cassandra	
  Server	
  
3.3	
  Million	
  writes/sec	
  at	
  0.014ms	
  –	
  14	
  microseconds	
  
Per	
  Node	
  AcRvity	
  
          Per	
  Node	
               48	
  Nodes	
         96	
  Nodes	
         144	
  Nodes	
           288	
  Nodes	
  
Per	
  Server	
  Writes/s	
           10,900	
  w/s	
       11,460	
  w/s	
          11,900	
  w/s	
            11,456	
  w/s	
  
Mean	
  Server	
  Latency	
            0.0117	
  ms	
        0.0134	
  ms	
           0.0148	
  ms	
             0.0139	
  ms	
  
Mean	
  CPU	
  %Busy	
                      74.4	
  %	
           75.4	
  %	
              72.5	
  %	
                81.5	
  %	
  
Disk	
  Read	
                        5,600	
  KB/s	
       4,590	
  KB/s	
          4,060	
  KB/s	
            4,280	
  KB/s	
  
Disk	
  Write	
                      12,800	
  KB/s	
   11,590	
  KB/s	
            10,380	
  KB/s	
           10,080	
  KB/s	
  
Network	
  Read	
                    22,460	
  KB/s	
   23,610	
  KB/s	
            21,390	
  KB/s	
           23,640	
  KB/s	
  
Network	
  Write	
                   18,600	
  KB/s	
   19,600	
  KB/s	
            17,810	
  KB/s	
           19,770	
  KB/s	
  


           Node	
  specificaRon	
  –	
  Xen	
  Virtual	
  Images,	
  AWS	
  US	
  East,	
  three	
  zones	
  
           •  Cassandra	
  0.8.6,	
  CentOS,	
  SunJDK6	
  
           •  AWS	
  EC2	
  m1	
  Extra	
  Large	
  –	
  Standard	
  price	
  $	
  0.68/Hour	
  
           •  15	
  GB	
  RAM,	
  4	
  Cores,	
  1Gbit	
  network	
  
           •  4	
  internal	
  disks	
  (total	
  1.6TB,	
  striped	
  together,	
  md,	
  XFS)	
  
Time	
  is	
  Money	
  
                                   48	
  nodes	
        96	
  nodes	
                  144	
  nodes	
                      288	
  nodes	
  
Writes	
  Capacity	
              174373	
  w/s	
       366828	
  w/s	
                   537172	
  w/s	
                1,099,837	
  w/s	
  
Storage	
  Capacity	
                  12.8	
  TB	
           25.6	
  TB	
                         38.4	
  TB	
                        76.8	
  TB	
  
Nodes	
  Cost/hr	
                      $32.64	
                $65.28	
                            $97.92	
                          $195.84	
  
Test	
  Driver	
  Instances	
                  10	
                      20	
                                30	
                               60	
  
Test	
  Driver	
  Cost/hr	
             $20.00	
                $40.00	
                            $60.00	
                          $120.00	
  
Cross	
  AZ	
  Traffic	
                 5	
  TB/hr	
         10	
  TB/hr	
                       15	
  TB/hr	
                       301	
  TB/hr	
  
Traffic	
  Cost/10min	
                     $8.33	
               $16.66	
                            $25.00	
                            $50.00	
  
Setup	
  DuraRon	
                15	
  minutes	
       22	
  minutes	
                    31	
  minutes	
                    662	
  minutes	
  
AWS	
  Billed	
  DuraRon	
                    1hr	
                    1hr	
                              1	
  hr	
                          2	
  hr	
  
Total	
  Test	
  Cost	
                 $60.97	
             $121.94	
                           $182.92	
                            $561.68	
  
                                                         1	
  EsRmate	
  two	
  thirds	
  of	
  total	
  network	
  traffic	
  	
  
                                                         2	
  Workaround	
  for	
  a	
  tooling	
  bug	
  slowed	
  setup	
  
Availability	
  and	
  Resilience	
  
Chaos	
  Monkey	
  
•  Computers	
  (Datacenter	
  or	
  AWS)	
  randomly	
  die	
  
    –  Fact	
  of	
  life,	
  but	
  too	
  infrequent	
  to	
  test	
  resiliency	
  
•  Test	
  to	
  make	
  sure	
  systems	
  are	
  resilient	
  
    –  Allow	
  any	
  instance	
  to	
  fail	
  without	
  customer	
  impact	
  
•  Chaos	
  Monkey	
  hours	
  
    –  Monday-­‐Thursday	
  9am-­‐3pm	
  random	
  instance	
  kill	
  
•  ApplicaRon	
  configuraRon	
  opRon	
  
    –  Apps	
  now	
  have	
  to	
  opt-­‐out	
  from	
  Chaos	
  Monkey	
  
Responsibility	
  and	
  Experience	
  
•  Make	
  developers	
  responsible	
  for	
  failures	
  
    –  Then	
  they	
  learn	
  and	
  write	
  code	
  that	
  doesn’t	
  fail	
  
•  Use	
  Incident	
  Reviews	
  to	
  find	
  gaps	
  to	
  fix	
  
    –  Make	
  sure	
  its	
  not	
  about	
  finding	
  “who	
  to	
  blame”	
  
•  Keep	
  Rmeouts	
  short,	
  fail	
  fast	
  
    –  Don’t	
  let	
  cascading	
  Rmeouts	
  stack	
  up	
  
•  Make	
  configuraRon	
  opRons	
  dynamic	
  
    –  You	
  don’t	
  want	
  to	
  push	
  code	
  to	
  tweak	
  an	
  opRon	
  
Resilient	
  Design	
  –	
  Circuit	
  Breakers	
  
h>p://techblog.ne:lix.com/2012/02/fault-­‐tolerance-­‐in-­‐high-­‐volume.html	
  
PaaS	
  OperaRonal	
  Model	
  -­‐	
  NoOps	
  
•  Developers	
  
   –  Provision	
  and	
  run	
  their	
  own	
  code	
  in	
  producRon	
  
   –  Take	
  turns	
  to	
  be	
  on	
  call	
  if	
  it	
  breaks	
  (pagerduty)	
  
   –  Configure	
  autoscalers	
  to	
  handle	
  capacity	
  needs	
  

•  Difference	
  between	
  DevOps	
  and	
  NoOps	
  
   –  DevOps	
  is	
  about	
  Dev	
  and	
  Ops	
  working	
  together	
  
   –  NoOps	
  constrains	
  Dev	
  to	
  use	
  automaRon	
  instead	
  
   –  NoOps	
  puts	
  more	
  responsibility	
  on	
  Dev,	
  with	
  tools	
  
ImplicaRons	
  for	
  IT	
  OperaRons	
  
•  Cloud	
  is	
  run	
  by	
  developer	
  organizaRon	
  
    –  Our	
  IT	
  department	
  is	
  the	
  AWS	
  API	
  
    –  We	
  have	
  no	
  IT	
  staff	
  working	
  on	
  cloud	
  (they	
  do	
  corp	
  IT)	
  

•  Cloud	
  capacity	
  is	
  10x	
  bigger	
  than	
  Datacenter	
  
    –  Datacenter	
  oriented	
  IT	
  staffing	
  is	
  flat	
  
    –  We	
  have	
  moved	
  a	
  few	
  people	
  out	
  of	
  IT	
  to	
  write	
  code	
  

•  TradiRonal	
  IT	
  Roles	
  are	
  going	
  away	
  
    –  Don’t	
  need	
  SA,	
  DBA,	
  Storage,	
  Network	
  admins	
  
    –  Developers	
  deploy	
  and	
  run	
  what	
  they	
  wrote	
  in	
  producRon	
  
Ne:lix	
  “NoOps”	
  OrganizaRon	
  
  Developer	
  Org	
  ReporRng	
  into	
  Product	
  Development,	
  not	
  ITops                                                                        	
  

                 Ne:lix	
  Cloud	
  Pla:orm	
  Team	
  
 Cloud	
  Ops	
                                      Build	
  Tools	
  
                             Database	
                                             Pla:orm	
                Cloud	
                   Cloud	
  
 Reliability	
                                           and	
  
                            Engineering	
                                         Development	
           Performance	
               SoluRons	
  
Engineering	
                                        AutomaRon	
  


                                                     Perforce	
  Jenkins	
          Pla:orm	
  jars	
        Cassandra	
  
                                                     ArRfactory	
  JIRA	
                                  Benchmarking	
              Monitoring	
  
  Alert	
  RouRng	
                                                                   Key	
  store	
  
                              Cassandra	
                                                                                               Monkeys	
  
Incident	
  Lifecycle	
                             Base	
  AMI,	
  Bakery	
         Zookeeper	
           JVM	
  GC	
  Tuning	
  
                                                   Ne:lix	
  App	
  Console	
                               Wiresharking	
             Entrypoints	
  
                                                                                      Astyanix	
  




    PagerDuty	
             AWS	
  Instances	
            AWS	
  API	
             AWS	
  Instances	
      AWS	
  Instances	
        AWS	
  Instances	
  
Wrap	
  Up	
  
                             	
  
   Answer	
  your	
  remaining	
  quesRons…	
  
                             	
  
What	
  was	
  missing	
  that	
  you	
  wanted	
  to	
  cover?	
  
Takeaway	
  
                                                     	
  
 Ne5lix	
  has	
  built	
  and	
  deployed	
  a	
  scalable	
  global	
  Pla5orm	
  as	
  a	
  Service.	
  
                                                     	
  
Key	
  components	
  of	
  the	
  Ne5lix	
  PaaS	
  are	
  being	
  released	
  as	
  Open	
  Source	
  
                   projects	
  so	
  you	
  can	
  build	
  your	
  own	
  custom	
  PaaS.	
  
                                                     	
  
                                  h>p://github.com/Ne:lix	
  
                                 h>p://techblog.ne:lix.com	
  
                                 h>p://slideshare.net/Ne:lix	
  
                                               	
  
                          h>p://www.linkedin.com/in/adriancockcro6	
  
                                  @adrianco	
  #ne:lixcloud	
  
                                               	
  
                                    End	
  of	
  Part	
  3	
  of	
  3	
  

More Related Content

What's hot

What's hot (20)

Netflix Global Applications - NoSQL Search Roadshow
Netflix Global Applications - NoSQL Search RoadshowNetflix Global Applications - NoSQL Search Roadshow
Netflix Global Applications - NoSQL Search Roadshow
 
Netflix in the Cloud
Netflix in the CloudNetflix in the Cloud
Netflix in the Cloud
 
Dystopia as a Service
Dystopia as a ServiceDystopia as a Service
Dystopia as a Service
 
Netflix in the Cloud at SV Forum
Netflix in the Cloud at SV ForumNetflix in the Cloud at SV Forum
Netflix in the Cloud at SV Forum
 
Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3) Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3)
 
Netflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconNetflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at Gluecon
 
Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
Gluecon 2013 - NetflixOSS Cloud Native Tutorial IntroductionGluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
 
NetflixOSS Meetup
NetflixOSS MeetupNetflixOSS Meetup
NetflixOSS Meetup
 
AmebaPico 裏側の技術やAWSの利用について
AmebaPico 裏側の技術やAWSの利用についてAmebaPico 裏側の技術やAWSの利用について
AmebaPico 裏側の技術やAWSの利用について
 
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
 
AWS for Start-ups - Architectural Best Practices & Automating Your Infrastruc...
AWS for Start-ups - Architectural Best Practices & Automating Your Infrastruc...AWS for Start-ups - Architectural Best Practices & Automating Your Infrastruc...
AWS for Start-ups - Architectural Best Practices & Automating Your Infrastruc...
 
Netflix Global Cloud Architecture
Netflix Global Cloud ArchitectureNetflix Global Cloud Architecture
Netflix Global Cloud Architecture
 
Gluecon keynote
Gluecon keynoteGluecon keynote
Gluecon keynote
 
Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...
Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...
Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...
 
Building Web Scale Applications with AWS
Building Web Scale Applications with AWSBuilding Web Scale Applications with AWS
Building Web Scale Applications with AWS
 
AWS Webcast - Design for Availability
AWS Webcast - Design for AvailabilityAWS Webcast - Design for Availability
AWS Webcast - Design for Availability
 
Global Netflix Platform
Global Netflix PlatformGlobal Netflix Platform
Global Netflix Platform
 
Building Cost-Aware Cloud Architectures - Jinesh Varia (AWS) and Adrian Cockc...
Building Cost-Aware Cloud Architectures - Jinesh Varia (AWS) and Adrian Cockc...Building Cost-Aware Cloud Architectures - Jinesh Varia (AWS) and Adrian Cockc...
Building Cost-Aware Cloud Architectures - Jinesh Varia (AWS) and Adrian Cockc...
 
AWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at NetflixAWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at Netflix
 
Netflix and Open Source
Netflix and Open SourceNetflix and Open Source
Netflix and Open Source
 

Viewers also liked

Best Practices for Architecting in the Cloud - Jeff Barr
Best Practices for Architecting in the Cloud - Jeff BarrBest Practices for Architecting in the Cloud - Jeff Barr
Best Practices for Architecting in the Cloud - Jeff Barr
Amazon Web Services
 

Viewers also liked (20)

Speeding Up Innovation
Speeding Up InnovationSpeeding Up Innovation
Speeding Up Innovation
 
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
 
Cloud Architecture: Patterns and Best Practices
Cloud Architecture: Patterns and Best PracticesCloud Architecture: Patterns and Best Practices
Cloud Architecture: Patterns and Best Practices
 
Cloud Architecture best practices
Cloud Architecture best practicesCloud Architecture best practices
Cloud Architecture best practices
 
Enterprise Cloud Architecture Best Practices
Enterprise Cloud Architecture Best PracticesEnterprise Cloud Architecture Best Practices
Enterprise Cloud Architecture Best Practices
 
(ARC401) Cloud First: New Architecture for New Infrastructure
(ARC401) Cloud First: New Architecture for New Infrastructure(ARC401) Cloud First: New Architecture for New Infrastructure
(ARC401) Cloud First: New Architecture for New Infrastructure
 
Perform a Cloud Readiness Assessment for Your Own Company
Perform a Cloud Readiness Assessment for Your Own CompanyPerform a Cloud Readiness Assessment for Your Own Company
Perform a Cloud Readiness Assessment for Your Own Company
 
A cloud readiness assessment framework
A cloud readiness assessment frameworkA cloud readiness assessment framework
A cloud readiness assessment framework
 
Best Practices for Architecting in the Cloud - Jeff Barr
Best Practices for Architecting in the Cloud - Jeff BarrBest Practices for Architecting in the Cloud - Jeff Barr
Best Practices for Architecting in the Cloud - Jeff Barr
 
Microsoft's New Platform
Microsoft's New PlatformMicrosoft's New Platform
Microsoft's New Platform
 
Nuts and bolts of running a popular site in the aws cloud
Nuts and bolts of running a popular site in the aws cloudNuts and bolts of running a popular site in the aws cloud
Nuts and bolts of running a popular site in the aws cloud
 
Millicomputing Usenix 2008
Millicomputing Usenix 2008Millicomputing Usenix 2008
Millicomputing Usenix 2008
 
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
 
Cloud Architecture
Cloud ArchitectureCloud Architecture
Cloud Architecture
 
Openstack Silicon Valley - Vendor Lock In
Openstack Silicon Valley - Vendor Lock InOpenstack Silicon Valley - Vendor Lock In
Openstack Silicon Valley - Vendor Lock In
 
When Developers Operate and Operators Develop
When Developers Operate and Operators DevelopWhen Developers Operate and Operators Develop
When Developers Operate and Operators Develop
 
Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013
 
Java Microservices with Netflix OSS & Spring
Java Microservices with Netflix OSS & Spring Java Microservices with Netflix OSS & Spring
Java Microservices with Netflix OSS & Spring
 
Cloud Trends Nov2015 Structure
Cloud Trends Nov2015 StructureCloud Trends Nov2015 Structure
Cloud Trends Nov2015 Structure
 

Similar to Cloud Architecture Tutorial - Running in the Cloud (3of3)

AWS Cloud Kata | Manila - Getting to Scale on AWS
AWS Cloud Kata | Manila - Getting to Scale on AWSAWS Cloud Kata | Manila - Getting to Scale on AWS
AWS Cloud Kata | Manila - Getting to Scale on AWS
Amazon Web Services
 

Similar to Cloud Architecture Tutorial - Running in the Cloud (3of3) (20)

Create cloud service on AWS
Create cloud service on AWSCreate cloud service on AWS
Create cloud service on AWS
 
Scaling the Platform for Your Startup - Startup Talks June 2015
Scaling the Platform for Your Startup - Startup Talks June 2015Scaling the Platform for Your Startup - Startup Talks June 2015
Scaling the Platform for Your Startup - Startup Talks June 2015
 
Svc 202-netflix-open-source
Svc 202-netflix-open-sourceSvc 202-netflix-open-source
Svc 202-netflix-open-source
 
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...
 
Scaling the Platform for Your Startup
Scaling the Platform for Your StartupScaling the Platform for Your Startup
Scaling the Platform for Your Startup
 
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...
 
Scaling the Platform for Your Startup
Scaling the Platform for Your StartupScaling the Platform for Your Startup
Scaling the Platform for Your Startup
 
(CMP404) Cloud Rendering at Walt Disney Animation Studios
(CMP404) Cloud Rendering at Walt Disney Animation Studios(CMP404) Cloud Rendering at Walt Disney Animation Studios
(CMP404) Cloud Rendering at Walt Disney Animation Studios
 
Scaling on AWS for the First 10 Million Users
Scaling on AWS for the First 10 Million Users Scaling on AWS for the First 10 Million Users
Scaling on AWS for the First 10 Million Users
 
AWS Cloud Kata | Manila - Getting to Scale on AWS
AWS Cloud Kata | Manila - Getting to Scale on AWSAWS Cloud Kata | Manila - Getting to Scale on AWS
AWS Cloud Kata | Manila - Getting to Scale on AWS
 
MariaDB on Docker
MariaDB on DockerMariaDB on Docker
MariaDB on Docker
 
Netflix presents at MassTLC Cloud Summit 2013
Netflix presents at MassTLC Cloud Summit 2013Netflix presents at MassTLC Cloud Summit 2013
Netflix presents at MassTLC Cloud Summit 2013
 
Web Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud PlatformWeb Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud Platform
 
Migrating Your Databases to AWS Deep Dive on Amazon RDS and AWS
Migrating Your Databases to AWS Deep Dive on Amazon RDS and AWSMigrating Your Databases to AWS Deep Dive on Amazon RDS and AWS
Migrating Your Databases to AWS Deep Dive on Amazon RDS and AWS
 
AWS re:Invent 2013 Recap
AWS re:Invent 2013 RecapAWS re:Invent 2013 Recap
AWS re:Invent 2013 Recap
 
AWS Summit London 2014 | Scaling on AWS for the First 10 Million Users (200)
AWS Summit London 2014 | Scaling on AWS for the First 10 Million Users (200)AWS Summit London 2014 | Scaling on AWS for the First 10 Million Users (200)
AWS Summit London 2014 | Scaling on AWS for the First 10 Million Users (200)
 
Aws webcast - Scaling on AWS 13 08-20
Aws webcast - Scaling on AWS 13 08-20Aws webcast - Scaling on AWS 13 08-20
Aws webcast - Scaling on AWS 13 08-20
 
AWS Summit Auckland 2014 | Scaling on AWS for the First 10 Million Users
 AWS Summit Auckland 2014 | Scaling on AWS for the First 10 Million Users AWS Summit Auckland 2014 | Scaling on AWS for the First 10 Million Users
AWS Summit Auckland 2014 | Scaling on AWS for the First 10 Million Users
 
Scaling Up to Your First 10 Million Users
Scaling Up to Your First 10 Million UsersScaling Up to Your First 10 Million Users
Scaling Up to Your First 10 Million Users
 
Building a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for AnalystsBuilding a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for Analysts
 

More from Adrian Cockcroft

More from Adrian Cockcroft (7)

Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)
Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)
Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)
 
Architectures for High Availability - QConSF
Architectures for High Availability - QConSFArchitectures for High Availability - QConSF
Architectures for High Availability - QConSF
 
Cassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSCassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWS
 
Migrating Netflix from Datacenter Oracle to Global Cassandra
Migrating Netflix from Datacenter Oracle to Global CassandraMigrating Netflix from Datacenter Oracle to Global Cassandra
Migrating Netflix from Datacenter Oracle to Global Cassandra
 
Migrating to Public Cloud
Migrating to Public CloudMigrating to Public Cloud
Migrating to Public Cloud
 
Cmg06 utilization is useless
Cmg06 utilization is uselessCmg06 utilization is useless
Cmg06 utilization is useless
 
NoSQL for Netflix
NoSQL for NetflixNoSQL for Netflix
NoSQL for Netflix
 

Recently uploaded

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 

Recently uploaded (20)

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
 

Cloud Architecture Tutorial - Running in the Cloud (3of3)

  • 1. Cloud  Architecture  Tutorial   Running  in  the  Cloud   Qcon  London  March  5th,  2012   Adrian  Cockcro6   @adrianco  #ne:lixcloud   h>p://www.linkedin.com/in/adriancockcro6   Part  3  of  3  
  • 2. Running  in  the  Cloud   Bring-­‐up  Strategy  for  Developers  and  TesRng   Capacity  Planning  and  Workloads   Running  Cassandra   Monitoring  and  Scalability   Availability  and  Resilience   OrganizaRonal  Structure        
  • 3. Cloud  Bring-­‐Up  Strategy   Simplest  and  Soonest  
  • 4. Shadow  Traffic  RedirecRon   •  Early  a>empt  to  send  traffic  to  cloud   –  Real  traffic  stream  to  validate  cloud  back  end   –  Uncovered  lots  of  process  and  tools  issues   –  Uncovered  Service  latency  issues   •  TV  Device  calls  Datacenter  API  and  Cloud  API   –  Returns  Genre/movie  list  for  a  customer   –  Asynchronously  duplicate  request  to  cloud   –  Start  with  send-­‐and-­‐forget  mode,  ignore  response  
  • 5. Shadow  Redirect  Instances   Modified   Datacenter   Datacenter   Service   Instances   Modified  Cloud   Cloud  Service   One  request  per   Instances   visit   Data  Sources   queueservice   videometadata  
  • 6. Video  Metadata  Server   •  VMS  instance  isolates  new  pla:orm  from  old  codebase   –  Isolate/unblock  cloud  team  from  metadata  team  schedule   –  Datacenter  code  supports  obsolete  movie  object   –  VMS  ESL  is  designed  to  support  new  video  facet  object   •  VMS  subsets  and  pre-­‐processes  the  metadata   –  Only  load  data  used  by  cloud  services   –  Fast  bulk  loads  for  VMS  clients  speed  startup  Rmes   –  Explore  next  generaRon  metadata  cache  architecture     Pa$ern  –  Add  services  to  isolate  old  and  new  code  base  
  • 7. First  Web  Pages  in  the  Cloud  
  • 8. First  Page   •  First  full  page  –  Basic  Genre   –  Simplest  page,  no  sub-­‐genres,  minimal  personalizaRon   –  Lots  of  investment  in  new  Struts  based  page  design   –  Uses  idenRty  cookie  to  lookup  in  member  info  svc   •  New  “merchweb”  front  end  instance   –  movies.ne:lix.com  points  to  merchweb  instance   •  Uncovered  lots  of  latency  issues   –  Used  memcached  to  hide  S3  and  SimpleDB  latency   –  Improved  from  slower  to  faster  than  Datacenter  
  • 9. Genre  Page  Cloud  Instances   Front  End   merchweb   mulRple  requests   Middle  Tier   genre    memcached   per  visit   Data  Sources   queueservice   rentalhistory   videometadata  
  • 10. Controlled  Cloud  TransiRon   •  WWW  calling  code  chooses  who  goes  to  cloud   –  Filter  out  corner  cases,  send  percentage  of  users   –  The  URL  that  customers  see  is   h>p://movies.ne:lix.com/WiContentPage?csid=1   –  If  problem,  redirect  to  old  Datacenter  page   h>p://www.ne:lix.com/WiContentPage?csid=1   •  Play  Bu>on  and  Star  RaRng  AcRon  redirect   –  Point  URLs  for  acRons  that  create/modify  data   back  to  datacenter  to  start  with  
  • 11. Cloud  Development  and  TesRng  Issues  
  • 12. Boot  Camp   •  One  day  “Ne:lix  Cloud  Training”  class   –  Has  been  run  6  Rmes  for  20-­‐45  people  each  Rme   •  Half  day  of  presentaRons   •  Half  day  hands-­‐on   –  Create  your  own  hello  world  app   –  Launch  in  AWS  test  account   –  Login  to  your  cloud  instances   –  Find  monitoring  data  on  your  cloud  instances   –  Connect  to  Cassandra  and  read/write  data  
  • 13. Very  First  Boot  Camp   •  Pathfinder  Bootstrap  Mission   –  Room  full  of  engineers  sharing  the  pain  for  1-­‐2  days   –  Built  a  very  rough  prototype  working  web  site   •  Get  everyone  hands-­‐on  with  a  new  code  base   –  Debug  lots  of  tooling  and  conceptual  issues  very  fast   –  Used  SimpleDB  to  create  mock  data  sources   •  Cloud  Specific  Key  Setup   –  Needed  to  integrate  with  AWS  security  model   –  New  concepts  for  datacenter  developers  
  • 14. Developer  Instances  Collision   Sam  and  Rex  both  want  to  deploy  web  front  end  for   development   Sam   Rex   web  in   test   account  
  • 15. Per-­‐Service  Namespace  Stack  RouRng   Developers  choose  what  to  share   Sam   Rex   Mike   web-­‐sam   web-­‐rex   web-­‐dev   backend-­‐dev   backend-­‐dev   backend-­‐mike  
  • 16. Developer  Namespace  Stacks   •  Developer  specific  service  instances   –  Configured  via  Java  properRes  at  runRme   –  RouRng  implemented  by  REST  client  library   •  Server  ConfiguraRon   –  Configure  discovery  service  version  string   –  Registers  as  <appname>-­‐<namespace>   •  Client  ConfiguraRon   –  Route  traffic  on  per-­‐service  basis  including   namespace  
  • 17. Capacity  Planning  Metrics  and   Methods  
  • 18. What  is  Capacity  Planning   •  We  care  about   –  CPU,  Memory,  Network  and  Disk  resource  uRlizaRon   –  ApplicaRon  response  Rmes  and  throughput   •  We  need  to  know   –  how  much  of  each  resource  we  are  using  now,  and  will  use  in   the  future   –  how  much  headroom  we  have  to  handle  higher  loads   •  We  want  to  understand   –  how  headroom  varies   –  how  it  relates  to  applicaRon  response  Rmes  and  throughput  
  • 19. Capacity  Planning  Norms   •  Capacity  is  expensive   •  Capacity  takes  Rme  to  buy  and  provision   •  Capacity  only  increases,  can’t  be  shrunk  easily   •  Capacity  comes  in  big  chunks,  paid  up  front   •  Planning  errors  can  cause  big  problems   •  Systems  are  clearly  defined  assets   •  Systems  can  be  instrumented  in  detail   •  Depreciate  assets  over  3  years  
  • 20. Capacity  Planning  in  Clouds   (a  few  things  have  changed…)   •  Capacity  is  expensive   •  Capacity  takes  Rme  to  buy  and  provision   •  Capacity  only  increases,  can’t  be  shrunk  easily   •  Capacity  comes  in  big  chunks,  paid  up  front   •  Planning  errors  can  cause  big  problems   •  Systems  are  clearly  defined  assets   •  Systems  can  be  instrumented  in  detail   •  Depreciate  assets  over  3  years  (reservaRons!)  
  • 21. Capacity  is  expensive   h>p://aws.amazon.com/s3/  &  h>p://aws.amazon.com/ec2/   •  Storage  (Amazon  S3)     –  $0.125  per  GB  –  first  50  TB  /  month  of  storage  used   –  $0.055  per  GB  –  storage  used  /  month  over  5  PB   •  Data  Transfer  (Amazon  S3)     –  $0.000  per  GB  –  all  data  transfer  in  is  free,  first  GB  out  is  free   –  $0.120  per  GB  –  first  10  TB  /  month  data  transfer  out   –  $0.050  per  GB  –  data  transfer  out  /  month  over  350  TB   •  Requests  (Amazon  S3  Storage  access  is  via  h>p)   –  $0.01  per  1,000  PUT,  COPY,  POST,  or  LIST  requests   –  $0.01  per  10,000  GET  and  all  other  requests   –  $0  per  DELETE   •  CPU  (Amazon  EC2)   –  Small  (Default)  $0.085/hour,  Extra  Large  $0.68/hour,  Four  XL  $2.00/hour   –  Small  (Default)  $0.08/hour,  Extra  Large  $0.64/hour,  Four  XL  $1.80/hour   •  Network  (Amazon  EC2)   –  Inbound/Outbound  around  $0.10  per  GB  
  • 22. Capacity  comes  in  big  chunks,  paid  up  front   •  Capacity  takes  Rme  to  buy  and  provision   –  No  minimum  price,  monthly  billing   –  “Amazon  EC2  enables  you  to  increase  or  decrease   capacity  within  minutes,  not  hours  or  days.  You  can   commission  one,  hundreds  or  even  thousands  of   server  instances  simultaneously”   •  Capacity  only  increases,  can’t  be  shrunk  easily   –  Pay  for  what  is  actually  used   •  Planning  errors  can  cause  big  problems   –  Size  only  for  what  you  need  now  
  • 23. Systems  are  clearly  defined  assets   •  You  are  running  in  a  “stateless”  mulR-­‐ tenanted  virtual  image  that  can  die  or  be   taken  away  and  replaced  at  any  Rme   •  You  don’t  know  exactly  where  it  is,  you  can   choose  to  locate  “US-­‐East”  or  “Europe”  etc.   •  You  can  specify  zones  that  will  not  share   components  to  avoid  common  mode  failures  
  • 24. Systems  can  be  instrumented  in  detail   •  Each  cloud  node  allocaRon  is  unique   –  So  elasRc  usage  pa>erns  keep  creaRng  new  nodes   –  “garbage  collect”  nodes  that  won’t  be  seen  again   –  Need  to  map  EIP  and  Cassandra  tokens  to  instances   •  Ne:lix  SoluRon  –  Entrypoints  Slots   –  Each  Autoscale  Group  has  a  size   –  Each  instance  is  given  a  slot  number  up  to  size   –  Replacements  pick  empty  slots  
  • 25. Depreciate  assets  over  3  years   (reservaRons!)   •  Reduced  costs  in  return  for  commitment   •  One  or  three  years,  upfront  payment   •  Payment  can  be  depreciated  as  capital  asset   •  Low,  medium  or  high  usage  reservaRons   –  Save  more  if  you  use  them  more   •  Spot  market  instances   –  Unused  reservaRons  sold  to  other  users  cheap   –  Will  be  yanked  at  any  Rme  if  needed  
  • 26. A  Discussion  of  Workloads  and   How  They  Behave  
  • 27. Workload  CharacterisRcs   •  A  quick  tour  through  a  taxonomy  of   workload  types   •  Start  with  the  easy  ones  and  work  up   •  Why  personalized  workloads  are  different   and  hard   •  Some  examples  and  coping  strategies   3/12/12   Slide  176  
  • 28. Simple  Random  Arrivals   •  Random  arrival  of  transacRons  with  fixed  mean   service  Rme   –  Li>le’s  Law:  QueueLength  =  Throughput  *  Response   –  URlizaRon  Law:  URlizaRon  =  Throughput  *  ServiceTime   •  Complex  models  are  o6en  reduced  to  this  model   –  By  averaging  over  longer  Rme  periods  since  the  formulas   only  work  if  you  have  stable  averages   –  By  wishful  thinking  (i.e.  how  to  fool  yourself)   3/12/12   Slide  177  
  • 29. Mixed  random  arrivals  of  transacRons   with  stable  mean  service  Rmes   •  Think  of  the  grocery  store  checkout  analogy   –  Trolleys  full  of  shopping  vs.  baskets  full  of  shopping   –  Baskets  are  quick  to  service,  but  get  stuck  behind  carts   –  RelaRve  mixture  of  transacRon  types  starts  to  ma>er   •  Many  transacRonal  systems  handle  a  mixture   –  Databases,  web  services   •  Consider  separaRng  fast  and  slow  transacRons   –  So  that  we  have  a  “10  items  or  less”  line  just  for  baskets   –  Separate  pools  of  servers  for  different  services   –  The  old  rule  -­‐  don’t  mix  OLTP  with  DSS  queries  in  databases   •  Performance  is  o6en  thread-­‐limited   –  Thread  limit  and  slow  transacRons  constrains  maximum  throughput   •  Model  mix  using  analyRcal  solvers  (e.g.  PDQ  perfdynamics.com)   3/12/12   Slide  178  
  • 30. Load  dependent  servers  –  varying   mean  service  Rmes   •  Mean  service  Rme  may  increase  at  high  throughput   –  Due  to  non-­‐scalable  algorithms,  lock  contenRon   –  System  runs  out  of  memory  and  starts  paging  or  frequent  GC   •  Mean  service  Rme  may  also  decrease  at  high  throughput   –  Elevator  seek  and  write  cancellaRon  opRmizaRons  in  storage   –  Load  shedding  and  simplified  fallback  modes   •  Systems  have  “Rpping  points”  if  the  service  Rme  increases   –  Hysteresis  means  they  don’t  come  back  when  load  drops   –  This  is  why  you  have  to  kill  catatonic  systems   –  Best  designs  shed  load  to  be  stable  at  the  limit  –  circuit  breaker  pa>ern   –  PracRcal  opRon  is  to  try  to  avoid  Rpping  points  by  reducing  variance     •  Model  using  discrete  event  simulaRon  tools   –  Behaviour  is  non-­‐linear  and  hard  to  model   3/12/12   Slide  179  
  • 31. Self-­‐similar  /  fractal  workloads   •  Bursty  rather  than  random  arrival  rates   •  Self-­‐similar   –  Looks  “random”  at  close  up,  stays  “random”  as  you  zoom  out   –  Work  arrives  in  bursts,  transacRons  aren’t  independent   –  Bursts  cluster  together  in  super-­‐bursts,  etc.   •  Network  packet  streams  tend  to  be  fractal   •  Common  in  pracRce,  too  hard  to  model   –  Probably  the  most  common  reason  why  your  model  is  wrong!   3/12/12   Slide  180  
  • 32. State  Dependent  Service  Workloads   •  Personalized  services  that  store  user  state/history   –  TransacRons  for  new  users  are  quick   –  TransacRons  for  users  with  lots  of  state/history  are  slower   –  As  user  base  builds  state  and  ages  you  get  into  trouble…   •  Social  Networks,  RecommendaRon  Services   –  Facebook,  Flickr,  Ne:lix,  Twi>er  etc.   •  “Abandon  hope  all  ye  who  enter  here”   –  Not  tractable  to  model,  repeatable  tests  are  tricky   –  Long  fat  tail  response  Rme  distribuRon  and  Rmeouts   •  Try  to  transform  workloads  to  more  tractable  forms   3/12/12   Slide  181  
  • 33. Example  -­‐  Twi>er  Workload   •  @adrianco  tweets  –  copy  to  3600  or  so  other  users   •  @zoecello  tweets  many  Rmes  a  day    –  to  over  1M  users   •  @barackobama  tweets  every  few  days  –  to  over  12M  users   •  It’s  the  same  transacRon,  but  the  service  Rme  varies  by  several   orders  of  magnitude   •  The  best  (most  acRve  and  connected  =  most  valuable)  users   trigger  a  “denial  of  service  a>ack”  on  the  systems  when  they   tweet   •  Cascading  effect  as  many  others  re-­‐tweet   3/12/12   Slide  182  
  • 34. Example  -­‐  Ne:lix  Movie  Choosing   •  “Pick  24  genres/subgenres  etc.  of  75  movies  each  for  me”   –  used  by  TV  based  devices  like  Xbox360,  PS/3,  iPhone  app   •  New  user   –  No  history  of  what  they  have  rented  (DVD)  or  streamed   –  No  star  raRngs  for  movies,  possibly  some  genre  raRngs   –  Basic  demographic  info   –  Fast  to  calculate,  easy  to  find  many  good  choices  to  return   •  User  with  several  years  tenure   –  Thousands  of  movies  rented  or  streamed,  “seen  it  already”   –  Hundreds  to  thousands  of  star  raRngs,  lots  of  genre  raRngs   –  Requests  may  Rme  out  and  return  fewer  or  worse  choices   3/12/12   Slide  183  
  • 35. Workload  Modelling  Survival   Methods   •  Simplify  the  workload  algorithms   –  move  from  hard  or  impossible  to  simpler  models   –  decouple,  cache  and  pre-­‐compute  to  get  constant  service  Rmes   •  Stand  further  away   –  averaging  is  your  friend  –  gets  rid  of  complex  fluctuaRons   •  Minimalist  Models   –  most  models  are  far  too  complex  –  the  classic  beginners  error…   –  the  art  of  modelling  is  to  only  model  what  really  ma>ers   •  Don’t  model  details  you  don’t  use   –  model  peak  hour  of  the  week,  not  day  to  day  fluctuaRons   –  e.g.  “Will  the  web  site  survive  next  Sunday  night?”   3/12/12   Slide  184  
  • 37. Cassandra  Use  Cases   •  Key  by  Customer  –  Cross-­‐region  clusters   –  Many  app  specific  Cassandra  clusters,  read-­‐intensive   –  Keys+Rows  in  memory  using  m2.4xl  Instances   •  Key  by  Customer:Movie  –  e.g.  Viewing  History   –  Growing  fast,  write  intensive  –  m1.xl  instances   –  Keys  cached  in  memory,  one  cluster  per  region   •  Large  scale  data  logging  –  lots  of  writes   –  Column  data  expires  a6er  Rme  period   –  Distributed  counters,  one  cluster  per  region  
  • 38. Ne:lix  Pla:orm  Cassandra  AMI   •  Tomcat  server  with  Priam   –  Always  running,  registers  with  pla:orm   –  Manages  Cassandra  state,  tokens,  backups   •  Removed  Root  Disk  Dependency  on  EBS   –  Use  S3  backed  AMI  for  stateful  services   –  Normally  use  EBS  backed  AMI  for  fast  provisioning  
  • 39. Ne:lix  ContribuRons  to  Cassandra   •  Cassandra  as  a  mutable  toolkit   –  Cassandra  is  in  Java,  pluggable,  well  structured   –  Ne:lix  has  a  building  full  of  Java  engineers….   –  We  changed  Cassandra  to  make  it  run  much  be>er  on  AWS   •  ContribuRons  delivered  to  Cassandra   –  0.8  Prototype  off-­‐heap  row  cache,  SSTable  write  callback   –  1.x  OpRmizaRons  reduced  impact  of  repair  &  compacRon   –  January  2012  –  Ne:lix  engineer  becomes  core  commi>er   •  Cassandra  Based  Projects  on  github.com/Ne:lix   –  Priam  AWS  integraRon  and  backup  using  Tomcat  helper   –  Astyanax    Java  client  library   –  CassJMeter  for  performance  and  regression  tesRng  
  • 41. Monitoring  Vision   •  Problem   –  Too  many  tools,  each  with  a  good  reason  to  exist   –  Hard  to  get  an  integrated  view  of  a  problem   –  Too  much  manual  work  building  dashboards   –  Tools  are  not  discoverable,  views  are  not  filtered   •  SoluRon   –  Get  vendors  to  add  deep  linking  and  embedding   –  IntegraRon  “portal”  Res  everything  together   –  Dynamic  portal  generaRon,  relevant  data,  all  tools  
  • 42. Cloud  Monitoring  Mechanisms   •  Keynote  or  Gomez  etc.   –  External  URL  monitoring   •  Amazon  CloudWatch   –  Metrics  for  ELB  and  Instances   •  AppDynamics   –  End  to  end  transacRon  view  showing  resources  used   –  Powerful  real  Rme  debug  tools  for  latency,  CPU  and  Memory   •  Epic  (Ne:lix  in-­‐house  project)   –  Flexible  and  easy  to  use  to  extend  and  embed  plots   •  Logs   –  High  capacity  logging  and  analysis  framework   –  Hadoop  (log4j  -­‐>  Honu  -­‐>  EMR)  
  • 43. Using  AppDynamics   (simple  example  from  early  2010)  
  • 44. AppDynamics  Monitoring  of  Cassandra  –  AutomaRc  Discovery  
  • 45. Scalability  TesRng   •  Cloud  Based  TesRng  –  fricRonless,  elasRc   –  Create/destroy  any  sized  cluster  in  minutes   –  Many  test  scenarios  run  in  parallel   •  Test  Scenarios   –  Internal  app  specific  tests   –  Simple  “stress”  tool  provided  with  Cassandra   •  Scale  test,  keep  making  the  cluster  bigger   –  Check  that  tooling  and  automaRon  works…   –  How  many  ten  column  row  writes/sec  can  we  do?  
  • 47. Scale-­‐Up  Linearity   h>p://techblog.ne:lix.com/2011/11/benchmarking-­‐cassandra-­‐scalability-­‐on.html   Client  Writes/s  by  node  count  –  Replica:on  Factor  =  3   1200000   1099837   1000000   800000   600000   537172   400000   366828   200000   174373   0   0   50   100   150   200   250   300   350  
  • 48. Stress  Client  Latency   Includes  ~10ms  Scheduling  Overhead  –  for  be>er  latency  data  see    h>p://techblog.ne:lix.com/2012/03/jmeter-­‐plugin-­‐for-­‐cassandra.html  
  • 49. Measured  at  the  Cassandra  Server   3.3  Million  writes/sec  at  0.014ms  –  14  microseconds  
  • 50. Per  Node  AcRvity   Per  Node   48  Nodes   96  Nodes   144  Nodes   288  Nodes   Per  Server  Writes/s   10,900  w/s   11,460  w/s   11,900  w/s   11,456  w/s   Mean  Server  Latency   0.0117  ms   0.0134  ms   0.0148  ms   0.0139  ms   Mean  CPU  %Busy   74.4  %   75.4  %   72.5  %   81.5  %   Disk  Read   5,600  KB/s   4,590  KB/s   4,060  KB/s   4,280  KB/s   Disk  Write   12,800  KB/s   11,590  KB/s   10,380  KB/s   10,080  KB/s   Network  Read   22,460  KB/s   23,610  KB/s   21,390  KB/s   23,640  KB/s   Network  Write   18,600  KB/s   19,600  KB/s   17,810  KB/s   19,770  KB/s   Node  specificaRon  –  Xen  Virtual  Images,  AWS  US  East,  three  zones   •  Cassandra  0.8.6,  CentOS,  SunJDK6   •  AWS  EC2  m1  Extra  Large  –  Standard  price  $  0.68/Hour   •  15  GB  RAM,  4  Cores,  1Gbit  network   •  4  internal  disks  (total  1.6TB,  striped  together,  md,  XFS)  
  • 51. Time  is  Money   48  nodes   96  nodes   144  nodes   288  nodes   Writes  Capacity   174373  w/s   366828  w/s   537172  w/s   1,099,837  w/s   Storage  Capacity   12.8  TB   25.6  TB   38.4  TB   76.8  TB   Nodes  Cost/hr   $32.64   $65.28   $97.92   $195.84   Test  Driver  Instances   10   20   30   60   Test  Driver  Cost/hr   $20.00   $40.00   $60.00   $120.00   Cross  AZ  Traffic   5  TB/hr   10  TB/hr   15  TB/hr   301  TB/hr   Traffic  Cost/10min   $8.33   $16.66   $25.00   $50.00   Setup  DuraRon   15  minutes   22  minutes   31  minutes   662  minutes   AWS  Billed  DuraRon   1hr   1hr   1  hr   2  hr   Total  Test  Cost   $60.97   $121.94   $182.92   $561.68   1  EsRmate  two  thirds  of  total  network  traffic     2  Workaround  for  a  tooling  bug  slowed  setup  
  • 53. Chaos  Monkey   •  Computers  (Datacenter  or  AWS)  randomly  die   –  Fact  of  life,  but  too  infrequent  to  test  resiliency   •  Test  to  make  sure  systems  are  resilient   –  Allow  any  instance  to  fail  without  customer  impact   •  Chaos  Monkey  hours   –  Monday-­‐Thursday  9am-­‐3pm  random  instance  kill   •  ApplicaRon  configuraRon  opRon   –  Apps  now  have  to  opt-­‐out  from  Chaos  Monkey  
  • 54. Responsibility  and  Experience   •  Make  developers  responsible  for  failures   –  Then  they  learn  and  write  code  that  doesn’t  fail   •  Use  Incident  Reviews  to  find  gaps  to  fix   –  Make  sure  its  not  about  finding  “who  to  blame”   •  Keep  Rmeouts  short,  fail  fast   –  Don’t  let  cascading  Rmeouts  stack  up   •  Make  configuraRon  opRons  dynamic   –  You  don’t  want  to  push  code  to  tweak  an  opRon  
  • 55. Resilient  Design  –  Circuit  Breakers   h>p://techblog.ne:lix.com/2012/02/fault-­‐tolerance-­‐in-­‐high-­‐volume.html  
  • 56. PaaS  OperaRonal  Model  -­‐  NoOps   •  Developers   –  Provision  and  run  their  own  code  in  producRon   –  Take  turns  to  be  on  call  if  it  breaks  (pagerduty)   –  Configure  autoscalers  to  handle  capacity  needs   •  Difference  between  DevOps  and  NoOps   –  DevOps  is  about  Dev  and  Ops  working  together   –  NoOps  constrains  Dev  to  use  automaRon  instead   –  NoOps  puts  more  responsibility  on  Dev,  with  tools  
  • 57. ImplicaRons  for  IT  OperaRons   •  Cloud  is  run  by  developer  organizaRon   –  Our  IT  department  is  the  AWS  API   –  We  have  no  IT  staff  working  on  cloud  (they  do  corp  IT)   •  Cloud  capacity  is  10x  bigger  than  Datacenter   –  Datacenter  oriented  IT  staffing  is  flat   –  We  have  moved  a  few  people  out  of  IT  to  write  code   •  TradiRonal  IT  Roles  are  going  away   –  Don’t  need  SA,  DBA,  Storage,  Network  admins   –  Developers  deploy  and  run  what  they  wrote  in  producRon  
  • 58. Ne:lix  “NoOps”  OrganizaRon   Developer  Org  ReporRng  into  Product  Development,  not  ITops   Ne:lix  Cloud  Pla:orm  Team   Cloud  Ops   Build  Tools   Database   Pla:orm   Cloud   Cloud   Reliability   and   Engineering   Development   Performance   SoluRons   Engineering   AutomaRon   Perforce  Jenkins   Pla:orm  jars   Cassandra   ArRfactory  JIRA   Benchmarking   Monitoring   Alert  RouRng   Key  store   Cassandra   Monkeys   Incident  Lifecycle   Base  AMI,  Bakery   Zookeeper   JVM  GC  Tuning   Ne:lix  App  Console   Wiresharking   Entrypoints   Astyanix   PagerDuty   AWS  Instances   AWS  API   AWS  Instances   AWS  Instances   AWS  Instances  
  • 59. Wrap  Up     Answer  your  remaining  quesRons…     What  was  missing  that  you  wanted  to  cover?  
  • 60. Takeaway     Ne5lix  has  built  and  deployed  a  scalable  global  Pla5orm  as  a  Service.     Key  components  of  the  Ne5lix  PaaS  are  being  released  as  Open  Source   projects  so  you  can  build  your  own  custom  PaaS.     h>p://github.com/Ne:lix   h>p://techblog.ne:lix.com   h>p://slideshare.net/Ne:lix     h>p://www.linkedin.com/in/adriancockcro6   @adrianco  #ne:lixcloud     End  of  Part  3  of  3