• Like
[B3]couchbase server for speed and scale with interactive applications   sdec deview 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

[B3]couchbase server for speed and scale with interactive applications sdec deview 2012

  • 1,088 views
Published

 

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,088
On SlideShare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
44
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Couchbase  Server  for  Speed  and  Scale  with  Interac8ve  Applica8ons   Perry  Krug   Presenter  Title   1  
  • 2. Agenda  (just  an  outline)   •  Needs  for  Interac8ve  Apps   –  Speed   •  Speed  of  access   •  Speed  of  app  development   –  Scale   •  Built  to  scale   –  Unique  architectural  principals   •  Clustered  system   –  Simple  for  applica8ons   –  Automa8c  sharding  of  data  across  cluster   •  Single  index  view   –  No  need  to  define  sharding  keys   –  Automa8c,  online  transfer  of  data  across  cluster   2  
  • 3. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again. EXAMPLE  OF  AN   INTERACTIVE  APP   3  
  • 4. Instagrowth:  Android  Launch  E xample •  Instagram  gained  nearly  1  million  users  overnight   when  they  expanded  to  Android   4  
  • 5. xampleE 1 Instagram = 7.5M MAU* 5  
  • 6. Draw  Something  -­‐  Social  Game   •  50  million  downloads  in  50  days   •  That’s  almost  5  Instagrams.  In  2  months.   6  
  • 7. Goes  Viral  3  Weeks  AZer  Launch   Draw  Something  by  OMGPOP   Daily  Ac8ve  Users  (millions)   16   14   12   10   8   6   4   2   2/6   8   10   12   14   16   18   20   22   24   26   28   3/1   3   5   7   9   11   13   15   17   19   21   7  
  • 8.  Database  Opera8ons   Draw  Something  by  OMGPOP   Daily  Ac8ve  Users  (millions)   16   14   12   10   8   By  March  19,  there  were     over  30,000,000  downloads  of  the  app,   6   over  5,000  drawings  being  stored  per  second,   over  2,200,000,000  drawings  stored,   over  105,000  database  transac8ons  per  second,   4   and  over  3.3  terabytes  of  data  stored.   2   2/6   8   10   12   14   16   18   20   22   24   26   28   3/1   3   5   7   9   11   13   15   17   19   21   8  
  • 9. Social  Gaming  on  Couchbase   iki   9  
  • 10. Speed   •  Modern,  interac8ve  applica8ons  require  speed   –  Web  pages  are  dynamic,  interac8ve  and  may  incrementally   fetch  content  for  display   –  Mobile  applica8ons  cannot  carry  all  of  the  content,  so  they   need  to  be  able  to  fetch  data  on  demand   •  Couchbase  Speed  via…   –  Managed,  built  in  cache   •  memcached  compa8ble,  built  on  memcached   –  Protocol  design   •  Client  can  directly  access  data  objects  within  the  cluster   •  No  quorum  reads  needed  to  determine  version,  always  consistent   10  
  • 11. Scale   •  Modern  app  deployments  are  always  on,  available   24x7x365   –  Some  applica8ons  may  have  a  lifecycle  which  requires   scaling  down,  in  addi8onal  to  scaling  up   •  Couchbase  scales  via…   –  Automa8cally  sharding  data  among  the  nodes  of  the  cluster   and  atomically  transferring  vbuckets   –  Coordina8ng  these  changes  with  client  libraries  accessing   the  cluster   11  
  • 12. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again. CLUSTERING  ARCHITECTURE   12  
  • 13. Basic  Opera8on   APP  SERVER  1   APP  SERVER  2        Docs  distributed  evenly  across       COUCHBASE  CLIENT  LIBRARY   servers  in  the  cluster   COUCHBASE  CLIENT  LIBRARY                Each  server  stores  both  ac/ve   CLUSTER  MAP     CLUSTER  MAP             &  replica  docs         Only  one  server  ac8ve  at  a  8me    Client  library  provides  app  with   Read/Write/Update   Read/Write/Update   simple  interface  to  database    Cluster  map  provides  map  to   which  server  doc  is  on     App  never  needs  to  know   SERVER  1   SERVER  2   SERVER  3     App  reads,  writes,  updates   Ac8ve  Docs   Ac8ve  Docs   Ac8ve  Docs         docs     Doc  5   DOC     Doc  4   DOC     Doc  1   DOC           Mul8ple  App  Servers  can     Doc  2   DOC     Doc  7   DOC     Doc  3   DOC   access  same  document  at           Doc  9   DOC     Doc  8   DOC     Doc  6   DOC   same  8me             Replica  Docs     Replica  Docs     Replica  Docs           Doc  4   DOC     Doc  6   DOC     Doc  7   DOC           Doc  1   DOC     Doc  3   DOC     Doc  9   DOC           Doc  8   DOC     Doc  2   DOC     Doc  5   DOC   COUCHBASE  SERVER    CLUSTER  User  Configured  Replica  Count  =  1   13  
  • 14. Add  Nodes   APP  SERVER  1   APP  SERVER  2             Two  servers  added  to   COUCHBASE  CLIENT  LIBRARY   COUCHBASE  CLIENT  LIBRARY   cluster             One-­‐click  opera8on     CLUSTER  MAP     CLUSTER  MAP         Docs  automa8cally               rebalanced  across   cluster     Even  distribu8on  of   docs   Read/Write/Update   Read/Write/Update     Minimum  doc   movement     Cluster  map  updated     App  database  calls  now   distributed  over  larger  #   SERVER  1   SERVER  2   SERVER  3   SERVER  4   SERVER  5   of  servers   Ac8ve  Docs       Ac8ve  Docs    Ac8ve  Docs  ocs   Ac8ve  Docs     Ac8ve  Docs     Ac8ve  D   Doc  5   DOC     Doc  4   DOC     Doc  1   DOC             Doc  3         Doc  2   DOC     Doc  7   DOC     Doc  3   DOC             Doc  6         Doc  9   DOC     Doc  8   DOC     Doc  6   DOC                   Replica  Docs     Replica  Docs    Replica  Docs     Replica  Docs     Replica  Docs         Replica  Docs         Doc  4   DOC     Doc  6   DOC     Doc  7   7   DOC   Doc                   Doc  1   DOC     Doc  3   DOC     Doc  9   9   DOC   Doc                   Doc  8   DOC     Doc  2   DOC     Doc  5   DOC       COUCHBASE  SERVER    CLUSTER  User  Configured  Replica  Count  =  1   14  
  • 15. Fail  Over  Node   APP  SERVER  1   APP  SERVER  2     App  servers  happily  accessing  docs       on  Server  3       COUCHBASE  CLIENT  LIBRARY     Server  fails   COUCHBASE  CLIENT  LIBRARY           App  server  requests  to  server  3  fail       CLUSTER  MAP     CLUSTER  MAP       Cluster  detects  server  has  failed               Promotes  replicas  of  docs  to  ac/ve         Updates  cluster  map     App  server  requests  for  docs  now   go  to  appropriate  server     Typically  rebalance    would  follow     SERVER  1   SERVER  2   SERVER  3   SERVER  4   SERVER  5   Ac8ve  Docs       Ac8ve  Docs    Ac8ve  Docs  ocs   Ac8ve  Docs     Ac8ve  Docs     Ac8ve  D   Doc  5   DOC     Doc  4   DOC     Doc  1   DOC     Doc  9   DOC     Doc  6   DOC         Doc  3         Doc  2   DOC     Doc  7   DOC     Doc  3     Doc  8     DOC         Doc  6           DOC                     Replica  Docs     Replica  Docs    Replica  Docs     Replica  Docs     Replica  Docs         Replica  Docs         Doc  4   DOC     Doc  6   DOC     Doc  7   7   DOC   Doc     Doc  5   DOC     Doc  8   DOC               Doc  1   DOC     Doc  3   DOC     Doc  9   9   DOC   Doc     Doc  2     DOC                       COUCHBASE  SERVER    CLUSTER  User  Configured  Replica  Count  =  1   15  
  • 16. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again. SINGLE  INDEX  VIEW   16  
  • 17. Couchbase  Server  Views   •  Couchbase  has  a  method   of  building  and   map:   function(doc) maintaining  views  over  the   { set  of  documents  within  a   emit(doc.name, [doc.city, doc.salary]); bucket   }   •  Can  be  used  for:   reduce:   –  Secondary  indexes   function (key, values, rereduce) { var sum = 0; –  Aggrega8ons  and  stats   for(i=0; i < values.length; i++) { –  Analyzing  datasets   sum = sum + values[i]; } •  Built  for  speed   } return(sum); –  Allow  relaxed  consistency   for  speed,  or  consistent  if   required     17  
  • 18. Views  are  B+Trees  Over  the  Data   Key: "f" Key: "f" Sum: 17 Sum: 17 bits 0 0 0 0 bits 0 0 0 0 Key: "f" Key: "f" Key: "f" Key: "f" Sum: 17 Sum: 17 Sum: 17 Sum: 17 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 Key: "f" Key: "f" Key: "f" Key: "f" Key: "f" Key: "f" Key: "f" Key: "f" Sum: 17 Sum: 17 Sum: 17 Sum: 17 Sum: 17 Sum: 17 Sum: 17 Sum: 17 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 Key: "a" Key: "b" Key: "c" Key: "d" Key: "e" Key: "f" Key: "j" Key: "k" Key: "l" Key: "m" Key: "r" Key: "s" Key: "t" Key: "u" Key: "v" Key: "w" Key: "x" Key: "y" Key: "z" Value: 4 Value: 2 Value: 0 Value: 6 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4vBucket: 1 vBucket: 2 vBucket: 3 vBucket: 4 vBucket: 5 vBucket: 6 vBucket: 10 vBucket: 11 vBucket: 12 vBucket: 13 vBucket: 18 vBucket: 19 vBucket: 20 vBucket: 21 vBucket: 22 vBucket: 23 vBucket: 24 vBucket: 25 vBucket: 26 Key: "n" Key: "o" Key: "p" Key: "q" Key: "g" Key: "h" Key: "i" Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 vBucket: 14 vBucket: 15 vBucket: 16 vBucket: 17 vBucket: 7 vBucket: 8 vBucket: 9 Views  are  Built  Atop  Incremental  Map  Reduce   •  Distributed  throughout  the  cluster   •  Query  API  abstracts  the  cluster  topology,  and  automa8cally  gathers  the  results   •  Rebuilds  subsets  of  the  B+Tree  based  on  updates   18  
  • 19. Key: "m" Key: "z" Sum: 92 Sum: 17 bits 0 0 0 0 bits 0 Key: "f" Key: "m" Sum: 21 Sum: 71 bits 0 0 0 0 bits 0 0 0 0 Key: "d" Key: "f" Key: "i" Key: "m" Sum: 10 Sum: 11 Sum: 24 Sum: 47 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 Key: "a" Key: "b" Key: "c" Key: "d" Key: "e" Key: "f" Key: "j" Key: "k" Key: "l" Ke Value: 4 Value: 2 Value: 0 Value: 6 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 VavBucket: 1 vBucket: 2 vBucket: 3 vBucket: 4 vBucket: 5 vBucket: 6 vBucket: 10 vBucket: 11 vBucket: 12 vBuc Key: "g" Key: "h" Key: "i" Value: 4 Value: 4 Value: 4 19   vBucket: 7 vBucket: 8 vBucket: 9
  • 20. Data  is  Distributed  Using  vbuckets,  but…   •  Couchbase  Server  has  1024  vbuckets  stored   throughout  the  cluster   20  
  • 21. B-­‐Superstar:  a  new  B+Tree  ready  for  vbuckets   Only  one  B+Tree  per   node  with  masking  of   ac8ve  vbucket  Data.   S8ll  allows  concurrent  Read  more  on  Damien’s  blog:   access  and  even  hmp://damienkatz.net/2012/05/stabilizing_couchbase_server_2.html   materializing  tree  during   rebalance   21  
  • 22. Thanks   Q&A   Perry  Krug   22