Social Gaming & Gambling Summit - London Chris Anderson
Upcoming SlideShare
Loading in...5
×
 

Social Gaming & Gambling Summit - London Chris Anderson

on

  • 400 views

#sgsconf

#sgsconf

Statistics

Views

Total Views
400
Views on SlideShare
400
Embed Views
0

Actions

Likes
2
Downloads
14
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Social Gaming & Gambling Summit - London Chris Anderson Social Gaming & Gambling Summit - London Chris Anderson Presentation Transcript

  • NoSQL  Landscape &  Grid  Compu7ng Chris  Anderson @jchris 1Saturday, October 6, 12
  • Growth  is  the  New  Reality • 2.2  Billion  internet  users • 50%  Americans  use   smartphones • Your  app  can  grow   overnight • Are  you  ready? 2Saturday, October 6, 12
  • Instagrowth:  Android  Launch E xa mple • Instagram  gained  nearly  1  million  users  overnight  when  they   expanded  to  Android 3Saturday, October 6, 12
  • Instagrowth:  Android  Launch E xa mple 1  Instagram = 7.5M  MAU* 4Saturday, October 6, 12
  • Draw  Something  -­‐  Social  Game 35 million monthly active users in 1 month about 5 Instagrams (Instagram today is waaaay more than 1 Instagram) 5Saturday, October 6, 12
  • Goes  Viral  3  Weeks  aOer  Launch Draw  Something  by  OMGPOP Daily  Ac)ve  Users  (millions) 16 14 35+M  MAU 12 at  peak 10 8 6 4 2 2/6 8 10 12 14 16 18 20 22 24 26 28 3/1 3 5 7 9 11 13 15 17 19 21 6Saturday, October 6, 12
  • By  Contrast,  at  1/2  an  Instagram The  Simpson’s:  Tapped  Out Daily  Ac)ve  Users  (millions) 7Saturday, October 6, 12
  • GETTING  IT  RIGHT 8Saturday, October 6, 12
  • Scalable  Data  Layer ●On-­‐demand  cluster  sizing ● Grow  or  shrink  with  workload ●Easy  node  provisioning ● All  nodes  are  the  same ●MulA-­‐master  Cross-­‐Datacenter  ReplicaAon ● For  a  fast  and  reliable  user  experience  worldwide ●EffecAve  Auto-­‐sharding ● Should  avoid  cluster  hot  spots 9Saturday, October 6, 12
  • Old  School  Hits  a  Scale  Wall Application Scales Out Just add more commodity web servers Database Scales Up Get a bigger, more complex server Expensive & disruptive sharding, doesn’t perform at web scale 10Saturday, October 6, 12
  • Tradi^onal  MySQL  +  Memcached  Architecture ● Run as many MySQL machines as you need www.example.com ● Data sharded evenly across the machines using client code App  Servers ● Memcached used to provide faster response time for users Memcached  Tier and reduce load on the database MySQL  Tier 11Saturday, October 6, 12
  • Limita^ons  of  MySQL  +  Memcached ● To scale you need to start using MySQL more simply ● Scale by hand ● Replication / Sharding is a black art ● Code overhead to manage keeping memcache and mysql in sync ● Lots of components to deploy Learn  From  Others  -­‐  This  Scenario  Costs  Time  and  Money.  Scaling  SQL  is   poten^ally  disastrous  when  going  Viral:  very  risky  ^me  for  major  code   changes  and  migra^ons...  you  have  no  Time  when  skyrocke^ng  up. 12Saturday, October 6, 12
  • NoSQL  Architectural  Promise • High Performance data access • Scale Up/Down Horizontally www.example.com • 24x7x365 Always-On Availability • Flexible Schema Document Model App  Servers Couchbase  Database  Servers 13Saturday, October 6, 12
  • NOSQL  TAXONOMY 14Saturday, October 6, 12
  • The Key-Value Store – the foundation of NoSQL Key 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 Opaque 101100101000100010011101 101100101000100010011101 Binary 101100101000100010011101 101100101000100010011101 Value 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 15Saturday, October 6, 12
  • Memcached – the NoSQL precursor Key 101100101000100010011101 memcached 101100101000100010011101 101100101000100010011101 101100101000100010011101 In-­‐memory  only 101100101000100010011101 Limited  set  of  opera^ons Opaque 101100101000100010011101 Blob  Storage:  Set,  Add,  Replace,  CAS 101100101000100010011101 Binary 101100101000100010011101 Retrieval:  Get 101100101000100010011101 Structured  Data:  Append,  Increment Value 101100101000100010011101 101100101000100010011101 “Simple  and  fast.” 101100101000100010011101 101100101000100010011101 Challenges:  cold  cache,  disrup^ve  elas^city 101100101000100010011101 101100101000100010011101 16Saturday, October 6, 12
  • Redis  –  More  “Structured  Data”  commands Key 101100101000100010011101 redis 101100101000100010011101 101100101000100010011101 101100101000100010011101 In-­‐memory  only “Data  Structures” 101100101000100010011101 Vast  set  of  opera^ons Blob 101100101000100010011101 Blob  Storage:  Set,  Add,  Replace,  CAS 101100101000100010011101 List 101100101000100010011101 Retrieval:  Get,  Pub-­‐Sub 101100101000100010011101 Structured  Data:  Strings,  Hashes,  Lists,  Sets, Set 101100101000100010011101 Sorted  lists Hash 101100101000100010011101 101100101000100010011101 … 101100101000100010011101 Example  opera7ons  for  a  Set 101100101000100010011101 Add,  count,  subtract  sets,  intersec^on,  is   101100101000100010011101 member?,  atomic  move  from  one  set  to  another 17Saturday, October 6, 12
  • NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis 18Saturday, October 6, 12
  • Membase  –  From  key-­‐value  cache  to  database Key 101100101000100010011101 membase 101100101000100010011101 101100101000100010011101 101100101000100010011101 Disk-­‐based  with  built-­‐in  memcached  cache 101100101000100010011101 Cache  refill  on  restart Opaque 101100101000100010011101 Memcached  compa^ble  (drop  in  replacement) 101100101000100010011101 Highly-­‐available  (data  replica^on) Binary 101100101000100010011101 Add  or  remove  capacity  to  live  cluster 101100101000100010011101 Value 101100101000100010011101 101100101000100010011101 “Simple,  fast,  elas^c.” 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 19Saturday, October 6, 12
  • NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase Database 20Saturday, October 6, 12
  • Couchbase  –  document-­‐oriented  database Key Couchbase {        “string”  :  “string”,        “string”  :  value, Auto-­‐sharding        “string”  :   Disk-­‐based  with  built-­‐in  memcached  cache                      {    “string”  :  “string”, JSON Cache  refill  on  restart                              “string”  :  value  },        “string”  :OBJECT Memcached  compa^ble  (drop  in  replace)  [  array  ] Highly-­‐available  (data  replica^on) } (“DOCUMENT”) Add  or  remove  capacity  to  live  cluster When  values  are  JSON  objects  (“documents”): Create  indices,  views  and  query  against  the  views 21Saturday, October 6, 12
  • NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase couchbase Database 22Saturday, October 6, 12
  • MongoDB  –  Document-­‐oriented  database Key MongoDB {        “string”  :  “string”,        “string”  :  value, Disk-­‐based  with  in-­‐memory  “caching”        “string”  :   BSON BSON  (“binary  JSON”)  format  and  wire  protocol                      {    “string”  :  “string”, Master-­‐slave  replica^on OBJECT                              “string”  :  value  }, Auto-­‐sharding        “string”  :  [  array  ] } (“DOCUMENT”) Values  are  BSON  objects Supports  ad  hoc  queries  –  best  when  indexed 23Saturday, October 6, 12
  • NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase couchbase Database mongoDB 24Saturday, October 6, 12
  • Cassandra  –  Column  overlays Cassandra Column  1 Disk-­‐based  system Clustered   Column  2 External  caching  required  for  low-­‐latency  reads “Columns”  are  overlaid  on  the  data Not  all  rows  must  have  all  columns Column  3   Supports  efficient  queries  on  columns (not  present)   Restart  required  when  adding  columns Good  cross-­‐datacenter  support 25Saturday, October 6, 12
  • NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase couchbase cassandra Database mongoDB 26Saturday, October 6, 12
  • Neo4j  –  Graph  database Neo4j Disk-­‐based  system External  caching  required  for  low-­‐latency  reads Nodes,  rela^onships  and  paths Proper^es  on  nodes Delete,  Insert,  Traverse,  etc. 27Saturday, October 6, 12
  • NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase couchbase cassandra Neo4j Database mongoDB 28Saturday, October 6, 12
  • The  Landscape S3 HBaseScale Riak Cassandra Couchbase SimpleDB RDBMS Datomic MongoDB memcached CouchDB Neo4j Redis Speed 29Saturday, October 6, 12
  • Datomic  -­‐  immutable  func^onal  data 30Saturday, October 6, 12
  • Hello  Couchbase  Server  2.0 31Saturday, October 6, 12
  • Couchbase  Server  2.0  beta 32Saturday, October 6, 12
  • Couchbase  handles  real  world  scale 33Saturday, October 6, 12
  • (Really)  High  Performance Cisco and Solarflare benchmark of Couchbase Server Latency Throughput less than 1/2 ms grows linearly with cluster size 5 Nodes -- 1.75M operations per second 34Saturday, October 6, 12
  • How  fast? hrp://www.slideshare.net/renatko/couchbase-­‐performance-­‐benchmarking 35Saturday, October 6, 12
  • LatencySaturday, October 6, 12
  • LatencySaturday, October 6, 12
  • COMPLEXITY  IS  THE  ENEMY 37Saturday, October 6, 12
  • Couchbase  Server  Basic  Opera^on APP  SERVER  1 APP  SERVER  2 §Docs  distributed  evenly  across   servers  in  the  cluster COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY §Each  server  stores  both  ac)ve  &   CLUSTER  MAP CLUSTER  MAP replica  docs § Only  one  server  ac^ve  at  a  ^me §Client  library  provides  app  with   simple  interface  to  database §Cluster  map  provides  map  to  which   server  doc  is  on § App  never  needs  to  know SERVER  1 SERVER  2 SERVER  3 § App  reads,  writes,  updates  docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs § Mul^ple  App  Servers  can  access  same   document  at  same  ^me Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  6 DOC COUCHBASE  SERVER  CLUSTER 38Saturday, October 6, 12
  • Couchbase  Server  Basic  Opera^on APP  SERVER  1 APP  SERVER  2 §Docs  distributed  evenly  across   servers  in  the  cluster COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY §Each  server  stores  both  ac)ve  &   CLUSTER  MAP CLUSTER  MAP replica  docs § Only  one  server  ac^ve  at  a  ^me §Client  library  provides  app  with   simple  interface  to  database §Cluster  map  provides  map  to  which   server  doc  is  on § App  never  needs  to  know SERVER  1 SERVER  2 SERVER  3 § App  reads,  writes,  updates  docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs § Mul^ple  App  Servers  can  access  same   document  at  same  ^me Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  2 DOC Doc  5 DOC COUCHBASE  SERVER  CLUSTER 38Saturday, October 6, 12
  • Couchbase  Server  Basic  Opera^on APP  SERVER  1 APP  SERVER  2 §Docs  distributed  evenly  across   servers  in  the  cluster COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY §Each  server  stores  both  ac)ve  &   CLUSTER  MAP CLUSTER  MAP replica  docs § Only  one  server  ac^ve  at  a  ^me §Client  library  provides  app  with   simple  interface  to  database Read/Write/Update Read/Write/Update §Cluster  map  provides  map  to  which   server  doc  is  on § App  never  needs  to  know SERVER  1 SERVER  2 SERVER  3 § App  reads,  writes,  updates  docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs § Mul^ple  App  Servers  can  access  same   document  at  same  ^me Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  2 DOC Doc  5 DOC COUCHBASE  SERVER  CLUSTER 38Saturday, October 6, 12
  • Add  Nodes  to  the  Cluster APP  SERVER  1 APP  SERVER  2 § Two  servers  added  to  cluster § One-­‐click  opera^on § Docs  automa^cally   COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY rebalanced  across  cluster CLUSTER  MAP CLUSTER  MAP § Even  distribu^on  of  docs § Minimum  doc  movement § Cluster  map  updated § App  database  calls  now   distributed  over  larger  #  of   servers SERVER  1 SERVER  2 SERVER  3 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  6 Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  9 Doc  8 DOC Doc  2 DOC Doc  5 DOC COUCHBASE  SERVER  CLUSTER 39Saturday, October 6, 12
  • Add  Nodes  to  the  Cluster APP  SERVER  1 APP  SERVER  2 § Two  servers  added  to  cluster § One-­‐click  opera^on § Docs  automa^cally   COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY rebalanced  across  cluster CLUSTER  MAP CLUSTER  MAP § Even  distribu^on  of  docs § Minimum  doc  movement § Cluster  map  updated § App  database  calls  now   distributed  over  larger  #  of   servers SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  6 Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  9 Doc  8 DOC Doc  2 DOC Doc  5 DOC COUCHBASE  SERVER  CLUSTER 39Saturday, October 6, 12
  • Add  Nodes  to  the  Cluster APP  SERVER  1 APP  SERVER  2 § Two  servers  added  to  cluster § One-­‐click  opera^on § Docs  automa^cally   COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY rebalanced  across  cluster CLUSTER  MAP CLUSTER  MAP § Even  distribu^on  of  docs § Minimum  doc  movement § Cluster  map  updated § App  database  calls  now   distributed  over  larger  #  of   servers SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC Doc  6 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 DOC Doc  8 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 39Saturday, October 6, 12
  • Add  Nodes  to  the  Cluster APP  SERVER  1 APP  SERVER  2 § Two  servers  added  to  cluster § One-­‐click  opera^on § Docs  automa^cally   COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY rebalanced  across  cluster CLUSTER  MAP CLUSTER  MAP § Even  distribu^on  of  docs § Minimum  doc  movement § Cluster  map  updated § App  database  calls  now   distributed  over  larger  #  of   servers Read/Write/Update Read/Write/Update SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC Doc  6 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 DOC Doc  8 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 39Saturday, October 6, 12
  • Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC Doc  6 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 DOC Doc  8 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  • Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC DOC Doc  6 Doc  1 Doc  3 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 Doc  8 DOC Doc  7 DOC DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  • Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC DOC Doc  6 Doc  1 Doc  3 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 Doc  8 DOC Doc  7 DOC DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  • Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC DOC Doc  6 Doc  1 Doc  3 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 Doc  8 DOC Doc  7 DOC DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  • Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  9 DOC Doc  6 DOC Doc  2 DOC Doc  7 DOC Doc  8 DOC DOC Doc  1 Doc  3 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  5 Doc  8 DOC DOC DOC Doc  2 DOC COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  • War  Story:  EBS  Outage ●Suddenly,disk writes all began to time out ●Many services experienced outages: ● FourSquare, Reddit, Quora, among others ●With memory buffered writes, a scalable data layer keeps working ● When EBS came back online, Couchbase wrote all the updated data to disk without missing a beat. 41Saturday, October 6, 12
  • Cross  Data  Center  Replica^on US  DATA  CENTER EUROPE  DATA  CENTER ASIA  DATA  CENTER Replica7on Replica7on Replica7on §Data  close  to  users §Mul^ple  loca^ons  for  disaster  recovery §Independently  managed  clusters  serving  local  data 42Saturday, October 6, 12
  • Built  for  Produc^on 43Saturday, October 6, 12
  • JSON  DOCUMENT  DATABASE 44Saturday, October 6, 12
  • Document  Database  as  Aggregate  Database hrp://mar^nfowler.com/bliki/AggregateOrientedDatabase.html 45Saturday, October 6, 12
  • Document  Database o::1001 { uid:  ji22jd, customer:  Ann, line_items:  [   {  sku:  0321293533,  quan:  3,    unit_price:  48.0  }, {  sku:  0321601912,  quan:  1,  unit_price:  39.0  }, {  sku:  0131495054,  quan:  1,  unit_price:  51.0  }   ], payment:  {  type:  Amex,  expiry:  04/2001,   last5:  12345  } } This synergy between the programming model and the distribution model is very valuable. It allows the database to use its knowledge of how the application programmer clusters the data to help performance across the cluster. hrp://mar^nfowler.com/bliki/AggregateOrientedDatabase.html 46Saturday, October 6, 12
  • Developers  <3  JSON 47Saturday, October 6, 12
  • LET’S  GET  POST-­‐RELATIONAL! 4Saturday, October 6, 12
  • JSON  Documents • Maps  more  closely  to  external  API • CRUD  Opera^ons,  lightweight  schema { “fields” : [“with basic types”, 3.14159, true], “like” : “your favorite language” } • Stored  under  an  iden^fier  key client.set(“mydocumentid”, myDocument); mySavedDocument = client.get(“mydocumentid”); 49Saturday, October 6, 12
  • Meta  +  Document  Body { Document      "brewery":  "New  Belgium  Brewing", user data,      "name":  "1554  Enlightened  Black  Ale", can be anything      "abv":  5.5,      "descrip7on":  "Born  of  a  flood...",      "category":  "Belgian  and  French  Ale",      "style":  "Other  Belgian-­‐Style  Ales",      "updated":  "2010-­‐07-­‐22  20:00:20" } unique ID “vintage” date format from an SQL dump { >_<      "id"  :  "beer_Enlightened_Black_Ale”, Metadata          ... identifier, { expiration, etc 50Saturday, October 6, 12
  • { Add  comments  to  the  beer      "brewery":  "New  Belgium        "name":  "1554  Enlightene      "abv":  5.5, {      "descrip7on":  "Born  of  a  fl      "type":  "comment",      "category":  "Belgian  and  F      "about_id":  "beer_Enlightened_Black_Ale",link to      "style":  "Other  Belgian-­‐Sty beer      "user_id":  525,      "updated":  "2010-­‐07-­‐22  2      "text":  "tastes  like  college!",    “ra7ngs”  :  {      "updated":  "2010-­‐07-­‐22  20:00:20"        “525”  :  5, }        “30”  :  4, {        “1044”  :  2      "id":  "f1e62"  }, }    “comment_ids”  :  [          “f1e62”, link to          “6ad8c” comments      ] }Saturday, October 6, 12
  • How  to:  look  up  comments  from  a  beer • SERIALIZED  LOOP   beer = client.get(“beer:A_cold_one”); beer.comment_ids.each { |id| comments.push(client.get(id)); } • FAST  MULTI-­‐KEY  LOOKUP beer = client.get(“beer:A_cold_one”); comments = client.multiGet(beer.comment_ids) • ASYNC  VIEW  QUERY comments = client.query(“myapp”,“by_comment_on”, {:key => “beer:A_cold_one”}); figure  hrp://www.ibm.com/developerworks/webservices/library/ws-­‐sdoarch/ 52Saturday, October 6, 12
  • Emergent  Schema • The  database  can  handle  it • Your  app  controls  the  schema "Capture  the  users  intent" JSON.org Github  API Twiqer  API 53Saturday, October 6, 12
  • Audience  par^cipa^on!  *that  means  you npm install twitterfight npm start twitterfight 54Saturday, October 6, 12
  • Thank  You! Chris  Anderson @jchris hrp://www.couchbase.com/ 55Saturday, October 6, 12
  • INCREMENTAL  MAP-­‐REDUCE FOR  REALTIME  ANALYTICS 5Saturday, October 6, 12
  • What  do  you  mean  “Incremental?” like: CREATE INDEX city ON brewery city; 57Saturday, October 6, 12
  • QUERY  PATTERN: FIND  BY  ATTRIBUTE 5Saturday, October 6, 12
  • Find  documents  by  a  specific  arribute • Lets  find  beers  by  brewery_id! 59Saturday, October 6, 12
  • The  index  defini^on 60Saturday, October 6, 12
  • The  result  set:  beers  keyed  by  brewery_id 61Saturday, October 6, 12
  • QUERY  PATTERN: BASIC  AGGREGATIONS 6Saturday, October 6, 12
  • Use  a  built-­‐in  reduce  func^on  with  a  group  query • Lets  find  average  abv  for  each  brewery! 63Saturday, October 6, 12
  • We  are  reducing  doc.abv  with  _stats 64Saturday, October 6, 12
  • Group  reduce  (reduce  by  unique  key) 65Saturday, October 6, 12
  • QUERY  PATTERN: TIME-­‐BASED  ROLLUPS WITH  KEY  GROUPING 6Saturday, October 6, 12
  • group_level=3  -­‐  daily  results  -­‐  great  for  graphing • Daily,  hourly,  minute  or  second  rollup  all  possible  with   the  same  index. • hrp://crate.im/posts/couchbase-­‐views-­‐reddit-­‐data/ 67Saturday, October 6, 12
  • GEO  INDEX  &   FULL  TEXT  INTEGRATION 6Saturday, October 6, 12
  • GeoCouch  R-­‐Tree  Index • Op^mized  for  bulk  loading  of  large  data  sets • Simplified  query  model  (bounding  box,  nearest  neighbor) 6Saturday, October 6, 12
  • Elas^c  Search  Adapter • Elas^c  Search  is  good  for  ad-­‐hoc  queries  and  faceted  browsing • Our  adapter  is  aware  of  changing  Couchbase  topology • Indexed  by  Elas^c  Search  aOer  stored  to  disk  in  Couchbase 70Saturday, October 6, 12
  • Thank  You! Chris  Anderson @jchris hrp://www.couchbase.com/ 71Saturday, October 6, 12