NoSQL	  Landscape                          &	  Grid	  Compu7ng                                     Chris	  Anderson       ...
Growth	  is	  the	  New	  Reality          •   2.2	  Billion	  internet	  users          •   50%	  Americans	  use	       ...
Instagrowth:	  Android	  Launch       E xa mple          • Instagram	  gained	  nearly	  1	  million	  users	  overnight	 ...
Instagrowth:	  Android	  Launch       E xa mple          1	  Instagram                                   =                ...
Draw	  Something	  -­‐	  Social	  Game              35 million monthly active users in 1 month                            ...
Goes	  Viral	  3	  Weeks	  aOer	  Launch                Draw	  Something	  by	  OMGPOP                Daily	  Ac)ve	  User...
By	  Contrast,	  at	  1/2	  an	  Instagram            The	  Simpson’s:	  Tapped	  Out            Daily	  Ac)ve	  Users	  (...
GETTING	  IT	  RIGHT                                                   8Saturday, October 6, 12
Scalable	  Data	  Layer   ●On-­‐demand	  cluster	  sizing         ● Grow	  or	  shrink	  with	  workload   ●Easy	  node	  ...
Old	  School	  Hits	  a	  Scale	  Wall                                                        Application Scales Out      ...
Tradi^onal	  MySQL	  +	  Memcached	  Architecture    ● Run  as many MySQL machines as       you need                      ...
Limita^ons	  of	  MySQL	  +	  Memcached   ●   To scale you need to start using       MySQL more simply   ●   Scale by hand...
NoSQL	  Architectural	  Promise   • High Performance data access   • Scale Up/Down Horizontally                           ...
NOSQL	  TAXONOMY                                              14Saturday, October 6, 12
The Key-Value Store – the foundation of NoSQL                          Key                          1011001010001000100111...
Memcached – the NoSQL precursor               Key                  101100101000100010011101                     memcached ...
Redis	  –	  More	  “Structured	  Data”	  commands               Key                  101100101000100010011101             ...
NoSQL	  catalog                   Key-­‐Value   Data	  Structure   Document   Column   Graph(memory	  only)   Cache       ...
Membase	  –	  From	  key-­‐value	  cache	  to	  database               Key                  101100101000100010011101      ...
NoSQL	  catalog                   Key-­‐Value   Data	  Structure   Document   Column   Graph(memory	  only)   Cache       ...
Couchbase	  –	  document-­‐oriented	  database              Key                                                           ...
NoSQL	  catalog                   Key-­‐Value   Data	  Structure   Document    Column   Graph(memory	  only)   Cache      ...
MongoDB	  –	  Document-­‐oriented	  database            Key                                                               ...
NoSQL	  catalog                   Key-­‐Value   Data	  Structure   Document    Column   Graph(memory	  only)   Cache      ...
Cassandra	  –	  Column	  overlays                                                 Cassandra Column	  1                    ...
NoSQL	  catalog                   Key-­‐Value   Data	  Structure   Document    Column      Graph(memory	  only)   Cache   ...
Neo4j	  –	  Graph	  database                                                   Neo4j                                      ...
NoSQL	  catalog                   Key-­‐Value   Data	  Structure   Document    Column      Graph(memory	  only)   Cache   ...
The	  Landscape                          S3                                          HBaseScale                           ...
Datomic	  -­‐	  immutable	  func^onal	  data                                                       30Saturday, October 6, 12
Hello	  Couchbase	  Server	  2.0                                                                31Saturday, October 6, 12
Couchbase	  Server	  2.0	  beta                                         32Saturday, October 6, 12
Couchbase	  handles	  real	  world	  scale                                                     33Saturday, October 6, 12
(Really)	  High	  Performance          Cisco and Solarflare benchmark of Couchbase Server                           Latenc...
How	  fast?                          hrp://www.slideshare.net/renatko/couchbase-­‐performance-­‐benchmarking              ...
LatencySaturday, October 6, 12
LatencySaturday, October 6, 12
COMPLEXITY	  IS	  THE	  ENEMY                                                             37Saturday, October 6, 12
Couchbase	  Server	  Basic	  Opera^on                      APP	  SERVER	  1                                  APP	  SERVER	...
Couchbase	  Server	  Basic	  Opera^on                       APP	  SERVER	  1                                    APP	  SERV...
Couchbase	  Server	  Basic	  Opera^on                       APP	  SERVER	  1                                      APP	  SE...
Add	  Nodes	  to	  the	  Cluster                    APP	  SERVER	  1                                      APP	  SERVER	  2...
Add	  Nodes	  to	  the	  Cluster                    APP	  SERVER	  1                                      APP	  SERVER	  2...
Add	  Nodes	  to	  the	  Cluster                    APP	  SERVER	  1                                      APP	  SERVER	  2...
Add	  Nodes	  to	  the	  Cluster                    APP	  SERVER	  1                                      APP	  SERVER	  2...
Fail	  Over	  Node                                                                                                        ...
Fail	  Over	  Node                                                                                                        ...
Fail	  Over	  Node                                                                                                        ...
Fail	  Over	  Node                                                                                                        ...
Fail	  Over	  Node                                                                                                        ...
War	  Story:	  EBS	  Outage ●Suddenly,disk writes all began to time out ●Many services experienced outages:        ● FourS...
Cross	  Data	  Center	  Replica^on           US	  DATA	  CENTER                    EUROPE	  DATA	  CENTER                A...
Built	  for	  Produc^on                                43Saturday, October 6, 12
JSON	  DOCUMENT	  DATABASE                                                         44Saturday, October 6, 12
Document	  Database	  as	  Aggregate	  Database                          hrp://mar^nfowler.com/bliki/AggregateOrientedData...
Document	  Database                                             o::1001                                             {     ...
Developers	  <3	  JSON                               47Saturday, October 6, 12
LET’S	  GET	  POST-­‐RELATIONAL!                                                               4Saturday, October 6, 12
JSON	  Documents       • Maps	  more	  closely	  to	  external	  API       • CRUD	  Opera^ons,	  lightweight	  schema     ...
Meta	  +	  Document	  Body    {                                                                  Document    	  	  	  "bre...
{  Add	  comments	  to	  the	  beer                               	  	  	  "brewery":	  "New	  Belgium	                   ...
How	  to:	  look	  up	  comments	  from	  a	  beer             • SERIALIZED	  LOOP	                    beer = client.get(“...
Emergent	  Schema       • The	  database	  can	  handle	  it       • Your	  app	  controls	  the	  schema   "Capture	  the...
Audience	  par^cipa^on!	  *that	  means	  you                   npm install twitterfight                   npm start twitt...
Thank	  You!                                 Chris	  Anderson                                      @jchris                ...
INCREMENTAL	  MAP-­‐REDUCE                           FOR	  REALTIME	  ANALYTICS                                           ...
What	  do	  you	  mean	  “Incremental?”        like:        CREATE INDEX city ON brewery city;                            ...
QUERY	  PATTERN:                          FIND	  BY	  ATTRIBUTE                                                    5Saturd...
Find	  documents	  by	  a	  specific	  arribute       • Lets	  find	  beers	  by	  brewery_id!                              ...
The	  index	  defini^on                               60Saturday, October 6, 12
The	  result	  set:	  beers	  keyed	  by	  brewery_id                                                                  61S...
QUERY	  PATTERN:                          BASIC	  AGGREGATIONS                                                  6Saturday,...
Use	  a	  built-­‐in	  reduce	  func^on	  with	  a	  group	  query       • Lets	  find	  average	  abv	  for	  each	  brewe...
We	  are	  reducing	  doc.abv	  with	  _stats                                                         64Saturday, October ...
Group	  reduce	  (reduce	  by	  unique	  key)                                                         65Saturday, October ...
QUERY	  PATTERN:                          TIME-­‐BASED	  ROLLUPS                          WITH	  KEY	  GROUPING           ...
group_level=3	  -­‐	  daily	  results	  -­‐	  great	  for	  graphing   • Daily,	  hourly,	  minute	  or	  second	  rollup	...
GEO	  INDEX	  &	                            FULL	  TEXT	  INTEGRATION                                                     ...
GeoCouch	  R-­‐Tree	  Index       • Op^mized	  for	  bulk	  loading	  of	  large	  data	  sets       • Simplified	  query	 ...
Elas^c	  Search	  Adapter       • Elas^c	  Search	  is	  good	  for	  ad-­‐hoc	  queries	  and	  faceted	  browsing       ...
Thank	  You!                                 Chris	  Anderson                                      @jchris                ...
Upcoming SlideShare
Loading in...5
×

Social Gaming & Gambling Summit - London Chris Anderson

302

Published on

#sgsconf

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
302
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Social Gaming & Gambling Summit - London Chris Anderson

  1. 1. NoSQL  Landscape &  Grid  Compu7ng Chris  Anderson @jchris 1Saturday, October 6, 12
  2. 2. Growth  is  the  New  Reality • 2.2  Billion  internet  users • 50%  Americans  use   smartphones • Your  app  can  grow   overnight • Are  you  ready? 2Saturday, October 6, 12
  3. 3. Instagrowth:  Android  Launch E xa mple • Instagram  gained  nearly  1  million  users  overnight  when  they   expanded  to  Android 3Saturday, October 6, 12
  4. 4. Instagrowth:  Android  Launch E xa mple 1  Instagram = 7.5M  MAU* 4Saturday, October 6, 12
  5. 5. Draw  Something  -­‐  Social  Game 35 million monthly active users in 1 month about 5 Instagrams (Instagram today is waaaay more than 1 Instagram) 5Saturday, October 6, 12
  6. 6. Goes  Viral  3  Weeks  aOer  Launch Draw  Something  by  OMGPOP Daily  Ac)ve  Users  (millions) 16 14 35+M  MAU 12 at  peak 10 8 6 4 2 2/6 8 10 12 14 16 18 20 22 24 26 28 3/1 3 5 7 9 11 13 15 17 19 21 6Saturday, October 6, 12
  7. 7. By  Contrast,  at  1/2  an  Instagram The  Simpson’s:  Tapped  Out Daily  Ac)ve  Users  (millions) 7Saturday, October 6, 12
  8. 8. GETTING  IT  RIGHT 8Saturday, October 6, 12
  9. 9. Scalable  Data  Layer ●On-­‐demand  cluster  sizing ● Grow  or  shrink  with  workload ●Easy  node  provisioning ● All  nodes  are  the  same ●MulA-­‐master  Cross-­‐Datacenter  ReplicaAon ● For  a  fast  and  reliable  user  experience  worldwide ●EffecAve  Auto-­‐sharding ● Should  avoid  cluster  hot  spots 9Saturday, October 6, 12
  10. 10. Old  School  Hits  a  Scale  Wall Application Scales Out Just add more commodity web servers Database Scales Up Get a bigger, more complex server Expensive & disruptive sharding, doesn’t perform at web scale 10Saturday, October 6, 12
  11. 11. Tradi^onal  MySQL  +  Memcached  Architecture ● Run as many MySQL machines as you need www.example.com ● Data sharded evenly across the machines using client code App  Servers ● Memcached used to provide faster response time for users Memcached  Tier and reduce load on the database MySQL  Tier 11Saturday, October 6, 12
  12. 12. Limita^ons  of  MySQL  +  Memcached ● To scale you need to start using MySQL more simply ● Scale by hand ● Replication / Sharding is a black art ● Code overhead to manage keeping memcache and mysql in sync ● Lots of components to deploy Learn  From  Others  -­‐  This  Scenario  Costs  Time  and  Money.  Scaling  SQL  is   poten^ally  disastrous  when  going  Viral:  very  risky  ^me  for  major  code   changes  and  migra^ons...  you  have  no  Time  when  skyrocke^ng  up. 12Saturday, October 6, 12
  13. 13. NoSQL  Architectural  Promise • High Performance data access • Scale Up/Down Horizontally www.example.com • 24x7x365 Always-On Availability • Flexible Schema Document Model App  Servers Couchbase  Database  Servers 13Saturday, October 6, 12
  14. 14. NOSQL  TAXONOMY 14Saturday, October 6, 12
  15. 15. The Key-Value Store – the foundation of NoSQL Key 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 Opaque 101100101000100010011101 101100101000100010011101 Binary 101100101000100010011101 101100101000100010011101 Value 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 15Saturday, October 6, 12
  16. 16. Memcached – the NoSQL precursor Key 101100101000100010011101 memcached 101100101000100010011101 101100101000100010011101 101100101000100010011101 In-­‐memory  only 101100101000100010011101 Limited  set  of  opera^ons Opaque 101100101000100010011101 Blob  Storage:  Set,  Add,  Replace,  CAS 101100101000100010011101 Binary 101100101000100010011101 Retrieval:  Get 101100101000100010011101 Structured  Data:  Append,  Increment Value 101100101000100010011101 101100101000100010011101 “Simple  and  fast.” 101100101000100010011101 101100101000100010011101 Challenges:  cold  cache,  disrup^ve  elas^city 101100101000100010011101 101100101000100010011101 16Saturday, October 6, 12
  17. 17. Redis  –  More  “Structured  Data”  commands Key 101100101000100010011101 redis 101100101000100010011101 101100101000100010011101 101100101000100010011101 In-­‐memory  only “Data  Structures” 101100101000100010011101 Vast  set  of  opera^ons Blob 101100101000100010011101 Blob  Storage:  Set,  Add,  Replace,  CAS 101100101000100010011101 List 101100101000100010011101 Retrieval:  Get,  Pub-­‐Sub 101100101000100010011101 Structured  Data:  Strings,  Hashes,  Lists,  Sets, Set 101100101000100010011101 Sorted  lists Hash 101100101000100010011101 101100101000100010011101 … 101100101000100010011101 Example  opera7ons  for  a  Set 101100101000100010011101 Add,  count,  subtract  sets,  intersec^on,  is   101100101000100010011101 member?,  atomic  move  from  one  set  to  another 17Saturday, October 6, 12
  18. 18. NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis 18Saturday, October 6, 12
  19. 19. Membase  –  From  key-­‐value  cache  to  database Key 101100101000100010011101 membase 101100101000100010011101 101100101000100010011101 101100101000100010011101 Disk-­‐based  with  built-­‐in  memcached  cache 101100101000100010011101 Cache  refill  on  restart Opaque 101100101000100010011101 Memcached  compa^ble  (drop  in  replacement) 101100101000100010011101 Highly-­‐available  (data  replica^on) Binary 101100101000100010011101 Add  or  remove  capacity  to  live  cluster 101100101000100010011101 Value 101100101000100010011101 101100101000100010011101 “Simple,  fast,  elas^c.” 101100101000100010011101 101100101000100010011101 101100101000100010011101 101100101000100010011101 19Saturday, October 6, 12
  20. 20. NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase Database 20Saturday, October 6, 12
  21. 21. Couchbase  –  document-­‐oriented  database Key Couchbase {        “string”  :  “string”,        “string”  :  value, Auto-­‐sharding        “string”  :   Disk-­‐based  with  built-­‐in  memcached  cache                      {    “string”  :  “string”, JSON Cache  refill  on  restart                              “string”  :  value  },        “string”  :OBJECT Memcached  compa^ble  (drop  in  replace)  [  array  ] Highly-­‐available  (data  replica^on) } (“DOCUMENT”) Add  or  remove  capacity  to  live  cluster When  values  are  JSON  objects  (“documents”): Create  indices,  views  and  query  against  the  views 21Saturday, October 6, 12
  22. 22. NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase couchbase Database 22Saturday, October 6, 12
  23. 23. MongoDB  –  Document-­‐oriented  database Key MongoDB {        “string”  :  “string”,        “string”  :  value, Disk-­‐based  with  in-­‐memory  “caching”        “string”  :   BSON BSON  (“binary  JSON”)  format  and  wire  protocol                      {    “string”  :  “string”, Master-­‐slave  replica^on OBJECT                              “string”  :  value  }, Auto-­‐sharding        “string”  :  [  array  ] } (“DOCUMENT”) Values  are  BSON  objects Supports  ad  hoc  queries  –  best  when  indexed 23Saturday, October 6, 12
  24. 24. NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase couchbase Database mongoDB 24Saturday, October 6, 12
  25. 25. Cassandra  –  Column  overlays Cassandra Column  1 Disk-­‐based  system Clustered   Column  2 External  caching  required  for  low-­‐latency  reads “Columns”  are  overlaid  on  the  data Not  all  rows  must  have  all  columns Column  3   Supports  efficient  queries  on  columns (not  present)   Restart  required  when  adding  columns Good  cross-­‐datacenter  support 25Saturday, October 6, 12
  26. 26. NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase couchbase cassandra Database mongoDB 26Saturday, October 6, 12
  27. 27. Neo4j  –  Graph  database Neo4j Disk-­‐based  system External  caching  required  for  low-­‐latency  reads Nodes,  rela^onships  and  paths Proper^es  on  nodes Delete,  Insert,  Traverse,  etc. 27Saturday, October 6, 12
  28. 28. NoSQL  catalog Key-­‐Value Data  Structure Document Column Graph(memory  only) Cache memcached redis(memory/disk) membase couchbase cassandra Neo4j Database mongoDB 28Saturday, October 6, 12
  29. 29. The  Landscape S3 HBaseScale Riak Cassandra Couchbase SimpleDB RDBMS Datomic MongoDB memcached CouchDB Neo4j Redis Speed 29Saturday, October 6, 12
  30. 30. Datomic  -­‐  immutable  func^onal  data 30Saturday, October 6, 12
  31. 31. Hello  Couchbase  Server  2.0 31Saturday, October 6, 12
  32. 32. Couchbase  Server  2.0  beta 32Saturday, October 6, 12
  33. 33. Couchbase  handles  real  world  scale 33Saturday, October 6, 12
  34. 34. (Really)  High  Performance Cisco and Solarflare benchmark of Couchbase Server Latency Throughput less than 1/2 ms grows linearly with cluster size 5 Nodes -- 1.75M operations per second 34Saturday, October 6, 12
  35. 35. How  fast? hrp://www.slideshare.net/renatko/couchbase-­‐performance-­‐benchmarking 35Saturday, October 6, 12
  36. 36. LatencySaturday, October 6, 12
  37. 37. LatencySaturday, October 6, 12
  38. 38. COMPLEXITY  IS  THE  ENEMY 37Saturday, October 6, 12
  39. 39. Couchbase  Server  Basic  Opera^on APP  SERVER  1 APP  SERVER  2 §Docs  distributed  evenly  across   servers  in  the  cluster COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY §Each  server  stores  both  ac)ve  &   CLUSTER  MAP CLUSTER  MAP replica  docs § Only  one  server  ac^ve  at  a  ^me §Client  library  provides  app  with   simple  interface  to  database §Cluster  map  provides  map  to  which   server  doc  is  on § App  never  needs  to  know SERVER  1 SERVER  2 SERVER  3 § App  reads,  writes,  updates  docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs § Mul^ple  App  Servers  can  access  same   document  at  same  ^me Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  6 DOC COUCHBASE  SERVER  CLUSTER 38Saturday, October 6, 12
  40. 40. Couchbase  Server  Basic  Opera^on APP  SERVER  1 APP  SERVER  2 §Docs  distributed  evenly  across   servers  in  the  cluster COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY §Each  server  stores  both  ac)ve  &   CLUSTER  MAP CLUSTER  MAP replica  docs § Only  one  server  ac^ve  at  a  ^me §Client  library  provides  app  with   simple  interface  to  database §Cluster  map  provides  map  to  which   server  doc  is  on § App  never  needs  to  know SERVER  1 SERVER  2 SERVER  3 § App  reads,  writes,  updates  docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs § Mul^ple  App  Servers  can  access  same   document  at  same  ^me Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  2 DOC Doc  5 DOC COUCHBASE  SERVER  CLUSTER 38Saturday, October 6, 12
  41. 41. Couchbase  Server  Basic  Opera^on APP  SERVER  1 APP  SERVER  2 §Docs  distributed  evenly  across   servers  in  the  cluster COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY §Each  server  stores  both  ac)ve  &   CLUSTER  MAP CLUSTER  MAP replica  docs § Only  one  server  ac^ve  at  a  ^me §Client  library  provides  app  with   simple  interface  to  database Read/Write/Update Read/Write/Update §Cluster  map  provides  map  to  which   server  doc  is  on § App  never  needs  to  know SERVER  1 SERVER  2 SERVER  3 § App  reads,  writes,  updates  docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs § Mul^ple  App  Servers  can  access  same   document  at  same  ^me Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  2 DOC Doc  5 DOC COUCHBASE  SERVER  CLUSTER 38Saturday, October 6, 12
  42. 42. Add  Nodes  to  the  Cluster APP  SERVER  1 APP  SERVER  2 § Two  servers  added  to  cluster § One-­‐click  opera^on § Docs  automa^cally   COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY rebalanced  across  cluster CLUSTER  MAP CLUSTER  MAP § Even  distribu^on  of  docs § Minimum  doc  movement § Cluster  map  updated § App  database  calls  now   distributed  over  larger  #  of   servers SERVER  1 SERVER  2 SERVER  3 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  6 Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  9 Doc  8 DOC Doc  2 DOC Doc  5 DOC COUCHBASE  SERVER  CLUSTER 39Saturday, October 6, 12
  43. 43. Add  Nodes  to  the  Cluster APP  SERVER  1 APP  SERVER  2 § Two  servers  added  to  cluster § One-­‐click  opera^on § Docs  automa^cally   COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY rebalanced  across  cluster CLUSTER  MAP CLUSTER  MAP § Even  distribu^on  of  docs § Minimum  doc  movement § Cluster  map  updated § App  database  calls  now   distributed  over  larger  #  of   servers SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  6 Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  9 Doc  8 DOC Doc  2 DOC Doc  5 DOC COUCHBASE  SERVER  CLUSTER 39Saturday, October 6, 12
  44. 44. Add  Nodes  to  the  Cluster APP  SERVER  1 APP  SERVER  2 § Two  servers  added  to  cluster § One-­‐click  opera^on § Docs  automa^cally   COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY rebalanced  across  cluster CLUSTER  MAP CLUSTER  MAP § Even  distribu^on  of  docs § Minimum  doc  movement § Cluster  map  updated § App  database  calls  now   distributed  over  larger  #  of   servers SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC Doc  6 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 DOC Doc  8 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 39Saturday, October 6, 12
  45. 45. Add  Nodes  to  the  Cluster APP  SERVER  1 APP  SERVER  2 § Two  servers  added  to  cluster § One-­‐click  opera^on § Docs  automa^cally   COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY rebalanced  across  cluster CLUSTER  MAP CLUSTER  MAP § Even  distribu^on  of  docs § Minimum  doc  movement § Cluster  map  updated § App  database  calls  now   distributed  over  larger  #  of   servers Read/Write/Update Read/Write/Update SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC Doc  6 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 DOC Doc  8 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 39Saturday, October 6, 12
  46. 46. Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC Doc  6 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 DOC Doc  8 DOC Doc  7 Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  47. 47. Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC DOC Doc  6 Doc  1 Doc  3 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 Doc  8 DOC Doc  7 DOC DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  48. 48. Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC DOC Doc  6 Doc  1 Doc  3 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 Doc  8 DOC Doc  7 DOC DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  49. 49. Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  9 DOC Doc  6 DOC Doc  3 Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  8 DOC DOC Doc  6 Doc  1 Doc  3 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  5 Doc  8 DOC Doc  7 DOC DOC Doc  9 DOC Doc  2 DOC Doc  9 COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  50. 50. Fail  Over  Node § App  servers  happily  accessing  docs  on   Server  3 § Server  fails § App  server  requests  to  server  3  fail § Cluster  detects  server  has  failed COUCHBASE  CLIENT  LIBRARY COUCHBASE  CLIENT  LIBRARY CLUSTER  MAP CLUSTER  MAP § Promotes  replicas  of  docs  to  ac)ve § Updates  cluster  map § App  server  requests  for  docs  now  go  to   appropriate  server § Typically  rebalance  would  follow SERVER  1 SERVER  2 SERVER  3 SERVER  4 SERVER  5 Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Ac^ve  Docs Doc  5 DOC Doc  4 DOC Doc  9 DOC Doc  6 DOC Doc  2 DOC Doc  7 DOC Doc  8 DOC DOC Doc  1 Doc  3 Replica  Docs Replica  Docs Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  5 Doc  8 DOC DOC DOC Doc  2 DOC COUCHBASE  SERVER  CLUSTER 40Saturday, October 6, 12
  51. 51. War  Story:  EBS  Outage ●Suddenly,disk writes all began to time out ●Many services experienced outages: ● FourSquare, Reddit, Quora, among others ●With memory buffered writes, a scalable data layer keeps working ● When EBS came back online, Couchbase wrote all the updated data to disk without missing a beat. 41Saturday, October 6, 12
  52. 52. Cross  Data  Center  Replica^on US  DATA  CENTER EUROPE  DATA  CENTER ASIA  DATA  CENTER Replica7on Replica7on Replica7on §Data  close  to  users §Mul^ple  loca^ons  for  disaster  recovery §Independently  managed  clusters  serving  local  data 42Saturday, October 6, 12
  53. 53. Built  for  Produc^on 43Saturday, October 6, 12
  54. 54. JSON  DOCUMENT  DATABASE 44Saturday, October 6, 12
  55. 55. Document  Database  as  Aggregate  Database hrp://mar^nfowler.com/bliki/AggregateOrientedDatabase.html 45Saturday, October 6, 12
  56. 56. Document  Database o::1001 { uid:  ji22jd, customer:  Ann, line_items:  [   {  sku:  0321293533,  quan:  3,    unit_price:  48.0  }, {  sku:  0321601912,  quan:  1,  unit_price:  39.0  }, {  sku:  0131495054,  quan:  1,  unit_price:  51.0  }   ], payment:  {  type:  Amex,  expiry:  04/2001,   last5:  12345  } } This synergy between the programming model and the distribution model is very valuable. It allows the database to use its knowledge of how the application programmer clusters the data to help performance across the cluster. hrp://mar^nfowler.com/bliki/AggregateOrientedDatabase.html 46Saturday, October 6, 12
  57. 57. Developers  <3  JSON 47Saturday, October 6, 12
  58. 58. LET’S  GET  POST-­‐RELATIONAL! 4Saturday, October 6, 12
  59. 59. JSON  Documents • Maps  more  closely  to  external  API • CRUD  Opera^ons,  lightweight  schema { “fields” : [“with basic types”, 3.14159, true], “like” : “your favorite language” } • Stored  under  an  iden^fier  key client.set(“mydocumentid”, myDocument); mySavedDocument = client.get(“mydocumentid”); 49Saturday, October 6, 12
  60. 60. Meta  +  Document  Body { Document      "brewery":  "New  Belgium  Brewing", user data,      "name":  "1554  Enlightened  Black  Ale", can be anything      "abv":  5.5,      "descrip7on":  "Born  of  a  flood...",      "category":  "Belgian  and  French  Ale",      "style":  "Other  Belgian-­‐Style  Ales",      "updated":  "2010-­‐07-­‐22  20:00:20" } unique ID “vintage” date format from an SQL dump { >_<      "id"  :  "beer_Enlightened_Black_Ale”, Metadata          ... identifier, { expiration, etc 50Saturday, October 6, 12
  61. 61. { Add  comments  to  the  beer      "brewery":  "New  Belgium        "name":  "1554  Enlightene      "abv":  5.5, {      "descrip7on":  "Born  of  a  fl      "type":  "comment",      "category":  "Belgian  and  F      "about_id":  "beer_Enlightened_Black_Ale",link to      "style":  "Other  Belgian-­‐Sty beer      "user_id":  525,      "updated":  "2010-­‐07-­‐22  2      "text":  "tastes  like  college!",    “ra7ngs”  :  {      "updated":  "2010-­‐07-­‐22  20:00:20"        “525”  :  5, }        “30”  :  4, {        “1044”  :  2      "id":  "f1e62"  }, }    “comment_ids”  :  [          “f1e62”, link to          “6ad8c” comments      ] }Saturday, October 6, 12
  62. 62. How  to:  look  up  comments  from  a  beer • SERIALIZED  LOOP   beer = client.get(“beer:A_cold_one”); beer.comment_ids.each { |id| comments.push(client.get(id)); } • FAST  MULTI-­‐KEY  LOOKUP beer = client.get(“beer:A_cold_one”); comments = client.multiGet(beer.comment_ids) • ASYNC  VIEW  QUERY comments = client.query(“myapp”,“by_comment_on”, {:key => “beer:A_cold_one”}); figure  hrp://www.ibm.com/developerworks/webservices/library/ws-­‐sdoarch/ 52Saturday, October 6, 12
  63. 63. Emergent  Schema • The  database  can  handle  it • Your  app  controls  the  schema "Capture  the  users  intent" JSON.org Github  API Twiqer  API 53Saturday, October 6, 12
  64. 64. Audience  par^cipa^on!  *that  means  you npm install twitterfight npm start twitterfight 54Saturday, October 6, 12
  65. 65. Thank  You! Chris  Anderson @jchris hrp://www.couchbase.com/ 55Saturday, October 6, 12
  66. 66. INCREMENTAL  MAP-­‐REDUCE FOR  REALTIME  ANALYTICS 5Saturday, October 6, 12
  67. 67. What  do  you  mean  “Incremental?” like: CREATE INDEX city ON brewery city; 57Saturday, October 6, 12
  68. 68. QUERY  PATTERN: FIND  BY  ATTRIBUTE 5Saturday, October 6, 12
  69. 69. Find  documents  by  a  specific  arribute • Lets  find  beers  by  brewery_id! 59Saturday, October 6, 12
  70. 70. The  index  defini^on 60Saturday, October 6, 12
  71. 71. The  result  set:  beers  keyed  by  brewery_id 61Saturday, October 6, 12
  72. 72. QUERY  PATTERN: BASIC  AGGREGATIONS 6Saturday, October 6, 12
  73. 73. Use  a  built-­‐in  reduce  func^on  with  a  group  query • Lets  find  average  abv  for  each  brewery! 63Saturday, October 6, 12
  74. 74. We  are  reducing  doc.abv  with  _stats 64Saturday, October 6, 12
  75. 75. Group  reduce  (reduce  by  unique  key) 65Saturday, October 6, 12
  76. 76. QUERY  PATTERN: TIME-­‐BASED  ROLLUPS WITH  KEY  GROUPING 6Saturday, October 6, 12
  77. 77. group_level=3  -­‐  daily  results  -­‐  great  for  graphing • Daily,  hourly,  minute  or  second  rollup  all  possible  with   the  same  index. • hrp://crate.im/posts/couchbase-­‐views-­‐reddit-­‐data/ 67Saturday, October 6, 12
  78. 78. GEO  INDEX  &   FULL  TEXT  INTEGRATION 6Saturday, October 6, 12
  79. 79. GeoCouch  R-­‐Tree  Index • Op^mized  for  bulk  loading  of  large  data  sets • Simplified  query  model  (bounding  box,  nearest  neighbor) 6Saturday, October 6, 12
  80. 80. Elas^c  Search  Adapter • Elas^c  Search  is  good  for  ad-­‐hoc  queries  and  faceted  browsing • Our  adapter  is  aware  of  changing  Couchbase  topology • Indexed  by  Elas^c  Search  aOer  stored  to  disk  in  Couchbase 70Saturday, October 6, 12
  81. 81. Thank  You! Chris  Anderson @jchris hrp://www.couchbase.com/ 71Saturday, October 6, 12
  1. Gostou de algum slide específico?

    Recortar slides é uma maneira fácil de colecionar informações para acessar mais tarde.

×