Couchbase 101 - Installation of Couchbase

2,673 views
2,519 views

Published on

Learn about the Architecture of Couchbase and why it’s so fast, installation of Couchbase on various platforms, setup and configuration parameters, monitoring and the Admin console, and scaling Couchbase.

What will be covered during this training:

Couchbase Architecture
Setup parameters and options
Admin Console
Monitoring Graphs and what they mean
Buckets
RAM/CPU/IO Writers
Scaling, Clustering and Rebalancing
Couchbase in the Cloud
Using AMI installations
References to Docs, Communities, etc.

Published in: Technology

Couchbase 101 - Installation of Couchbase

  1. 1. Tuesday, October 8, 13
  2. 2. Technical  Evangelist twi0er:  @scalabl3email:  jasdeep@couchbase.com Jasdeep  Jaitla Couchbase  101  -­‐  The  Server Tuesday, October 8, 13
  3. 3. Technical  Evangelist twi0er:  @scalabl3email:  jasdeep@couchbase.com Jasdeep  Jaitla Couchbase  101  -­‐  The  Server Tuesday, October 8, 13
  4. 4. Tuesday, October 8, 13
  5. 5. ARCHITECTURE Tuesday, October 8, 13
  6. 6. Evolution from memcached •Founders  were  key  contributors  to  memcached •Evolved  into  Membase,  a  distributed  and  persisted  key-­‐value   store •Evolved  into  Couchbase  Document  Store  with  JSON  support   and  Map-­‐Reduce  Indexes,  ElasOc  Search  IntegraOon,  and   Cross-­‐Data  Center  ReplicaOon Tuesday, October 8, 13
  7. 7. Couchbase Server Core Principles Easy   Scalability Consistent  High   Performance Always  On   24x365 Grow  cluster  without   applica9on  changes,  without   down9me  with  a  single  click Consistent  sub-­‐millisecond   read  and  write  response  9mes   with  consistent  high  throughput No  down9me  for  soBware   upgrades,  hardware   maintenance,  etc. Flexible  Data   Model JSON  Anywhere  document   model  with  no  fixed  schema. JSON JSON JSON JSONJSON PERFORMANCE Tuesday, October 8, 13
  8. 8. Couchbase Server 2.0 Architecture Heartbeat Process  monitor Global  singleton  supervisor Configura9on  manager on  each  node Rebalance  orchestrator Node  health  monitor one  per  cluster vBucket  state  and  replica9on  manager h"p REST  management  API/Web  UI HTTP 8091 Erlang  port  mapper 4369 Distributed  Erlang 21100  -­‐  21199 Erlang/OTP storage  interface Couchbase  EP  Engine 11210 Memcapable    2.0 Moxi 11211 Memcapable    1.0 Memcached New  Persistence  Layer 8092 Query  APIQuery  Engine Data  Manager Cluster  Manager Tuesday, October 8, 13
  9. 9. Couchbase Server 2.0 Architecture New  Persistence  Layer storage  interface Couchbase  EP  Engine 11210 Memcapable    2.0 Moxi 11211 Memcapable    1.0 Object-­‐level  Cache Disk  Persistence 8092 Query  API Query  Engine HTTP 8091 Erlang  port  mapper 4369 Distributed  Erlang 21100  -­‐  21199 Heartbeat Process  monitor Global  singleton  supervisor Configura9on  manager on  each  node Rebalance  orchestrator Node  health  monitor one  per  cluster vBucket  state  and  replica9on  manager h"p REST  management  API/Web  UI Erlang/OTP Server/Cluster   Management  &   Communica6on (Erlang) RAM  Cache,   Indexing  &   Persistence   Management (C) Tuesday, October 8, 13
  10. 10. Couchbase  OrganizaOon • Couchbase  operates  like  a  Key-­‐Value  Document  Store • Key  is  a  UTF-­‐8  string  up  to  256  Bytes • Values  can  be:   -­‐ Simple  Datatypes:  strings,  numbers,  dateOme,  boolean,  and  binary   data  can  be  stored  -­‐-­‐  they  are  stored  as  Base64  encoded  strings -­‐ Complex  Datatypes:  dicOonaries/hashes,  arrays/lists,  can  be  stored   in  JSON  format  (simple  lists  can  be  string  based  with  delimiter) -­‐ JSON  is  a  special  class  of  string  with  a  specific  format  for  encoding   simple  and  complex  data  structures • Schema  is  unenforced  and  implicit,  schema  changes  are   programmaOc,  done  online,  and  can  vary  from  Document  to   Document Tuesday, October 8, 13
  11. 11. Metadata and Documents meta { “id”:  “u::jasdeep@couchbase.com”, “rev”:  “1-­‐0002bce0000000000”, “flags”:  0, “expiraUon”:  0, “type”:  “json” } document { “uid”:  123456, “firstname”:  “jasdeep”, “lastname”:  “Jaitla”, “age”:  22, “favorite_colors”:  [“blue”,  “black”], “email”:  “jasdeep@couchbase.com” } Meta  Informa6on   Including  Key  (id) All  Keys  Unique  and   Kept  in  RAM Document  Value Most  Recent  In  RAM   And  Persisted  To  Disk Tuesday, October 8, 13
  12. 12. Retrieval Operations Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine Tuesday, October 8, 13
  13. 13. Retrieval Operations Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine get Tuesday, October 8, 13
  14. 14. Retrieval Operations Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine get Tuesday, October 8, 13
  15. 15. Storage Operations Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine Tuesday, October 8, 13
  16. 16. Storage Operations Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine set/add/replace Tuesday, October 8, 13
  17. 17. Storage Operations Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine set/add/replace Tuesday, October 8, 13
  18. 18. Storage Operations Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine set/add/replace Tuesday, October 8, 13
  19. 19. Storage Operations Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine set/add/replace Tuesday, October 8, 13
  20. 20. Consistency Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine Tuesday, October 8, 13
  21. 21. Consistency Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine get Tuesday, October 8, 13
  22. 22. Ejection, NRU, Cache Miss Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine Tuesday, October 8, 13
  23. 23. Ejection, NRU, Cache Miss Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine set/add/replaceset/add/replaceset/add/replace Tuesday, October 8, 13
  24. 24. Ejection, NRU, Cache Miss Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine FULL  (90%) NRU Documents Ejected Tuesday, October 8, 13
  25. 25. Ejection, NRU, Cache Miss Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine Tuesday, October 8, 13
  26. 26. Ejection, NRU, Cache Miss Couchbase Server EP EngineRAM Cache Disk Write Queue Replication Queue Application Server Replica Couchbase Cluster Machine get Non-­‐Resident   Document   ("Cache  Miss") Tuesday, October 8, 13
  27. 27. Clients Connect Directly to Couchbase Nodes Application Servers MAP 1024 8 GB RAM Partitions 3 IO Workers Tuesday, October 8, 13
  28. 28. Clients Connect Directly to Couchbase Nodes Application Servers MAP 1024 8 GB RAM Partitions 3 IO Workers Tuesday, October 8, 13
  29. 29. Clients Connect Directly to Couchbase Nodes Application Servers MAP MAP MAP 1024 8 GB RAM Partitions 3 IO Workers Tuesday, October 8, 13
  30. 30. Key Hash-Partitioning Application Servers MAP MAP MAP 1024 8 GB RAM Partitions 3 IO Workers ClientHashFunc6on("jasdeep@couchbase.com")  =>  Par66on[0..1023]  {25} ClusterMap[P(25)]  =>  [x.x.x.x]  =>  IP  of  Server  Responsible  for  Par66on  25 Tuesday, October 8, 13
  31. 31. Application Servers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers Horizontal Scale-Rebalance TOTAL 16 GB RAM 6 IO Workers 1024 Partitions 1024 Partitions TOTAL 8 GB RAM 3 IO Workers 1024 Partitions Tuesday, October 8, 13
  32. 32. Application Servers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers Horizontal Scale-Rebalance TOTAL 16 GB RAM 6 IO Workers 1024 Partitions 1024 Partitions 512 Partitions 512 Partitions Tuesday, October 8, 13
  33. 33. Horizontal Scale-Rebalance TOTAL 32 GB RAM 12 IO Workers 1024 Partitions Application Servers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 512 Partitions 512 Partitions TOTAL 16 GB RAM 6 IO Workers 1024 Partitions MAP MAP MAP MAP Tuesday, October 8, 13
  34. 34. Horizontal Scale-Rebalance TOTAL 32 GB RAM 12 IO Workers 1024 Partitions Application Servers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 512 Partitions 512 Partitions 256 Partitions 256 Partitions 256 Partitions 256 Partitions TOTAL 16 GB RAM 6 IO Workers 1024 Partitions MAP MAP MAP MAP MAP MAP MAP MAP Tuesday, October 8, 13
  35. 35. Horizontal Scale-Rebalance TOTAL 32 GB RAM 12 IO Workers 1024 Partitions Application Servers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 8 GB RAM 3 IO Workers 512 Partitions 512 Partitions 256 Partitions 256 Partitions 256 Partitions 256 Partitions MAP MAP MAP MAP MAP MAP MAP MAP Tuesday, October 8, 13
  36. 36. All  Metadata  for  All   Documents   (64  bytes  +  Key  Length) Document  Values (NRU  Ejected  if  RAM  Quota   Used  >  90%) Also  Leave  RAM  For  OS:   [Filesystem  Cache  >>  Views] Document  Indexing Monitoring XDCR Recommended:   minimum  4  Cores +  1  core  per  design  document +  1  core  per  XDCR  replicated   bucket Persisted  Documents All  Indexes  for  Design   Documents/Views Append-­‐Only  Disk  Format &  CompacUon Performance: MulUple  EBS  Volumes  High   IOPS  Raid  0  on  Amazon RAM, CPU and IO Guidelines RAM CPU Disk  IO Tuesday, October 8, 13
  37. 37. Binary Socket Operations (11210/1) • get (key) – Retrieve a document • set (key, value) – Store a document, overwrites if exists • add (key, value) – Store a document, error/exception if exists • replace (key, value) – Store a document, error/exception if doesn’t exist • incr (key) – Create/Increment Atomic Counter • decr (key) – Decrement Atomic Counter • cas (key, value, cas) – Compare and swap, set document only if it hasn’t changed (Optimistic Lock) Tuesday, October 8, 13
  38. 38. HTTP Operations (8092) • View Querying – Range Queries – Index-Key Match Queries – Set Match Queries – Aggregate Reduces – Group Level + Grouping Queries Tuesday, October 8, 13
  39. 39. Tuesday, October 8, 13
  40. 40. INSTALL  SERVER Tuesday, October 8, 13
  41. 41. Download Couchbase www.couchbase.com/download Tuesday, October 8, 13
  42. 42. Install on Amazon via AMI Tuesday, October 8, 13
  43. 43. Install on Amazon via AMI Tuesday, October 8, 13
  44. 44. Run Couchbase and Open Browser localhost:8091  or  ec2-­‐xx-­‐xx-­‐xx-­‐xx.compute-­‐1.amazonaws.com:8091 Tuesday, October 8, 13
  45. 45. Setup: Step 1 - Storage Paths & RAM Maximum Tuesday, October 8, 13
  46. 46. Setup: Step 1 - Storage Paths & RAM Maximum Tuesday, October 8, 13
  47. 47. Setup: Step 2 - Sample Buckets Tuesday, October 8, 13
  48. 48. Setup: Step 3 - Configure Default Bucket Tuesday, October 8, 13
  49. 49. Setup: Step 3 - Configure Default Bucket Tuesday, October 8, 13
  50. 50. Setup: Step 4 - Register & Agree Tuesday, October 8, 13
  51. 51. Setup: Step 5 - Admin password Tuesday, October 8, 13
  52. 52. Setup: Complete Tuesday, October 8, 13
  53. 53. Tuesday, October 8, 13
  54. 54. SETUP  SDK Tuesday, October 8, 13
  55. 55. Supported SDK's www.couchbase.com/communiUes • Each  supported  SDK  page  has  instrucOons  for  setup • PHP,  Ruby,  NodeJS  and  Python  clients  are  wrappers  around   libcouchbase  C  library,  so  libcouchbase  must  be  installed  first • For  other  community  clients,  click  on  "All  Clients"  on  lei  nav,  scroll   down  the  page  and  you  can  see  clients  for  Go,  Erlang,  Clojure,  TCL,   other  nodejs  and  Perl. Tuesday, October 8, 13
  56. 56. Installing Libcouchbase • Mac  Tips  before  Libcouchbase  &  SDK  Install • Make  sure  you  have  XCode  &  Command  Line  Tools  Installed • Install  Homebrew  if  you  don't  have  it  already • Do  a  $  brew  update,  $  brew  upgrade  and  $  brew  doctor  to  be  sure  you're  up   to  date www.couchbase.com/communiUes/c/gekng-­‐started Tuesday, October 8, 13
  57. 57. Installing Libcouchbase • Mac  Via  Homebrew • $  brew  install  libcouchbase • PC-­‐Windows   • Download  appropriate  Zip  from  website • Redhat/CentOS • wget  the  yum  repositories • $  sudo  yum  install  -­‐y  libcouchbase2-­‐libevent  libcouchbase-­‐devel • Ubuntu • wget  ubuntu  repositories • $  sudo  apt-­‐get  install  libcouchbase2-­‐libevent  libcouchbase-­‐dev www.couchbase.com/communiUes/c/gekng-­‐started Tuesday, October 8, 13
  58. 58. More On Development Training:  Couchbase  102  -­‐  Developing www.couchbase.com/webinars Tuesday, October 8, 13
  59. 59. Tuesday, October 8, 13
  60. 60. SERVER  MONITORING Tuesday, October 8, 13
  61. 61. Store and Retrieve Operations Tuesday, October 8, 13
  62. 62. Store and Retrieve Operations Tuesday, October 8, 13
  63. 63. Store and Retrieve Operations Storage  Ops/sRetrieve  Ops/sRetrieve  off  DiskTotal  Ops/s Tuesday, October 8, 13
  64. 64. Store and Retrieve Operations Storage  Ops/sRetrieve  Ops/sRetrieve  off  DiskTotal  Ops/s Item  CountDelete  Ops/s CAS  Ops/s %  Documents  in  RAM Tuesday, October 8, 13
  65. 65. Store and Retrieve Operations Tuesday, October 8, 13
  66. 66. Store and Retrieve Operations RAM  FULL  +  Very   High  Velocity  Writes! Tuesday, October 8, 13
  67. 67. Store and Retrieve Operations RAM  FULL  +  Very   High  Velocity  Writes! RAM  Used  for   Metadata  +  Data Tuesday, October 8, 13
  68. 68. Store and Retrieve Operations RAM  FULL  +  Very   High  Velocity  Writes! Begin  EjecUon  of   Replicas  from  RAM Begin  EjecUon  of   AcUve  from  RAM RAM  Used  for   Metadata  +  Data Tuesday, October 8, 13
  69. 69. Store and Retrieve Operations Tuesday, October 8, 13
  70. 70. Store and Retrieve Operations Disk  Creates Disk  Updates Disk  Queue  Size Disk  Reads   (Cache  Miss) Tuesday, October 8, 13
  71. 71. Store and Retrieve Operations Disk  Creates Disk  Updates Disk  Queue  Size Disk  Reads   (Cache  Miss) Total  Disk  Usage Docs  Data  Size   (Compressed) Docs  Size  on  Disk FragmentaUon (from  Appends) Tuesday, October 8, 13
  72. 72. Store and Retrieve Operations Disk  Queue  Size Tuesday, October 8, 13
  73. 73. Store and Retrieve Operations Disk  Queue  Size Disk  Queue  Size (Need  More  IO!) Tuesday, October 8, 13
  74. 74. Partition (vbucket) Details Tuesday, October 8, 13
  75. 75. Partition (vbucket) Details Tuesday, October 8, 13
  76. 76. Partition (vbucket) Details AcUve  ParUUons Tuesday, October 8, 13
  77. 77. Partition (vbucket) Details AcUve  ParUUons Replica  ParUUons Tuesday, October 8, 13
  78. 78. Partition (vbucket) Details AcUve  ParUUons Replica  ParUUons Pending  ParUUons   (Rebalance) Tuesday, October 8, 13
  79. 79. Partition (vbucket) Details Tuesday, October 8, 13
  80. 80. Partition (vbucket) Details Item  Count Tuesday, October 8, 13
  81. 81. Partition (vbucket) Details %  Documents  in  RAM Item  Count Tuesday, October 8, 13
  82. 82. Partition (vbucket) Details %  Documents  in  RAM CreaUons Item  Count Tuesday, October 8, 13
  83. 83. Partition (vbucket) Details EjecUons  (RAM  Full) %  Documents  in  RAM CreaUons Item  Count Tuesday, October 8, 13
  84. 84. Partition (vbucket) Details Documents EjecUons  (RAM  Full) %  Documents  in  RAM CreaUons Item  Count Tuesday, October 8, 13
  85. 85. Partition (vbucket) Details Metadata Documents EjecUons  (RAM  Full) %  Documents  in  RAM CreaUons Item  Count Tuesday, October 8, 13
  86. 86. Partition (vbucket) Details Tuesday, October 8, 13
  87. 87. Partition (vbucket) Details Note:  64  Par66ons  on  Mac  OS  X!   This  is  simply  to  avoid  altering   default  file  descriptor  limits,   don't  XDCR  with  other  Opera6ng   Systems Tuesday, October 8, 13
  88. 88. Disk IO Information Tuesday, October 8, 13
  89. 89. Disk IO Information AcUve  ParUUons Tuesday, October 8, 13
  90. 90. Disk IO Information AcUve  ParUUons Replica  ParUUons Tuesday, October 8, 13
  91. 91. Disk IO Information AcUve  ParUUons Replica  ParUUons Pending  ParUUons   (Rebalance) Tuesday, October 8, 13
  92. 92. Disk IO Information Tuesday, October 8, 13
  93. 93. Disk IO Information Queue  Size Tuesday, October 8, 13
  94. 94. Disk IO Information Queue  Fill  Rate Queue  Size Tuesday, October 8, 13
  95. 95. Disk IO Information Queue  Fill  Rate Queue  Drain  Rate Queue  Size Tuesday, October 8, 13
  96. 96. Disk IO Information Average  Age  of   Queue  Items Queue  Fill  Rate Queue  Drain  Rate Queue  Size Tuesday, October 8, 13
  97. 97. Tuesday, October 8, 13
  98. 98. QUICK  DEMO Tuesday, October 8, 13
  99. 99. Tuesday, October 8, 13
  100. 100. Q  &  A Tuesday, October 8, 13
  101. 101. Focused  on  Beginning  Development This  Thursday! www.couchbase.com/webinars Tuesday, October 8, 13
  102. 102. COUCHBASE  102 Focused  on  Beginning  Development This  Thursday! www.couchbase.com/webinars Tuesday, October 8, 13

×