Developing with Java and Couchbase Server


Published on

NoSQL has garnered tremendous attention in recent years as the basis of architectures for building large linearly scalable systems. As a document oriented database, Couchbase Server presents the user with a simple approach to storing and retrieving data across a cluster.

Couchbase Server 2.0 has rich querying capabilities of the distributed database through views built atop an always updating incremental map-reduce. Accessing Couchbase Server is also very easy, as there are client SDKs for Java, .NET, PHP, Ruby and many other common languages. Currently in developer preview, Couchbase Server builds upon Apache CouchDB and Memcached.

This presentation gives quick overview of Couchbase NoSQL, the ecosystem and what it means to Java developers and implementers. Raghavan "Rags" Srinivas, Developer Advocate at Couchbase, will demonstrate with examples of how to use the Java APIs on Couchbase to drive home some of these concepts.

Published in: Technology

Developing with Java and Couchbase Server

  1. 1. A  Technical  Overview  of  Couchbase   using  Java   Raghavan  “Rags”  Srinivas   Developer  Advocate   Simple.  Fast.  Elas.c.   1  
  2. 2. Speaker  CredenKals  •  Developer  Advocate  at  Couchbase  working  with   developers  •  Speaker  at  JavaOne,  RSA  conferences,  Sun  Tech  Days,   JUGs  and  other  developer  conferences  •  Taught  undergrad  and  grad  courses  •  Technology  Evangelist  for  Java  and  Sun  Microsystems   for  10+  years  •  SKll  trying  to  understand  and  work  more  effecKvely  on   Java  and  distributed  systems  •  Philosophy:  “Be#er  to  have  an  unanswered  ques1on   than  a  unques1oned  answer”   2  
  3. 3. Agenda  •  Distributed  Systems  and  Performance  •  Couchbase  Server  •  GePng  Started  •  DemonstraKon  and  Case  Study  •  Resources  and  Summary   3  
  5. 5. Web Application Architecture and Performance Application Scales Out   Just add more commodity web   servers Load  Balancer   Web  Servers   RelaKonal     Database Scales Up   Get a bigger, more complex Database   server-­‐  Expensive  and  disrupKve  sharding  -­‐  Doesn’t  perform  at  Web  Scale   5  
  6. 6. A  Distributed  Hash  Table   ApplicaKon   set(key,  value)   get(key)   returns  value   Database   DData   Cluster   6  
  7. 7. Eight  Fallacies  of  Distributed  CompuKng  •  The  Network  is  reliable  •  Latency  is  zero  •  Bandwidth  is  infinite  •  The  Network  is  secure  •  Topology  doesn’t  change  •  There  is  one  administrator  •  Transport  cost  is  zero  •  The  Network  is  homogenous   7  
  8. 8. Couchbase data layer scales like application logic tier
Data layer now scales with linear cost and constant performance. Application Scales Out   Just add more commodity web servers Load  Balancer   Web  Servers   Couchbase    Servers   Database Scales Out   Just add more commodity data servers Horizontally  scalable,  schema-­‐less,  auto-­‐ sharding,  high-­‐performance  at  Web  Scale   Scaling out flattens the cost and performance curves. 8  
  9. 9. COUCHBASE  SERVER   9  
  10. 10. Couchbase  “write”  Data  Flow  –  applicaKon  view   User  acKon  results  in  the  need  to   change  the  VALUE  of  KEY   1   ApplicaKon  updates  key’s  VALUE,   2   performs  SET  operaKon     4   Couchbase  client  hashes  KEY,  idenKfies   3   KEY’s  master  server   SET  request  sent  over   network  to  master  server   5   Couchbase  replicates  KEY-­‐VALUE  pair,  caches  it   in  memory  and  stores  it  to  disk   10  
  11. 11. Couchbase  Data  Flow  –  under  the  hood   SET  request  arrives  at  KEY’s   SET  acknowledgement   master  server   1   3   returned  to  applicaKon   2   2   Listener-­‐Sender   RAM*   Couchbase  storage  engine   2   Disk   Disk   Disk   4   Disk   Disk   Disk  Replica  Server  1  for  KEY   Master  server  for  KEY   Replica  Server  2  for  KEY   11  
  12. 12. Proven  at  Small,  and  Extra  Large  Scale   •  Leading cloud service (PAAS) •  Social game leader – FarmVille, provider   Mafia Wars, Empires and Allies, •  Over 150,000 hosted applications   Café World, FishVille   •  Couchbase Server serving over •  Over 230 million monthly users   6,200 Heroku customers •  Couchbase Server is the primary database behind key Zynga properties 12  
  13. 13. GETTING  STARTED   13  
  14. 14. Couchbase  Client  Java  SDK   Java  Client   SDK   User  Code     List  <URI>  servers  =  new  ArrayList<URI>();   .Net  SDK   Java  client  API   servers.add(oneserver);   servers.add(twoserver);     CouchbaseClient  cb  =  new  CouchbaseClient(servers,   "aBucket",  "letmein");   spymemcached   HTTP  couchDB     PHP  SDK   ConnecKon   connecKon   //  this  is  all  the  same  as  before   cb.set("hello",  0,  "world");   cb.get("hello");   Map<String,  Object>  manyThings  =   cb.getBulk(CollecKon<String>  keys);     Ruby  SDK   //  accessing  a  view     Couchbase  Server   View  view  =  cb.getView("design_document",   "my_view");   Query  query  =  new  Query();   query.getRange("abegin",  "theend");     …   hvp://   14  
  15. 15. Client  at  RunKme:  Adding  a  node   Couchbase  Client   New  node     Cluster coming  online   Topology Update Couchbase   Couchbase   Couchbase   Couchbase   Couchbase   Server   Server   Server   Server   Server   Node   Node   Node   Node   Node   15  
  16. 16. Asynchronous  OperaKons  •  Set  OperaKon              // Do an asynchronous set! OperationFutureBoolean setOp = client.set(KEY, EXP_TIME, VALUE);!! // Block and check to see if our set succeeded! try {! if (setOp.get().booleanValue()) {! System.out.println(Set Succeeded);! } else {! System.err.println(Set failed: + ! setOp.getStatus().getMessage());! }! } catch (Exception e) {! System.err.println(Exception while doing set: +! e.getMessage());! }! 16  
  17. 17. OperaKons  Available  to  a  Client  of  Couchbase  Server  •  Store  OperaKons   –  Add:  Store  the  document  if  it  does  not  yet  exist   –  Set:  Store  the  document,  overwriKng  exisKng  if  necessary  •  Retrieve  OperaKons   –  Get:  Fetch  the  document   –  Get  and  touch:  Fetch  the  document  and  update  the  TTL   –  MulKget:  Fetch  mulKple  documents  at  the  same  Kme  •  Update  operaKons   –  Append/prepend:  Add  data  in  front  of  or  on  the  end  of  a  document   –  Delete:  Remove  the  document  from  the  store   –  Compare  and  Swap  (CAS):  Replace  the  current  document,  if  CAS  matches   –  Replace:  Replace  the  document  if  it  exists,  otherwise  do  not   –  Touch:  Update  the  TTL  for  a  document   17  
  18. 18. Metadata   Document  Metadata:   •  TTL   •  CAS  value   •  Flags  Q:  What  happens  to  a  document  persisted  aer  it’s  TTL?  A:  A  regular  background  job  will  remove  expired  documents.    Q:  How  do  I  use  flags?  A:  Frequently  used  to  idenKfy  data  type  or  other  avributes,  such  as  compression.    Behavior  varies  from  client  to  client.   18  
  19. 19. Distributed  System  Design:  Concurrency  Controls   Actor  1   Actor  2   •  Compare  and  Swap  OperaKons   –  Owen  referred  to  as  “CAS”   –  OpKmisKc  concurrency  control   Success   CAS  mismatch   –  Available  with  many  mutaKon   operaKons,  depending  on  client.   •  Get  with  Lock   Couchbase  Server   –  Owen  referred  to  as  “GETL”   –  PessimisKc  concurrency  control   –  Locks  have  a  short  TTL   A   –  Locks  released  with  CAS   operaKons   –  Useful  when  working  with  object   B   C   D   graphs   F   E   19  
  20. 20. DEMO   20  
  21. 21. Demo:  The  next  big  social  game  3  Objects  (documents)  within  game:   • Players   • Monsters   • Items    Gameplay:   • Players  fight  monsters   • Monsters  drop  items   • Players  own  items   21  
  22. 22. Player  Document  {    _id:  Keith4540,    _rev:  1-­‐ab354009ce09f198c555b693e057adce,    jsonType:  player,    uuid:  35767d02-­‐a958-­‐4b83-­‐8179-­‐616816692de1,    name:  Keith4540,    hitpoints:  75,    experience:  663,    level:  4,    loggedIn:  false  }     22  
  23. 23. Item  Document  {    _id:  Katana_e5890c94-­‐11c6-­‐48-­‐65746ce6c560,    _rev:  1-­‐d6bbd5e814c32c66e22db2918a2efcd9,    jsonType:  item,    name:  Katana_e5890c94-­‐11c6-­‐65746ce6c560,    uuid:  e5890c94-­‐11c6-­‐4856-­‐a7a6-­‐65746ce6c560,    ownerId:  Dale9887  }   Player  “_id”   23  
  24. 24. Monster  Document  {    _id:  Bauchan9932,    _rev:  1-­‐5c90be58be58134a0fc5e7db77dab5f2,    jsonType:  monster,    name:  Bauchan9932,    uuid:  d10dfc1b-­‐0412-­‐4140-­‐b4ec-­‐affdbf2aa5ec,    hitpoints:  370,    experienceWhenKilled:  52,  }   24  
  25. 25. GA M E O N! thanks  for  the  photo!  hvp://         2 25   25  
  26. 26. CASE  STUDY:  TRIBAL  CROSSING     26  
  27. 27. Tribal  Crossing:  RepresenKng  Game  Data  in  Couchbase  Use  case  -­‐  simple  farming  game:    •  A  player  can  have  a  variety  of  plants  on  their  farm.•  A  player  can  add  or  remove  plants  from  their  farm.•  A  Player  can  see  what  plants  are  on  another  players   farm.   27  
  28. 28. Tribal  Crossing:  Challenges   ●  Write-­‐heavy  requests –  Caching  does  not  help –  MySQL  /  InnoDB  limitaKon  (Percona) ●  Need  to  scale  drasKcally  over  night –  My  Polls  –  100  to  1m  users  over  a  weekend ●  Small  team,  no  dedicated  sysadmin –  Focus  on  what  we  do  best  –  making  games ●  Keeping  cost  down   28  
  29. 29. Tribal  Crossing:  RepresenKng  Game  Data  in  Couchbase  RepresenKng  Objects●  Simply  treat  an  object  as  a  Map/dicKonary/JSON  object●  Determine  the  key  for  an  object  using  the  class  name  (or   type)  of  the  object  and  an  unique  IDRepresenKng  Object  Lists●  DenormalizaKon●  Save  a  comma  separated  list  or  an  array  of  object  IDs   29  
  30. 30. Tribal  Crossing:  RepresenKng  Game  Data  in  Couchbase   Player  Object Key: Player1 Plant  Object JSON Key: Plant201 { “_id” : “Player1”, JSON “nid” : 1, { “name” : “Shawn” “_id” : “Plant201”, } “nid” : 201, “player_id” : 1PlayerPlant  List “name” : “Starflower”Key: Player1_PlantList }JSON{ “_id” : “Player1_Plantlist”, “plants” : [201, 202, 204]} 30  
  31. 31. Tribal  Crossing:  Schema-­‐less  Game  Data  ●  No  need  to  “ALTER  TABLE”●  Add  new  “fields”  all  objects  at  any  Kme –  Specify  default  value  for  missing  fields –  Increased  development  speed●  Using  JSON  for  data  objects  though,     ●  Offers  the  ability  to  query  and  analyze  arbitrary  fields  in   Couchbase  2.0   31  
  32. 32. Tribal  Crossing:  Accessing  Game  Data  in  Couchbase   Get  all  plants  that  belong  to  a  given  player Request: GET /player/1/farm // Create a new PlantList from players PlantList playersPlants = new PlantList(cbclient.get(Player1_PlantList); for (Plant plant : playersPlants) { aPlayer.addPlant(plant); 32  
  33. 33. Tribal  Crossing:  Modifying  Game  Data  in  Couchbase   Give  a  player  a  new  plant // Create the new plant Plant givenPlant = new Plant(100, Mushroom); cbclient.set(Plant100, givenPlant); // Update the player plant list Player thePlayer = player.fetch(cbclient.get(Player1); // Add the plant to the player thePlayer.receivePlant(givenPlant); // Store the players new plant list cbclient.set(Player1_PlantList, thePlayer.getPlantsArray()); 33  
  34. 34. Tribal  Crossing:  Conclusion  •  Significantly  reduced  the  cost  incurred  by  scaling  up   database  servers  and  managing  them.  •  Achieved  significant  improvements  in  various   performance  metrics  (e.g.,  read,  write,  latency,  etc.)  •  Allowed  focus  more  on  game  development  and   opKmizing  key  metrics  •  Plan  to  use  real-­‐Kme  Map-­‐Reduce,  querying,  and   indexing  abiliKes  provided  by  Couchbase  Server  2.0   34  
  35. 35. ROADMAP   35  
  36. 36. Couchbase  Server  2.0  •  Next  major  release  of  Couchbase  Server  •  Currently  in  Developer  Preview  What’s  new:  •  Indexing  and  Querying  •  Incremental  Map  Reduce  •  Cross  Data  Center  ReplicaKon  •  Fully  backwards  compaKble  with  exisKng   Couchbase  Server   36  
  37. 37. Accessing  a  View  in  Couchbase  Server  2.0   Query query = new Query(); query.setReduce(false); query.setIncludeDocs(true); View view = cbclient.getView(dev_players, player_by_level); HttpFutureViewResponse future = client.asyncQuery(view, query); ViewResponse response = future.get(); if (!future.getStatus().isSuccess()) { // do something to handle the errror } IteratorViewRow itr = response.iterator(); while (itr.hasNext()) { ViewRow row =; if (ITEMS.containsKey(row.getId())) { assert ITEMS.get(row.getId()).equals(row.getDocument()); } } 37  
  38. 38. RESOURCES  AND  SUMMARY   38  
  39. 39. Resources,  Summary  and  Call  For  AcKon  •  Couchbase  Server  Downloads   –  hvp://­‐all   –  hvp://­‐server/overview  •  Developing  with  Java  Client  libraries   –  hvp://  •  Couchbase  forums   –  hvp://  •  Sqoop-­‐based  Hadoop  Connector  for  Couchbase   –  hvp://  •  Couch  Conference  (coming  to  a  city  near  you)   –  hvp://­‐world-­‐tour  •  Tribal  Crossing  blog   –  hvp://­‐world-­‐nosql-­‐membase-­‐at-­‐tribal-­‐crossing.html       39  
  40. 40. THANKS  -­‐  QA  Raghavan “Rags” Srinivas, Couchbase Inc.  (, @ragss) 40