How	
  Companies	
  use	
  	
  
NoSQL	
  and	
  Couchbase	
  	
  
Dip7	
  Borkar	
  
Director,	
  Product	
  Management	
  
Couchbase	
  Server	
  
NoSQL	
  Document	
  Database	
  
Couchbase	
  Open	
  Source	
  Project	
  
•  Leading	
  NoSQL	
  database	
  project	
  
focused	
  on	
  distributed	
  ...
OLTP	
  
Analy7cs	
  
2	
  kinds	
  of	
  database	
  management	
  system	
  	
  
	
  
NoSQL	
  +	
  Big	
  Data	
  
Map-­‐reduce	
  against	
  
huge	
  datasets	
  to	
  
analyze	
  and	
  find	
  insights	
  ...
Common	
  Use	
  Cases	
  
Social	
  Gaming	
  
	
  
•  Couchbase	
  stores	
  
player	
  and	
  game	
  
data	
  	
  
•  ...
• Applica7on	
  objects	
  
• Popular	
  search	
  query	
  
results	
  
• Session	
  informa7on	
  
• Heavily	
  accessed...
• Session	
  values	
  or	
  Cookies	
  
(stored	
  as	
  key-­‐value	
  pairs)	
  
• Examples	
  include:	
  items	
  in	...
User	
  ID	
  /	
  Profile	
  Store	
  
Use	
  Case:	
  Globally	
  Distributed	
  User	
  Profile	
  Store	
  
• Extremely	...
• Content	
  metadata	
  
• Content:	
  Ar7cles,	
  text	
  	
  
• Landing	
  pages	
  for	
  website	
  
• Digital	
  con...
• Social	
  media	
  feeds:	
  Twiger,	
  
Facebook,	
  LinkedIn	
  
• Blogs,	
  news,	
  press	
  ar7cles	
  
• Data	
  s...
Common	
  Use	
  Cases	
  
Social	
  Gaming	
  
	
  
•  Couchbase	
  stores	
  
player	
  and	
  game	
  
data	
  	
  
•  ...
McGraw	
  Hill	
  Educa7on	
  Labs	
  	
  
Learning	
  portal	
  
	
  
Use	
  Case:	
  Content	
  and	
  metadata	
  store	
  
Building	
  a	
  self-­‐adapAng,	
  
interacAve	
  learning	
  por...
As learning move online in great numbers
Growing need to build interactive learning environments that
Scale!!
Scale	
  to	...
• Allow	
  for	
  elasAc	
  scaling	
  under	
  spike	
  periods	
  
• Ability	
  to	
  catalog	
  &	
  deliver	
  content...
The Learning Portal	
  
•  Designed and built as a
collaboration between MHE Labs
and Couchbase
•  Serves as proof-of-conc...
•  Document	
  Modeling	
  
•  Metadata	
  &	
  Content	
  Storage	
  
•  View	
  Querying	
  to	
  support	
  Content	
  ...
Couchbase	
  2.0	
  	
  	
  	
  	
  +	
  	
  	
  	
  	
  	
  Elas7csearch	
  
Store	
  full-text articles	
  as	
  well	
 ...
Data	
  Model	
  
Content Metadata 
Bucket
User Profiles
Bucket
Content Stats
Bucket
•  Stores content metadata for
media o...
Architecture	
  
• Social	
  media	
  feeds:	
  Twiger,	
  
Facebook,	
  LinkedIn	
  
• Blogs,	
  news,	
  press	
  ar7cles	
  
• Data	
  s...
LivePerson	
  –	
  Real	
  7me	
  visitor	
  
engagement	
  
Use	
  Case:	
  3rd	
  party	
  data	
  aggrega7on	
  with	
  
analy7cs	
  
Real	
  Ame	
  AnalyAcs	
  for	
  
LivePerson'...
LivePerson:	
  Leading	
  customer	
  engagement	
  
plaeorm	
  
Requirements Requirements Requirements
•  High	
  throughput,	
  really	
  fast	
  
•  Linear	
  scale	
  	
  
•  Searchab...
Couchbase	
  Java	
  SDK	
  
ApplicaAon	
  server	
  
Tomcat	
  
M/R	
  views	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  ...
Visitor	
  
Stream	
  Event	
  
Processing	
  
Visitor	
  Feed	
  -­‐	
  
Storm	
  
Topology	
  
Customer	
  
RepresentaAv...
Document	
  Structurestructure	
  
{
"accountId": "64302875",
"id": 121640710013,
"rtSessionId": "643028754295878498",
"ev...
Couchbase	
  Server	
  
Easy	
  
Scalability	
  
Consistent	
  High	
  
Performance	
  
Always	
  On	
  
24x365	
  
Grow	
  cluster	
  without	
  ...
Features	
  in	
  Couchbase	
  Server	
  2.0	
  
JSON	
  support	
   Indexing	
  and	
  Querying	
  
Cross	
  data	
  cent...
Addi7onal	
  Features	
  
Built-­‐in	
  clustering	
  –	
  All	
  nodes	
  equal	
  
	
  
Data	
  replicaAon	
  with	
  au...
Ques7ons?	
  
Thank	
  you!	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
dip7@couchbase.com	
  
How companies use NoSQL and Couchbase - NoSQL Now 2013
Upcoming SlideShare
Loading in …5
×

How companies use NoSQL and Couchbase - NoSQL Now 2013

2,790 views

Published on

My presentation at NoSQL Now 2013

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,790
On SlideShare
0
From Embeds
0
Number of Embeds
287
Actions
Shares
0
Downloads
46
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

How companies use NoSQL and Couchbase - NoSQL Now 2013

  1. 1. How  Companies  use     NoSQL  and  Couchbase     Dip7  Borkar   Director,  Product  Management  
  2. 2. Couchbase  Server   NoSQL  Document  Database  
  3. 3. Couchbase  Open  Source  Project   •  Leading  NoSQL  database  project   focused  on  distributed  database   technology  and  surrounding   ecosystem   •  Supports  both  key-­‐value  and   document-­‐oriented  use  cases   •  All  components  are  available   under  the  Apache  2.0  Public   License   •  Obtained  as  packaged  so?ware  in   both  enterprise  and  community   ediAons.   Couchbase Open Source Project
  4. 4. OLTP   Analy7cs   2  kinds  of  database  management  system      
  5. 5. NoSQL  +  Big  Data   Map-­‐reduce  against   huge  datasets  to   analyze  and  find  insights   and  answers   Opera7onal  database  for   web  and  mobile  apps  with   high  performance  at  scale  
  6. 6. Common  Use  Cases   Social  Gaming     •  Couchbase  stores   player  and  game   data     •  Examples   customers  include:   Zynga   •  Tapjoy,  Ubiso?,   Tencent       Mobile  Apps     •  Couchbase  stores  user   info  and  app  content   •  Examples  customers   include:  Kobo,  PlayAka         Ad  Targe7ng     •  Couchbase  stores   user  informaAon  for   fast  access   •  Examples  customers   include:  AOL,   Mediamind,   Convertro       Session  store     •  Couchbase  Server  as  a  key-­‐ value  store   •  Examples  customers  include:   Concur,  Sabre     User  Profile  Store     •  Couchbase  Server  as  a   key-­‐value  store   •  Examples  customers   include:  Tunewiki    High  availability  cache     •  Couchbase  Server  used  as  a  cache  Aer  replacement   •  Examples  customers  include:  Orbitz   Content  &  Metadata   Store   •  Couchbase  document  store   with  ElasAc  Search   •  Examples  customers   include:  McGraw  Hill       3rd  party  data    aggrega7on       •  Couchbase  stores  social  media  and   data  feeds   •  Examples  customers  include:   LivePerson    
  7. 7. • Applica7on  objects   • Popular  search  query   results   • Session  informa7on   • Heavily  accessed  web   landing  pages   High  availability  caching   Use  Case:  High-­‐Availability  Caching   • Speed  up  RDBMS   • Consistently  low  response  7mes   for  document  /  key  lookups   • High-­‐availability  24x7x365   • Replacement  for  en7re  caching   7er     • Low  latency  in  sub-­‐milliseconds  with  consistently  high  read  /   write  throughput  using  built-­‐in  cache   • Always-­‐on  opera7ons  even  for  database  upgrades  and   maintenance  with  zero  down  7me   • memcached  compa7bility  for  easy  migra7on  to  Couchbase   without  any  applica7on  changes   • High  availability  and  disaster  replica7on  with  intra-­‐cluster  and   cross-­‐cluster  replica7on  (XDCR)   Data  Cached  in  Couchbase   Applica7on  Requirements   Why  NoSQL  and  Couchbase     RDBMS   Applica7on  Layer   User  Requests   Cache     Misses     and     Write   Requests   Read-­‐Write   Requests   Couchbase  Distributed  Cache   Key  Ver7cals     -­‐  E-­‐commerce                -­‐  Travel   -­‐  High-­‐tech      -­‐  HR  (ADP)  
  8. 8. • Session  values  or  Cookies   (stored  as  key-­‐value  pairs)   • Examples  include:  items  in  a   shopping  cart,  flights   selected,  search  results,  etc.   Session  Store   Use  Case:  Session  Store   • Extremely  fast  access  to  session   data  using  unique  session  ID   • Easy  scalability  to  handle  fast   growing  number  of  users  and   user-­‐generated  data   • Always-­‐on  func7onality  for   global  user  base   • Low  latency  in  sub-­‐milliseconds  with  consistently  high  read  /   write  throughput  for  session  data  via  the  built-­‐in  object-­‐level   cache   • Linear  throughput  scalability  to  grow  the  database  as  user  and   data  volume  grow   • Always-­‐on  opera7ons  even  par7cularly  high  availability  using   Couchbase  replica7on  and  failover   • Intra  cluster  and  cross  cluster  (XDCR)  replica7on  for  globally   distributed  ac7ve-­‐ac7ve  plaeorm   Data  Stored  in  Couchbase   Applica7on  Requirements   Why  NoSQL  and  Couchbase   Key  Ver7cals     •  Ad  Targe7ng   •  Travel   •  E-­‐commerce  
  9. 9. User  ID  /  Profile  Store   Use  Case:  Globally  Distributed  User  Profile  Store   • Extremely  fast  access  to   individual  profiles   • Always  online  system  as  mul7ple   applica7ons  access  user  profiles   • Flexibility  to  add  and  update   user  agributes   • Easy  scalability  to  handle  fast   growing  number  of  users     • Low  latency  and  high  throughput  for  very  quick  lookups  for   millions  of  concurrent  users  using  built-­‐in  cache   • Intra  cluster  and  cross  cluster  (XDCR)  replica7on  for  high   availability  and  disaster  recovery   • Ac7ve-­‐ac7ve  geo-­‐distributed  system  to  handle  globally   distributed  user  base     • Online  admin  opera7ons  eliminate  system  down7me   Data  Stored  in  Couchbase   Applica7on  Requirements   Why  NoSQL  and  Couchbase     • User  profile  with  unique  ID   • User  sehng  /  preferences   • User’s  network   • User  applica7on  state   Key  Ver7cals     -­‐  E-­‐commerce                -­‐  Social  gaming   -­‐  Sojware  as  a  service  
  10. 10. • Content  metadata   • Content:  Ar7cles,  text     • Landing  pages  for  website   • Digital  content:  eBooks,   magazine,  research  material     Content  and  Metadata  Store   Use  Case:  Content  and  Metadata  Store   •  Flexibility  to  store  any  kind  of   content   •  Fast  access  to  content  metadata   (most  accessed  objects)  and   content     •  Full-­‐text  Search  across  data  set   •  Scales  horizontally  as  more  content   gets  added  to  the  system   • Fast  access  to  metadata  and  content  via  object-­‐managed  cache   • JSON  provides  schema  flexibility  to  store  all  types  of  content  and   metadata   • Indexing  and  querying  provides  real-­‐7me  analy7cs  capabili7es   across  dataset     • Integra7on  with  Elas7cSearch  for  full-­‐text  search   • Ease  of  scalability  ensures  that  the  data  cluster  can  be  grown   seamlessly  as  the  amount  of  user  and  ad  data  grows   Data  Stored  in  Couchbase   Applica7on  Requirements   Why  NoSQL  and  Couchbase     Key  Ver7cals     -­‐  Media  &  Publishing    -­‐  High-­‐tech   -­‐   Social            -­‐  Fin  serv.  
  11. 11. • Social  media  feeds:  Twiger,   Facebook,  LinkedIn   • Blogs,  news,  press  ar7cles   • Data  service  feeds:   Hoovers,  Reuters   • Data  form  other  systems   Data  Aggrega7on   Use  Case:  Data  Aggrega7on   • Flexibility  to  store  any  kind  of   content   • Flexibility  to  handle  schema   changes     • Full-­‐text  Search  across  data  set   • High  speed  data  inges7on   • Scales  horizontally  as  more  content   gets  added  to  the  system   • JSON  provides  schema  flexibility  to  store  all  types  of  content  and   metadata   • Fast  access  to  individual  documents  via  built-­‐in  cache,  high  write   throughput   • Indexing  and  querying  provides  real-­‐7me  analy7cs  capabili7es  across   dataset     • Integra7on  with  Elas7cSearch  for  full-­‐text  search   • Ease  of  scalability  ensures  that  the  data  cluster  can  be  grown   seamlessly  as  the  amount  of  user  and  ad  data  grows   Data  Stored  in  Couchbase   Applica7on  Requirements   Why  NoSQL  and  Couchbase     Key  Ver7cals     -­‐  Ad  targe7ng        -­‐  High-­‐tech   -­‐  Media  &  Publishing        
  12. 12. Common  Use  Cases   Social  Gaming     •  Couchbase  stores   player  and  game   data     •  Examples   customers  include:   Zynga   •  Tapjoy,  Ubiso?,   Tencent       Mobile  Apps     •  Couchbase  stores  user   info  and  app  content   •  Examples  customers   include:  Kobo,  PlayAka         Ad  Targe7ng     •  Couchbase  stores   user  informaAon  for   fast  access   •  Examples  customers   include:  AOL,   Mediamind,   Convertro       Session  store     •  Couchbase  Server  as  a  key-­‐ value  store   •  Examples  customers  include:   Concur,  Sabre     User  Profile  Store     •  Couchbase  Server  as  a   key-­‐value  store   •  Examples  customers   include:  Tunewiki    High  availability  cache     •  Couchbase  Server  used  as  a  cache  Aer  replacement   •  Examples  customers  include:  Orbitz   Content  &  Metadata   Store   •  Couchbase  document  store   with  ElasAc  Search   •  Examples  customers   include:  McGraw  Hill       3rd  party  data    aggrega7on       •  Couchbase  stores  social  media  and   data  feeds   •  Examples  customers  include:   LivePerson    
  13. 13. McGraw  Hill  Educa7on  Labs     Learning  portal    
  14. 14. Use  Case:  Content  and  metadata  store   Building  a  self-­‐adapAng,   interacAve  learning  portal  with   Couchbase  
  15. 15. As learning move online in great numbers Growing need to build interactive learning environments that Scale!! Scale  to  millions  of   learners   Serve  MHE  as  well  as  third-­‐party   content   Including   open  content   Support   learning  apps   0101001001 1101010101 0101001010 101010   Self-­‐adapt  via   usage  data   The Problem  
  16. 16. • Allow  for  elasAc  scaling  under  spike  periods   • Ability  to  catalog  &  deliver  content  from  many   sources   • Consistent  low-­‐latency  for  metadata  and  stats  access   • Require  full-­‐text  search  support  for  content   discovery   • Offer  tunable  content  ranking  &  recommendaAon   funcAons     Backend is an Interactive Content Delivery Cloud that must: XML  Databases   SQL/MR  Engines   In-­‐memory  Data  Grids   Enterprise  Search  Servers   Experimented with a combination of: The Challenge  
  17. 17. The Learning Portal   •  Designed and built as a collaboration between MHE Labs and Couchbase •  Serves as proof-of-concept and testing harness for Couchbase + ElasticSearch integration •  Available for download and further development as open source code
  18. 18. •  Document  Modeling   •  Metadata  &  Content  Storage   •  View  Querying  to  support  Content  Browsing   •  ElasAc  Search  IntegraAon  (Full  Text  Search)   -­‐  Content  Updated  in  near  Real-­‐Time   -­‐  Search  Content  Summaries   -­‐  Relevancy  boosted  based  on  User  Preferences   •  Real-­‐Time  Content  Updates   •  Event  Logging  for  offline  analysis   Techniques  Used  
  19. 19. Couchbase  2.0          +            Elas7csearch   Store  full-text articles  as  well   as  document metadata  for   image,  video  and  text  content  in   Couchbase   Combine  user  preferences   staAsAcs  with  custom relevancy scoring  to  provide   personalized search results Logs  user behavior  to  calculate   user  preference  staAsAcs  (e.g.   video  >  text)   1   2   4   ConAnuously  accept updates from  Couchbase  with  new   content  &  stats   3  
  20. 20. Data  Model   Content Metadata Bucket User Profiles Bucket Content Stats Bucket •  Stores content metadata for media objects and content for articles •  Includes tags, contributors, type information •  Includes pointer to the media •  Stores user view details per type •  Updated every time a user views a doc with running count •  To be used for customizing ES search results per user preference •  Stores content view details •  Updated for every time a document is viewed •  To be used for boosting ES search results based on popularity
  21. 21. Architecture  
  22. 22. • Social  media  feeds:  Twiger,   Facebook,  LinkedIn   • Blogs,  news,  press  ar7cles   • Data  service  feeds:   Hoovers,  Reuters   3rd  Party  Data  Aggrega7on   Use  Case:  3rd  party  data  aggrega7on   • Flexibility  to  store  any  kind  of   content   • Flexibility  to  handle  schema   changes     • Full-­‐text  Search  across  data  set   • High  speed  data  inges7on   • Scales  horizontally  as  more  content   gets  added  to  the  system   • JSON  provides  schema  flexibility  to  store  all  types  of  content  and   metadata   • Fast  access  to  individual  documents  via  built-­‐in  cache,  high  write   throughput   • Indexing  and  querying  provides  real-­‐7me  analy7cs  capabili7es  across   dataset     • Integra7on  with  Elas7cSearch  for  full-­‐text  search   • Ease  of  scalability  ensures  that  the  data  cluster  can  be  grown   seamlessly  as  the  amount  of  user  and  ad  data  grows   Types  of  Data   Applica7on  Requirements   Why  NoSQL  and  Couchbase    
  23. 23. LivePerson  –  Real  7me  visitor   engagement  
  24. 24. Use  Case:  3rd  party  data  aggrega7on  with   analy7cs   Real  Ame  AnalyAcs  for   LivePerson's  customers   LiveEngage  DASHBOARD  
  25. 25. LivePerson:  Leading  customer  engagement   plaeorm  
  26. 26. Requirements Requirements Requirements •  High  throughput,  really  fast   •  Linear  scale     •  Searchable  (Views  and  M/R)     •  Supports  both  K/V  &  Document  store   •  Cross  data  center  replicaAon   •  “Always  on”,  Resilience  soluAon     The Problem   13   TB   per  month   ~1  PB   In  total   1.8   B   Visits  per  month   VOLUME  
  27. 27. Couchbase  Java  SDK   ApplicaAon  server   Tomcat   M/R  views                                  cluster   M/R  views                                  cluster   XDCR   REST  API   Couchbase  Java  SDK   Storm  Topology   Couchbase  Java  SDK   Storm  Topology   Architecture  
  28. 28. Visitor   Stream  Event   Processing   Visitor  Feed  -­‐   Storm   Topology   Customer   RepresentaAve   Kala   Couchbase   Visitor   Monitoring     Service   (1)  Visitor   browsing   (2)   Visitor   events   (4)  Write   event  to   user   document   (6)  Return   relevant   visitors   (7)  Return   relevant   visitors   (5)  Get   visitors   List   Every  3   sec    Visitor  Feed   API   (3)  Analyze   relevant  events   and  persist     Data  flow  
  29. 29. Document  Structurestructure   { "accountId": "64302875", "id": 121640710013, "rtSessionId": "643028754295878498", "eventSequence": 5104, "ipAddress": { "fieldValue": "194.39.63.10", "seq": 1 }, "browser": { "fieldValue": "Chrome 27.0.1453.116", "seq": 1 }, "state": { "fieldValue": "LEFT_SITE", "seq": 5104 } ...................................... } MulA  tenant   DB     Basic  visitor   informaAon         Sequence   use  due  to   Kala    
  30. 30. Couchbase  Server  
  31. 31. Easy   Scalability   Consistent  High   Performance   Always  On   24x365   Grow  cluster  without   applicaAon  changes,  without   downAme  with  a  single  click   Consistent  sub-­‐millisecond     read  and  write  response  Ames     with  consistent  high  throughput   No  downAme  for  so?ware   upgrades,  hardware   maintenance,  etc.   JSON JSON JSON JSONJSON PERFORMANCE Flexible  Data   Model   JSON  document  model  with   no  fixed  schema.   Couchbase  Server  
  32. 32. Features  in  Couchbase  Server  2.0   JSON  support   Indexing  and  Querying   Cross  data  center  replica7on  Incremental  Map  Reduce   JSON JSON JSON JSONJSON
  33. 33. Addi7onal  Features   Built-­‐in  clustering  –  All  nodes  equal     Data  replicaAon  with  auto-­‐failover     Zero-­‐downAme  maintenance       Built-­‐in  managed  cached       Append-­‐only  storage  layer     Online  compacAon     Monitoring  and  admin  API  &  UI     SDK  for  a  variety  of  languages  
  34. 34. Ques7ons?  
  35. 35. Thank  you!                   dip7@couchbase.com  

×