• Like

Dfw big-data-talk-couchbase-overview

  • 556 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • This was a great meetup experience. Talking to 'Big Data' guys about how to best present their analytics results. If Hadoop is OLAP, Couchbase is OLTP...an interactive application's run-time layer for working with data.
    Are you sure you want to
    Your message goes here
    Be the first to like this
No Downloads

Views

Total Views
556
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
9
Comments
1
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • CAPI interface – basic Couch API of which some goes through the caching layer (CRUD), some goes directly to Couch (Views)
  • GSM/Wireless Intelligence study, The Global Cellular Industry Balance Sheethttp://gigaom.com/mobile/global-mobile-revenues-will-be-1-1-trillion-in-2012-here-is-why/GigaOM: Mobile Internet user to eclipse wireline users by 2015

Transcript

  • 1. Perry Krug, Sr. Solutions Architect, Couchbase Inc. 1
  • 2. Couchbase - Company overview • Elastic Data Management Software – For web applications and cloud computing environments – Most mature, reliable and widely deployed NoSQL database solution – Fully featured, open source document datastore – Headquartered in Silicon Valley – Couchbase team has authored majority of memcached and CouchDB codebase Membase1 CouchOne1 CouchBase3 + =The market’s leading caching and The most reliable and full-featured The fastest, most complete and most clustering technology. document database. reliable NoSQL database on the planet. 2
  • 3. Product family spans device to data center Couchbase Mobile Couchbase Single Server Couchbase Server (iPhone and Android) (Single Node) (Multi-Node) Most full featured Simple, Fast, Elastic Document Database NoSQL Datastore 3
  • 4. Automated synchronization via CouchSync CouchSync CouchSync Couchbase Mobile Couchbase Couchbase Server (iPhone and Android) (Single Node) (Multi-Node) Most full featured Simple, Fast, Elastic Document Database NoSQL Datastore 4
  • 5. Paid Production Deployment Examples 5
  • 6. WHY NOSQL?“ Relational database technology has served us well for 40 years, and will likely continue to do so for the foreseeable future to support transactions requiring ACID guarantees. But a large, and increasingly dominant, class of software systems and data do not need those guarantees. Much of the data manipulated by Web applications have less strict transactional requirements but, for lack of a practical alternative, many IT teams continue to use relational technology, needlessly tolerating its cost and scalability limitations. For these applications and data, distributed document cache and database technologies such as Couchbase’s provide a promising alternative. ” Carl Olofson IDC Research Vice President, Information and Data Management 6
  • 7. Modern interactive software architecture Application Scales Out Just add more commodity web servers Database Scales Up Get a bigger, more complex server-Expensive and disruptive sharding-Doesn’t perform at Web Scale 7
  • 8. Things Have Changed• Number of users of apps growing rapidly – Was thousands, now often millions or more• Amount of data stored in apps growing rapidly – Was GBs, now often 1000s of GBs or more• Types of data stored in apps is different – Was structured, now often unstructured & user-generated – Many apps including social features to increase engagement• Data you want to store is changing rapidly – Fixed data model was okay, now its not flexible enough• High-speed networking is inexpensive – Central computing was okay, now distributed computing better 8
  • 9. Data Layer Matches Application Logic Tier Architecture Application Scales Out Just add more commodity web servers Database Scales Out Just add more commodity data servers • Horizontally scalable with auto-sharding • High performance at web scale • Schema-less for flexibility Scaling out flattens the cost and performance curves. 9
  • 10. Couchbase is Simple, Fast, Elastic NoSQL • Simple to: Elastic Couchbase – Deploy (Membase ServerTemplate) – Develop (memcached) – Manage (UI and RESTful API) • Fast: – Predictable low latency – Sub-ms response times – Built-in memcached technology • Zero-downtime Elasticity: – Spread I/O and data across instances – Consistent performance with linear cost – Dynamic rebalancing of a live cluster 10
  • 11. Couchbase Community Worldwide • Vibrant developer community • Full documentation and resources available • Worldwide training events and meet up groups • CouchConf – User Conference in San Francisco and other major cities 11
  • 12. CouchConf World Tourhttp://www.couchbase.com/couchconf-world-tour 12
  • 13. Couchbase: Leading Open Source NoSQL • Community Edition – Open source build – Free forum support • Enterprise Edition – Free for non-production use – Certified, QA tested version of open source – Case tracking and guaranteed SLA for production environments – Annual subscriptions starting at $999/node – Hourly cloud pricing via RightScale • NoSQL Consulting Services and QuickStart Services available 13
  • 14. Couchbase Server 2.0 = Simple. Fast. Elastic. 14
  • 15. Couchbase Server 2.0: Overview = Simple. Fast. Elastic. • Membase + CouchDB • Managed memory caching layer = super high performance • Clustering and online data redistribution (Rebalancing) • Indexing and Querying via JSON Map-Reduce • New SDK’s and client libraries • Developer Preview available today! • http://www.couchbase.com/downloads 15
  • 16. Couch API Moxi Database Operations Data Manager Membase EP Engine storage interface REST management API/Web UI http Heartbeat Couchbase Server 2.0 Architecture Process monitor Configuration manager on each node Global singleton supervisor Erlang/OTP Rebalance orchestrator Cluster Management Node health monitor vBucket state and replication manager Cluster Manager one per cluster16
  • 17. Couchbase Server 2.0 Architecture 5984 11211 11210 Couch View Memcapable 1.0 Memcapable 2.0 Moxi REST management API/Web UI vBucket state and replication manager Couch API Memcached Global singleton supervisor Rebalance orchestrator Configuration manager Node health monitor Process monitor Heartbeat Membase EP Engine Membase storage interface CouchDB http on each node one per cluster Erlang/OTP HTTP Erlang port mapper Distributed Erlang 8091 4369 21100 - 21199 17
  • 18. Clustering With Couchbase SET request arrives at KEY’s SET acknowledgement master server 1 2 returned to application 3 3 Listener-Sender RAM Couchbase storage engine 4 Disk Disk Disk Disk Disk DiskReplica Server 1 for KEY Master server for KEY Replica Server 2 for KEY 18
  • 19. Couchbase Client SDKsJava ClientSDK User Code.Net SDK Java client API CouchbaseClient cb = new CouchbaseClient(listURIs, "aBucket", "letmein"); // this is all the same as before cb.set("hello", 0, "world"); cb.get("hello"); spymemcached HTTP couchDB Map<String, Object> manyThings =PHP SDK Connection connection cb.getBulk(Collection<String> keys); /* accessing a view View view = cb.getView("design_document", "my_view"); Query query = new Query(); query.getRange("abegin", "theend");Ruby SDK Couchbase ServerPython SDK http://www.couchbase.org/code 19
  • 20. Partitioning The Data – vbucket map 20
  • 21. Basic Operation APP SERVER 1 APP SERVER 2  Docs distributed evenly across COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY servers in the cluster  Each server stores both active CLUSTER MAP CLUSTER MAP & replica docs  Only one server active at a time  Client library provides app with Read/Write/Update Read/Write/Update simple interface to database  Cluster map provides map to which server doc is on  App never needs to know SERVER 1 SERVER 2 SERVER 3  App reads, writes, updates Active Docs Active Docs Active Docs docs Doc 5 DOC Doc 4 DOC Doc 1 DOC  Multiple App Servers can Doc 2 DOC Doc 7 DOC Doc 3 DOC access same document at Doc 9 DOC Doc 8 DOC Doc 6 DOC same time Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 8 DOC Doc 2 DOC Doc 5 DOC COUCHBASE SERVER CLUSTERUser Configured Replica Count = 1 21
  • 22. Indexing and Querying APP SERVER 1 APP SERVER 2 APP SERVER 1 APP SERVER 2  Indexing work is distributed COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY amongst nodes  Large data set possible CLUSTER MAP MAP CLUSTER CLUSTER MAPMAP CLUSTER  Parallelize the effort  Each node has index for data stored on it Read/Write/Update  Queries combined the results Read/Write/Update from required nodes SERVER 1 SERVER 2 SERVER 3 Active Docs Active Docs Active Docs Doc 5 DOC Doc 4 DOC Doc 1 DOC Doc 2 DOC Doc 7 DOC Doc 3 DOC Doc 9 DOC Doc 8 DOC Doc 6 DOC Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 8 DOC Doc 2 DOC Doc 5 DOCUser Configured Replica Count = 1 22
  • 23. Add Nodes APP SERVER 1 APP SERVER 2  Two servers added to COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY cluster  One-click operation CLUSTER MAP CLUSTER MAP  Docs automatically rebalanced across cluster  Even distribution of docs Read/Write/Update Read/Write/Update  Minimum doc movement  Cluster map updated  App database calls now distributed over larger # SERVER 1 SERVER 2 SERVER 3 SERVER 4 SERVER 5 of servers Active Docs Active Docs Active Docs Active Docs Active Docs Active Docs Doc 5 DOC Doc 4 DOC Doc 1 DOC Doc 3 Doc 2 DOC Doc 7 DOC Doc 3 DOC Doc 6 Doc 9 DOC Doc 8 DOC Doc 6 DOC Replica Docs Replica Docs Replica Docs Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 7 Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 9 Doc 8 DOC Doc 2 DOC Doc 5 DOC COUCHBASE SERVER CLUSTERUser Configured Replica Count = 1 23
  • 24. Fail Over Node APP SERVER 1 APP SERVER 2  App servers happily accessing docs on Server 3 COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY  Server fails  App server requests to server 3 fail CLUSTER MAP CLUSTER MAP  Cluster detects server has failed  Promotes replicas of docs to active  Updates cluster map  App server requests for docs now go to appropriate server  Typically rebalance would follow SERVER 1 SERVER 2 SERVER 3 SERVER 4 SERVER 5 Active Docs Active Docs Active Docs Active Docs Active Docs Active Docs Doc 5 DOC Doc 4 DOC Doc 1 DOC Doc 9 DOC Doc 6 DOC Doc 3 Doc 2 DOC Doc 7 DOC Doc 3 Doc 8 DOC Doc 6 DOC Replica Docs Replica Docs Replica Docs Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 5 DOC Doc 8 DOC Doc 7 Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 2 DOC Doc 9 COUCHBASE SERVER CLUSTERUser Configured Replica Count = 1 24
  • 25. Cross Data Center ReplicationUS DATA CENTER EUROPE DATA CENTER ASIA DATA CENTER CouchSync CouchSync CouchSync  Want data close to user  Want multiple locations for disaster recovery  Can write to same document in all different regions & it will sync  FTW: Support for uni-directional and bi-directional replication. Too much detail? 25
  • 26. Simple Monitoring and Operations 26
  • 27. Couchbase and Hadoop Integration• Support large-scale analytics on application data by streaming data from Couchbase to Hadoop – Real-time integration using Flume – Batch integration using Sqoop• Examples – Various game statistics (e.g., monthly / daily / hourly rankings) – Analyze game patterns from users to enhance various game metrics Flume memcached TAP protocol listener/sender Sqoop engine interface Couchbase Storage Engine 2 27
  • 28. Couchbase Mobile• Couchbase on Mobile devices• Data available offline• Real-time multi- device synchronization • Including to the Cloud• iOS and Android support • SDKs provided (CouchCocoa, Ektorp)Check it out today at:http://www.couchbase.org/get/couchbase-mobile-for-ios/current 28
  • 29. COUCHBASE USE CASES 29
  • 30. Proven at small, and extra large scale• Leading cloud service (PAAS) • Social game leader – FarmVille, provider Mafia Wars, Café World• Over 65,000 hosted applications • Over 230 million monthly users• Membase Server serving over • Membase Server is the primary 2,000 Heroku customers database behind key Zynga properties 30
  • 31. Ad and offer targeting 40 milliseconds to respond with the decision. profiles, real time campaign 3 statistics 2 1 profiles, campaigns events 31
  • 32. Social and online gaming 1 2 3 32
  • 33. Cloud data services 1 4 PROVISIONING 2 PROVISIONING 3 33
  • 34. SQL query caching RDBMS 34
  • 35. Session state storage 35
  • 36. Mobile to cloud data synchronizationgroup.complete 36
  • 37. DEMO TIME 37
  • 38. Demo: The next big social gameDemo will show:-Multiple Couchbase servers under moderate load(10k-15k operations per second)-Dynamic scalability – adding nodes on the fly-Secondary indices – Generation, querying, updatingTake-away?Couchbase serves fast-changing data to users inreal-time, all while scaling without disruption. 38
  • 39. Demo: The next big social game 3 Objects (documents) within game: • Players • Monsters • Items Gameplay: • Players fight monsters • Monsters drop items • Players own items 39
  • 40. Player Document { "_id": "Keith4540", "_rev": "1-ab354009ce09f198c555b693e057adce", "jsonType": "player", "uuid": "35767d02-a958-4b83-8179-616816692de1", "name": "Keith4540", "hitpoints": 75, "experience": 663, "level": 4, "loggedIn": false } 40
  • 41. Item Document { "_id": "Katana_e5890c94-11c6-48-65746ce6c560", "_rev": "1-d6bbd5e814c32c66e22db2918a2efcd9", "jsonType": "item", "name": "Katana_e5890c94-11c6-65746ce6c560", "uuid": "e5890c94-11c6-4856-a7a6-65746ce6c560", "ownerId": "Dale9887" } Player “_id” 41
  • 42. Monster Document { "_id": "Bauchan9932", "_rev": "1-5c90be58be58134a0fc5e7db77dab5f2", "jsonType": "monster", "name": "Bauchan9932", "uuid": "d10dfc1b-0412-4140-b4ec-affdbf2aa5ec", "hitpoints": 370, "experienceWhenKilled": 52, "itemProbability": 0.5050581341872865 } 42
  • 43. GAME ON! 43
  • 44. THANK YOU! Get Couchbase Server 2.0 athttp://www.couchbase.com/downloads Give us feedback at: http://www.couchbase.org/forums 44
  • 45. 45