Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

London web performance-Couchbase meetup

572

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
572
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • CAPI interface – basic Couch API of which some goes through the caching layer (CRUD), some goes directly to Couch (Views)
  • CAPI interface – basic Couch API of which some goes through the caching layer (CRUD), some goes directly to Couch (Views)
  • CAPI interface – basic Couch API of which some goes through the caching layer (CRUD), some goes directly to Couch (Views)
  • Not yet enabled in current DP, will be available for Beta
  • Transcript

    • 1. CouchbaseDistributed Document Database Perry Krug Sr. Solutions Architect 1
    • 2. Company Background • Leading NoSQL database company • Open source development and distribution model • Provide easy-to-develop and -deploy, high-performance, easily scalable, document database • Focused on internet and mobile applications and cloud computing environments • Most mature, reliable and widely deployed solution – 1000s of production deployments worldwide • Located in Silicon Valley (Mountain View, CA) – 80 employees including 50 in engineering/product – Round C company, VCs includes Accel partners, Mayfield, North Bridge, and Ignition 2
    • 3. Paid Production Deployments (partial list) 3
    • 4. Couchbase Server (a.k.a. Membase) Simple. Fast. Elastic. NoSQL. Couchbase automatically distributes data across commodity servers. Built-in caching enables apps to read and write data with sub-millisecond latency. And with no schema to manage, Couchbase effortlessly accommodates changing data management requirements. 4
    • 5. “ Relational database technology has served us well for 40 years, and will likely continue to do so for the foreseeable future to support transactions requiring ACID guarantees. But a large, and increasingly dominant, class of software systems and data do not need those guarantees. Much of the data manipulated by Web applications have less strict transactional requirements but, for lack of a practical alternative, many IT teams continue to use relational technology, needlessly tolerating its cost and scalability limitations. For these applications and data, distributed document cache and database technologies such as Couchbase’s provide a promising alternative. ” Carl Olofson IDC Research Vice President, Information and Data Management RDBMS HAS DOMINATED FOR 40 YEARS BUT NO LONGER BEST SOLUTION FOR MANY APPS 5
    • 6. Modern Interactive Software Architecture Application Scales Out Just add more commodity web servers Database Scales Up Get a bigger, more complex server Expensive & disruptive sharding, doesn’t perform at web scale 6
    • 7. Data Layer Matches Application Logic TierArchitecture Application Scales Out Just add more commodity web servers Database Scales Out Just add more commodity data servers • Horizontally scalable with auto-sharding • High performance at web scale • Schema-less for flexibility Scaling out flattens the cost and performance curves 7
    • 8. Couchbase NoSQL: Simple, Fast, Elastic • Easily scale apps to an “infinite” number of users – Simply add nodes with a single click – Never need to change your application to scale – Simple development with memcached API • High performance with predictably low latency – Sub millisecond reads and writes – No drop in performance as app scales • Schema-less document database – Flexibility to meet rapidly changing market requirements – Roadmap: Indexing, querying similar to RDBMS capabilities • Low cost solution that economically scales with app 8
    • 9. PERFORMANCE 9
    • 10. Key results of Cisco and Solarflare BenchmarkCouchbase Server demonstrates• Consistent sub-millisecond latency for mixed workload• High throughput• Linear scalability http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-708169.pdf 10
    • 11. Your secret weapon: Sub-millisecond AND consistent latencyLatency (micro seconds) Consistently low latencies in microseconds for varying documents sizes with a mixed workload Object size (Bytes) 11
    • 12. Your secret weapon: Sub-millisecond AND consistent latency High throughput with 1.4 GB/sec data transfer rate using 4 serversOperations per second Linear throughput scalability Number of servers in cluster 12
    • 13. SCALE 13
    • 14. Draw Something by OMGPOP 14
    • 15. Draw Something “goes viral” 3 weeks after launch Draw Something by OMGPOP Daily Active Users (millions)161412108642 2/6 8 10 12 14 16 18 20 22 24 26 28 3/1 3 5 7 9 11 13 15 17 19 21 15
    • 16. As usage grew, game data went non-linear. Draw Something by OMGPOP Daily Active Users (millions) 16 14 12 10 8 By March 19, there were 6 over 30,000,000 downloads of the app, over 5,000 drawings being stored per second, 4 over 2,200,000,000 drawings stored,over 105,000 database transactions per second, Instagram (7.5M in 5 wks) 2 and over 3.3 terabytes of data stored. 2/6 8 10 12 14 16 18 20 22 24 26 28 3/1 3 5 7 9 11 13 15 17 19 21 16
    • 17. The game exploded. But Couchbase did not. February February February February March March March 6 13 20 27 5 12 19Drawings/second 0 3 50 333 1660 3000 5400 Total drawings 0 5 12 50 500 1 2.2 million million million million billion billion R/W latency (usec) 30 40 32 31 38 29 34 Servers 6 6 6 18 54 72 90 Without a second of downtime, and while sustaining front-end performance, the cluster was continuously expanded to support growth, absorbing frequent server hardware failures. 17
    • 18. In contrast. The Simpson’s: Tapped Out Daily Active Users (millions)16141210864 #2 Free app on iPad2 #3 Free app on iPhone 2/6 8 10 12 14 16 18 20 22 24 26 28 3/1 3 5 7 9 11 13 15 17 19 21 18
    • 19. NO SCHEMA 19
    • 20. Document database• Json objects• Flexible schema { "_id": "brewery_Cleveland_ChopHouse_and_Brewery", "_rev": "1-00000061480b50910000000000000000", "city": "Cleveland", "updated": "2010-07-22 20:00:20", "code": "44113", "name": "Cleveland ChopHouse and Brewery", "country": "United States", "phone": "1-216-623-0909", "state": "Ohio", "address": [ "824 West St.Clair Avenue" ], "geo": { "loc": [ "-81.6994", "41.4995" ], "accuracy": "ROOFTOP" }, "$expiration": 0, "$flags": 0 } 20
    • 21. COUCHBASE SOLUTION “THE BASICS” 21
    • 22. Basic Operation – scale out APP SERVER 1 APP SERVER 2  Docs distributed evenly across COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY servers in the cluster  Each server stores both active CLUSTER MAP CLUSTER MAP & replica docs  Only one server active at a time  Client library provides app with Read/Write/Update Read/Write/Update simple interface to database  Cluster map provides map to which server doc is on  App never needs to know SERVER 1 SERVER 2 SERVER 3  App reads, writes, updates Active Docs Active Docs Active Docs docs Doc 5 DOC Doc 4 DOC Doc 1 DOC  Multiple App Servers can Doc 2 DOC Doc 7 DOC Doc 3 DOC access same document at Doc 9 DOC Doc 8 DOC Doc 6 DOC same time Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 8 DOC Doc 2 DOC Doc 5 DOC COUCHBASE SERVER CLUSTERUser Configured Replica Count = 1 22
    • 23. Add Nodes APP SERVER 1 APP SERVER 2  Two servers added to COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY cluster  One-click operation CLUSTER MAP CLUSTER MAP  Docs automatically rebalanced across cluster  Even distribution of docs Read/Write/Update Read/Write/Update  Minimum doc movement  Cluster map updated  App database calls now distributed over larger # SERVER 1 SERVER 2 SERVER 3 SERVER 4 SERVER 5 of servers Active Docs Active Docs Active Docs Active Docs Active Docs Active Docs Doc 5 DOC Doc 4 DOC Doc 1 DOC Doc 3 Doc 2 DOC Doc 7 DOC Doc 3 DOC Doc 6 Doc 9 DOC Doc 8 DOC Doc 6 DOC Replica Docs Replica Docs Replica Docs Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 7 Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 9 Doc 8 DOC Doc 2 DOC Doc 5 DOC COUCHBASE SERVER CLUSTERUser Configured Replica Count = 1 23
    • 24. Fail Over Node APP SERVER 1 APP SERVER 2  App servers happily accessing docs on Server 3 COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY  Server fails  App server requests to server 3 fail CLUSTER MAP CLUSTER MAP  Cluster detects server has failed  Promotes replicas of docs to active  Updates cluster map  App server requests for docs now go to appropriate server  Typically rebalance would follow SERVER 1 SERVER 2 SERVER 3 SERVER 4 SERVER 5 Active Docs Active Docs Active Docs Active Docs Active Docs Active Docs Doc 5 DOC Doc 4 DOC Doc 1 DOC Doc 9 DOC Doc 6 DOC Doc 3 Doc 2 DOC Doc 7 DOC Doc 3 Doc 8 DOC Doc 6 DOC Replica Docs Replica Docs Replica Docs Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 5 DOC Doc 8 DOC Doc 7 Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 2 DOC Doc 9 COUCHBASE SERVER CLUSTERUser Configured Replica Count = 1 24
    • 25. Couchbase Server 2.0 • Next major release of Couchbase Server • Currently in Developer Preview, approaching Beta and GA. What’s new: • New storage engine technology (Append only b-tree) • Indexing and Querying • Incremental Map Reduce • Cross Data Center Replication • Better memory management, large data sets, and other technological improvments • Fully backwards compatible with existing Couchbase Server 25
    • 26. Couchbase Server 2.0 Architecture 8092 11211 11210 Couch View Memcapable 1.0 Memcapable 2.0 Moxi REST management API/Web UI vBucket state and replication manager Couch API Memcached Global singleton supervisor Rebalance orchestrator Configuration manager Node health monitor Process monitor Heartbeat Membase EP Engine Data Manager Cluster Manager Membase storage interface Distributed CouchStore Indexing Auto compaction http on each node one per cluster CouchBase Erlang/OTP HTTP Erlang port mapper Distributed Erlang 8091 4369 21100 - 21199 26
    • 27. Couchbase Server 2.0 Architecture 8092 11211 11210 Couch View Memcapable 1.0 Memcapable 2.0 Moxi REST management API/Web UI vBucket state and replication manager Memcached Interface Couch API Global singleton supervisor Rebalance orchestrator Configuration manager Node health monitor Process monitor Heartbeat Membase EP Engine Cluster Manager Membase storage interface Distributed CouchStore Indexing Auto compaction http on each node one per cluster CouchBase Erlang/OTP HTTP Erlang port mapper Distributed Erlang 8091 4369 21100 - 21199 27
    • 28. Couchbase Server 2.0 Architecture 8092 11211 11210 Couch View Memcapable 1.0 Memcapable 2.0 Moxi REST management API/Web UI vBucket state and replication manager Memcached Interface Couch API Global singleton supervisor Rebalance orchestrator Configuration manager Node health monitor Process monitor Heartbeat EP Engine storage interface Distributed CouchStore Indexing Auto compaction http on each node one per cluster CouchBase Erlang/OTP HTTP Erlang port mapper Distributed Erlang 8091 4369 21100 - 21199 28
    • 29. Partitioning The Data – vbucket map 29
    • 30. Indexing and querying • Build in incremental map reduce • Map functions are written and executed on Java Script (V8) • Index is built incrementally as mutation streams in • Query in a scatter/gather fashion 30
    • 31. Incremental Map reduce using javascript• Document from our sample built in beer database { "_id": "brewery_Cleveland_ChopHouse_and_Brewery", "_rev": "1-00000061480b50910000000000000000", "city": "Cleveland", "updated": "2010-07-22 20:00:20", "code": "44113", "name": "Cleveland ChopHouse and Brewery", "country": "United States", "phone": "1-216-623-0909", "state": "Ohio", "address": [ "824 West St.Clair Avenue" ], "geo": { "loc": [ "-81.6994", "41.4995" ], "accuracy": "ROOFTOP" }, "$expiration": 0, "$flags": 0 } 31
    • 32. Map function • Map functionsfunction (doc) { if (doc.country, doc.state, doc.city) { emit([doc.country, doc.state, doc.city], 1); } else if (doc.country, doc.state) { emit([doc.country, doc.state], 1); } else if (doc.country) { emit([doc.country], 1); }} REST call: http://db1.couchbase.com:8092/beer-sample/_design/dev_beer/_view/by_location?limit=10 32
    • 33. Reduce functions• Built in reduce functions • _count • _sum • _stats ({“sum”: 1411, “count”: 1411, “min”: 1, “max”: 1, “sumsqr”:1411})• Developing procedure • Develop against a subset of the data • Built the index on the entire cluster • Promote a dev_ view to production 33
    • 34. Indexing and Querying APP SERVER 1 APP SERVER 2 APP SERVER 1 APP SERVER 2  Indexing work is distributed COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY COUCHBASE CLIENT LIBRARY amongst nodes  Large data set possible CLUSTER MAP MAP CLUSTER CLUSTER MAPMAP CLUSTER  Parallelize the effort  Each node has index for data stored on it Query Response  Queries combine the results from required nodes SERVER 1 SERVER 2 SERVER 3 Active Docs Active Docs Active Docs Doc 5 DOC Doc 4 DOC Doc 1 DOC Doc 2 DOC Doc 7 DOC Doc 3 DOC Doc 9 DOC Doc 8 DOC Doc 6 DOC Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 8 DOC Doc 2 DOC Doc 5 DOCUser Configured Replica Count = 1 34
    • 35. Cross Data Center Replication US DATA EUROPE DATA ASIA DATA CENTER CENTER CENTER Replication Replication Replication  Data close to users  Multiple locations for disaster recovery  Independently managed clusters serving local data 35
    • 36. Integration to Analytics systems COUCHBASE SERVER CLUSTER  Use the cross data center SERVER 1 SERVER 2 SERVER 3 interface Active Docs Active Docs Active Docs  Agnostic to topology changes Doc 5 DOC Doc 4 DOC Doc 1 DOC  De-duplication Doc 2 DOC Doc 7 DOC Doc 3 DOC  Effective changes feed of the Doc 9 DOC Doc 8 DOC Doc 6 DOC entire cluster Replica Docs Replica Docs Replica Docs Doc 4 DOC Doc 6 DOC Doc 7 DOC Doc 1 DOC Doc 3 DOC Doc 9 DOC Doc 8 DOC Doc 2 DOC Doc 5 DOC CROSS DATA CENTER CONNETROR Changes feed to consumed by Any destinationUser Configured Replica Count = 1 36
    • 37. Couchbase and Hadoop Integration• Support large-scale analytics on application data by streaming data from Couchbase to Hadoop – Real-time integration using Flume – Batch integration using Sqoop• Examples – Various game statistics (e.g., monthly / daily / hourly rankings) – Analyze game patterns from users to enhance various game metrics memcached Sqoop TAP protocol listener/sender engine interface Couchbase Storage Engine 37
    • 38. Couchbase Client SDKsJava ClientSDK User Code.Net SDK Java client API CouchbaseClient cb = new CouchbaseClient(listURIs, "aBucket", "letmein"); // this is all the same as before cb.set("hello", 0, "world"); cb.get("hello"); spymemcached HTTP couchDB Map<String, Object> manyThings =PHP SDK Connection connection cb.getBulk(Collection<String> keys); /* accessing a view View view = cb.getView("design_document", "my_view"); Query query = new Query(); query.getRange("abegin", "theend");Ruby SDK Couchbase ServerPython SDK http://www.couchbase.org/code 38
    • 39. Couchbase Demonstration• Couchbase ServerTemplate Demo – Starting with one database node under load Application user – Dynamically scaling to two database nodes – Easy management and Web application server monitoring – Not possible any other Couchbase Servers database technology In the EC2 or Datacenter 39
    • 40. THANK YOU COUCHBASE SIMPLE, FAST, ELASTIC NOSQL QUESTIONS?@couchbase perry@couchbase.com 40
    • 41. COUCHBASE CUSTOMERS 41
    • 42. Paid Production Deployments – Social Gaming iki 42
    • 43. Paid Production Deployments – Key SegmentsAd PlatformsSocial Networks 43
    • 44. Production Deployments – Key Industry SegmentsOnline Biz Services E-CommerceOnline Media 44
    • 45. Production Deployments – Key Industry Segments HealthCare Military/ GovernmentCommunications 45
    • 46. Production Deployments – Key Industry Segments Online Education Web Design Financial Services 46
    • 47. Production Deployments – Key Industry Segments Software Security 47
    • 48. Production Deployments – Enterprises 48

    ×