• Like


Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Membase East Coast Meetups


These are the slides from the Membase meetups in NYC and Boston.

These are the slides from the Membase meetups in NYC and Boston.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Boston
  • 2. Tonight • Membase Overview • Use Cases and Deployment Examples • Membase Architecture • Demo! • Developing with Membase • A Glimpse into the Future 2
  • 3. What is Membase?
  • 4. Membase is a distributed database Application user Web application server Web application server Web application server Membase Servers In the data center On the administrator console 4
  • 5. Membase is Simple, Fast, Elastic Five minutes or less to a working cluster • Downloads for Linux and Windows • Start with a single node • One button press joins nodes to a cluster Easy to develop against • Just SET and GET – no schema required • Drop it in. 10,000+ existing applications already “speak membase” (via memcached) • Practically every language and application framework is supported, out of the box Easy to manage • One-click failover and cluster rebalancing • Graphical and programmatic interfaces • Configurable alerting 5
  • 6. Membase is Simple, Fast, Elastic Predictable • “Never keep an application waiting” • Quasi-deterministic latency and throughput Low latency • Built-in Memcached technology High throughput • Multi-threaded • Low lock contention • Asynchronous wherever possible • Automatic write de-duplication 6
  • 7. Membase is Simple, Fast, Elastic Zero-downtime elasticity • Spread I/O and data across commodity servers (or VMs) • Consistent performance with linear cost • Dynamic rebalancing of a live cluster All nodes are created equal • No special case nodes • Any node can replace any other node, online • Clone to grow Extensible • Filtered TAP interface provides hook points for external systems (e.g. full-text search, backup, warehouse) • Data bucket – engine API for specialized container types 7
  • 8. Built-in Memcached Caching Layer Memcached Membase Cache Membase Database Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project. 8
  • 9. Use Cases
  • 10. Deployments Leading Membase Leading cloud service (PAAS) Social game leader – FarmVille, provider Mafia Wars, Café World Over 65,000 hosted Over 230 million monthly users applications Membase Server Membase Server serving over is the 500,000 ops-per-second 1,200 Heroku customers (as of database behind FarmVille and June 10, 2010) Café World 10
  • 11. Ad targeting 40 milliseconds to come up with an answer. profiles, real time campaign 3 statistics 2 1 profiles, campaigns events 11
  • 12. Search and Gaming Portal Database 12
  • 13. Membase Architecture
  • 14. Clustering • Underlying cluster functionality based on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 14
  • 15. TAP • A generic, scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Data Mutations Working set Data Mutations Working set Working set 16
  • 16. Membase data flow – under the hood SET request arrives at SET acknowledgement KEY’s master server 1 2 returned to application 3 Listener-Sender 3 RAM membase storage engine 4 Disk Disk Disk Disk Disk Disk Replica Server 1 for KEY Master server for KEY Replica Server 2 for KEY 17
  • 17. Clients, nodes and other nodes Client moxi + Client port 11210 port 11211 memcached operations REST/comet memcached operations cluster topology and vbucket map memcached operations moxi ns_server membase ns_server (memcached + membase engine) TAP memcached operations with tap commands vbucketmigrator 18
  • 18. Data buckets are secure membase “slices” Application user Web application server Bucket 1 Bucket 2 Aggregate Cluster Memory and Disk Capacity Membase data servers In the data center On the administrator console 19
  • 19. vBucket mapping 20
  • 20. Disk > Memory Dataset may have many memory quota Bucket Configuration items infrequently accessed. mem_high_wat However, memcached has mem_low_wat different behavior (LRU) than wanted with membase. Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important. 21
  • 21. Membase Demo
  • 22. Thanks! 23
  • 23. Key-Value Patterns
  • 24. Key-Value Items have: Operations include: (with a replica Key Value Get/Set Increment/Decrement ) Expiration Append/Prepend Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/ Flags CAS (more on this later) 25
  • 25. Membase Datatypes • byte[] – Does your data have 1s and 0s? • Items do have flags – Many clients use flags – Data type options • Google protobuf “Any customer can have a car painted any colour • Thrift that he wants so long as • Avro it is black.” 26
  • 26. Transactions • Lock == slow me down • CAS operations User 1 User 2 – Optimistic locking Su • Very useful with complex cc il! es Fa datatypes s – Imagine two clients trying to update a complex item • You’re likely using CAS already... if you use a CPU 27
  • 27. Common Use: Sessions • Web user sessions – Highly read, less writes in many case – Protocol advantage of memcached • Options already for PHP, Ruby and Java • Application state – Not necessarily “entity” style things – May be appropriate for a “cache” pool 28
  • 28. Common Use (cache): Rate Limiting • Want to provide API calls into the system Your Users – Twitter search h ! uc – Google search services ¡O • Use the atomic increment Your App – Set an item with a unique ID – Upon API request, increment and check • HTTP 420: go away and come back later 29
  • 29. Looking Ahead: NodeCode Frank Weigel, Membase
  • 30. NodeCode – Motivation Beyond key-value • Indexing/Range Queries • Advanced Data Structures • Sub-object direct manipulation Validation and In-flight transformation • Block mutations failing validation • Enrich or transform objects Connectors (Integrate easily with other systems) • Solr • Hadoop • MySQL 31
  • 31. NodeCode - What is it? Method for extending & customizing Membase Separate code modules Defined interface to datapath and cluster manager Notification on events • Synchronous • Asynchronous 32
  • 32. NodeCode – Drivers Simple • Packaged modules for easy install and enable • Library of “off the shelf” modules • Module monitoring • Straight forward development and debugging Fast • Low latency/high-throughput • Per-bucket process isolation • Don’t break data manager performance/correctness Elastic • Automatically migrate and instantiate on rebalance • Provide support for migration of internal data • Leverage native Membase engine for internal data storage 33
  • 33. Block-level architecture 34
  • 34. NodeCode 1.0 Plans Java only – jar format Must implement minimal module API • Initial module startup • Module removal • Association with bucket NodeCode library helper functions • Register synchronous & asynchronous listeners/callbacks • Register protocol extension/callbacks • Register rebalance callback • Register cluster manager event callbacks • Membase data access 35
  • 35. Q&A 37
  • 36. Attributions • http://commons.wikimedia.org/wiki/ File:Flag_of_China.png • http://commons.wikimedia.org/wiki/ File:Flag_of_South_Korea.svg • http://commons.wikimedia.org/wiki/ File:Flag_of_Japan.svg 38