• Like

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Membase East Coast Meetups

  • 1,644 views
Uploaded on

These are the slides from the Membase meetups in NYC and Boston.

These are the slides from the Membase meetups in NYC and Boston.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,644
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
31
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Boston
  • 2. Tonight • Membase Overview • Use Cases and Deployment Examples • Membase Architecture • Demo! • Developing with Membase • A Glimpse into the Future 2
  • 3. What is Membase?
  • 4. Membase is a distributed database Application user Web application server Web application server Web application server Membase Servers In the data center On the administrator console 4
  • 5. Membase is Simple, Fast, Elastic Five minutes or less to a working cluster • Downloads for Linux and Windows • Start with a single node • One button press joins nodes to a cluster Easy to develop against • Just SET and GET – no schema required • Drop it in. 10,000+ existing applications already “speak membase” (via memcached) • Practically every language and application framework is supported, out of the box Easy to manage • One-click failover and cluster rebalancing • Graphical and programmatic interfaces • Configurable alerting 5
  • 6. Membase is Simple, Fast, Elastic Predictable • “Never keep an application waiting” • Quasi-deterministic latency and throughput Low latency • Built-in Memcached technology High throughput • Multi-threaded • Low lock contention • Asynchronous wherever possible • Automatic write de-duplication 6
  • 7. Membase is Simple, Fast, Elastic Zero-downtime elasticity • Spread I/O and data across commodity servers (or VMs) • Consistent performance with linear cost • Dynamic rebalancing of a live cluster All nodes are created equal • No special case nodes • Any node can replace any other node, online • Clone to grow Extensible • Filtered TAP interface provides hook points for external systems (e.g. full-text search, backup, warehouse) • Data bucket – engine API for specialized container types 7
  • 8. Built-in Memcached Caching Layer Memcached Membase Cache Membase Database Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project. 8
  • 9. Use Cases
  • 10. Deployments Leading Membase Leading cloud service (PAAS) Social game leader – FarmVille, provider Mafia Wars, Café World Over 65,000 hosted Over 230 million monthly users applications Membase Server Membase Server serving over is the 500,000 ops-per-second 1,200 Heroku customers (as of database behind FarmVille and June 10, 2010) Café World 10
  • 11. Ad targeting 40 milliseconds to come up with an answer. profiles, real time campaign 3 statistics 2 1 profiles, campaigns events 11
  • 12. Search and Gaming Portal Database 12
  • 13. Membase Architecture
  • 14. Clustering • Underlying cluster functionality based on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 14
  • 15. TAP • A generic, scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Data Mutations Working set Data Mutations Working set Working set 16
  • 16. Membase data flow – under the hood SET request arrives at SET acknowledgement KEY’s master server 1 2 returned to application 3 Listener-Sender 3 RAM membase storage engine 4 Disk Disk Disk Disk Disk Disk Replica Server 1 for KEY Master server for KEY Replica Server 2 for KEY 17
  • 17. Clients, nodes and other nodes Client moxi + Client port 11210 port 11211 memcached operations REST/comet memcached operations cluster topology and vbucket map memcached operations moxi ns_server membase ns_server (memcached + membase engine) TAP memcached operations with tap commands vbucketmigrator 18
  • 18. Data buckets are secure membase “slices” Application user Web application server Bucket 1 Bucket 2 Aggregate Cluster Memory and Disk Capacity Membase data servers In the data center On the administrator console 19
  • 19. vBucket mapping 20
  • 20. Disk > Memory Dataset may have many memory quota Bucket Configuration items infrequently accessed. mem_high_wat However, memcached has mem_low_wat different behavior (LRU) than wanted with membase. Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important. 21
  • 21. Membase Demo
  • 22. Thanks! 23
  • 23. Key-Value Patterns
  • 24. Key-Value Items have: Operations include: (with a replica Key Value Get/Set Increment/Decrement ) Expiration Append/Prepend Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/ Flags CAS (more on this later) 25
  • 25. Membase Datatypes • byte[] – Does your data have 1s and 0s? • Items do have flags – Many clients use flags – Data type options • Google protobuf “Any customer can have a car painted any colour • Thrift that he wants so long as • Avro it is black.” 26
  • 26. Transactions • Lock == slow me down • CAS operations User 1 User 2 – Optimistic locking Su • Very useful with complex cc il! es Fa datatypes s – Imagine two clients trying to update a complex item • You’re likely using CAS already... if you use a CPU 27
  • 27. Common Use: Sessions • Web user sessions – Highly read, less writes in many case – Protocol advantage of memcached • Options already for PHP, Ruby and Java • Application state – Not necessarily “entity” style things – May be appropriate for a “cache” pool 28
  • 28. Common Use (cache): Rate Limiting • Want to provide API calls into the system Your Users – Twitter search h ! uc – Google search services ¡O • Use the atomic increment Your App – Set an item with a unique ID – Upon API request, increment and check • HTTP 420: go away and come back later 29
  • 29. Looking Ahead: NodeCode Frank Weigel, Membase
  • 30. NodeCode – Motivation Beyond key-value • Indexing/Range Queries • Advanced Data Structures • Sub-object direct manipulation Validation and In-flight transformation • Block mutations failing validation • Enrich or transform objects Connectors (Integrate easily with other systems) • Solr • Hadoop • MySQL 31
  • 31. NodeCode - What is it? Method for extending & customizing Membase Separate code modules Defined interface to datapath and cluster manager Notification on events • Synchronous • Asynchronous 32
  • 32. NodeCode – Drivers Simple • Packaged modules for easy install and enable • Library of “off the shelf” modules • Module monitoring • Straight forward development and debugging Fast • Low latency/high-throughput • Per-bucket process isolation • Don’t break data manager performance/correctness Elastic • Automatically migrate and instantiate on rebalance • Provide support for migration of internal data • Leverage native Membase engine for internal data storage 33
  • 33. Block-level architecture 34
  • 34. NodeCode 1.0 Plans Java only – jar format Must implement minimal module API • Initial module startup • Module removal • Association with bucket NodeCode library helper functions • Register synchronous & asynchronous listeners/callbacks • Register protocol extension/callbacks • Register rebalance callback • Register cluster manager event callbacks • Membase data access 35
  • 35. Q&A 37
  • 36. Attributions • http://commons.wikimedia.org/wiki/ File:Flag_of_China.png • http://commons.wikimedia.org/wiki/ File:Flag_of_South_Korea.svg • http://commons.wikimedia.org/wiki/ File:Flag_of_Japan.svg 38