This document provides an overview and demonstration of Membase, a distributed database. It discusses how Membase can be deployed in five minutes or less on a single node or cluster. It is simple to develop applications using Membase's key-value interface without requiring a schema. The document also summarizes several use cases for Membase in domains like social games, advertising, and search/gaming portals. It provides a high-level overview of Membase's architecture, including its clustering functionality and TAP interface. Finally, it briefly previews Membase's NodeCode feature for extending the database with custom modules.

1. Boston

2. Tonight
2
• Membase Overview
• Use Cases and Deployment Examples
• Membase Architecture
• Demo!
• Developing with Membase
• A Glimpse into the Future

3. What is Membase?

4. Membase is a distributed database
4
Membase Servers
In the data center
Web application server
Application user
On the administrator console
Web application serverWeb application server

5. Five minutes or less to a working
cluster
• Downloads for Linux and Windows
• Start with a single node
• One button press joins nodes to a cluster
Easy to develop against
• Just SET and GET – no schema required
• Drop it in. 10,000+ existing applications
already “speak membase” (via memcached)
• Practically every language and application
framework is supported, out of the box
Easy to manage
• One-click failover and cluster rebalancing
• Graphical and programmatic interfaces
• Configurable alerting
Membase is Simple, Fast, Elastic
5

6. Membase is Simple, Fast, Elastic
6
Predictable
• “Never keep an application waiting”
• Quasi-deterministic latency and throughput
Low latency
• Built-in Memcached technology
High throughput
• Multi-threaded
• Low lock contention
• Asynchronous wherever possible
• Automatic write de-duplication

7. Membase is Simple, Fast, Elastic
7
Zero-downtime elasticity
• Spread I/O and data across commodity
servers (or VMs)
• Consistent performance with linear cost
• Dynamic rebalancing of a live cluster
All nodes are created equal
• No special case nodes
• Any node can replace any other node, online
• Clone to grow
Extensible
• Filtered TAP interface provides hook points
for external systems (e.g. full-text search,
backup, warehouse)
• Data bucket – engine API for specialized
container types

8. Built-in Memcached Caching Layer
8
Memcached
Membase Database
Membase Cache
Membase Database
Memcached Mode Membase Mode
Fact: Membase development team has also contributed over
half of the code to the Memcached project.

9. Use Cases

10. Leading cloud service (PAAS)
provider
Over 65,000 hosted
applications
Membase Server serving over
1,200 Heroku customers (as of
June 10, 2010)
Deployments Leading Membase
10
Social game leader – FarmVille,
Mafia Wars, Café World
Over 230 million monthly users
Membase Server
is the 500,000 ops-per-second
database behind FarmVille and
Café World

11. Ad targeting
11
events
profiles, campaigns
profiles, real time campaign
statistics
40 milliseconds to come
up with an answer.
2
3
1

12. Search and Gaming Portal
12
Database

13. Membase Architecture

14. Clustering
• Underlying cluster
functionality based on
erlang OTP
• Have a custom, vector
clock based way of storing
and propagating...
– Cluster topology
– vBucket mapping
• Collect statistics from many
nodes of the cluster
– Identify hot keys, resource
utilization
14

16. TAP
• A generic, scalable method of streaming mutations
from a given server
– As data operations arrive, they can be sent to arbitrary TAP
receivers
• Leverages the existing memcached engine interface,
and the non-blocking IO interfaces to send data
• Three modes of operation
Working set
Data
Mutations
Working set
Data
Mutations
Working set
16

17. Membase data flow – under the hood
17
SET request arrives at
KEY’s master server
Listener-Sender
Master server for KEY Replica Server 2 for KEYReplica Server 1 for KEY
3 3
1
SET acknowledgement
returned to application2
DiskDisk Disk
RAM
membasestorageengine
DiskDisk Disk
4

18. ns_server
membase
(memcached + membase engine)
moxi ns_server
vbucketmigrator
TAP
memcached operations
with tap commands
memcached operations
Client
port 11211
memcached operations
moxi + Client
port 11210
memcached operations REST/comet
cluster topology
and vbucket map
Clients, nodes and other nodes
18

19. Data buckets are secure membase “slices”
19
Membase data servers
In the data center
Web application server
Application user
On the administrator console
Bucket 1
Bucket 2
Aggregate Cluster Memory and Disk Capacity

20. vBucket mapping
20

21. Disk > Memory
BucketConfiguration
mem_high_wat
mem_low_wat
memory quota
21
Dataset may have many
items infrequently accessed.
However, memcached has
different behavior (LRU) than
wanted with membase.
Still, traditional (most)
RDBMS implementations are
not 100% correct for us
either. The speed of a miss
is very, very important.

22. Membase Demo

23. 23
Thanks!

24. Key-Value Patterns

25. Key-Value
Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/
(with a replica )
25
Items have:
Key
Value
Expiration
Flags
CAS (more on this later)
Operations include:
Get/Set
Increment/Decrement
Append/Prepend

26. Membase Datatypes
• byte[]
– Does your data have
1s and 0s?
26
“Any customer can have
a car painted any colour
that he wants so long as
it is black.”
• Items do have flags
– Many clients use flags
–Data type options
• Google protobuf
• Thrift
• Avro

27. Transactions
• Lock == slow me down
• CAS operations
– Optimistic locking
• Very useful with complex
datatypes
– Imagine two clients trying to
update a complex item
• You’re likely using CAS
already... if you use a CPU
27
User 1
Fail!
User 2
Success

28. Common Use: Sessions
• Web user sessions
– Highly read, less writes in many case
– Protocol advantage of memcached
• Options already for PHP, Ruby and Java
• Application state
– Not necessarily “entity” style things
– May be appropriate for a “cache” pool
28

29. Common Use (cache): Rate Limiting
• Want to provide API calls
into the system
– Twitter search
– Google search services
• Use the atomic increment
– Set an item with a unique ID
– Upon API request,
increment and check
• HTTP 420: go away and come
back later
29
Your Users
Your App
¡Ouch!

30. Looking Ahead: NodeCode
Frank Weigel, Membase

31. Beyond key-value
• Indexing/Range Queries
• Advanced Data Structures
• Sub-object direct manipulation
Validation and In-flight transformation
• Block mutations failing validation
• Enrich or transform objects
Connectors (Integrate easily with other systems)
• Solr
• Hadoop
• MySQL
NodeCode – Motivation
31

32. NodeCode - What is it?
Method for extending & customizing Membase
Separate code modules
Defined interface to datapath and cluster manager
Notification on events
• Synchronous
• Asynchronous
32

33. Simple
• Packaged modules for easy install and enable
• Library of “off the shelf” modules
• Module monitoring
• Straight forward development and debugging
Fast
• Low latency/high-throughput
• Per-bucket process isolation
• Don’t break data manager performance/correctness
Elastic
• Automatically migrate and instantiate on rebalance
• Provide support for migration of internal data
• Leverage native Membase engine for internal data storage
NodeCode – Drivers
33

34. Block-level architecture
34

35. Java only
– jar format
Must implement minimal module API
• Initial module startup
• Module removal
• Association with bucket
NodeCode library helper functions
• Register synchronous & asynchronous listeners/callbacks
• Register protocol extension/callbacks
• Register rebalance callback
• Register cluster manager event callbacks
• Membase data access
NodeCode 1.0 Plans
35

37. 37
Q&A

38. Attributions
• http://commons.wikimedia.org/wiki/
File:Flag_of_China.png
• http://commons.wikimedia.org/wiki/
File:Flag_of_South_Korea.svg
• http://commons.wikimedia.org/wiki/
File:Flag_of_Japan.svg
38