Membase Meetup - Silicon Valley

Silicon Valley

2
Tonight
Membase Overview
Use Cases and Deployment Examples
Membase Architecture
Demo!
Developing with Membase
A Glimpse into the Future

What is Membase?

Before: Application scales linearly, data hits wall
Application Scales Out
Just add more commodity web servers
Database Scales Up
Get a bigger, more complex server
4

Membase is a distributed database
5
Application user
Web application server
Membase Servers
In the data center
On the administrator console

Built-in Memcached Caching Layer
6
Memcached
Memcached
Membase Database
Membase Database
Memcached Mode
Membase Mode
Membase development team has contributed over half of the source code to the Memcached project.

Deployment options
7

Secure multitenant support
8
Application user
Bucket 1
Bucket 2
Aggregate Cluster Memory and Disk Capacity
Membase data servers
In the data center

Five minutes or less to a working cluster
Downloads for Linux and Windows
Start with a single node
One button press joins nodes to a cluster
Easy to develop against
Just SET and GET – no schema required
Drop it in. 10,000+ existing applications already “speak membase” (via memcached)
Practically every language and application framework is supported, out of the box
Easy to manage
One-click failover and cluster rebalancing
Graphical and programmatic interfaces
Configurable alerting
Membase is Simple, Fast, Elastic
9

10
Predictable
“Never keep an application waiting”
Quasi-deterministic latency and throughput
Low latency
Built-in Memcached technology
Auto-migration of hot data to lowest latency storage technology (RAM, SSD, Disk)
Selectable write behavior – asynchronous, synchronous (on replication, persistence)
High throughput
Multi-threaded
Low lock contention
Asynchronous wherever possible
Automatic write de-duplication

11
Zero-downtime elasticity
Spread I/O and data across commodity servers (or VMs)
Consistent performance with linear cost
Dynamic rebalancing of a live cluster
All nodes are created equal
No special case nodes
Clone to grow
Extensible
Filtered TAP interface provides hook points for external systems (e.g. full-text search, backup, warehouse)
Data bucket – engine API for specialized container types
Membase NodeCode[FUTURE]

Proven at small, and extra large scale
Leading cloud service (PAAS) provider
Over 65,000 hosted applications
Membase Server supporting over 3,000 Heroku customers
Social game leader – FarmVille, Mafia Wars, Café World
Over 230 million monthly users
Zynga is a core contributor to and large scale user of Membase Server
12

After: Data layer scales like application logic layerData layer now scales with linear cost and constant performance.
Application Scales Out
Just add more commodity web servers
Membase Servers
Database Scales Out
Just add more commodity data servers
Scaling out flattens the cost and performance curves.
13

Membase - A practical path to “NoSQL” adoption
14

Use Cases

16
Where does Membase Fit?
Applications with many, many users
Online, dynamic applications
Applications with growing datasets which need quick access, growing audience

17
Deployments Leading Membase
Leading cloud service (PAAS) provider
Over 65,000 hosted applications
Membase Server serving over 1,200 Heroku customers (as of June 10, 2010)
Social game leader – FarmVille, Mafia Wars, Café World
Over 230 million monthly users
Membase Serveris the 500,000 ops-per-second database behind FarmVille and Café World

Use case – Ad targeting
18
40 milliseconds to come up with an answer.
profiles, real time campaign statistics
3
2
1
profiles, campaigns
events

19
Search and Gaming Portal
Database

Targeting atShareThis

Largest integrated sharing network
We make sharing simple, engaging & valuable
Powerful Social Analytics & Audience Monetization
About ShareThis
450/momillion consumers
~850thousand sites
50+social channels

This is how it works
Sharing Behavior
Log Files
Search Keywords
Page Views
HDFS
Map/Reduce
Content Analysis
Taxonomy
Ad Server
User
Membase
2

ShareThis Ad Products

25
Clustering
Underlying cluster functionality based on erlang OTP
Have a custom, vector clock based way of storing and propagating...
Cluster topology
vBucket mapping
Collect statistics from many nodes of the cluster
Identify hot keys, resource utilization
25

27
TAP
A generic, scalable method of streaming mutations from a given server
As data operations arrive, they can be sent to arbitrary TAP receivers
Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data
Three modes of operation

11211
11210
memcapable 1.0
memcapable 2.0
moxi
memcached
protocol listener/sender
REST management API/Web UI
vBucket state and replication manager
Node health monitor
Rebalance orchestrator
Heartbeat
Process monitor
Global singleton supervisor
Configuration manager
engine interface
membase storage engine
http
on each node
one per cluster
Erlang/OTP
HTTP
distributed erlang
erlang port mapper
21100 – 21199
4369
8080

29
Clients, nodes and other nodes

30
Data buckets are secure membase “slices”
Application user
Bucket 1
Bucket 2
Aggregate Cluster Memory and Disk Capacity
Membase data servers
In the data center

11211
11210
memcapable 1.0
memcapable 2.0
moxi
memcached
protocol listener/sender
REST management API/Web UI
vBucket state and replication manager
Rebalance orchestrator
Node health monitor
Data Manager
Cluster Manager
Global singleton supervisor
Configuration manager
Heartbeat
Process monitor
engine interface
membase storage engine
http
on each node
one per cluster
Erlang/OTP
HTTP
distributed erlang
erlang port mapper
21100 – 21199
4369
8080

32
vBucket mapping

33
Membase “write” data flow – application view
User action results in the need to change the VALUE of KEY
1
Application updates key’s VALUE, performs SET operation
2
4
Membase (memcached) client hashes KEY, identifies KEY’s master server
3
SET request sent over network to master server
5
Membase replicates KEY-VALUE pair, caches it in memory and stores it to disk

34
Disk > Memory
Dataset may have many items infrequently accessed. However, memcached has different behavior (LRU) than wanted with membase.
Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important.

Membase Demo

36
Thanks!

Key-Value Patterns

38
Key-Value
(with a replica)
Items have:
Key
Value
Expiration
Flags
CAS (more on this later)
Operations include:
Get/Set
Increment/Decrement
Append/Prepend

39
Membase Datatypes
“Any customer can have a car painted any colour that he wants so long as it is black.”
byte[]
Does your data have 1s and 0s?
Items do have flags
Many clients use flags
Data type options
Google protobuf
Thrift
Avro

40
Transactions
Lock == slow me down
CAS operations
Optimistic locking
Very useful with complex datatypes
Imagine two clients trying to update a complex item
You’re likely using CAS already... if you use a CPU
User 1
User 2
Success
Fail!

41
Common Use: Sessions
Web user sessions
Highly read, less writes in many case
Protocol advantage of memcached
Options already for PHP, Ruby and Java
Application state
Not necessarily “entity” style things
May be appropriate for a “cache” pool

42
Common Use (cache): Rate Limiting
Want to provide API calls into the system
Twitter search
Google search services
Use the atomic increment
Set an item with a unique ID
Upon API request, increment and check
HTTP 420: go away and come back later
Your Users
¡Ouch!
Your App

Looking Ahead: NodeCodeFrank Weigel, Membase

44
Beyond key-value
Indexing/Range Queries
Advanced Data Structures
Sub-object direct manipulation
Validation and In-flight transformation
Block mutations failing validation
Enrich or transform objects
Connectors (Integrate easily with other systems)
Solr
Hadoop
MySQL
NodeCode – Motivation

45
NodeCode - What is it?
Method for extending & customizing Membase
Separate code modules
Defined interface to datapath and cluster manager
Notification on events
Synchronous
Asynchronous

46
Simple
Packaged modules for easy install and enable
Library of “off the shelf” modules
Module monitoring
Straight forward development and debugging
Fast
Low latency/high-throughput
Per-bucket process isolation
Don’t break data manager performance/correctness
Elastic
Automatically migrate and instantiate on rebalance
Provide support for migration of internal data
Leverage native Membase engine for internal data storage
NodeCode – Drivers

47
Block-level architecture

48
Java only
jar format
Must implement minimal module API
Initial module startup
Module removal
Association with bucket
NodeCode library helper functions
Register synchronous & asynchronous listeners/callbacks
Register protocol extension/callbacks
Register rebalance callback
Register cluster manager event callbacks
Membase data access
NodeCode 1.0 Plans

50
Q&A

51
Attributions
http://commons.wikimedia.org/wiki/File:Flag_of_China.png
http://commons.wikimedia.org/wiki/File:Flag_of_South_Korea.svg
http://commons.wikimedia.org/wiki/File:Flag_of_Japan.svg

http://blog.couchbase.com	14
http://blog.membase.com	10
http://www.stage.couchbase.com	1

Membase Meetup - Silicon Valley

by Membase

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 25

Statistics

Membase Meetup - Silicon Valley Presentation Transcript