0
Boston
Tonight
2
• Membase Overview
• Use Cases and Deployment Examples
• Membase Architecture
• Demo!
• Developing with Membase
...
What is Membase?
Membase is a distributed database
4
Membase Servers
In the data center
Web application server
Application user
On the admi...
Five minutes or less to a working
cluster
• Downloads for Linux and Windows
• Start with a single node
• One button press ...
Membase is Simple, Fast, Elastic
6
Predictable
• “Never keep an application waiting”
• Quasi-deterministic latency and thr...
Membase is Simple, Fast, Elastic
7
Zero-downtime elasticity
• Spread I/O and data across commodity
servers (or VMs)
• Cons...
Built-in Memcached Caching Layer
8
Memcached
Membase Database
Membase Cache
Membase Database
Memcached Mode Membase Mode
F...
Use Cases
Leading cloud service (PAAS)
provider
Over 65,000 hosted
applications
Membase Server serving over
1,200 Heroku customers (...
Ad targeting
11
events
profiles, campaigns
profiles, real time campaign
statistics
40 milliseconds to come
up with an answ...
Search and Gaming Portal
12
Database
Membase Architecture
Clustering
• Underlying cluster
functionality based on
erlang OTP
• Have a custom, vector
clock based way of storing
and p...
TAP
• A generic, scalable method of streaming mutations
from a given server
– As data operations arrive, they can be sent ...
Membase data flow – under the hood
17
SET request arrives at
KEY’s master server
Listener-Sender
Master server for KEY Repl...
ns_server
membase
(memcached + membase engine)
moxi ns_server
vbucketmigrator
TAP
memcached operations
with tap commands
m...
Data buckets are secure membase “slices”
19
Membase data servers
In the data center
Web application server
Application use...
vBucket mapping
20
Disk > Memory
BucketConfiguration
mem_high_wat
mem_low_wat
memory quota
21
Dataset may have many
items infrequently accesse...
Membase Demo
23
Thanks!
Key-Value Patterns
Key-Value
Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/
(with a replica )
25...
Membase Datatypes
• byte[]
– Does your data have
1s and 0s?
26
“Any customer can have
a car painted any colour
that he wan...
Transactions
• Lock == slow me down
• CAS operations
– Optimistic locking
• Very useful with complex
datatypes
– Imagine t...
Common Use: Sessions
• Web user sessions
– Highly read, less writes in many case
– Protocol advantage of memcached
• Optio...
Common Use (cache): Rate Limiting
• Want to provide API calls
into the system
– Twitter search
– Google search services
• ...
Looking Ahead: NodeCode
Frank Weigel, Membase
Beyond key-value
• Indexing/Range Queries
• Advanced Data Structures
• Sub-object direct manipulation
Validation and In-fli...
NodeCode - What is it?
Method for extending & customizing Membase
Separate code modules
Defined interface to datapath and c...
Simple
• Packaged modules for easy install and enable
• Library of “off the shelf” modules
• Module monitoring
• Straight ...
Block-level architecture
34
Java only
– jar format
Must implement minimal module API
• Initial module startup
• Module removal
• Association with buck...
37
Q&A
Attributions
• http://commons.wikimedia.org/wiki/
File:Flag_of_China.png
• http://commons.wikimedia.org/wiki/
File:Flag_of...
Membase East Coast Meetups
Membase East Coast Meetups
Membase East Coast Meetups
Upcoming SlideShare
Loading in...5
×

Membase East Coast Meetups

1,729

Published on

These are the slides from the Membase meetups in NYC and Boston.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,729
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
31
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Membase East Coast Meetups"

  1. 1. Boston
  2. 2. Tonight 2 • Membase Overview • Use Cases and Deployment Examples • Membase Architecture • Demo! • Developing with Membase • A Glimpse into the Future
  3. 3. What is Membase?
  4. 4. Membase is a distributed database 4 Membase Servers In the data center Web application server Application user On the administrator console Web application serverWeb application server
  5. 5. Five minutes or less to a working cluster • Downloads for Linux and Windows • Start with a single node • One button press joins nodes to a cluster Easy to develop against • Just SET and GET – no schema required • Drop it in. 10,000+ existing applications already “speak membase” (via memcached) • Practically every language and application framework is supported, out of the box Easy to manage • One-click failover and cluster rebalancing • Graphical and programmatic interfaces • Configurable alerting Membase is Simple, Fast, Elastic 5
  6. 6. Membase is Simple, Fast, Elastic 6 Predictable • “Never keep an application waiting” • Quasi-deterministic latency and throughput Low latency • Built-in Memcached technology High throughput • Multi-threaded • Low lock contention • Asynchronous wherever possible • Automatic write de-duplication
  7. 7. Membase is Simple, Fast, Elastic 7 Zero-downtime elasticity • Spread I/O and data across commodity servers (or VMs) • Consistent performance with linear cost • Dynamic rebalancing of a live cluster All nodes are created equal • No special case nodes • Any node can replace any other node, online • Clone to grow Extensible • Filtered TAP interface provides hook points for external systems (e.g. full-text search, backup, warehouse) • Data bucket – engine API for specialized container types
  8. 8. Built-in Memcached Caching Layer 8 Memcached Membase Database Membase Cache Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project.
  9. 9. Use Cases
  10. 10. Leading cloud service (PAAS) provider Over 65,000 hosted applications Membase Server serving over 1,200 Heroku customers (as of June 10, 2010) Deployments Leading Membase 10 Social game leader – FarmVille, Mafia Wars, Café World Over 230 million monthly users Membase Server is the 500,000 ops-per-second database behind FarmVille and Café World
  11. 11. Ad targeting 11 events profiles, campaigns profiles, real time campaign statistics 40 milliseconds to come up with an answer. 2 3 1
  12. 12. Search and Gaming Portal 12 Database
  13. 13. Membase Architecture
  14. 14. Clustering • Underlying cluster functionality based on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 14
  15. 15. TAP • A generic, scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Working set Data Mutations Working set Data Mutations Working set 16
  16. 16. Membase data flow – under the hood 17 SET request arrives at KEY’s master server Listener-Sender Master server for KEY Replica Server 2 for KEYReplica Server 1 for KEY 3 3 1 SET acknowledgement returned to application2 DiskDisk Disk RAM membasestorageengine DiskDisk Disk 4
  17. 17. ns_server membase (memcached + membase engine) moxi ns_server vbucketmigrator TAP memcached operations with tap commands memcached operations Client port 11211 memcached operations moxi + Client port 11210 memcached operations REST/comet cluster topology and vbucket map Clients, nodes and other nodes 18
  18. 18. Data buckets are secure membase “slices” 19 Membase data servers In the data center Web application server Application user On the administrator console Bucket 1 Bucket 2 Aggregate Cluster Memory and Disk Capacity
  19. 19. vBucket mapping 20
  20. 20. Disk > Memory BucketConfiguration mem_high_wat mem_low_wat memory quota 21 Dataset may have many items infrequently accessed. However, memcached has different behavior (LRU) than wanted with membase. Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important.
  21. 21. Membase Demo
  22. 22. 23 Thanks!
  23. 23. Key-Value Patterns
  24. 24. Key-Value Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/ (with a replica ) 25 Items have: Key Value Expiration Flags CAS (more on this later) Operations include: Get/Set Increment/Decrement Append/Prepend
  25. 25. Membase Datatypes • byte[] – Does your data have 1s and 0s? 26 “Any customer can have a car painted any colour that he wants so long as it is black.” • Items do have flags – Many clients use flags –Data type options • Google protobuf • Thrift • Avro
  26. 26. Transactions • Lock == slow me down • CAS operations – Optimistic locking • Very useful with complex datatypes – Imagine two clients trying to update a complex item • You’re likely using CAS already... if you use a CPU 27 User 1 Fail! User 2 Success
  27. 27. Common Use: Sessions • Web user sessions – Highly read, less writes in many case – Protocol advantage of memcached • Options already for PHP, Ruby and Java • Application state – Not necessarily “entity” style things – May be appropriate for a “cache” pool 28
  28. 28. Common Use (cache): Rate Limiting • Want to provide API calls into the system – Twitter search – Google search services • Use the atomic increment – Set an item with a unique ID – Upon API request, increment and check • HTTP 420: go away and come back later 29 Your Users Your App ¡Ouch!
  29. 29. Looking Ahead: NodeCode Frank Weigel, Membase
  30. 30. Beyond key-value • Indexing/Range Queries • Advanced Data Structures • Sub-object direct manipulation Validation and In-flight transformation • Block mutations failing validation • Enrich or transform objects Connectors (Integrate easily with other systems) • Solr • Hadoop • MySQL NodeCode – Motivation 31
  31. 31. NodeCode - What is it? Method for extending & customizing Membase Separate code modules Defined interface to datapath and cluster manager Notification on events • Synchronous • Asynchronous 32
  32. 32. Simple • Packaged modules for easy install and enable • Library of “off the shelf” modules • Module monitoring • Straight forward development and debugging Fast • Low latency/high-throughput • Per-bucket process isolation • Don’t break data manager performance/correctness Elastic • Automatically migrate and instantiate on rebalance • Provide support for migration of internal data • Leverage native Membase engine for internal data storage NodeCode – Drivers 33
  33. 33. Block-level architecture 34
  34. 34. Java only – jar format Must implement minimal module API • Initial module startup • Module removal • Association with bucket NodeCode library helper functions • Register synchronous & asynchronous listeners/callbacks • Register protocol extension/callbacks • Register rebalance callback • Register cluster manager event callbacks • Membase data access NodeCode 1.0 Plans 35
  35. 35. 37 Q&A
  36. 36. Attributions • http://commons.wikimedia.org/wiki/ File:Flag_of_China.png • http://commons.wikimedia.org/wiki/ File:Flag_of_South_Korea.svg • http://commons.wikimedia.org/wiki/ File:Flag_of_Japan.svg 38
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×