Membase Introduction

membase.org: The Simple, Fast, Elastic NoSQL Database Membase, Inc. Matt Ingenthron matt@membase.com

Membase is an Open Source distributed, key-value database management system optimized for storing data behind interactive web applications. All aspects of membase are simple, fast and elastic by design. 2

Value image courtesy http://www.flickr.com/photos/vintagedept/3617706196/ 3

Simple Image courtesy http://www.flickr.com/photos/brenda-starr/ 3509344100/sizes/m/in/photostream/ 4

Simple (with a replica ) Image courtesy http://www.flickr.com/photos/brenda-starr/ 3509344100/sizes/m/in/photostream/ 4

Fast • Original use case: speed up access to authoritative data as a distributed hashtable • Must be at at least as fast as a highly tuned DBMS • Designed for modern datacenter substrate – Designed for VM and cloud deployments 5

Elastic • Add nodes without losing access to data • Maintain consistency when accessing data – membase is a CP type system • Scale linearly by just adding more nodes 6

Before: Application scales linearly, data hits wall Application Scales Out Just add more commodity web servers Database Scales Up Get a bigger, more complex server 7

Membase is a distributed database Application user Web application server Membase Servers In the data center On the administrator console 8

Built-in Memcached Caching Layer Memcached Memcached Membase Database Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project. 9

Proven at small, and extra large scale Leading cloud service (PAAS) Social game leader – FarmVille, provider Maﬁa Wars, Café World Over 65,000 hosted Over 230 million monthly users applications Membase Server is the Over 2,000 users to date 500,000 ops-per-second Membase Server serving over database behind FarmVille and 3,000 Heroku customers Café World 10

After: Data layer scales like application logic layer Data layer now scales with linear cost and constant performance. Application Scales Out Just add more commodity web servers Membase Servers Database Scales Out Just add more commodity data servers Scaling out ﬂattens the cost and performance curves. 11

Who? 12

Fault-tolerant memcached Cluster at NHN the biggest web portal in Korea

What is Project Arcus? • Memcached – Common protocol across PHP, Java, C applications • Moxi (Memcached proxy) based • In-house automatic fault-detection and failover solution • Collectd-based monitoring • Proxy and cache server administration UI • Private cloud service 14

Previous Deployments • A few individual memcached installations • Problems – No fault-tolerance • Hardware failures are common (heat, network switch failure, etc) – No automatic scalability • To add / remove a memcached server, they need to rebuild code, distribute, and restart all clients 15

Today • Memcached clusters – Fault-tolerance transparent to clients • Consistent hashing in moxi (memcached proxy) – Cache As A Service (CaaS) • All major services in NHN started using cache • Multitenancy across cache services 16

Performance impact Performance DB Load X 16.6 50 % X 10 34 % Response Time Throughput

Membase-Cloudera Partnership “AOL serves more than 5 billion impressions per day from our ad serving platforms, and any incremental improvement in processing time translates to huge benefits in our ability to more effectively serve the ads to needed meet our contractual commitments. Traditional databases like MySQL lack the scalability required to support our goal of five milliseconds per read/write. Creating user profiles with Hadoop, then serving them from Membase, reduces profile read and write access to under a millisecond, leaving the bulk of the processing time budget for improved targeting and customization.” Pero Subasic Chief Architect, AOL

Membase-Cloudera Partnership Joint development of bi-directional software integration between Membase and Hadoop • Membase NodeCode Module streaming interface to Cloudera Distribution for Hadoop via Flume interface • Sqoop-derived command line utility for bi- directional batch movement of data between Membase and Cloudera Distribution for Hadoop Joint marketing and sales of integrated distributed OLTP-OLAP solution • Membase – the distributed OLTP solution • Cloudera – the distributed OLAP solution Cloudera to distribute integration

Customer use case – Ad targeting 40 milliseconds to come up with an answer. profiles, real time campaign 3 statistics 2 1 profiles, campaigns events 20

Demo 21

The Guts Photo Courtesy http://www.flickr.com/photos/pellis/76804760/ 23

Clustering • Underlying cluster functionality based on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 24

vBucket mapping 26

TAP • A generic, scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Data Mutations Working set Data Mutations Working set Working set 27

Disk > Memory Dataset may have many memory quota Bucket Conﬁguration items infrequently accessed. mem_high_wat However, memcached has mem_low_wat different behavior (LRU) than wanted with membase. Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important. 28

Clients, nodes and other nodes Client moxi + Client port 11210 port 11211 memcached operations REST/comet memcached operations cluster topology and vbucket map memcached operations moxi ns_server membase ns_server (memcached + membase engine) TAP memcached operations with tap commands vbucketmigrator 29

http://www.nosqldatabases.com	47
http://liulizhi.info	5
http://www.liulizhi.info	2

Membase Introduction

by Membase on Oct 27, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 54

Statistics

Membase Introduction — Presentation Transcript