Your SlideShare is downloading. ×
0
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Membase Introduction
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Membase Introduction

6,265

Published on

Introduction to Membase, presented at SD Forum Cloud SIG on Oct. 26, 2010.

Introduction to Membase, presented at SD Forum Cloud SIG on Oct. 26, 2010.

Published in: Technology
0 Comments
13 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,265
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
120
Comments
0
Likes
13
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. membase.org: The Simple, Fast, Elastic NoSQL Database Membase, Inc. Matt Ingenthron matt@membase.com
  • 2. Membase is an Open Source distributed, key-value database management system optimized for storing data behind interactive web applications. All aspects of membase are simple, fast and elastic by design. 2
  • 3. Value image courtesy http://www.flickr.com/photos/vintagedept/3617706196/ 3
  • 4. Simple Image courtesy http://www.flickr.com/photos/brenda-starr/ 3509344100/sizes/m/in/photostream/ 4
  • 5. Simple Image courtesy http://www.flickr.com/photos/brenda-starr/ 3509344100/sizes/m/in/photostream/ (with a replica ) 4
  • 6. Fast 5 • Original use case: speed up access to authoritative data as a distributed hashtable • Must be at at least as fast as a highly tuned DBMS • Designed for modern datacenter substrate – Designed forVM and cloud deployments
  • 7. Elastic • Add nodes without losing access to data • Maintain consistency when accessing data – membase is a CP type system • Scale linearly by just adding more nodes 6
  • 8. Before: Application scales linearly, data hits wall Application Scales Out Just add more commodity web servers Database Scales Up Get a bigger, more complex server 7
  • 9. Membase is a distributed database 8 Membase Servers In the data center Web application server Application user On the administrator console
  • 10. Built-in Memcached Caching Layer 9 Memcached Membase Database Memcached Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project.
  • 11. Leading cloud service (PAAS) provider Over 65,000 hosted applications Over 2,000 users to date Membase Server serving over 3,000 Heroku customers Proven at small, and extra large scale 10 Social game leader – FarmVille, Mafia Wars, Café World Over 230 million monthly users Membase Server is the 500,000 ops-per-second database behind FarmVille and Café World
  • 12. After: Data layer scales like application logic layer Data layer now scales with linear cost and constant performance. Application Scales Out Just add more commodity web servers 11 Database Scales Out Just add more commodity data servers Scaling out flattens the cost and performance curves. Membase Servers
  • 13. Who? 12
  • 14. Fault-tolerant memcached Cluster at  NHN the  biggest  web  portal  in  Korea
  • 15. What is Project Arcus? • Memcached – Common protocol across PHP, Java, C applications • Moxi (Memcached proxy) based • In-house automatic fault-detection and failover solution • Collectd-based monitoring • Proxy and cache server administration UI • Private cloud service 14
  • 16. Previous Deployments • A few individual memcached installations • Problems – No fault-tolerance • Hardware failures are common (heat, network switch failure, etc) – No automatic scalability • To add / remove a memcached server, they need to rebuild code, distribute, and restart all clients 15
  • 17. Today • Memcached clusters – Fault-tolerance transparent to clients • Consistent hashing in moxi (memcached proxy) – Cache As A Service (CaaS) • All major services in NHN started using cache • Multitenancy across cache services 16
  • 18. Performance impact X 16.6 Throughput X 10 Response Time Performance 50 % 34 % DB Load
  • 19. Membase-Cloudera Partnership “AOL serves more than 5 billion impressions per day from our ad serving platforms, and any incremental improvement in processing time translates to huge benefits in our ability to more effectively serve the ads to needed meet our contractual commitments. Traditional databases like MySQL lack the scalability required to support our goal of five milliseconds per read/write. Creating user profiles with Hadoop, then serving them from Membase, reduces profile read and write access to under a millisecond, leaving the bulk of the processing time budget for improved targeting and customization.” Pero Subasic Chief Architect, AOL
  • 20. Joint development of bi-directional software integration between Membase and Hadoop • Membase NodeCode Module streaming interface to Cloudera Distribution for Hadoop via Flume interface • Sqoop-derived command line utility for bi- directional batch movement of data between Membase and Cloudera Distribution for Hadoop Joint marketing and sales of integrated distributed OLTP-OLAP solution • Membase – the distributed OLTP solution • Cloudera – the distributed OLAP solution Cloudera to distribute integration Membase-Cloudera Partnership
  • 21. Customer use case – Ad targeting 20 events profiles, campaigns profiles, real time campaign statistics 40 milliseconds to come up with an answer. 2 3 1
  • 22. 21 Demo
  • 23. The Guts Photo Courtesy http://www.flickr.com/photos/pellis/76804760/ 23
  • 24. Clustering • Underlying cluster functionality based on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 24
  • 25. vBucket mapping 26
  • 26. TAP • A generic, scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Working set Data Mutations Working set Data Mutations Working set 27
  • 27. Disk > Memory BucketConfiguration mem_high_wat mem_low_wat memory quota 28 Dataset may have many items infrequently accessed. However, memcached has different behavior (LRU) than wanted with membase. Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important.
  • 28. ns_server membase (memcached + membase engine) moxi ns_server vbucketmigrator TAP memcached operations with tap commands memcached operations Client port 11211 memcached operations moxi + Client port 11210 memcached operations REST/comet cluster topology and vbucket map Clients, nodes and other nodes 29

×