Download as PDF, PPTX
Uploaded byMembase
Membase Introduction
Membase is an open source, distributed key-value database management system optimized for storing data behind interactive web applications. It is designed to be simple, fast, and elastic. Membase allows data to scale out linearly by just adding more nodes, maintaining consistency when accessing data. It also includes a built-in Memcached caching layer.
More Related Content
Membase Introduction
- 1. membase.org: The Simple, Fast,Elastic NoSQL Database Membase, Inc. Matt Ingenthron matt@membase.com
- 2. Membase is anOpen Source distributed, key-value database management system optimized for storing data behind interactive web applications. All aspects of membase are simple, fast and elastic by design. 2
- 3.
- 4.
- 5.
- 6. Fast 5 • Original usecase: speed up access to authoritative data as a distributed hashtable • Must be at at least as fast as a highly tuned DBMS • Designed for modern datacenter substrate – Designed forVM and cloud deployments
- 7. Elastic • Add nodeswithout losing access to data • Maintain consistency when accessing data – membase is a CP type system • Scale linearly by just adding more nodes 6
- 8. Before: Application scaleslinearly, data hits wall Application Scales Out Just add more commodity web servers Database Scales Up Get a bigger, more complex server 7
- 9. Membase is adistributed database 8 Membase Servers In the data center Web application server Application user On the administrator console
- 10. Built-in Memcached CachingLayer 9 Memcached Membase Database Memcached Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project.
- 11. Leading cloud service(PAAS) provider Over 65,000 hosted applications Over 2,000 users to date Membase Server serving over 3,000 Heroku customers Proven at small, and extra large scale 10 Social game leader – FarmVille, Mafia Wars, Café World Over 230 million monthly users Membase Server is the 500,000 ops-per-second database behind FarmVille and Café World
- 12. After: Data layerscales like application logic layer Data layer now scales with linear cost and constant performance. Application Scales Out Just add more commodity web servers 11 Database Scales Out Just add more commodity data servers Scaling out flattens the cost and performance curves. Membase Servers
- 13.
- 14. Fault-tolerant memcached Cluster at NHN the biggest web portal in Korea
- 15. What is ProjectArcus? • Memcached – Common protocol across PHP, Java, C applications • Moxi (Memcached proxy) based • In-house automatic fault-detection and failover solution • Collectd-based monitoring • Proxy and cache server administration UI • Private cloud service 14
- 16. Previous Deployments • Afew individual memcached installations • Problems – No fault-tolerance • Hardware failures are common (heat, network switch failure, etc) – No automatic scalability • To add / remove a memcached server, they need to rebuild code, distribute, and restart all clients 15
- 17. Today • Memcached clusters –Fault-tolerance transparent to clients • Consistent hashing in moxi (memcached proxy) – Cache As A Service (CaaS) • All major services in NHN started using cache • Multitenancy across cache services 16
- 18. Performance impact X 16.6 Throughput X10 Response Time Performance 50 % 34 % DB Load
- 19. Membase-Cloudera Partnership “AOL servesmore than 5 billion impressions per day from our ad serving platforms, and any incremental improvement in processing time translates to huge benefits in our ability to more effectively serve the ads to needed meet our contractual commitments. Traditional databases like MySQL lack the scalability required to support our goal of five milliseconds per read/write. Creating user profiles with Hadoop, then serving them from Membase, reduces profile read and write access to under a millisecond, leaving the bulk of the processing time budget for improved targeting and customization.” Pero Subasic Chief Architect, AOL
- 20. Joint development ofbi-directional software integration between Membase and Hadoop • Membase NodeCode Module streaming interface to Cloudera Distribution for Hadoop via Flume interface • Sqoop-derived command line utility for bi- directional batch movement of data between Membase and Cloudera Distribution for Hadoop Joint marketing and sales of integrated distributed OLTP-OLAP solution • Membase – the distributed OLTP solution • Cloudera – the distributed OLAP solution Cloudera to distribute integration Membase-Cloudera Partnership
- 21. Customer use case– Ad targeting 20 events profiles, campaigns profiles, real time campaign statistics 40 milliseconds to come up with an answer. 2 3 1
- 22.
- 24. The Guts Photo Courtesyhttp://www.flickr.com/photos/pellis/76804760/ 23
- 25. Clustering • Underlying cluster functionalitybased on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 24
- 30.
- 31. TAP • A generic,scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Working set Data Mutations Working set Data Mutations Working set 27
- 32. Disk > Memory BucketConfiguration mem_high_wat mem_low_wat memoryquota 28 Dataset may have many items infrequently accessed. However, memcached has different behavior (LRU) than wanted with membase. Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important.
- 33. ns_server membase (memcached + membaseengine) moxi ns_server vbucketmigrator TAP memcached operations with tap commands memcached operations Client port 11211 memcached operations moxi + Client port 11210 memcached operations REST/comet cluster topology and vbucket map Clients, nodes and other nodes 29