0
Chicago
What is Membase?
Membase is a distributed database                             Application user Web application server   Web application se...
Membase is Simple, Fast, Elastic               Five minutes or less to a working               cluster               • Dow...
Membase is Simple, Fast, Elastic               Predictable               • “Never keep an application waiting”            ...
Membase is Simple, Fast, Elastic               Zero-downtime elasticity               • Spread I/O and data across commodi...
Built-in Memcached Caching Layer      Memcached                    Membase Cache   Membase Database              Membase D...
Use Cases
Ad targeting                             40 milliseconds to come                             up with an answer.           ...
Search and Gaming Portal                           Database                                      10
Membase Architecture
Clustering             • Underlying cluster               functionality based on               erlang OTP             • Ha...
TAP     • A generic, scalable method of streaming mutations       from a given server            – As data operations arri...
Membase data flow – under the hood                   SET request arrives at          SET acknowledgement                   ...
A Membase Node  memcached operations   memcached smart      (any client)        client operations   REST Interface        ...
A Membase Node: Component View            moxi                         ns_server                    membase             ns...
Clients, nodes and other nodes                                 Client                      moxi + Client                  ...
Data buckets are secure membase “slices”               Application user            Web application server Bucket 1 Bucket ...
vBucket mapping                  19
Disk > MemoryDataset may have many                                 memory quota                                Bucket Confi...
(Intermission)
Thanks!          22
Membase Demo
Key-Value Patterns
Key-Value            25
Key-Value Items have:            Operations include: Key                    Get/Set Value                  Increment/Decre...
Key-Value            25
Key-Value       (with a replica                                                                        )Image courtesy htt...
Membase Datatypes                    26
Membase Datatypes• byte[]  – Does your data have    1s and 0s?                          26
Membase Datatypes• byte[]  – Does your data have    1s and 0s?                          “Any customer can have            ...
Membase Datatypes• byte[]  – Does your data have    1s and 0s?• Items do have flags  – Many clients use flags – Data type ...
Transactions• Lock == slow me down• CAS operations                    User 1          User 2  – Optimistic locking        ...
Common Use: Sessions• Web user sessions  – Highly read, less writes in many case  – Protocol advantage of memcached     • ...
Common Use (cache): Rate Limiting• Want to provide API calls  into the system                   Your Users  – Twitter sear...
Looking Ahead
Frequent Requests: New Access Methods  Beyond key-value  • Indexing/Range Queries  • Advanced Data Structures  • Sub-objec...
Feedback: your turn!• Querying and indexing?• Use cases you can share?                             32
NodeCode - What is it? Method for extending & customizing Membase Separate code modules Defined interface to datapath and c...
NodeCode – Drivers Simple •   Packaged modules for easy install and enable •   Library of “off the shelf” modules •   Modu...
Block-level architecture                           35
Q&A Matt Ingenthron  matt@membase.com  Twitter: ingenthr  IRC: #membase                      36
Attributions•   http://commons.wikimedia.org/wiki/File:Flag_of_China.png•   http://commons.wikimedia.org/wiki/File:Flag_of...
Membase Meetup Chicago - january 2011
Upcoming SlideShare
Loading in...5
×

Membase Meetup Chicago - january 2011

1,649

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,649
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
25
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Membase Meetup Chicago - january 2011"

  1. 1. Chicago
  2. 2. What is Membase?
  3. 3. Membase is a distributed database Application user Web application server Web application server Web application server Membase Servers In the data center On the administrator console 3
  4. 4. Membase is Simple, Fast, Elastic Five minutes or less to a working cluster • Downloads for Linux and Windows • Start with a single node • One button press joins nodes to a cluster Easy to develop against • Just SET and GET – no schema required • Drop it in. 10,000+ existing applications already “speak membase” (via memcached) • Practically every language and application framework is supported, out of the box Easy to manage • One-click failover and cluster rebalancing • Graphical and programmatic interfaces • Configurable alerting 4
  5. 5. Membase is Simple, Fast, Elastic Predictable • “Never keep an application waiting” • Quasi-deterministic latency and throughput Low latency • Built-in Memcached technology High throughput • Multi-threaded • Low lock contention • Asynchronous wherever possible • Automatic write de-duplication 5
  6. 6. Membase is Simple, Fast, Elastic Zero-downtime elasticity • Spread I/O and data across commodity servers (or VMs) • Consistent performance with linear cost • Dynamic rebalancing of a live cluster All nodes are created equal • No special case nodes • Any node can replace any other node, online • Clone to grow Extensible • Filtered TAP interface provides hook points for external systems (e.g. full-text search, backup, warehouse) • Data bucket – engine API for specialized container types 6
  7. 7. Built-in Memcached Caching Layer Memcached Membase Cache Membase Database Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project. 7
  8. 8. Use Cases
  9. 9. Ad targeting 40 milliseconds to come up with an answer. profiles, real time campaign 3 statistics 2 1 profiles, campaigns events 9
  10. 10. Search and Gaming Portal Database 10
  11. 11. Membase Architecture
  12. 12. Clustering • Underlying cluster functionality based on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 12
  13. 13. TAP • A generic, scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Data Mutations Data set DataMutations Data set Data set 13
  14. 14. Membase data flow – under the hood SET request arrives at SET acknowledgement KEY’s master server 1 2 returned to application 3 Listener-Sender 3 RAM membase storage engine 4 Disk Disk Disk Disk Disk DiskReplica Server 1 for KEY Master server for KEY Replica Server 2 for KEY 14
  15. 15. A Membase Node memcached operations memcached smart (any client) client operations REST Interface Membase Node 15
  16. 16. A Membase Node: Component View moxi ns_server membase ns_server (memcached + membase engine) TAP vbucketmigrator 16
  17. 17. Clients, nodes and other nodes Client moxi + Client port 11210 port 11211 memcached operations REST/comet memcached operations cluster topology and vbucket map memcached operations moxi ns_server membase ns_server (memcached + membase engine) TAP memcached operations with tap commands vbucketmigrator 17
  18. 18. Data buckets are secure membase “slices” Application user Web application server Bucket 1 Bucket 2 Aggregate Cluster Memory and Disk Capacity Membase data servers In the data center On the administrator console 18
  19. 19. vBucket mapping 19
  20. 20. Disk > MemoryDataset may have many memory quota Bucket Configurationitems infrequently accessed. mem_high_watHowever, memcached has mem_low_watdifferent behavior (LRU) thanwanted with membase.Still, traditional (most)RDBMS implementations arenot 100% correct for useither. The speed of a missis very, very important. 20
  21. 21. (Intermission)
  22. 22. Thanks! 22
  23. 23. Membase Demo
  24. 24. Key-Value Patterns
  25. 25. Key-Value 25
  26. 26. Key-Value Items have: Operations include: Key Get/Set Value Increment/Decrement Expiration Append/Prepend Flags CAS (more on this later) 25
  27. 27. Key-Value 25
  28. 28. Key-Value (with a replica )Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/ 25
  29. 29. Membase Datatypes 26
  30. 30. Membase Datatypes• byte[] – Does your data have 1s and 0s? 26
  31. 31. Membase Datatypes• byte[] – Does your data have 1s and 0s? “Any customer can have a car painted any colour that he wants so long as it is black.” 26
  32. 32. Membase Datatypes• byte[] – Does your data have 1s and 0s?• Items do have flags – Many clients use flags – Data type options • Google protobuf “Any customer can have a car painted any colour • Thrift that he wants so long as • Avro it is black.” 26
  33. 33. Transactions• Lock == slow me down• CAS operations User 1 User 2 – Optimistic locking Su• Very useful with complex cc il! es Fa datatypes s – Imagine two clients trying to update a complex item• You’re likely using CAS already... if you use a CPU 27
  34. 34. Common Use: Sessions• Web user sessions – Highly read, less writes in many case – Protocol advantage of memcached • Options already for PHP, Ruby and Java• Application state – Not necessarily “entity” style things – May be appropriate for a “cache” pool 28
  35. 35. Common Use (cache): Rate Limiting• Want to provide API calls into the system Your Users – Twitter search h ! uc ¡O – Google search services• Use the atomic increment Your App – Set an item with a unique ID – Upon API request, increment and check • HTTP 420: go away and come back later 29
  36. 36. Looking Ahead
  37. 37. Frequent Requests: New Access Methods Beyond key-value • Indexing/Range Queries • Advanced Data Structures • Sub-object direct manipulation Validation and In-flight transformation • Block mutations failing validation • Enrich or transform objects Connectors (Integrate easily with other systems) • Solr • Hadoop 31
  38. 38. Feedback: your turn!• Querying and indexing?• Use cases you can share? 32
  39. 39. NodeCode - What is it? Method for extending & customizing Membase Separate code modules Defined interface to datapath and cluster manager Notification on events • Synchronous • Asynchronous 33
  40. 40. NodeCode – Drivers Simple • Packaged modules for easy install and enable • Library of “off the shelf” modules • Module monitoring • Straight forward development and debugging Fast • Low latency/high-throughput • Per-bucket process isolation • Don’t break data manager performance/correctness Elastic • Automatically migrate and instantiate on rebalance • Provide support for migration of internal data • Leverage native Membase engine for internal data storage 34
  41. 41. Block-level architecture 35
  42. 42. Q&A Matt Ingenthron matt@membase.com Twitter: ingenthr IRC: #membase 36
  43. 43. Attributions• http://commons.wikimedia.org/wiki/File:Flag_of_China.png• http://commons.wikimedia.org/wiki/File:Flag_of_South_Korea.svg• http://commons.wikimedia.org/wiki/File:Flag_of_Japan.svg 37
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×