1
<Insert Picture Here>The Native NDB Engine for MemcachedJohn David Duncanjohn.duncan@oracle.com                           ...
Program Agenda                                   <Insert Picture Here>•   MySQL Cluster Today•   A little bit about memcac...
The following is intended to outline our generalproduct direction. It is intended for informationpurposes only, and may no...
<Insert Picture Here>MySQL Cluster Today                                              5
The Basics                      API Nodes     Node Group 1                Node Group 2                    NDB Data Nodes  ...
The Basics – Data Storage  • High Read and Write Performance  – Automatic data partitioning, multi-master, parallel execut...
The Basics – Data Access                         API Nodes • High throughput – tens-to-hundreds of thousands of transactio...
Recent History• Cluster 6.3 (2008)  – multi-master replication, key-distribution awareness• Cluster 7.0 (2009)  – Multi-th...
ClusterJ ExampleEmployee findEmployee(long id) {   Employee employee = session.find(Employee.class, id);   return employee;}...
MySQL Cluster 7.2 Beta• Push-Down Joins  – Many “SPJ” (Select/Project/Join) operations can be executed    at the data node...
<Insert Picture Here>About memcache           • It’s a cache!           • It’s not your data store!           • If it fail...
Two levels of hashing                                           memcached  httpd                                          ...
Cache hit  httpd                                                                    hash keyPHP/Perl                      ...
Cache miss (1): fetch from DB  httpd                                                       hash keyPHP/Perl               ...
Cache miss (2): manage cache  httpd                                                               hash keyPHP/Perl        ...
Data change (1): Write to DB httpdPHP/Perl mysql            INSERT INTO user_friends            (user_id, friend_id)      ...
Data change (2): manage cache httpdPHP/Perl                                   memcached mysql                             ...
Expected Latency & Throughput  httpd                                ons/sec.                      f operati           10 ,...
Cost per n active users  httpdPHP/Perl                    $1memcache mysql                                 $10            ...
A little bit more history• Memcached 1.2 (2007)  – The 2000-line Facebook patch     • UDP support     • Vector I/O and oth...
<Insert Picture Here>Design Decisions                                           22
Goals• Access NDB data from memcache clients – Memcached perspective:   • NDB is a reliable, write-scalable, replicated da...
Goals• Support existing schemas and all MySQL data types• Cache NDB data inside memcached  – with automatic cache manageme...
Which codebase?                  memcached.org libmemcached   build our own Text Protocol         ✔Binary Protocol        ...
Server Architecture• Memcached separate from ndbd        • Memcached inside ndbd  – Definitely!                       – Ma...
Decisions• Use memcached 1.6 tree• Isolate the NDB-specific code into an ndb_engine  – which currently exists outside the ...
Other decisions• Memcache INCR & DECR  – Fully supported  – Atomic operations  – Performed at the data node• Memcache CAS ...
Limitations• The size of stored values is limited to the NDB row size  – currently just less than 8 KB  – vs. 1 MB typical...
Architecture Overview       Application                     Application memcached                      mysqld             ...
<Insert Picture Here>Configuring It                                         31
What’s Configurable• Does it use local cache?• Does it use NDB?• What columns hold the keys?• What columns hold the data?•...
A Key Prefix                   user:1248      the prefix               the database key                                   ...
Fundamentals• memcached command line specifies a connectstring for  a primary cluster• primary = “where the config is stor...
Standard Tables in ndbmemcache• meta    – stores configuration schema version (for upgrade      compatibility); consider i...
ndb_clusters• cluster_id  – int, referenced by key_prefixes• ndb_connectstring  – varchar(128) – how to reach this cluster...
containers•   name•   schema_name•   table_name (the existing table where your data lives)•   key_columns (comma-separated...
cache_policies• policy_name• get_policy  – enum (cache_only, ndb_only, caching, disabled)• set_policy  – enum (cache_only,...
A key-prefix mappingMemcache                                              Cache   key                  Cluster   Container...
A memcache server role                      key                Con-    Cache                              Cluster         ...
key_prefixes•   server_role_id•   key_prefix•   cluster_id•   policy•   container                     41
demo_table• Not really part of the configuration ...• mkey  – varchar(250) NOT NULL PRIMARY KEY• math_value  – bigint unsi...
<Insert Picture Here>Performance                                      43
Measured Latency            memcachetest -t 2 -M 7000 -c 250001000500                                                     ...
Links• This Presentation  – http://en.oreilly.com/mysql2011/• Memcached 1.6 development tree  – git clone git://github.com...
Demo• What we’ll see:  – Memcache Server Window     • start memcached “sandbox”        – default role (NDB-only)        – ...
47
Upcoming SlideShare
Loading in...5
×

The Native NDB Engine for Memcached

5,669

Published on

From the O'Reilly MySQL Conference & Expo 2011. Integration between MySQL Cluster and memcached 1.6.

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,669
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
172
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

The Native NDB Engine for Memcached

  1. 1. 1
  2. 2. <Insert Picture Here>The Native NDB Engine for MemcachedJohn David Duncanjohn.duncan@oracle.com 2
  3. 3. Program Agenda <Insert Picture Here>• MySQL Cluster Today• A little bit about memcached• Design Decisions• Configuration• Performance• Links• Demo 3
  4. 4. The following is intended to outline our generalproduct direction. It is intended for informationpurposes only, and may not be incorporated into anycontract. It is not a commitment to deliver anymaterial, code, or functionality, and should not berelied upon in making purchasing decisions.The development, release, and timing of any featuresor functionality described for Oracle’s productsremains at the sole discretion of Oracle. 4
  5. 5. <Insert Picture Here>MySQL Cluster Today 5
  6. 6. The Basics API Nodes Node Group 1 Node Group 2 NDB Data Nodes 6
  7. 7. The Basics – Data Storage • High Read and Write Performance – Automatic data partitioning, multi-master, parallel execution • High Availability • Fast Failover – sub-second fault detection and reconfiguration • Scalable – Using commodity hardware Node Group 1 Node Group 2 NDB Data Nodes 7
  8. 8. The Basics – Data Access API Nodes • High throughput – tens-to-hundreds of thousands of transactions per second • Low latency – sub-millisecond response • Multiple Acceess Methods – SQL and NoSQL 8
  9. 9. Recent History• Cluster 6.3 (2008) – multi-master replication, key-distribution awareness• Cluster 7.0 (2009) – Multi-threaded NDB nodes – Ability to add nodes online to a running cluster• Cluster 7.1 (2010) – ClusterJ – MySQL Cluster Manager – ndbinfo (and then MEM support for Cluster) – Windows support• Plus – Extensive improvements to BLOB performance, locking behavior, node restart times, etc. in monthly releases 9
  10. 10. ClusterJ ExampleEmployee findEmployee(long id) { Employee employee = session.find(Employee.class, id); return employee;}• Runs faster than JDBC• Almost as fast as the C++ NDB API• 3 lines of code 10
  11. 11. MySQL Cluster 7.2 Beta• Push-Down Joins – Many “SPJ” (Select/Project/Join) operations can be executed at the data nodes rather than the MySQL server – Long-term effort (presented at this conference last year) – 50x improvement for some queries• MySQL privilege tables can be stored in cluster• Memcache API 11
  12. 12. <Insert Picture Here>About memcache • It’s a cache! • It’s not your data store! • If it fails, you get a cache miss! 12
  13. 13. Two levels of hashing memcached httpd hash keyPHP/Perl r memcached k serve to find data y to picMemcache ha sh ke friends:12389 memcached memcache Key 13
  14. 14. Cache hit httpd hash keyPHP/Perl r memcached k serve to find data y to picMemcache ha sh ke friends:12389 VALUE friends:12389 0 31rn 101, 11009, 11150, 55881, 77798 rn 14
  15. 15. Cache miss (1): fetch from DB httpd hash keyPHP/Perl r memcached k serve to find data y to picMemcache ha sh ke F OUND NOT mysql SELECT friend_id FROM u MySQL ser_frien WHERE ds user_id = Slave ? 15
  16. 16. Cache miss (2): manage cache httpd hash keyPHP/Perl memcached to find dataMemcache n n 9 31r 9 8 r : 1238 81 , 777 iends 1150, 558 set fr 09, 1 10 1, 110 16
  17. 17. Data change (1): Write to DB httpdPHP/Perl mysql INSERT INTO user_friends (user_id, friend_id) VALUES ( 12389, 999101);MySQLMaster 17
  18. 18. Data change (2): manage cache httpdPHP/Perl memcached mysql n 3 89 r nds:12 dele te frieMySQLMaster 18
  19. 19. Expected Latency & Throughput httpd ons/sec. f operati 10 ,000s o u nd tripPHP/Perl ~2 00 µs ro memcachedmemcache mysql 1,000s of operatio ~ 2 ms ro ns/sec. MySQL und trip Slave 19
  20. 20. Cost per n active users httpdPHP/Perl $1memcache mysql $10 20
  21. 21. A little bit more history• Memcached 1.2 (2007) – The 2000-line Facebook patch • UDP support • Vector I/O and other “details” – adding up to 25% improved CPU efficiency • Multithreaded (for a small number of libevent threads) • Compare-And-Set operation (CAS)• Memcached 1.4 (2009): – Binary Protocol – SASL Authentication• Memcached 1.6 (upcoming) – Storage Engines – Logging modules – Windows platform support 21
  22. 22. <Insert Picture Here>Design Decisions 22
  23. 23. Goals• Access NDB data from memcache clients – Memcached perspective: • NDB is a reliable, write-scalable, replicated data store – MySQL Cluster perspective: • memcache is an easy-to-use high performance API 23
  24. 24. Goals• Support existing schemas and all MySQL data types• Cache NDB data inside memcached – with automatic cache management – and flexibility to fine-tune (or disable) the cache policies• Support the whole memcache protocol• Acheive superior performance – latency as expected from memcached – throughput as expected from memcached 24
  25. 25. Which codebase? memcached.org libmemcached build our own Text Protocol ✔Binary Protocol ✔ ✔ TCP ✔ ✔ UDP ✔Authentication ✔ ✔ 25
  26. 26. Server Architecture• Memcached separate from ndbd • Memcached inside ndbd – Definitely! – Maybe! – “m:n” ratio of memcache servers – we know how to do it to data nodes – “embedded” API-to-TC channel – lower latency – network round trip + tens of microseconds – Not as flexible memcached memcached memcached 26
  27. 27. Decisions• Use memcached 1.6 tree• Isolate the NDB-specific code into an ndb_engine – which currently exists outside the memcached source tree – and runs in an unmodified memcached server• Build a standalone server binary• Possibly also build an ndbmtd-embedded server• Share code with InnoDB Memcache team – on basic configuration and user-visible concepts – and on changes that would allow a memcache server module to be loaded into either ndbmtd or mysqld 27
  28. 28. Other decisions• Memcache INCR & DECR – Fully supported – Atomic operations – Performed at the data node• Memcache CAS (version id check) – Fully Supported • either at local cache or at database – CAS check is pushed down to the data node – Ideally an update that comes from a non-memcache API node should invalidate the CAS • we still need to decide the best design for this 28
  29. 29. Limitations• The size of stored values is limited to the NDB row size – currently just less than 8 KB – vs. 1 MB typical limit for memcached – due to the lack of BLOB support in the async NDB API 29
  30. 30. Architecture Overview Application Application memcached mysqld Wide-area Local Cache Replication NDB Engine InnoDB NDB Engine binlog stream NDB Data Data Data Data Cluster Node Node Node Node 30
  31. 31. <Insert Picture Here>Configuring It 31
  32. 32. What’s Configurable• Does it use local cache?• Does it use NDB?• What columns hold the keys?• What columns hold the data?• Are memcache commands like DELETE and FLUSH allowed to delete records from the database?• You decide all of this ...• on a “per-key-prefix” basis. 32
  33. 33. A Key Prefix user:1248 the prefix the database key 33
  34. 34. Fundamentals• memcached command line specifies a connectstring for a primary cluster• primary = “where the config is stored”• The NDB Engine reads the configuration tables from the ndbmemcache database on the primary cluster• You, the administrator, manage the config via SQL• An NDB Memcache server can operate on data in the primary cluster or in other clusters• Different NDB Memcache servers can fetch different configurations 34
  35. 35. Standard Tables in ndbmemcache• meta – stores configuration schema version (for upgrade compatibility); consider it to be read-only• ndb_clusters• containers – where data is stored• cache_policies – how it can be accessed• key_prefixes• memcache_server_roles• last_memcached_signon• demo_table 35
  36. 36. ndb_clusters• cluster_id – int, referenced by key_prefixes• ndb_connectstring – varchar(128) – how to reach this cluster• microsec_rtt – default 250; used for internal performance tuning 36
  37. 37. containers• name• schema_name• table_name (the existing table where your data lives)• key_columns (comma-separated)• value_columns (comma-separated)• flags (either a constant number, or a column name)• increment_column (optional, for INCR/DECR)• cas_column (optional)• expire_time_column (optional) 37
  38. 38. cache_policies• policy_name• get_policy – enum (cache_only, ndb_only, caching, disabled)• set_policy – enum (cache_only, ndb_only, caching, disabled)• delete_policy – enum (cache_only, ndb_only, caching, disabled)• flush_from_db – enum(false, true) 38
  39. 39. A key-prefix mappingMemcache Cache key Cluster Container Policy prefix 39
  40. 40. A memcache server role key Con- Cache Cluster prefix tainer Policy Server key Con- Cache Role Cluster prefix tainer Policy ID key Con- Cache Cluster prefix tainer Policy key Con- Cache Cluster prefix tainer Policy 40
  41. 41. key_prefixes• server_role_id• key_prefix• cluster_id• policy• container 41
  42. 42. demo_table• Not really part of the configuration ...• mkey – varchar(250) NOT NULL PRIMARY KEY• math_value – bigint unsigned• cas_value – bigint unsigned• string_value – varchar(7500) 42
  43. 43. <Insert Picture Here>Performance 43
  44. 44. Measured Latency memcachetest -t 2 -M 7000 -c 250001000500 95th 351 374 avg. 184 min. 98!sec Memcached NDB NDB Write ICMP Default 1-Node 2-Node ping Read Engine Cluster Cluster 44
  45. 45. Links• This Presentation – http://en.oreilly.com/mysql2011/• Memcached 1.6 development tree – git clone git://github.com/memcached/memcached – cd memcached – git checkout -t origin/engine-pu• MySQL Cluster 7.2 with Memcached Engine – bzr clone lp:~mysql/mysql-server/mysql-cluster-7.2-labs-memcached – or look for it at labs.mysql.com• Mailing List – cluster@lists.mysql.com 45
  46. 46. Demo• What we’ll see: – Memcache Server Window • start memcached “sandbox” – default role (NDB-only) – mc-only role (plain memcached) – ndb-caching role (NDB with local cache) – Client Window • run memcapable – MyQSL Window • examine data and configuration 46
  47. 47. 47
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×