Your SlideShare is downloading. ×
Tricking out your Memcached Setup
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Tricking out your Memcached Setup

4,184
views

Published on

OSCON 2010 presentation on tips and tricks for Memcached.

OSCON 2010 presentation on tips and tricks for Memcached.

Published in: Technology

0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,184
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
72
Comments
0
Likes
11
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Tricking out your Memcached Setup Wednesday, July 28, 2010
  • 2. Chaos Factor Wednesday, July 28, 2010
  • 3. ASCII Protocol [brian@gaz libmemcached]$ memcp foo --servers=localhost [brian@gaz libmemcached]$ telnet localhost 11211 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. get foo VALUE foo 0 4 bar END version VERSION 1.4.5 Wednesday, July 28, 2010
  • 4. Binary Protocol MAGIC Opcode Key Length (1 byte) (1 byte) (2 bytes) Extra Length Data Type Reserved (1 byte) (1 byte) (2 bytes) Total Body Length (4 bytes) Opaque (4 bytes) CAS (Compare and Swap) (8 bytes) Wednesday, July 28, 2010
  • 5. Pipelining! GET FOO GET DOG GET HELP PACKET Wednesday, July 28, 2010
  • 6. New Record Item goes into nearest node Consistent Hashing Wednesday, July 28, 2010
  • 7. libhashkit • One_at_a_time • MD5 • HSIES (patents anyone?) • CRC • murmur (pretty simple) Wednesday, July 28, 2010
  • 8. Replica Node: Node: B C Node: A Node: D Store: Shopping_key Wednesday, July 28, 2010
  • 9. Replica Node: Node: B C Node: A Node: D GET: Shopping_key Wednesday, July 28, 2010
  • 10. STAT! • memstat --servers=localhost “args” • slab • items • sizes Wednesday, July 28, 2010
  • 11. slab STAT 1:chunk_size 104 STAT 1:chunks_per_page 10082 STAT 1:total_pages 1 STAT 1:total_chunks 10082 STAT 1:used_chunks 10081 STAT 1:free_chunks 1 STAT 1:free_chunks_end 10079 STAT 9:chunk_size 696 STAT 9:chunks_per_page 1506 STAT 9:total_pages 63 STAT 9:total_chunks 94878 STAT 9:used_chunks 94878 STAT 9:free_chunks 0 STAT 9:free_chunks_end 0 STAT active_slabs 2 STAT total_malloced 67083616 END Wednesday, July 28, 2010
  • 12. items STAT items:2:number 1 STAT items:2:age 452 STAT items:2:evicted 0 STAT items:2:evicted_nonzero 0 STAT items:2:evicted_time 2 STAT items:2:outofmemory 0 STAT items:2:tailrepairs 0 ... STAT items:27:number 1 STAT items:27:age 452 STAT items:27:evicted 0 STAT items:27:evicted_nonzero 0 STAT items:27:evicted_time 2 STAT items:27:outofmemory 0 STAT items:27:tailrepairs 0 Wednesday, July 28, 2010
  • 13. sizes 96 35 128 38 160 807 192 804 224 410 256 222 288 83 320 39 352 53 384 33 416 64 448 51 480 30 512 54 544 39 576 10065 Wednesday, July 28, 2010
  • 14. Dump your data? • memdump • memcached_dump(memcached_st *ptr, memcached_dump_fn *function, void *context, uint32_t number_of_callbacks); Wednesday, July 28, 2010
  • 15. Drain for (items= memdump()) { store(items); memcached_delete(); } Wednesday, July 28, 2010
  • 16. libmemcachedutil • libmemcached_util_ping(); • libmemcached_util_version_check(); • memcached_pool_st Wednesday, July 28, 2010
  • 17. memcached_pool_st* pool= memcached_pool_create(memc, 5, 10); memcached_st* mmc[10]; for (size_t x= 0; x < 10; ++x) { mmc[x]= memcached_pool_pop(pool, false, &rc); …; } memcached_pool_push(pool, mmc[x]) Wednesday, July 28, 2010
  • 18. ...and now for databases Wednesday, July 28, 2010
  • 19. Wednesday, July 28, 2010
  • 20. Setting via SELECT select id, url, memc_set(concat('feeds', md5(url)), url) from feeds; select memc_get(concat('feeds', md5(url))) from feeds; +-------------------------------------------------------+ | memc_get(concat('feeds', md5(url))) | +-------------------------------------------------------+ | http://feeds.feedburner.com/littlegreenfootballs/Ilds | | http://del.icio.us/rss/jacomien | | http://del.icio.us/rss/tags/rmj20 | | http://www.jihadwatch.org/index.rdf | | http://www.connotea.org/rss/user/rmj20 | | http://devblog.grazr.com/?feed=rss2x | | http://feeds.feedburner.com/littlegreenfootballs/kyeH | +-------------------------------------------------------+ Wednesday, July 28, 2010
  • 21. Trigger DROP TRIGGER IF EXISTS feed_insert; CREATE TRIGGER feed_insert BEFORE INSERT ON feeds FOR EACH ROW BEGIN SET @mm= memc_set(concat('feeds:',md5(NEW.url)), NEW.url);END | insert into feeds (url) values ('http://grazr.com/feedlist.xml'); select memc_get(concat('feeds:', md5('http://grazr.com/feedlist.xml'))); +------------------------------------------------------------------+| memc_get(concat('feeds:', md5('http://grazr.com/feedlist.xml'))) | +------------------------------------------------------------------+| http://grazr.com/feedlist.xml | +------------------------------------------------------------------+ Wednesday, July 28, 2010
  • 22. Memcached Replication via MySQL INSERT INTO table_a VALUES (“key”, “value”, “values”); INSERT INTO blackhole_table VALUES (memc_delete(“key”)); Wednesday, July 28, 2010
  • 23. drizzle> select table_name -> from information_schema.tables -> where table_name like '%MEMCACHED%'; +--------------------+ | table_name | +--------------------+ | MEMCACHED_STATS | | MEMCACHED_ANALYSIS | +--------------------+ 2 rows in set (0 sec) Wednesday, July 28, 2010
  • 24. drizzle> desc information_schema.memcached_stats; +-----------------------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------------------+-------------+------+-----+---------+-------+ | NAME | varchar(32) | NO | | | | | PORT_NUMBER | bigint | NO | | 0 | | | PROCESS_ID | bigint | NO | | 0 | | | UPTIME | bigint | NO | | 0 | | | TIME | bigint | NO | | 0 | | | VERSION | varchar(8) | NO | | | | | POINTER_SIZE | bigint | NO | | 0 | | | RUSAGE_USER | bigint | NO | | 0 | | | RUSAGE_SYSTEM | bigint | NO | | 0 | | | CURRENT_ITEMS | bigint | NO | | 0 | | Wednesday, July 28, 2010
  • 25. drizzle> desc information_schema.memcached_analysis; +--------------------------------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------------------------+-------------+------+-----+---------+-------+ | SERVERS_ANALYZED | bigint | NO | | 0 | | | AVERAGE_ITEM_SIZE | bigint | NO | | 0 | | | NODE_WITH_MOST_MEM_CONSUMPTION | varchar(32) | NO | | | | | USED_BYTES | bigint | NO | | 0 | | | NODE_WITH_LEAST_FREE_SPACE | varchar(32) | NO | | | | | FREE_BYTES | bigint | NO | | 0 | | | NODE_WITH_LONGEST_UPTIME | varchar(32) | NO | | | | | LONGEST_UPTIME | bigint | NO | | 0 | | | POOL_WIDE_HIT_RATIO | bigint | NO | | 0 | | +--------------------------------+-------------+------+-----+---------+-------+ Wednesday, July 28, 2010
  • 26. libmemcachedprotocol • Ascii • Binary Wednesday, July 28, 2010
  • 27. memcached_binary_protocol_callback_st interface_v1_impl= { .interface_version= MEMCACHED_PROTOCOL_HANDLER_V1, .interface.v1= { .add= add_handler, .append= append_handler, .decrement= decrement_handler, .delete= delete_handler, .flush= flush_handler, .get= get_handler, .increment= increment_handler, .noop= noop_handler, .prepend= prepend_handler, .quit= quit_handler, .replace= replace_handler, .set= set_handler, .stat= stat_handler, .version= version_handler } }; Wednesday, July 28, 2010
  • 28. ./examples/memcached_light (everyone needs their own persistent Memcached server) Wednesday, July 28, 2010
  • 29. Examples! • tests/mem_functions.c • man “libmemcached_examples” Wednesday, July 28, 2010
  • 30. Brian Aker (brian.aker@datadifferential.com) twitter:brianaker url: Krow.net http://DataDifferential.com Wednesday, July 28, 2010