Tokyo Cabinet / Tyrant @ Nosql Paris

Feedbacks: Tokyo Cabinet & Tyrant Florent Solt — Netvibes ﬂorent@solt.biz — twitter.com/ﬂorentsolt

Tokyo *what* ? Tokyo Cabinet http://1978th.net/tokyocabinet/ « Tokyo Cabinet is a library of routines for managing a database » 1 database = 1 ﬁle key-value storage written in C, ofﬁcial packages in Perl, Ruby, Java, Lua

Tokyo Cabinet Memory or Filesystem Compression bzip or gzip Dynamic defragmentation Hash, BTree, Fixed-Length, Table

Tokyo Cabinet — Hash

Tokyo Cabinet — BTree

Tokyo Cabinet — Fixed

Tokyo Cabinet — Table

Tokyo *what* ? Tokyo Tyrant http://1978th.net/tokyotyrant/ « Tokyo Tyrant is a package of network interface to the DBM called Tokyo Cabinet » binary, http and memcached protocols epoll / kqueue / eventports Written in C, ofﬁcial packages in Perl and Ruby

Tokyo Tyrant No conﬁguration ﬁles (command line only) Lua extention Master/Slave and Master/Master async replication Backup (via the copy fonction) 1 process = 1 database

Examples Run a memory hash server (same as memcached) ttserver Sample queries memcat --servers=127.0.0.1:1978 1 curl 127.0.0.1:1978/1

Examples Run a btree on port 11211 (to simulate a memcache server) and save all data compressed on the disk ttserver -dmn -port 11211 -log /path/to/server.log /path/ to/cache.tcb#opts=ld#bnum=1000000

Examples Run a table with some indexes and enable master- master replication Master 1 mkdir ulog1; ttserver -dmn -port 1978 -ulog ulog1 -sid 1 -mhost localhost -mport 1979 -rts 1.rts /path/to/ table1.tct#idx=foo:dec#idx=bar Master 2 mkdir ulog2; ttserver -dmn -port 1979 -ulog ulog2 -sid 2 -mhost localhost -mport 1978 -rts 2.rts /path/to/ table2.tct#idx=foo:dec#idx=bar

Benchmarks or not Benchmarks ?

Netvibes architecture Queue Feed Sort Item Subscriber Flag Enclosure User

Netvibes architecture Table + Hash + Lua Table BTree Shard Queue Feed Sort Item Hash + Lua Subscriber Flag Enclosure BTree + Lua + Table Shard User Table

Netvibes architecture Hundreds Millions Millions Millions of millions Queue Feed Sort Item Subscriber Flag Enclosure Hundreds Dozens of millions of millions User

Queue implementation — Example multi queue (via preﬁx) support « adddouble » function helps to avoid conﬂicts « putkeep » function prevent overwriting

Queue implementation — Example « fwmkeys » ﬁnds all items with the given preﬁx multiple dequeue support

Queue implementation — Netvibes One queue / minute Automatic slot change When a queue empty and not in the slot representing the current time No duplicates Check if an item is already in the queue

Strengths and Weaknesses

Strengths and Weaknesses Weaknesses No bug tracker, no public code repository The documentation is not good enough Under heavy load, the master-master replication failed, we fallback to Master-Slave (and still have some issues) Databases can be corrupted Process killed, System crash or reboot

Strengths and Weaknesses Weaknesses With big tables, queries need a lot of RAM and time Tables seems slow, we may avoid them in the future and prefer Hash or BTree Need a lots of time to understand all conﬁguration options and their impact No live backup, the copy function lock the database

Strengths and Weaknesses Strengths Easy to deploy and setup Easy to use It’s not a black box Good to very good performance for most of the time

Strengths and Weaknesses Strengths Small memory footprint A single Tyrant process can handle thousands of open connections Many command line tools Lua extensions

And so what ?

Links Google groups of english spoken users http://groups.google.com/group/tokyocabinet-users Wiki: unofﬁcial page to collect links http://tokyocabinetwiki.pbworks.com/ Blog of Ilya Grigorik http://www.igvita.com/2009/02/13/tokyo-cabinet- beyond-key-value-store http://www.igvita.com/2009/07/13/extending-tokyocabinet-db-with-lua/

http://nosql.mypopescu.com	627
http://www.slideshare.net	54
http://irr.posterous.com	6
http://www.papiersdundev.com	3
http://translate.googleusercontent.com	3
http://www.hanrss.com	2
http://webcache.googleusercontent.com	1
http://static.slidesharecdn.com	1
https://d4y2u9.appspot.com	1

Tokyo Cabinet / Tyrant @ Nosql Paris

by florentsolt

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 698

Statistics

Tokyo Cabinet / Tyrant @ Nosql Paris Presentation Transcript