1. Feedbacks:
Tokyo Cabinet & Tyrant
Florent Solt — Netvibes
florent@solt.biz — twitter.com/florentsolt

2. Tokyo *what* ?
Tokyo Cabinet
http://1978th.net/tokyocabinet/
« Tokyo Cabinet is a library of routines for managing a
database »
1 database = 1 file
key-value storage
written in C, official packages in Perl, Ruby, Java, Lua

3. Tokyo Cabinet
Memory or Filesystem
Compression bzip or gzip
Dynamic defragmentation
Hash, BTree, Fixed-Length, Table

4. Tokyo Cabinet — Hash

5. Tokyo Cabinet — BTree

6. Tokyo Cabinet — Fixed

7. Tokyo Cabinet — Table

8. Tokyo *what* ?
Tokyo Tyrant
http://1978th.net/tokyotyrant/
« Tokyo Tyrant is a package of network interface to
the DBM called Tokyo Cabinet »
binary, http and memcached protocols
epoll / kqueue / eventports
Written in C, official packages in Perl and Ruby

9. Tokyo Tyrant
No configuration files (command line only)
Lua extention
Master/Slave and Master/Master async replication
Backup (via the copy fonction)
1 process = 1 database

10. Examples
Run a memory hash server (same as memcached)
ttserver
Sample queries
memcat --servers=127.0.0.1:1978 1
curl 127.0.0.1:1978/1

11. Examples
Run a btree on port 11211 (to simulate a memcache
server) and save all data compressed on the disk
ttserver -dmn -port 11211 -log /path/to/server.log /path/
to/cache.tcb#opts=ld#bnum=1000000

12. Examples
Run a table with some indexes and enable master-
master replication
Master 1
mkdir ulog1; ttserver -dmn -port 1978 -ulog ulog1 -sid
1 -mhost localhost -mport 1979 -rts 1.rts /path/to/
table1.tct#idx=foo:dec#idx=bar
Master 2
mkdir ulog2; ttserver -dmn -port 1979 -ulog ulog2 -sid
2 -mhost localhost -mport 1978 -rts 2.rts /path/to/
table2.tct#idx=foo:dec#idx=bar

13. Benchmarks or not
Benchmarks ?

14. Netvibes architecture
Queue Feed Sort Item
Subscriber Flag Enclosure
User

15. Netvibes architecture
Table +
Hash + Lua Table BTree
Shard
Queue Feed Sort Item
Hash + Lua
Subscriber Flag Enclosure
BTree +
Lua + Table
Shard
User
Table

16. Netvibes architecture
Hundreds
Millions Millions Millions
of millions
Queue Feed Sort Item
Subscriber Flag Enclosure
Hundreds Dozens
of millions of millions
User

17. Queue implementation —
Example
multi queue (via prefix)
support
« adddouble » function
helps to avoid conflicts
« putkeep » function
prevent overwriting

18. Queue implementation —
Example
« fwmkeys » finds all
items with the given
prefix
multiple dequeue
support

19. Queue implementation —
Netvibes
One queue / minute
Automatic slot change
When a queue empty and not in the slot representing
the current time
No duplicates
Check if an item is already in the queue

20. Strengths and Weaknesses

21. Strengths and Weaknesses
Weaknesses
No bug tracker, no public code repository
The documentation is not good enough
Under heavy load, the master-master replication
failed, we fallback to Master-Slave (and still have
some issues)
Databases can be corrupted
Process killed, System crash or reboot

22. Strengths and Weaknesses
Weaknesses
With big tables, queries need a lot of RAM and time
Tables seems slow, we may avoid them in the future
and prefer Hash or BTree
Need a lots of time to understand all configuration
options and their impact
No live backup, the copy function lock the database

23. Strengths and Weaknesses
Strengths
Easy to deploy and setup
Easy to use
It’s not a black box
Good to very good performance for most of the time

24. Strengths and Weaknesses
Strengths
Small memory footprint
A single Tyrant process can handle thousands of
open connections
Many command line tools
Lua extensions

25. And so what ?

26. Links
Google groups of english spoken users
http://groups.google.com/group/tokyocabinet-users
Wiki: unofficial page to collect links
http://tokyocabinetwiki.pbworks.com/
Blog of Ilya Grigorik
http://www.igvita.com/2009/02/13/tokyo-cabinet-
beyond-key-value-store
http://www.igvita.com/2009/07/13/extending-tokyo-
cabinet-db-with-lua/