Tokyo Cabinet / Tyrant @ Nosql Paris

Uploaded on


More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Feedbacks: Tokyo Cabinet & Tyrant Florent Solt — Netvibes —
  • 2. Tokyo *what* ? Tokyo Cabinet « Tokyo Cabinet is a library of routines for managing a database » 1 database = 1 file key-value storage written in C, official packages in Perl, Ruby, Java, Lua
  • 3. Tokyo Cabinet Memory or Filesystem Compression bzip or gzip Dynamic defragmentation Hash, BTree, Fixed-Length, Table
  • 4. Tokyo Cabinet — Hash
  • 5. Tokyo Cabinet — BTree
  • 6. Tokyo Cabinet — Fixed
  • 7. Tokyo Cabinet — Table
  • 8. Tokyo *what* ? Tokyo Tyrant « Tokyo Tyrant is a package of network interface to the DBM called Tokyo Cabinet » binary, http and memcached protocols epoll / kqueue / eventports Written in C, official packages in Perl and Ruby
  • 9. Tokyo Tyrant No configuration files (command line only) Lua extention Master/Slave and Master/Master async replication Backup (via the copy fonction) 1 process = 1 database
  • 10. Examples Run a memory hash server (same as memcached) ttserver Sample queries memcat --servers= 1 curl
  • 11. Examples Run a btree on port 11211 (to simulate a memcache server) and save all data compressed on the disk ttserver -dmn -port 11211 -log /path/to/server.log /path/ to/cache.tcb#opts=ld#bnum=1000000
  • 12. Examples Run a table with some indexes and enable master- master replication Master 1 mkdir ulog1; ttserver -dmn -port 1978 -ulog ulog1 -sid 1 -mhost localhost -mport 1979 -rts 1.rts /path/to/ table1.tct#idx=foo:dec#idx=bar Master 2 mkdir ulog2; ttserver -dmn -port 1979 -ulog ulog2 -sid 2 -mhost localhost -mport 1978 -rts 2.rts /path/to/ table2.tct#idx=foo:dec#idx=bar
  • 13. Benchmarks or not Benchmarks ?
  • 14. Netvibes architecture Queue Feed Sort Item Subscriber Flag Enclosure User
  • 15. Netvibes architecture Table + Hash + Lua Table BTree Shard Queue Feed Sort Item Hash + Lua Subscriber Flag Enclosure BTree + Lua + Table Shard User Table
  • 16. Netvibes architecture Hundreds Millions Millions Millions of millions Queue Feed Sort Item Subscriber Flag Enclosure Hundreds Dozens of millions of millions User
  • 17. Queue implementation — Example multi queue (via prefix) support « adddouble » function helps to avoid conflicts « putkeep » function prevent overwriting
  • 18. Queue implementation — Example « fwmkeys » finds all items with the given prefix multiple dequeue support
  • 19. Queue implementation — Netvibes One queue / minute Automatic slot change When a queue empty and not in the slot representing the current time No duplicates Check if an item is already in the queue
  • 20. Strengths and Weaknesses
  • 21. Strengths and Weaknesses Weaknesses No bug tracker, no public code repository The documentation is not good enough Under heavy load, the master-master replication failed, we fallback to Master-Slave (and still have some issues) Databases can be corrupted Process killed, System crash or reboot
  • 22. Strengths and Weaknesses Weaknesses With big tables, queries need a lot of RAM and time Tables seems slow, we may avoid them in the future and prefer Hash or BTree Need a lots of time to understand all configuration options and their impact No live backup, the copy function lock the database
  • 23. Strengths and Weaknesses Strengths Easy to deploy and setup Easy to use It’s not a black box Good to very good performance for most of the time
  • 24. Strengths and Weaknesses Strengths Small memory footprint A single Tyrant process can handle thousands of open connections Many command line tools Lua extensions
  • 25. And so what ?
  • 26. Links Google groups of english spoken users Wiki: unofficial page to collect links Blog of Ilya Grigorik beyond-key-value-store cabinet-db-with-lua/