Tokyo Cabinet / Tyrant @ Nosql Paris


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

  • Tokyo Cabinet / Tyrant @ Nosql Paris

    1. 1. Feedbacks: Tokyo Cabinet & Tyrant Florent Solt — Netvibes —
    2. 2. Tokyo *what* ? Tokyo Cabinet « Tokyo Cabinet is a library of routines for managing a database » 1 database = 1 file key-value storage written in C, official packages in Perl, Ruby, Java, Lua
    3. 3. Tokyo Cabinet Memory or Filesystem Compression bzip or gzip Dynamic defragmentation Hash, BTree, Fixed-Length, Table
    4. 4. Tokyo Cabinet — Hash
    5. 5. Tokyo Cabinet — BTree
    6. 6. Tokyo Cabinet — Fixed
    7. 7. Tokyo Cabinet — Table
    8. 8. Tokyo *what* ? Tokyo Tyrant « Tokyo Tyrant is a package of network interface to the DBM called Tokyo Cabinet » binary, http and memcached protocols epoll / kqueue / eventports Written in C, official packages in Perl and Ruby
    9. 9. Tokyo Tyrant No configuration files (command line only) Lua extention Master/Slave and Master/Master async replication Backup (via the copy fonction) 1 process = 1 database
    10. 10. Examples Run a memory hash server (same as memcached) ttserver Sample queries memcat --servers= 1 curl
    11. 11. Examples Run a btree on port 11211 (to simulate a memcache server) and save all data compressed on the disk ttserver -dmn -port 11211 -log /path/to/server.log /path/ to/cache.tcb#opts=ld#bnum=1000000
    12. 12. Examples Run a table with some indexes and enable master- master replication Master 1 mkdir ulog1; ttserver -dmn -port 1978 -ulog ulog1 -sid 1 -mhost localhost -mport 1979 -rts 1.rts /path/to/ table1.tct#idx=foo:dec#idx=bar Master 2 mkdir ulog2; ttserver -dmn -port 1979 -ulog ulog2 -sid 2 -mhost localhost -mport 1978 -rts 2.rts /path/to/ table2.tct#idx=foo:dec#idx=bar
    13. 13. Benchmarks or not Benchmarks ?
    14. 14. Netvibes architecture Queue Feed Sort Item Subscriber Flag Enclosure User
    15. 15. Netvibes architecture Table + Hash + Lua Table BTree Shard Queue Feed Sort Item Hash + Lua Subscriber Flag Enclosure BTree + Lua + Table Shard User Table
    16. 16. Netvibes architecture Hundreds Millions Millions Millions of millions Queue Feed Sort Item Subscriber Flag Enclosure Hundreds Dozens of millions of millions User
    17. 17. Queue implementation — Example multi queue (via prefix) support « adddouble » function helps to avoid conflicts « putkeep » function prevent overwriting
    18. 18. Queue implementation — Example « fwmkeys » finds all items with the given prefix multiple dequeue support
    19. 19. Queue implementation — Netvibes One queue / minute Automatic slot change When a queue empty and not in the slot representing the current time No duplicates Check if an item is already in the queue
    20. 20. Strengths and Weaknesses
    21. 21. Strengths and Weaknesses Weaknesses No bug tracker, no public code repository The documentation is not good enough Under heavy load, the master-master replication failed, we fallback to Master-Slave (and still have some issues) Databases can be corrupted Process killed, System crash or reboot
    22. 22. Strengths and Weaknesses Weaknesses With big tables, queries need a lot of RAM and time Tables seems slow, we may avoid them in the future and prefer Hash or BTree Need a lots of time to understand all configuration options and their impact No live backup, the copy function lock the database
    23. 23. Strengths and Weaknesses Strengths Easy to deploy and setup Easy to use It’s not a black box Good to very good performance for most of the time
    24. 24. Strengths and Weaknesses Strengths Small memory footprint A single Tyrant process can handle thousands of open connections Many command line tools Lua extensions
    25. 25. And so what ?
    26. 26. Links Google groups of english spoken users Wiki: unofficial page to collect links Blog of Ilya Grigorik beyond-key-value-store cabinet-db-with-lua/