Tokyo Cabinet / Tyrant @ Nosql Paris

  • 4,478 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,478
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
100
Comments
0
Likes
13

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide


























Transcript

  • 1. Feedbacks: Tokyo Cabinet & Tyrant Florent Solt — Netvibes florent@solt.biz — twitter.com/florentsolt
  • 2. Tokyo *what* ? Tokyo Cabinet http://1978th.net/tokyocabinet/ « Tokyo Cabinet is a library of routines for managing a database » 1 database = 1 file key-value storage written in C, official packages in Perl, Ruby, Java, Lua
  • 3. Tokyo Cabinet Memory or Filesystem Compression bzip or gzip Dynamic defragmentation Hash, BTree, Fixed-Length, Table
  • 4. Tokyo Cabinet — Hash
  • 5. Tokyo Cabinet — BTree
  • 6. Tokyo Cabinet — Fixed
  • 7. Tokyo Cabinet — Table
  • 8. Tokyo *what* ? Tokyo Tyrant http://1978th.net/tokyotyrant/ « Tokyo Tyrant is a package of network interface to the DBM called Tokyo Cabinet » binary, http and memcached protocols epoll / kqueue / eventports Written in C, official packages in Perl and Ruby
  • 9. Tokyo Tyrant No configuration files (command line only) Lua extention Master/Slave and Master/Master async replication Backup (via the copy fonction) 1 process = 1 database
  • 10. Examples Run a memory hash server (same as memcached) ttserver Sample queries memcat --servers=127.0.0.1:1978 1 curl 127.0.0.1:1978/1
  • 11. Examples Run a btree on port 11211 (to simulate a memcache server) and save all data compressed on the disk ttserver -dmn -port 11211 -log /path/to/server.log /path/ to/cache.tcb#opts=ld#bnum=1000000
  • 12. Examples Run a table with some indexes and enable master- master replication Master 1 mkdir ulog1; ttserver -dmn -port 1978 -ulog ulog1 -sid 1 -mhost localhost -mport 1979 -rts 1.rts /path/to/ table1.tct#idx=foo:dec#idx=bar Master 2 mkdir ulog2; ttserver -dmn -port 1979 -ulog ulog2 -sid 2 -mhost localhost -mport 1978 -rts 2.rts /path/to/ table2.tct#idx=foo:dec#idx=bar
  • 13. Benchmarks or not Benchmarks ?
  • 14. Netvibes architecture Queue Feed Sort Item Subscriber Flag Enclosure User
  • 15. Netvibes architecture Table + Hash + Lua Table BTree Shard Queue Feed Sort Item Hash + Lua Subscriber Flag Enclosure BTree + Lua + Table Shard User Table
  • 16. Netvibes architecture Hundreds Millions Millions Millions of millions Queue Feed Sort Item Subscriber Flag Enclosure Hundreds Dozens of millions of millions User
  • 17. Queue implementation — Example multi queue (via prefix) support « adddouble » function helps to avoid conflicts « putkeep » function prevent overwriting
  • 18. Queue implementation — Example « fwmkeys » finds all items with the given prefix multiple dequeue support
  • 19. Queue implementation — Netvibes One queue / minute Automatic slot change When a queue empty and not in the slot representing the current time No duplicates Check if an item is already in the queue
  • 20. Strengths and Weaknesses
  • 21. Strengths and Weaknesses Weaknesses No bug tracker, no public code repository The documentation is not good enough Under heavy load, the master-master replication failed, we fallback to Master-Slave (and still have some issues) Databases can be corrupted Process killed, System crash or reboot
  • 22. Strengths and Weaknesses Weaknesses With big tables, queries need a lot of RAM and time Tables seems slow, we may avoid them in the future and prefer Hash or BTree Need a lots of time to understand all configuration options and their impact No live backup, the copy function lock the database
  • 23. Strengths and Weaknesses Strengths Easy to deploy and setup Easy to use It’s not a black box Good to very good performance for most of the time
  • 24. Strengths and Weaknesses Strengths Small memory footprint A single Tyrant process can handle thousands of open connections Many command line tools Lua extensions
  • 25. And so what ?
  • 26. Links Google groups of english spoken users http://groups.google.com/group/tokyocabinet-users Wiki: unofficial page to collect links http://tokyocabinetwiki.pbworks.com/ Blog of Ilya Grigorik http://www.igvita.com/2009/02/13/tokyo-cabinet- beyond-key-value-store http://www.igvita.com/2009/07/13/extending-tokyo- cabinet-db-with-lua/