My talk at conference, Tallinn, 1st of November 2012


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

My talk at conference, Tallinn, 1st of November 2012

  1. 1. Tarantool/Box: a use case withserving 2 billion queries a day November 1st, 2012, Tallinn, Estonia Konstantin Osipov, Mail.Ru
  2. 2. AgendaTarantool/Box: features and architectureCase study 1: advanced memcacheCase study 2: a message queue serverCase study 3: a reliable in-memory databaseProject future
  3. 3. Feature overview● Flexible data model, primary and secondary keys● Fully cached: 100% of data is cached in RAM● Persistent using a Write Ahead Log● Log shipping replication and online backup● extensible with Lua
  4. 4. Data model● A game of fields, tuples and spaces● HASH, TREE, BITMAP, partial, indexes● Secondary keys● Single-part, multi-part● STRING, NUM and NUM64 data types
  5. 5. Server architecture
  6. 6. Lua API Redis Tarantool (Lua)redis.set(key, value) box.insert(space, key, value)redis.get(key), 0, key, value)redis.getset(key, newkey) box.update(space, key, =p, 0, newkey)redis.incr(key) box.update(space, key, +p, 1, 1)redis.lpush(key, value) box.update(space, key, !p, 1, value)redis.rpush(key, value) You guess it...
  7. 7. Performance overviewIntel I5 , 4G RAM, 7200 RPM SATA10 threads, 200-300 bytes per tupleTarantool 1.4.6: 170k writes,260k reads
  8. 8. Memory footprint
  9. 9. Raw GET/SET performance
  10. 10. Use case 1: flexible memcache● You can create your own fibers● box.fiber.create(), box.fiber.yield()● A background fiber performs a customized expire● Session store: 20M online users, 200M monthly users,4 2-CPU units, 96GB RAM each● 60K RPS, CPU usage is below 20%
  11. 11. Use case 2: message queue● Reliable queues are a vital ingredient for buildingscalable applications● In Web Apps queues are used for delayed processing,load balancing, e-mail notifications● Our use case: prefetching avatars
  12. 12. Message queues: how● Rich Lua stored procedure environment ― box.fiber,, box.cfg● Specialized inter-procedure communication API: ― box.ipc● Specialized data structures ― Bitmaps and partial keys
  13. 13. Use case 3: reliable database● User profile: 500 bytes of key/value pairs● Math: 200M users * 500 = 100GB● At least 2x smaller memory footprint than in a RDBMS● Win: predictable response time, significantly higherperformance
  14. 14. Example: a FIFOfunction fifo_push(name, val) fifo = find_or_create_fifo(name) top = box.unpack(i, fifo[1]) bottom = box.unpack(i, fifo[2]) if top == fifomax+2 then -- % size top = 3 … end return box.update(0, name, =p=p=p, 1, top, 2, bottom, top, val)end
  15. 15. Conclusion● Tarantool/Box – a high performance data engine● Tarantool/Lua – a building block for your heavily loadedweb applications● Tarantool – our approach to an easy to use databasefor highly volatile data
  16. 16. Whats cooking in 1.5● Disk-based backend● Synchronous master-master replication● New data types (array, date, currency, json)● authentication
  17. 17. Thank you!
  18. 18. Links - source code - open source stored proceduresrepository - mailing list - always fresh .tar.gz and .rpm