My talk at Topconf.com conference, Tallinn, 1st of November 2012
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

My talk at Topconf.com conference, Tallinn, 1st of November 2012

  • 419 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
419
On Slideshare
419
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
3
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Tarantool/Box: a use case withserving 2 billion queries a day November 1st, 2012, Tallinn, Estonia Konstantin Osipov, Mail.Ru kostja@tarantool.org
  • 2. AgendaTarantool/Box: features and architectureCase study 1: advanced memcacheCase study 2: a message queue serverCase study 3: a reliable in-memory databaseProject future
  • 3. Feature overview● Flexible data model, primary and secondary keys● Fully cached: 100% of data is cached in RAM● Persistent using a Write Ahead Log● Log shipping replication and online backup● extensible with Lua
  • 4. Data model● A game of fields, tuples and spaces● HASH, TREE, BITMAP, partial, indexes● Secondary keys● Single-part, multi-part● STRING, NUM and NUM64 data types
  • 5. Server architecture
  • 6. Lua API Redis Tarantool (Lua)redis.set(key, value) box.insert(space, key, value)redis.get(key) box.select(space, 0, key, value)redis.getset(key, newkey) box.update(space, key, =p, 0, newkey)redis.incr(key) box.update(space, key, +p, 1, 1)redis.lpush(key, value) box.update(space, key, !p, 1, value)redis.rpush(key, value) You guess it...
  • 7. Performance overviewIntel I5 , 4G RAM, 7200 RPM SATA10 threads, 200-300 bytes per tupleTarantool 1.4.6: 170k writes,260k reads
  • 8. Memory footprint
  • 9. Raw GET/SET performance
  • 10. Use case 1: flexible memcache● You can create your own fibers● box.fiber.create(), box.fiber.yield()● A background fiber performs a customized expire● Session store: 20M online users, 200M monthly users,4 2-CPU units, 96GB RAM each● 60K RPS, CPU usage is below 20%
  • 11. Use case 2: message queue● Reliable queues are a vital ingredient for buildingscalable applications● In Web Apps queues are used for delayed processing,load balancing, e-mail notifications● Our use case: prefetching avatars
  • 12. Message queues: how● Rich Lua stored procedure environment ― box.fiber, box.space, box.cfg● Specialized inter-procedure communication API: ― box.ipc● Specialized data structures ― Bitmaps and partial keys
  • 13. Use case 3: reliable database● User profile: 500 bytes of key/value pairs● Math: 200M users * 500 = 100GB● At least 2x smaller memory footprint than in a RDBMS● Win: predictable response time, significantly higherperformance
  • 14. Example: a FIFOfunction fifo_push(name, val) fifo = find_or_create_fifo(name) top = box.unpack(i, fifo[1]) bottom = box.unpack(i, fifo[2]) if top == fifomax+2 then -- % size top = 3 … end return box.update(0, name, =p=p=p, 1, top, 2, bottom, top, val)end
  • 15. Conclusion● Tarantool/Box – a high performance data engine● Tarantool/Lua – a building block for your heavily loadedweb applications● Tarantool – our approach to an easy to use databasefor highly volatile data
  • 16. Whats cooking in 1.5● Disk-based backend● Synchronous master-master replication● New data types (array, date, currency, json)● authentication
  • 17. Thank you!
  • 18. Linkshttp://github.com/mailru/tarantool - source codehttp://github.com/mailru/tntlua - open source stored proceduresrepositoryhttp://groups.google.com/group/tarantool - mailing listhttp://tarantool.org/dist/ - always fresh .tar.gz and .rpm