Downloaded 41 times
Uploaded byTim Lossen
Memory: The New Disk
This document discusses using Redis as a database for the backend of a Facebook game application. It describes the requirements of supporting 1 million daily users with high write throughput needs. A Redis database was chosen because it provides fast in-memory performance suitable for the application's random access workload. Redis was able to meet the throughput requirements of 200,000 requests per minute and support storing 100KB of data per user in memory. The document provides advice to choose the right tool for the job and avoid sharding until necessary to keep the database configuration simple.
More Related Content
Similar to Memory: The New Disk
Memory: The New Disk
- 1. Memory: The New Disk BeuthHochschule für Technik 2010-11-09
- 2. Who? • Tim Lossen/ @tlossen • Ruby developer • backend developer at ....
- 4.
- 5. Requirements • backend forfacebook game
- 6. Requirements • backend forfacebook game • 1 mio. daily users • 100 KB data per user
- 7. Requirements • peak traffic: -10.000 concurrent users - 3.000 requests/second
- 8. Requirements • peak traffic: -10.000 concurrent users - 3.000 requests/second • write-heavy workload
- 9.
- 10. Scaling Reads • caching(memcached)
- 11. Scaling Reads • caching(memcached) • read-only slaves - horizontal scaling
- 12. Scaling Reads • caching(memcached) • read-only slaves - horizontal scaling • easy
- 13. Scaling Writes • limit:1000 writes/second on EC2
- 14. Scaling Writes • limit:1000 writes/second on EC2 • sharding - split database into parts
- 15. Scaling Writes • limit:1000 writes/second on EC2 • sharding - split database into parts • difficult
- 16.
- 17.
- 18.
- 19.
- 20. Relative Latency Memory 1 SSD1000 Disk x 1000
- 21. Relative Latency Memory 1 SSD1000 Disk 100 000 x 1000 x 100
- 22. Pizza Delivery Time Memory30 minutes SSD 3 weeks Disk 5.5 years x 1000 x 100
- 23. “Memory is thenew disk, disk is the new tape.” — Jim Gray
- 24. Rule of thumb •random access = memory • sequential access = disk
- 25. Euro per GB Memory20 SSD 2 Disk 0.05 ÷ 10 ÷ 40
- 26.
- 27. Architecture • key-value-store • in-memorydatabase - with virtual memory
- 28.
- 29. Durability • full databasedumps • append-only log
- 30. Killer Feature • high(write) throughput - 30 to 150 K operations / second
- 31. Other Features • interestingdata structures - lists, hashes, (sorted) sets - atomic operations
- 32.
- 33. Architecture • single Redismaster - with virtual memory - handles all reads / writes
- 34. Architecture • single Redismaster - with virtual memory - handles all reads / writes • single Redis slave - as hot standby (for failover)
- 35. Throughput • redis-benchmark - 60K ops / s = 3.6 mio ops / m
- 36. Throughput • redis-benchmark - 60K ops / s = 3.6 mio ops / m • monitoring (rpm, scout) - ca. 10 ops per request
- 37. Throughput • redis-benchmark - 60K ops / s = 3.6 mio ops / m • monitoring (rpm, scout) - ca. 10 ops per request • 200 K req / m = 2.0 mio ops / m ✔
- 38. Capacity 1 • 100KB / user (on disk) • 10.000 concurrent users (peak)
- 39. Capacity 1 • 100KB / user (on disk) • 10.000 concurrent users (peak) • 1 GB memory - (plus Redis overhead) ✔
- 40. Capacity 2 • Rediskeeps all keys in memory • 10 mio. total users • 20 GB / 100 mio. integer keys
- 41. Capacity 2 • Rediskeeps all keys in memory • 10 mio. total users • 20 GB / 100 mio. integer keys • 2 GB memory for keys ✔
- 42. Data model • oneRedis hash per user - key: facebook id
- 43. Data model • oneRedis hash per user - key: facebook id • store data as serialized JSON - booleans, strings, numbers, timestamps ...
- 44. Advantages • efficient toswap user data in / out
- 45. Advantages • efficient toswap user data in / out • turns Redis into “document db” - atomic ops on parts
- 46. Advantages • efficient toswap user data in / out • turns Redis into “document db” - atomic ops on parts • easy to dump / restore user data
- 47.
- 48. Advice • use theright tool for the job
- 49. Advice • use theright tool for the job • avoid sharding - as long as possible
- 50. Advice • use theright tool for the job • avoid sharding - as long as possible • keep it simple
- 51.
- 52. Nosql Night Berlin November17, 19:30 newthinking store Tucholskystr. 48
- 53. Links • "A Conversationwith Jim Gray" •redis.io •tim.lossen.de •wooga.com/jobs