1.
LESSONS LEARNED AT REDDIT Steve Huffman FOWA 2010

2.
Reddit.com

3.
A brief history of reddit <ul><li>Founded in June 2005 </li></ul><ul><li>Acquired by Condé Nast October 2007 </li></ul><ul><li>7.5 Million user / month </li></ul><ul><li>270 Million page views / month </li></ul><ul><li>Many mistakes along the way </li></ul>

4.
Lesson 1: Crash! <ul><li>… and restart. </li></ul><ul><li>Daemontools (supervise) </li></ul><ul><li>Single greatest improvement to uptime we ever made. </li></ul><ul><li>When in doubt, let it die. </li></ul><ul><li>Don’t forget to read the logs! </li></ul>

5.
Lesson 2: Separation of services <ul><li>Often, one->two machines more than doubles performance. </li></ul><ul><li>Group similar process together. </li></ul><ul><li>Group similar types of data together. </li></ul><ul><li>Better caching. </li></ul><ul><li>Less contention for CPU. </li></ul><ul><li>Avoid threads. Processes are easier to separate later. </li></ul>

6.
Lesson 3: Open Schema ID UPS DOWNS TITLE URL 12345 120 34 Buffins Create Zombie Dog! www.someaussiesite.co.au/dog.html 12346 3 24 Check out my new blog! noobspamer.blogspot.com 12347 509 167 Pee in a sink if you’ve ever voted up. self

7.
Lesson 3: Open Schema <ul><li>In the early days: </li></ul><ul><li>Too much time spent thinking about the database. </li></ul><ul><li>Every feature required a schema update. </li></ul><ul><li>Schema updates became more painful as we grew. </li></ul><ul><li>Maintaining replication was difficult. </li></ul><ul><li>Deployment was complex. </li></ul>

8.
Lesson 3: Open Schema THING_ID KEY VALUE 12345 Title Boffins Create Zombie Dog! 12345 URL www.someaussiesite.com.au/zombiedog.html 12346 Title Pee in a sink if you’ve ever voted up. 12346 URL self ID UPS DOWNS TYPE 12345 120 34 Link 12346 3 24 Link Thing Data

9.
Lesson 3: Open Schema <ul><li>With an open schema: </li></ul><ul><li>Faster development </li></ul><ul><li>Easier deployment </li></ul><ul><li>Maintainable database replication </li></ul><ul><li>No joins = easy to distribute </li></ul><ul><li>Must be careful to maintain consistency </li></ul>

10.
Lesson 4: Keep it stateless <ul><li>Goal: any app server can handle any request </li></ul><ul><li>App server failure/restart is no big deal </li></ul><ul><li>Scaling is straightforward </li></ul><ul><li>Caching must be independent from a specific app server. </li></ul>

11.
Lesson 5: Memcache everything <ul><li>Database data </li></ul><ul><li>Session data </li></ul><ul><li>Rendered pages </li></ul><ul><li>Memoizing internal functions </li></ul><ul><li>Rate-limiting (user actions, crawlers) </li></ul><ul><li>Storing pre-computing listings/pages </li></ul><ul><li>Global locking </li></ul><ul><li>Memcachedb for persistence </li></ul>

12.
Lesson 6: Store redundant data <ul><li>Recipe for slow: keep data normalized until you need it. </li></ul><ul><li>If data has multiple presentations, store it in multiple times in multiple formats. </li></ul><ul><li>Disk and memory is less costly than making your users wait. </li></ul>

13.
Lesson 7: Work offline <ul><li>Do the minimum amount of work to end the request. </li></ul><ul><li>Everything else can be done offline. </li></ul><ul><li>An architecture of queues is simple and easy to scale. </li></ul><ul><li>AMQP/RabbitMQ. </li></ul>

14.
Lesson 7: Work offline <ul><li>Pre-computing listings </li></ul><ul><li>Fetching thumbnails </li></ul><ul><li>Detecting cheating </li></ul><ul><li>Removing spam </li></ul><ul><li>Computing awards </li></ul><ul><li>Updating the “search” index </li></ul>

15.
Lesson 7: Work offline Master Databases App Servers Worker Databases Cache Precomputer Thumbnailer Spam Request Queue

16.
THANKS! QUESTIONS?