Downloaded 571 times
Uploaded byCarsonified Team
Steve Huffman - Lessons learned while at reddit.com
The document outlines key lessons learned by Reddit's co-founder Steve Huffman regarding the platform's early technical challenges and solutions. Significant insights include implementing crash recovery, the benefits of separation of services, adopting an open schema for flexibility in development, and the importance of statelessness for scalability. Additionally, it emphasizes strategies like caching data, storing redundant information, and enabling offline processing to enhance performance and user experience.
More Related Content
![[DockerCon 2019] Hardening Docker daemon with Rootless mode](https://cdn.slidesharecdn.com/ss_thumbnails/hardeningdockerdaemonwithrootlessmode-190501210234-thumbnail.jpg?width=640&height=640&fit=bounds)
![[OpenInfra Days Korea 2018] (Track 2) Neutron LBaaS 어디까지 왔니? - Octavia 소개](https://cdn.slidesharecdn.com/ss_thumbnails/26octavia-180704054917-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Steve Huffman - Lessons learned while at reddit.com
Steve Huffman - Lessons learned while at reddit.com
- 1. LESSONS LEARNED ATREDDIT Steve Huffman FOWA 2010
- 2.
- 3. A brief historyof reddit Founded in June 2005 Acquired by Condé Nast October 2007 7.5 Million user / month 270 Million page views / month Many mistakes along the way
- 4. Lesson 1: Crash!… and restart. Daemontools (supervise) Single greatest improvement to uptime we ever made. When in doubt, let it die. Don’t forget to read the logs!
- 5. Lesson 2: Separationof services Often, one->two machines more than doubles performance. Group similar process together. Group similar types of data together. Better caching. Less contention for CPU. Avoid threads. Processes are easier to separate later.
- 6. Lesson 3: OpenSchema ID UPS DOWNS TITLE URL 12345 120 34 Buffins Create Zombie Dog! www.someaussiesite.co.au/dog.html 12346 3 24 Check out my new blog! noobspamer.blogspot.com 12347 509 167 Pee in a sink if you’ve ever voted up. self
- 7. Lesson 3: OpenSchema In the early days: Too much time spent thinking about the database. Every feature required a schema update. Schema updates became more painful as we grew. Maintaining replication was difficult. Deployment was complex.
- 8. Lesson 3: OpenSchema THING_ID KEY VALUE 12345 Title Boffins Create Zombie Dog! 12345 URL www.someaussiesite.com.au/zombiedog.html 12346 Title Pee in a sink if you’ve ever voted up. 12346 URL self ID UPS DOWNS TYPE 12345 120 34 Link 12346 3 24 Link Thing Data
- 9. Lesson 3: OpenSchema With an open schema: Faster development Easier deployment Maintainable database replication No joins = easy to distribute Must be careful to maintain consistency
- 10. Lesson 4: Keepit stateless Goal: any app server can handle any request App server failure/restart is no big deal Scaling is straightforward Caching must be independent from a specific app server.
- 11. Lesson 5: Memcache everything Database data Session data Rendered pages Memoizing internal functions Rate-limiting (user actions, crawlers) Storing pre-computing listings/pages Global locking Memcachedb for persistence
- 12. Lesson 6: Storeredundant data Recipe for slow: keep data normalized until you need it. If data has multiple presentations, store it in multiple times in multiple formats. Disk and memory is less costly than making your users wait.
- 13. Lesson 7: Workoffline Do the minimum amount of work to end the request. Everything else can be done offline. An architecture of queues is simple and easy to scale. AMQP/RabbitMQ.
- 14. Lesson 7: Workoffline Pre-computing listings Fetching thumbnails Detecting cheating Removing spam Computing awards Updating the “search” index
- 15. Lesson 7: Workoffline Master Databases App Servers Worker Databases Cache Precomputer Thumbnailer Spam Request Queue
- 16.
Editor's Notes
- #3 Began as a way to share links Now we have thousands of communities, some for news, some behave like forums Premier way to waste time at work
- #5 Originally, we didn’t detect crashes very well. wake up every could of hours and check if things were working. friends would have to call me. I dreaded the sound of my phone ringing Ruined many a dinner trying to fix reddit. I had to run across the street to an Apple store to use a terminal to fix things. Bringing supervise into the mix fixed many of our woes. If the app died, supervise automatically restarts it. write scripts to detect weird states Running out of connections? Let the app server crash, all of a sudden you’ll have plenty of connections. Memory leak? Deadlocked thread?
- #6 One machine with web, app, db Things were slow, but it wasn’t clear why. Cpu wasn’t too high, memory was under control. Context switching killing us Adding that second machine was a huge breath of fresh air. learned this lesson multiple times over the years Not just separating services, but also data. Breaking apart links/comments gave a large performance increase. PostgreSQL is great, but it doesn’t like to share.
- #7 In the early days our schema looked like this. It’s fairly straightforward. Normalized Lots of foreign keys Complex many-to-many relationships A table for links, accounts, comments. Columns for each attribute.
- #9 Every type of data is divided into two parts: things and data. A thing table stores properties common to all types. Every type has ups, downs, creation date. The data table is just a list of key value pairs. To do queries against the data table, we have specific indices for each key type.
- #11 only one application server and developed all sorts of bad habits. The app server was an always-running Lisp process. We stored all sorts of state per user. We didn’t have memcached, just stored data in an in-memory hashtable. When we switched to Python, we preserved this. When we added multiple app servers, we were in a bad way Every app had to share this cache. We were duplicating our entire cache for each app server. Couldn’t use memcache right away because we had too many keys
- #12 All queries are generated by the same piece of code. Makes general caching simple. limited state we have gets put in memcache. Examples: password reset or captchas. Every element of every page is cached. group small elements into bigger pieces and cache blobs. Slow function? Memoize it. For example, the normalized hot page, or more complex database lookups. Memcachedb – Listings, comment trees, slow queries (by_url)
- #13 When we first began, we had a nice, consistent normalized database. This means we had to do a lot of work to get all of the right data together to render a page. This can be mitigated with caching, but it’s not a cure-all What we do now is more like pre-emptive caching. store complete listings, complete comment trees. A link might appear in a front-page listing or in a user’s profile page. A comment might appear with a link, or in a user’s inbox. Each of these is stored pre-computed and ready to go. For some listings we store as many as 15 versions with different sorts and different time periods.
- #14 Why we work offline
- #15 What we do offline