Scaling the Rails

893 views

Published on

How a little quiz achieved Olympic Glory

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
893
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Scaling the Rails

  1. 1. Nov 19, 2008 Scaling the Rails How a little quiz achieved Olympic Glory Josh Schwartzman, Newzwag
  2. 2. Get Ready For: • Database Enhancements & Gotchas • Hardware Basics • Load Testing • Rails Backends & Web Servers
  3. 3. From Humble Beginnings:
  4. 4. (on limited hardware) 30,000 concurrent users 9 million pageviews/hour
  5. 5. [question]
  6. 6. [answer]
  7. 7. Find Your MySQL Bottlenecks
  8. 8. Analyze Your Logs • Look for slowest requests • Verbose Logging • Detailed DB logging (a small script) Completed in 0.06763 (14 reqs/sec) | Rendering: 0.01377 (20%) | DB: 0.00587 (8%) 2.6k, 54 rows, 7 queries | 200 OK [http://url]
  9. 9. Database Enhancements (Reads) • Index all of the keys you are going to use to find your data • Turn on debugging and check the slowest queries • Use EXPLAIN on your SQL statements (Some MySQL-specific enhancements)
  10. 10. Database Enhancements (Reads) • Minimize your # of queries • Use Eager Loading
  11. 11. Database Enhancements (Writes) • Only save the data that has changed • Rails 2.1: 2x faster on some actions, due to: • Partial Updates (automatic performance) • Dirty Tracking
  12. 12. Database Enhancements (Writes) • Check your model validations • Transactions w/InnoDB
  13. 13. Database Enhancements (Writes) • Bulk Writes with AR-Extensions • 8-12x faster on InnoDB • 8-40x faster on MyISAM* • Master/Slave Replication • Master DB for writes with replication to read slaves • Distributes load to multiple machines *40x if you don’t use validations
  14. 14. Useful Plugins • ActiveRecord Extensions • Bulk writes • Dr. Nic’s magic_multi_connections • Offload reads/writes to another DB • Single-Write Multi-Read (SWMR) • Our plugin to write to one DB & use multiple read slaves
  15. 15. Hardware • Cache everything, so lots of RAM • Make your reads/writes fast • RAID helps: RAID 10 - faster writes RAID 5 - faster reads
  16. 16. Hardware • Minimize hits to your Rails Stack • Cache images or use a fast web server • Move all your routes.rb routing logic to dedicated hardware (load balancer)
  17. 17. Hardware • Keep App & DB on separate servers • Infrastructure specific to it’s purpose • Easy to isolate & debug bottlenecks: Graph from Cacti
  18. 18. Load Testing • Test the path a user will take • One Tool: JMeter • Session and full hardware stack testing • jmeter-server • Distributed testing • Test from any computer @ ~1500 threads per box
  19. 19. Load Testing • You’ll always find new problems under load • We discovered issues with lots of new users: checking login twice • Simple queries (especially joins) became slow • Tweaking how often we wrote scores to the database, and how many at a time
  20. 20. Load Testing • The type and number of web servers* is critical. Quick comparison: Mongrel Ebb Thin •Stable •Some bugs •Almost as stable •RAM & CPU heavy •RAM & CPU light as Mongrel (it uses •Slowest per action •Fastest per Mongrel’s parser) •Lightest hardware action (uses C for heavy lifting) load •Almost as fast as Ebb *Yeah, you still need a web server like Lighttpd or Nginx to handle your static files and proxy to your Rails backends
  21. 21. Load Testing Using 18 Mongrels # of questions 1 2 3 4 5 6 7 8 9 10 minutes Gruff graphs from our custom user actions
  22. 22. Load Testing Using 18 Ebbs # of questions 1 2 3 4 5 6 7 8 9 10 minutes 949 questions / minute: 40% faster than mongrel!
  23. 23. When You Need a Little Help • After all of that performance work... • Servers were crashing spontaneously • Nothing was apparent in the logs • Sleepless nights...
  24. 24. We Found Stability in God Simple monitoring for Rails apps:
  25. 25. What Was the Problem? • Post requests on older browsers (Firefox 1.5, Navigator 7) • Killed our backends, and as each failed, it passed on the request to another server • Solution: switch from Nginx to Lighttpd
  26. 26. A Summary • 200 (scaling) 5000 simultaneous users a 25x performance gain • Achieved by adding plugins, custom Memcached methods, optimizing the DB, changing to ebbs, and numerous other code tweaks (and a slight hardware upgrade)
  27. 27. Any Questions: Josh Schwartzman Lead Developer, Newzwag josh@newzwag.com
  28. 28. Resources •Active Record Extensions: http://www.continuousthinking.com/are/activerecord-extensions-0-0-5 •Detailed DB Logging: http://gurge.com/blog/2006/11/09/rails-sql-logging-improvements/ •Ebb: http://ebb.rubyforge.org/ •God (Monitoring): http://god.rubyforge.org/ •Gruff Graphs: http://nubyonrails.com/pages/gruff •Jmeter: http://jakarta.apache.org/jmeter/ •Magic Multi Connections: http://magicmodels.rubyforge.org/magic_multi_connections/ •Seattle.rb’s Memcache Client: http://github.com/fiveruns/memcache-client/tree/master •Thin: http://code.macournoyer.com/thin/

×