Successfully reported this slideshow.
Your SlideShare is downloading. ×

How Shopify Scales Rails

Advertisement

More Related Content

Advertisement
Advertisement

How Shopify Scales Rails

  1. 1. shopify How Shopify Scales Rails John Duff
  2. 2. • The Shopify stack • Knowing what to scale • How we cache • Scaling beyond caching • Splitting things up Overview
  3. 3. What is Shopify?
  4. 4. The Stack
  5. 5. • Ruby 1.9.3-p385 • Rails 3.2 • Percona MySQL 5.5 • Unicorn 4.5 • Memcached 1.4.14 • Redis 2.6 The Stack
  6. 6. The Stack • 53 App Servers • 1590 Unicorn Workers • 5 Job Servers • 370 Job Workers Nginx Unicorn Rails 3.2 Ruby 1.9.3-p385
  7. 7. The Stack Firewall Load Balancer App Servers Redis Job Servers Database Memcached Search
  8. 8. • 55,873 Lines of application code • 15,968 Lines of CoffeeScript application code • 81,892 Lines of test code • 211 Controllers • 468 Models The Stack
  9. 9. Current Scale
  10. 10. 9.9 M Orders An order every 3.2 seconds
  11. 11. 2,008 Sales per Minute Cyber Monday
  12. 12. 50,000 RPM 45 ms response time
  13. 13. 13.3 billion requests
  14. 14. Looking Back, to Look Ahead
  15. 15. • First line of code written in 2004 • Shopify released June, 2006 • Same codebase • Over 9 years of Rails upgrades, improvements and changes Looking Back, to Look Ahead
  16. 16. Looking Back, to Look Ahead • 6,702 Lines of application code (55,873) • 4,386 Lines of test code (81,892) • 38 Controllers (211) • 77 Models (468)
  17. 17. Looking Back, to Look Ahead • Ruby 1.8.2 • Rails 0.13.1 • MySQL 4.1 • Lighttpd • Memcached
  18. 18. Know The System
  19. 19. One Request, One Process
  20. 20. RPM = W * 1/R
  21. 21. RPM = 1590 * 60 / 0.072
  22. 22. 1,325,000 = 1172 * 60 / 0.072
  23. 23. ↑ Workers ↓ Response Time
  24. 24. Know The System • Avoid network calls during requests • Speed up unavoidable network calls • The Storefront and Checkout • The Chive
  25. 25. Chive Flash Sale
  26. 26. Measure ALL THE THINGS
  27. 27. Measure ALL THE THINGS • New Relic • Splunk • StatsD • Cacti • Conan
  28. 28. New Relic
  29. 29. Splunk
  30. 30. Caching
  31. 31. cacheable
  32. 32. cacheable • https://github.com/Shopify/cacheable • serve gzip’d content • ETag and 304 Not Modified • generational caching • no explicit expiry
  33. 33. cacheable class PostsController < ApplicationController def show response_cache do @post = @shop.posts.find(params[:id]) respond_with(@post) end end def cache_key_data { :action => action_name, :format => request.format, :params => params.slice(:id), :shop_version => @shop.version } end end
  34. 34. requests Caching Dynamic 404s
  35. 35. Identity Cache
  36. 36. Identity Cache • https://github.com/Shopify/identity_cache • cache full model objects in memcached • can include associated objects in cache • must opt in to the cache • explicit, but automatic expiry
  37. 37. Identity Cache class Product < ActiveRecord::Base include IdentityCache has_many :images cache_index [:shop_id, :id] cache_has_many :images, :embed => true end @product = Product.fetch_by_shop_id_and_id(shop_id, id) @images = @product.fetch_images
  38. 38. Identity Cache
  39. 39. Get Out of My Process
  40. 40. Delayed Job • Jobs stored in the db • Workers run in their own process • Workers poll for jobs periodically • https://github.com/collectiveidea/delayed_job
  41. 41. Resque • Redis backed • O(1) operation to pop jobs • Faster (300 jobs/sec vs 120 jobs/sec) • Extensible • https://github.com/defunkt/resque
  42. 42. Resque • Sending Email • Processing Payments • Geolocation • Import / Export • Indexing for Search • 86 Other things...
  43. 43. Background Payment Processing ms
  44. 44. Resque class AddressGeolocationJob max_retries 3 def self.perform(params) object = params[:model].constantize.find(params[:id]) object.latitude, object.longitude = Geocoder.geocode(object) object.save! end end Resque.enqueue(AddressGeolocationJob, :id => 1, :model => 'Address')
  45. 45. Redis • Inventory reservation system • Sessions • Theme uploads • Throttling • Carts
  46. 46. All Roads Lead To MySQL
  47. 47. MySQL Hardware • 4 x 8 Core Processor • SSD • 256 GB Ram • Full working set in memory
  48. 48. MySQL Query Optimization • pt-query-digest • Avoid queries that generate temp tables • Adding the right indexes • Forcing / Ignoring Indexes
  49. 49. MySQL Tuning • disable innodb_stats_on_metadata • increase table_open_cache • replace glibc memory allocator with tcmalloc • innodb_autoinc_lock_mode=‘interleaved’
  50. 50. after_commit db transactions best friend
  51. 51. after_commit • After transaction has been committed • Webhooks • Cache expiry • Update associated objects
  52. 52. after_commit class OrderObserver < ActiveRecord::Observer observe :order def after_save(order) if order.changes.keys.include?(:financial_status) order.flag_for_after_commit(:update_customer) end end def after_commit(order) if order.flagged_for_after_commit?(:update_customer) Resque.enqueue(UpdateCustomerJob, :id => order.id) end end end
  53. 53. Services
  54. 54. Services • Split out standalone services as needed • Independently scaled • Segmented metrics • Overall system is more complex • Limit to what is necessary
  55. 55. Imagery
  56. 56. Adapt and Evolve as Needed Using data and knowledge of the system to drive decisions
  57. 57. Summary • Know your application and infrastructure. • Keep slow IO or CPU tasks out of the main process. • Measure your optimizations. You can make it worse.
  58. 58. Thanks. @johnduff | john.duff@shopify.com

×