How Shopify Scales Rails

11,152 views
10,817 views

Published on

Published in: Technology
1 Comment
69 Likes
Statistics
Notes
No Downloads
Views
Total views
11,152
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
285
Comments
1
Likes
69
Embeds 0
No embeds

No notes for slide

How Shopify Scales Rails

  1. 1. shopifyHow Shopify Scales RailsJohn Duff
  2. 2. • The Shopify stack• Knowing what to scale• How we cache• Scaling beyond caching• Splitting things upOverview
  3. 3. What is Shopify?
  4. 4. The Stack
  5. 5. • Ruby 1.9.3-p385• Rails 3.2• Percona MySQL 5.5• Unicorn 4.5• Memcached 1.4.14• Redis 2.6The Stack
  6. 6. The Stack• 53 App Servers• 1590 Unicorn Workers• 5 Job Servers• 370 Job WorkersNginxUnicornRails 3.2Ruby 1.9.3-p385
  7. 7. The StackFirewallLoad BalancerApp ServersRedisJob ServersDatabaseMemcached Search
  8. 8. • 55,873 Lines of application code• 15,968 Lines of CoffeeScript application code• 81,892 Lines of test code• 211 Controllers• 468 ModelsThe Stack
  9. 9. Current Scale
  10. 10. 9.9 M OrdersAn order every 3.2 seconds
  11. 11. 2,008 Sales per MinuteCyber Monday
  12. 12. 50,000 RPM45 ms response time
  13. 13. 13.3 billion requests
  14. 14. Looking Back, to Look Ahead
  15. 15. • First line of code written in 2004• Shopify released June, 2006• Same codebase• Over 9 years of Rails upgrades, improvements and changesLooking Back, to Look Ahead
  16. 16. Looking Back, to Look Ahead• 6,702 Lines of application code (55,873)• 4,386 Lines of test code (81,892)• 38 Controllers (211)• 77 Models (468)
  17. 17. Looking Back, to Look Ahead• Ruby 1.8.2• Rails 0.13.1• MySQL 4.1• Lighttpd• Memcached
  18. 18. Know The System
  19. 19. One Request, One Process
  20. 20. RPM = W * 1/R
  21. 21. RPM = 1590 * 60 / 0.072
  22. 22. 1,325,000 = 1172 * 60 / 0.072
  23. 23. ↑ Workers↓ Response Time
  24. 24. Know The System• Avoid network calls during requests• Speed up unavoidable network calls• The Storefront and Checkout• The Chive
  25. 25. Chive Flash Sale
  26. 26. Measure ALL THE THINGS
  27. 27. Measure ALL THE THINGS• New Relic• Splunk• StatsD• Cacti• Conan
  28. 28. New Relic
  29. 29. Splunk
  30. 30. Caching
  31. 31. cacheable
  32. 32. cacheable• https://github.com/Shopify/cacheable• serve gzip’d content• ETag and 304 Not Modified• generational caching• no explicit expiry
  33. 33. cacheableclass PostsController < ApplicationControllerdef showresponse_cache do@post = @shop.posts.find(params[:id])respond_with(@post)endenddef cache_key_data{:action => action_name,:format => request.format,:params => params.slice(:id),:shop_version => @shop.version}endend
  34. 34. requestsCaching Dynamic 404s
  35. 35. Identity Cache
  36. 36. Identity Cache• https://github.com/Shopify/identity_cache• cache full model objects in memcached• can include associated objects in cache• must opt in to the cache• explicit, but automatic expiry
  37. 37. Identity Cacheclass Product < ActiveRecord::Baseinclude IdentityCachehas_many :imagescache_index [:shop_id, :id]cache_has_many :images, :embed => trueend@product = Product.fetch_by_shop_id_and_id(shop_id, id)@images = @product.fetch_images
  38. 38. Identity Cache
  39. 39. Get Out of My Process
  40. 40. Delayed Job• Jobs stored in the db• Workers run in their own process• Workers poll for jobs periodically• https://github.com/collectiveidea/delayed_job
  41. 41. Resque• Redis backed• O(1) operation to pop jobs• Faster (300 jobs/sec vs 120 jobs/sec)• Extensible• https://github.com/defunkt/resque
  42. 42. Resque• Sending Email• Processing Payments• Geolocation• Import / Export• Indexing for Search• 86 Other things...
  43. 43. Background Payment Processingms
  44. 44. Resqueclass AddressGeolocationJobmax_retries 3def self.perform(params)object = params[:model].constantize.find(params[:id])object.latitude, object.longitude = Geocoder.geocode(object)object.save!endendResque.enqueue(AddressGeolocationJob, :id => 1, :model => Address)
  45. 45. Redis• Inventory reservation system• Sessions• Theme uploads• Throttling• Carts
  46. 46. All Roads Lead To MySQL
  47. 47. MySQL Hardware• 4 x 8 Core Processor• SSD• 256 GB Ram• Full working set in memory
  48. 48. MySQL Query Optimization• pt-query-digest• Avoid queries that generate temp tables• Adding the right indexes• Forcing / Ignoring Indexes
  49. 49. MySQL Tuning• disable innodb_stats_on_metadata• increase table_open_cache• replace glibc memory allocator with tcmalloc• innodb_autoinc_lock_mode=‘interleaved’
  50. 50. after_commitdb transactions best friend
  51. 51. after_commit• After transaction has been committed• Webhooks• Cache expiry• Update associated objects
  52. 52. after_commitclass OrderObserver < ActiveRecord::Observerobserve :orderdef after_save(order)if order.changes.keys.include?(:financial_status)order.flag_for_after_commit(:update_customer)endenddef after_commit(order)if order.flagged_for_after_commit?(:update_customer)Resque.enqueue(UpdateCustomerJob, :id => order.id)endendend
  53. 53. Services
  54. 54. Services• Split out standalone services as needed• Independently scaled• Segmented metrics• Overall system is more complex• Limit to what is necessary
  55. 55. Imagery
  56. 56. Adapt and Evolve as NeededUsing data and knowledge of the system to drive decisions
  57. 57. Summary• Know your application and infrastructure.• Keep slow IO or CPU tasks out of the main process.• Measure your optimizations. You can make it worse.
  58. 58. Thanks.@johnduff | john.duff@shopify.com

×