Your SlideShare is downloading. ×

When rails hits the fan

1,894

Published on

What happens when site traffic outgrows the ability of your Ruby on Rails site (and related infrastructure) to handle it? What does Rails provide to solve this problem, and where do those built-ins …

What happens when site traffic outgrows the ability of your Ruby on Rails site (and related infrastructure) to handle it? What does Rails provide to solve this problem, and where do those built-ins break down? Where do you go from there?

Published in: Technology
0 Comments
13 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,894
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
25
Comments
0
Likes
13
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Proprietary andConfidentialWhen Rails hits the fanEric Saxby@sax @ecdysone @saxThursday, June 6, 13
  • 2. This talk is about:Proprietary andConfidentialUser traffic increasing weekover weekThursday, June 6, 13
  • 3. Proprietary andConfidentialDatabases outgrowing RAMThursday, June 6, 13
  • 4. Proprietary andConfidentialAvg site response time over same periodThursday, June 6, 13
  • 5. Who am I? Why should you care?Proprietary andConfidential■ Application developerMany and various technologies.Worked with Rails for ~5 years.■ Recent focus has beenoperationalChef, PostgreSQL, SmartOS,monitoring■ TDD, BDD, Agile, DevOps, SOA, etcI care about how code is organized, and always want tolearn how to do that better.■ Now I’m at WaneloThursday, June 6, 13
  • 6. What is Wanelo?Proprietary andConfidential■ Wanelo (“Wah-nee-lo” from Want, NeedLove) is a global platform for shopping.Thursday, June 6, 13
  • 7. Proprietary andConfidentialMarketing-free shopping across 100s ofthousands of unique storesThursday, June 6, 13
  • 8. Proprietary andConfidentialPersonal feed of products from any store on theinternetNo Il8n or l10n... yet!Thursday, June 6, 13
  • 9. Technology overviewProprietary andConfidential■ MRI Ruby 1.9.3 & Rails 3.2■ PostgreSQL 9.2.4, Solr 3.6■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, dTrace■ Circonus, NewRelic, BoundaryMonitoring, graphing, alerting■ Chef + Opscode■ Amazon S3 + Fastly CDN■ statsd, Graphite, nagiosThursday, June 6, 13
  • 10. How do you know whenyou are entering thebad place?Proprietary andConfidentialThursday, June 6, 13
  • 11. Lets go back to this for a secProprietary andConfidentialThursday, June 6, 13
  • 12. Proprietary andConfidentialThursday, June 6, 13
  • 13. What is the bad place?Proprietary andConfidentialgreen: disk reads, red: disk writes on DB serverIs this actually a problem?Thursday, June 6, 13
  • 14. What is the bad place?Proprietary andConfidentialCan be difficult to predict, using most of thedefault metrics we trackgreen: disk reads, red: disk writes on DB serverThursday, June 6, 13
  • 15. UtilizationProprietary andConfidential> iostat -xM 3extended device statisticsdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12Thursday, June 6, 13
  • 16. UtilizationProprietary andConfidential> iostat -xM 3extended device statisticsdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12Thursday, June 6, 13
  • 17. UtilizationProprietary andConfidential> iostat -xM 3extended device statisticsdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12Thursday, June 6, 13
  • 18. UtilizationProprietary andConfidential> iostat -xM 3extended device statisticsdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12Develop spidey senses over time, but problemsare highly anecdotalThursday, June 6, 13
  • 19. SaturationProprietary andConfidential> iostat -xM 3extended device statisticsdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12Thursday, June 6, 13
  • 20. SaturationProprietary andConfidential> iostat -xM 3extended device statisticsdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12Thursday, June 6, 13
  • 21. SaturationProprietary andConfidential> iostat -xM 3extended device statisticsdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12Thursday, June 6, 13
  • 22. Both are important!Proprietary andConfidential■ Tracking saturation helps you predictproblems■ Tracking utilization can tell you how tosolve that problem■ Basically, watch every video byBrendan Gregg on YouTubehttp://www.youtube.com/results?search_query=brendan+greggThursday, June 6, 13
  • 23. Know the limits of your dataProprietary andConfidentialAverages can be extremely useful, butdo not give a complete pictureThursday, June 6, 13
  • 24. Know the limits of your dataProprietary andConfidentialOutliers can cause severe problemseven when average is greatThursday, June 6, 13
  • 25. This is why we ❤ PostgreSQLProprietary andConfidential■ pg_stat_activity■ pg_stat_user_indexes■ pg_stat_user_tables■ pg_stat_statements■ PostgreSQL gives you tools to monitorit and operate at scaleThursday, June 6, 13
  • 26. Proprietary andConfidentialpg_stat_statementsThursday, June 6, 13
  • 27. Ok, so back to thebad placeProprietary andConfidentialThursday, June 6, 13
  • 28. Proprietary andConfidentialHit ratio of File System Cache ondatabase server degrading over timeThursday, June 6, 13
  • 29. Proprietary andConfidentialAverage response latencyincreasing over time, ties up CPU resourcesThursday, June 6, 13
  • 30. Step 1: Cache all the things!Proprietary andConfidential■ Reduce load on DB(s)■ Reduce rendering timeThursday, June 6, 13
  • 31. Cache invalidation?Proprietary andConfidential■ Use model updated_at in the cachekey■ Changing DB record invalidates allcaches for that recordThursday, June 6, 13
  • 32. What were the goals of caching?Proprietary andConfidential■ Using model attributes as cache keysmeans you need to fetch DB records■ At high scale, fetching DB records forevery page becomes problematic■ Cache sweepers are painful, buttheyre the only thing weve found tobe scalable and reliableThursday, June 6, 13
  • 33. Action cache all the things!Proprietary andConfidential■ Cache hits skip all rendering■ Still able to run before filters, forinstance when doing A/B testing■ Some cached pages can be putbehind a CDN■ Requires page personalization to beadded via AjaxThursday, June 6, 13
  • 34. Proprietary andConfidentialThursday, June 6, 13
  • 35. Proprietary andConfidentialThursday, June 6, 13
  • 36. Fragment cache the rest!Proprietary andConfidential■ Fragments can be shared between pages■ Can reduce rendering time■ Can remove queries for related recordsThursday, June 6, 13
  • 37. Proprietary andConfidential■ Difficult to remove top level query■ Joins/eager loading means some relatedrecords are still queried■ How many trips to memcached?Thursday, June 6, 13
  • 38. Proprietary andConfidentialdef multi_get_on_collection(objects,cache_options = {})cache_keys = objects.inject(ActiveSupport::OrderedHash.new) do |key_map, obj|key_map[obj] = cache_options[:cache_key_proc].call objkey_mapendpre_rendered_objects = Rails.cache.read_multi *cache_keys.valuescache_keys.map do |object, cache_key|cached_html = pre_rendered_objects[cache_key]if cached_html.present? && caching_enabled?cached_htmlelsecache_options[:render_proc].call(object).tap do |fragment|Rails.cache.write cache_key, fragment, cache_optionsendendendendThursday, June 6, 13
  • 39. Proprietary andConfidentialdef multi_get_on_collection(objects,cache_options = {})cache_keys = objects.inject(ActiveSupport::OrderedHash.new) do |key_map, obj|key_map[obj] = cache_options[:cache_key_proc].call objkey_mapendpre_rendered_objects = Rails.cache.read_multi *cache_keys.valuescache_keys.map do |object, cache_key|cached_html = pre_rendered_objects[cache_key]if cached_html.present? && caching_enabled?cached_htmlelsecache_options[:render_proc].call(object).tap do |fragment|Rails.cache.write cache_key, fragment, cache_optionsendendendendThursday, June 6, 13
  • 40. Proprietary andConfidentialdef multi_get_on_collection(objects,cache_options = {})cache_keys = objects.inject(ActiveSupport::OrderedHash.new) do |key_map, obj|key_map[obj] = cache_options[:cache_key_proc].call objkey_mapendpre_rendered_objects = Rails.cache.read_multi *cache_keys.valuescache_keys.map do |object, cache_key|cached_html = pre_rendered_objects[cache_key]if cached_html.present? && caching_enabled?cached_htmlelsecache_options[:render_proc].call(object).tap do |fragment|Rails.cache.write cache_key, fragment, cache_optionsendendendendThursday, June 6, 13
  • 41. MultiGet is your friendProprietary andConfidential■ Turns 100+ memcached calls into asingle requestcache_key = ->(product) do"products_thumb_#{product.id}"endrenderer = ->(product) dorender "products/thumb", model: productendmulti_get_on_collection(@products,cache_key_proc: cache_key,render_proc: renderer,expires_in: 6.hours).join("n").html_safeThursday, June 6, 13
  • 42. Proprietary andConfidentialIncreasing your cache hit ratio meansless queries against your databaseThursday, June 6, 13
  • 43. Proprietary andConfidentialdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100DB latency increases■ Even with highly efficient caches, iostatshows 100% disk busyThursday, June 6, 13
  • 44. count(*)Proprietary andConfidentialThursday, June 6, 13
  • 45. Proprietary andConfidentialThursday, June 6, 13
  • 46. Proprietary andConfidentialThursday, June 6, 13
  • 47. Proprietary andConfidentialThursday, June 6, 13
  • 48. Proprietary andConfidentialThursday, June 6, 13
  • 49. Proprietary andConfidentialThursday, June 6, 13
  • 50. Proprietary andConfidentialThursday, June 6, 13
  • 51. Proprietary andConfidentialThursday, June 6, 13
  • 52. TOO MANY COUNTSProprietary andConfidentialThursday, June 6, 13
  • 53. Rails has the answer!Proprietary andConfidential■ counter_cache column on table■ Adding records executes INCR■ Removing records executes DECRclass Product < ActiveRecord::Basebelongs_to :store, counter_cache: trueendThursday, June 6, 13
  • 54. But...Proprietary andConfidential■ On very write heavy applications,multiple requests will update the samerecord■ DEADLOCK errorsThursday, June 6, 13
  • 55. Use background jobsProprietary andConfidential■ Stop updating counter caches on save■ Queue a delayed job for the near future■ Job performs a complete recalculation ofthe counter cache and is idempotent■ The higher the count, the further wedelay the job (less likely users will notice)Thursday, June 6, 13
  • 56. Deduplicate delayed jobsProprietary andConfidential■ Sidekiq with UniqueJob plugin■ Updates are serialized via a fixednumber of workers■ Workers can be stopped to alleviate DBload in an emergency■ Same pattern can be applied elsewhere,like updating Solr indexesThursday, June 6, 13
  • 57. Deduplicate delayed jobsProprietary andConfidentialclass UpdateProductCountWorker < WaneloWorkersidekiq_options queue: :product_counts,unique: truewait 10.minutesdef perform!(params)id = params[:id].to_iActiveRecord::Base.connection.execute %Q{update stores set product_count =(select count(*)from productswhere store_id = stores.id)where stores.id = #{id}}endendThursday, June 6, 13
  • 58. STILL TOO MANY COUNTSProprietary andConfidentialThursday, June 6, 13
  • 59. Proprietary andConfidentialThursday, June 6, 13
  • 60. Proprietary andConfidentialThursday, June 6, 13
  • 61. Proprietary andConfidentialPagination gems run countsThursday, June 6, 13
  • 62. Proprietary andConfidential■ Kaminari executes a count(*) to determinetotal page countPagination gems run countsThursday, June 6, 13
  • 63. Proprietary andConfidential■ Kaminari executes a count(*) to determinetotal page countSELECT "stores".* FROM "stores"WHERE (state = approved)LIMIT 20 OFFSET 0SELECT COUNT(*) FROM "stores" WHERE (state = approved)Pagination gems run countsThursday, June 6, 13
  • 64. Proprietary andConfidential■ Kaminari executes a count(*) to determinetotal page countSELECT "stores".* FROM "stores"WHERE (state = approved)LIMIT 20 OFFSET 0SELECT COUNT(*) FROM "stores" WHERE (state = approved)Pagination gems run counts■ We paginate EVERYTHING■ We often already know total count fromcounter cache (or can hard code 10,000)Thursday, June 6, 13
  • 65. Proprietary andConfidentialmodule Kaminarimodule ActiveRecordRelationMethods# a workaround for AR 3.0.x that returns 0 for #count when page > 1# if +limit_value+ is specified, load all the records and count themif ActiveRecord::VERSION::STRING < 3.1def count(column_name = nil, options = {}) #:nodoc:limit_value ? length : super(column_name, options)endenddef total_count(column_name = nil, options = {}) #:nodoc:@total_count ||= beginc = except(:offset, :limit, :order)# Remove includes only if they are irrelevantc = c.except(:includes) unless references_eager_loaded_tables?# .group returns an OrderdHash that responds to #countc = c.count(column_name, options)if c.is_a?(ActiveSupport::OrderedHash)c.countelsec.respond_to?(:count) ? c.count(column_name, options) : cendendendendendThursday, June 6, 13
  • 66. Proprietary andConfidentialTime for some monkey patches!module ActiveRecordclass Relationdef custom_counter(count)@total_count ||= countselfendendendmodule Sunspotmodule Searchclass AbstractSearchdef custom_counter(count)@total ||= countselfendendendend@products = @store.products.custom_counter(@store.products_count).page(params[:page])Thursday, June 6, 13
  • 67. How do we know whatRails is really doing?Proprietary andConfidentialThursday, June 6, 13
  • 68. ruby-prof and pilferProprietary andConfidential■ Profile the entire stack trace of an actionor a set of classeshttps://github.com/eric/pilferhttps://github.com/ruby-prof/ruby-profThursday, June 6, 13
  • 69. Proprietary andConfidential■ Can work as Rack middleware■ Outputs HTML that will tell you...if Rails.env.profile?use Rack::RubyProf, :path => /tmp/profileendThursday, June 6, 13
  • 70. Proprietary andConfidential■ Can work as Rack middleware■ Outputs HTML that will tell you...if Rails.env.profile?use Rack::RubyProf, :path => /tmp/profileend■ How long are we spending generatingURLs???Thursday, June 6, 13
  • 71. Proprietary andConfidentialThursday, June 6, 13
  • 72. Proprietary andConfidentialmodule UrlHelpersdef product_path(product)"/p/#{product.id}/#{product.slug}"endendThursday, June 6, 13
  • 73. The importance offast JSON renderingProprietary andConfidentialThursday, June 6, 13
  • 74. HTML vs JSONProprietary andConfidential■ Since launching native iOS and Androidapps, the majority of Wanelo traffic isserved via a JSON API■ The longer we spend rendering JSON,the more CPUs are tied up, the moreservers we needThursday, June 6, 13
  • 75. Proprietary andConfidentialRemoving RABLThursday, June 6, 13
  • 76. Things we did not expectProprietary andConfidential■ RABL serializes, then deserializes JSONto do merges!■ ActiveSupport defines inefficient :to_jsonon every objectThursday, June 6, 13
  • 77. Proprietary andConfidential■ Rendering JSON partials shouldhash.merge!■ Fragment caching should marshal hashes,not JSON■ Caching should allow for MultiGet■ Should allow for arbitrary composition■ JSON conversion should use OJ tocall :to_json ONCEhttps://github.com/wanelo/compositorThursday, June 6, 13
  • 78. Proprietary andConfidentialrequire active_support/jsonrequire active_support/core_ext/object/to_json[Object, Array, FalseClass, Float,Hash, Integer, NilClass, String,TrueClass].each do |klass|klass.class_eval dodef to_json(options = nil)Oj.dump(self, options)endendendThursday, June 6, 13
  • 79. AsynchronousCommitsProprietary andConfidentialThursday, June 6, 13
  • 80. Do you read, or do you write?Proprietary andConfidential■ Tools like iostat, vmstat, kstat, collectdpg_stat_user_tables can show youutilization■ Reads much easier to scale than writes.Read/write splitting, caching.■ What do you do when writes become thebottleneck?Thursday, June 6, 13
  • 81. Writes committed to disk?Proprietary andConfidential■ Some workloads are more lenient fordelay/possible data loss■ Applicable to many technologies■ Even microsecond delays can reduceload■ Waiting for Solr to respond can keep DBtransactions open longerThursday, June 6, 13
  • 82. i.e. PostgreSQLProprietary andConfidentialThursday, June 6, 13
  • 83. i.e. SolrProprietary andConfidential<!-- Perform a <commit/> automatically under certainconditions --><autoCommit><!-- number of updates since last commit --><maxDocs>1000</maxDocs><!-- oldest uncommited update (in ms) long ago --><maxTime>30000</maxTime></autoCommit>Thursday, June 6, 13
  • 84. What happens when datagrows larger than asingle database?Proprietary andConfidentialThursday, June 6, 13
  • 85. Proprietary andConfidentialGrowth of one key DB table over 3 monthsThursday, June 6, 13
  • 86. Service OrientedArchitectureProprietary andConfidentialThursday, June 6, 13
  • 87. The only real way to scaleProprietary andConfidential■ Tune code, infrastructure for a veryparticular workload■ Hide sharding from other codebases■ Allow small teams to manage small(er)codebasesThursday, June 6, 13
  • 88. The only real way to scaleProprietary andConfidential■ Tune code, infrastructure for a veryparticular workload■ Hide sharding from other codebases■ Allow small teams to manage small(er)codebases■ Everyone talks about why, no-one talksabout how in the Rails worldThursday, June 6, 13
  • 89. Services are hard (the first time)Proprietary andConfidential■ Synchonous vs asynchronous datapersistence■ Message passing■ Testing■ Iterative developmentThursday, June 6, 13
  • 90. Iteration is the keyProprietary andConfidential■ Isolate data■ Isolate interface■ Extract interface with an databaseadapter■ Launch service layer■ Switch interface to user service adapterThursday, June 6, 13
  • 91. Proprietary andConfidentialclass Product < ActiveRecord::Basehas_many :savesendThursday, June 6, 13
  • 92. Proprietary andConfidentialclass Product < ActiveRecord::Basehas_many :savesend XThursday, June 6, 13
  • 93. Proprietary andConfidentialclass Product < ActiveRecord::Basehas_many :savesend Xclass Product < ActiveRecord::Basedef savesSave.where(product_id: self.id)endendThursday, June 6, 13
  • 94. Proprietary andConfidentialclass Product < ActiveRecord::Basehas_many :savesend Xclass Product < ActiveRecord::Basedef savesSave.where(product_id: self.id)endendDo this everywhere (you have tests, right?)Thursday, June 6, 13
  • 95. Proprietary andConfidentialclass Save < ActiveRecord::Baseestablish_connection "saves_#{Rails.env}"end■ Set up a read replica■ Take down site■ Promote replica to be a master■ Restart unicorns with new config■ Bring up site■ Clean up unnecessary tables on each DBThursday, June 6, 13
  • 96. Proprietary andConfidentialclass Save < ActiveRecord::Baseestablish_connection "saves_#{Rails.env}"def self.by_product(product)where(product_id: product.id)endendclass Product < ActiveRecord::Basedef savesSave.by_product(self)endend■ Reduce Ruby interface to minimum possible■ Easy to deploy thisThursday, June 6, 13
  • 97. Proprietary andConfidentialclass Saveinclude SavesClientendmodule SavesClientdef self.included(other)other.send(:attr_accessor, :id, :product_id)other.extend ClientClassMethodsendmodule ClientClassMethodsdef by_product(*args)adapter.new(self).by_product(*args)enddef adapter@adapter ||= SavesClient::DbAdapterendendendThursday, June 6, 13
  • 98. Proprietary andConfidentialclass Saveinclude SavesClientendmodule SavesClientdef self.included(other)other.send(:attr_accessor, :id, :product_id)other.extend ClientClassMethodsendmodule ClientClassMethodsdef by_product(*args)adapter.new(self).by_product(*args)enddef adapter@adapter ||= SavesClient::DbAdapterendendendThursday, June 6, 13
  • 99. Proprietary andConfidentialclass Saveinclude SavesClientendmodule SavesClientdef self.included(other)other.send(:attr_accessor, :id, :product_id)other.extend ClientClassMethodsendmodule ClientClassMethodsdef by_product(*args)adapter.new(self).by_product(*args)enddef adapter@adapter ||= SavesClient::DbAdapterendendendThursday, June 6, 13
  • 100. Proprietary andConfidentialmodule SavesClientclass DbAdapterdef self.close_connections# Check in the database connection,# since were shutting down this threadSavesService::Save.clear_active_connections!enddef by_product(*args)relation :by_product, *argsenddef relation(method, *args)SavesClient::AdapterRelation.new(self,SavesService::Save.send(method, *args))enddef all(scope)# Scope is an AR Relation instance returned from# SavesService::Save.by_product(product)scope.all.map { |m| client_class.new save_attrs_from(m) }endendendThursday, June 6, 13
  • 101. Proprietary andConfidentialmodule SavesClientclass DbAdapterdef self.close_connections# Check in the database connection,# since were shutting down this threadSavesService::Save.clear_active_connections!enddef by_product(*args)relation :by_product, *argsenddef relation(method, *args)SavesClient::AdapterRelation.new(self,SavesService::Save.send(method, *args))enddef all(scope)# Scope is an AR Relation instance returned from# SavesService::Save.by_product(product)scope.all.map { |m| client_class.new save_attrs_from(m) }endendendThursday, June 6, 13
  • 102. Proprietary andConfidentialmodule SavesClientclass DbAdapterdef self.close_connections# Check in the database connection,# since were shutting down this threadSavesService::Save.clear_active_connections!enddef by_product(*args)relation :by_product, *argsenddef relation(method, *args)SavesClient::AdapterRelation.new(self,SavesService::Save.send(method, *args))enddef all(scope)# Scope is an AR Relation instance returned from# SavesService::Save.by_product(product)scope.all.map { |m| client_class.new save_attrs_from(m) }endendendThread safety is EXTREMELY importantThursday, June 6, 13
  • 103. Proprietary andConfidentialmodule SavesClientclass AdapterRelationattr_reader :adapter, :scopedef initialize(adapter, scope)@adapter, @scope = adapter, scopeenddef limit(num); enddef order(order); enddef page(num); enddef first; enddef alladapter.all(scope)endendend■ Calling :by_product instantiates a Relation■ Calling :all executes the queryThursday, June 6, 13
  • 104. What executes the query?Proprietary andConfidential■ The ActiveRecord model moves into the gem■ The adapter translates the Save methods into AR calls,maps columns into attributes on our class■ Important to deploy this at this stage, as there manyproblems to solve, like:■ Getting your tests green, fixtures consistent■ Figuring out whether you really covered all accesspatternsmodule SavesServiceclass Save < ActiveRecord::BaseendendThursday, June 6, 13
  • 105. Minimize the Ruby accessProprietary andConfidential■ We were able to reduce everything to 7 scopes,i.e. :by_product, :by_user, etc■ Reduced Relation methods to these:■ limit■ page■ order■ count■ all■ first■ last■ pluck■ find_in_batchesThursday, June 6, 13
  • 106. Launch the service layerProprietary andConfidential■ Now that we have a small public Rubyinterface, we can pair that to a Sinatra app■ Sinatra can serve as a long-term fake forSelenium, even after service is re-writtenThursday, June 6, 13
  • 107. module SavesServiceclass Web < Sinatra::Baseset :environment, ENV[RACK_ENV] || "development"PAGE_SIZE = 50ActiveRecord::Base.include_root_in_json = falseregister Sinatra::ActiveRecordExtensionconfigure doset :database_file, SavesService.config.db_configset :root, File.expand_path("../../../", __FILE__)disable :raise_errorsdisable :show_exceptionsset :logger, nilenderror doe = env[sinatra.error]ActiveRecord::Base.logger.error ["#{e.class}: #{e.message}",*e.backtrace].join("n ")status 500body {"errors":":("}endbefore { content_type application/json }endendProprietary andConfidentialThursday, June 6, 13
  • 108. Proprietary andConfidentialget /products/:pid/saves dopaginate SavesService::Save.by_product(params[:pid])endprivatedef paginate(scope)return count(scope) if params[:count].present?order = params[:order] if params[:order] =~ /Adesc|ascZ/iscope = scope.order("created_at #{order || desc}")limit = params[:limit] ? params[:limit].to_i : PAGE_SIZEpage_number = [params[:page].to_i - 1, 0].max * limitscope = scope.offset(page_number).limit(limit)scope = scope.pluck(params[:pluck]) if params[:pluck]body Oj.dump(saves: scope)endThursday, June 6, 13
  • 109. Proprietary andConfidential#!/usr/bin/env rubyrequire optparseoptions = {:environment => development,:port => 3000}OptionParser.new do |opts|opts.banner = "Usage: saves_service [options]"opts.on("-d", "--dbconfig OPT", "path to database.yml") do |opt|options[:db_config] = File.expand_path(opt, Dir.pwd)endopts.on("-E", "--environment OPT", "RACK_ENV to use") do |opt|options[:environment] = optendopts.on("-p", "--port OPT", "port to use") do |opt|options[:port] = optendend.parse!cmd_env = { RACK_ENV => options[:environment],DB_CONFIG => options[:db_config],}.delete_if{|k,v| v.nil? }rackup_file = File.expand_path(../../config.ru, __FILE__)exec cmd_env, "unicorn -p #{options[:port]} #{rackup_file}"Thursday, June 6, 13
  • 110. DbAdapter vs HTTPAdapterProprietary andConfidential■ Maps Ruby interface to Net::HTTP::Persistent■ Deserializes JSON into values on class■ Scope becomes a mapby_product(1) => "/products/1/saves"■ Finder methods map paramslimit(10) => "?limit=10"■ AdapterRelation uses Adapter to fetch records,does not change at allThursday, June 6, 13
  • 111. Proprietary andConfidentialmodule SavesClientclass AdapterRelationattr_reader :adapter, :scopedef initialize(adapter, scope)@adapter, @scope = adapter, scopeenddef limit(num); enddef order(order); enddef page(num); enddef first; enddef alladapter.all(scope)endendendThursday, June 6, 13
  • 112. Deploy as a setting changeProprietary andConfidentialRails.application.config.after_initialize do |app|if Settings.saves_service.enabledSave.saves_base_url = Settings.saves_service.urlelserequire saves_client/db_adapterSave.adapter = SavesClient::DbAdapterrequire saves_service/savesaves_db = "saves_#{Rails.env}"SavesService::Save.establish_connection saves_dbendendThursday, June 6, 13
  • 113. TakeawaysProprietary andConfidentialThursday, June 6, 13
  • 114. Proprietary andConfidential■ Choose technologies that are easy tooperate and monitor■ Dont immediately break when they hitresource thresholds■ Sound replication strategies■ Assume that data should be tracked, even ifyou dont yet understand the relevance■ Small iterative performance improvementscan have massive payoff over timeThursday, June 6, 13
  • 115. Thanks!Proprietary andConfidential@sax@ecdysone@saxhttps://github.com/wanelohttps://github.com/wanelo-chefThursday, June 6, 13

×