Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cache is King - RailsConf 2019

253 views

Published on

Sometimes your fastest queries can cause the most problems. I will take you beyond the slow query optimization and instead zero in on the performance impacts surrounding the quantity of your datastore hits. Using real world examples dealing with datastores such as Elasticsearch, MySQL, and Redis, I will demonstrate how many fast queries can wreak just as much havoc as a few big slow ones. With each example I will make use of the simple tools available in Ruby and Rails to decrease and eliminate the need for these fast and seemingly innocuous datastore hits.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Cache is King - RailsConf 2019

  1. 1. @molly_struve
  2. 2. @molly_struve
  3. 3. @molly_struve
  4. 4. @molly_struve
  5. 5. @molly_struve
  6. 6. @molly_struve
  7. 7. @molly_struve
  8. 8. @molly_struve Adding Indexes Using SELECT statements Batch Processing
  9. 9. @molly_struve Elasticsearch::Transport::Errors::GatewayTimeout 504 { "statusCode": 200, "took": "100ms" }
  10. 10. @molly_struve Resque
  11. 11. @molly_struve
  12. 12. @molly_struve
  13. 13. @molly_struve
  14. 14. @molly_struve
  15. 15. @molly_struve The average company has... 60 thousand assets 24 million vulnerabilities
  16. 16. @molly_struve MySQL Elasticsearch Cluster
  17. 17. @molly_struve
  18. 18. @molly_struve MySQL Elasticsearch Cluster ActiveModelSerializers
  19. 19. @molly_struve class Vulnerability < ActiveModel::Serializer attributes :id, :client_id, :created_at, :updated_at, :priority, :details, :notes, :asset_id, :solution_id, :owner_id, :ticket_id end
  20. 20. @molly_struve 200 MILLION
  21. 21. @molly_struve
  22. 22. @molly_struve
  23. 23. @molly_struve
  24. 24. @molly_struve (1.6ms) (0.9ms) (4.1ms)(5.2ms) (5.2ms) (1.3ms) (3.1ms) (2.9ms) (2.2ms) (4.9ms) (6.0ms) (0.3ms) (1.6ms) (0.9ms) (2.2ms) (3.0ms) (2.1ms) (1.3ms) (2.1ms) (8.1ms) (1.4ms)
  25. 25. @molly_struve MySQL
  26. 26. @molly_struve
  27. 27. @molly_struve class BulkVulnerabilityCache attr_accessor :vulnerabilities, :client, :vulnerability_ids def initialize(vulns, client) self.vulnerabilities = vulns self.vulnerability_ids = vulns.map(&:id) self.client = client end # MySQL Lookups end
  28. 28. @molly_struve module Serializers class Vulnerability attr_accessor :vulnerability, :cache def initialize(vuln, bulk_cache) self.cache = bulk_cache self.vulnerability = vuln end end end self.cache = bulk_cache
  29. 29. @molly_struve class Vulnerability has_many :custom_fields end
  30. 30. @molly_struve CustomField.where(:vulnerability_id => vuln.id) cache.fetch('custom_fields', vuln.id)
  31. 31. @molly_struve (pry)> vulns = Vulnerability.limit(300); (pry)> Benchmark.realtime { vulns.each(&:serialize) } => 6.022452222998254 (pry)> Benchmark.realtime do > BulkVulnerability.new(vulns, [], client).serialize > end => 0.7267019419959979
  32. 32. @molly_struve Individual Serialization: Bulk Serialization: 2,100 7
  33. 33. @molly_struve 1k vulns 1k vulns 1k vulns
  34. 34. @molly_struve 1k vulns 1k vulns 1k vulns 7k 7
  35. 35. @molly_struve
  36. 36. @molly_struve
  37. 37. Process in Bulk
  38. 38. @molly_struve
  39. 39. @molly_struve
  40. 40. @molly_struve Elasticsearch Cluster + Redis MySQL Vulnerabilities
  41. 41. @molly_struve Redis.get Client 1 Index Client 2 Index Client 3 & 4 Index
  42. 42. @molly_struve indexing_hashes = vulnerability_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end
  43. 43. @molly_struve indexing_hashes = vulnerability_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end
  44. 44. @molly_struve (pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”) DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234" DEBUG -- : [Redis] call_time=1.07 ms GET
  45. 45. @molly_struve
  46. 46. @molly_struve
  47. 47. @molly_struve client_indexes = Hash.new do |h, client_id| h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”) end
  48. 48. @molly_struve indexing_hashes = vuln_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{client_id}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end client_indexes[hash[:client_id]],
  49. 49. @molly_struve 1 + 1 + 1 Client 1 Client 2 Client 3 1k 1k 1k
  50. 50. @molly_struve
  51. 51. @molly_struve 65% job speed up
  52. 52. @molly_struve
  53. 53. @molly_struve
  54. 54. Process in Bulk Hash Cache
  55. 55. @molly_struve CLIENT 1 CLIENT 2 CLIENT 3
  56. 56. @molly_struve Asset.using(:shard_1).find(1)
  57. 57. @molly_struve { 'client_123' => 'shard_123', 'client_456' => 'shard_456', 'client_789' => 'shard_789' }
  58. 58. @molly_struve { 'client_123' => 'shard_123', 'client_456' => 'shard_456', 'client_789' => 'shard_789' } sharding_config_hash[client_id]
  59. 59. @molly_struve
  60. 60. @molly_struve 20 bytes 1kb 13kb
  61. 61. @molly_struve
  62. 62. @molly_struve
  63. 63. @molly_struve ActiveRecord::Base.connection
  64. 64. @molly_struve (pry)> ActiveRecord::Base.connection => #<Octopus::Proxy:0x000055b38c697d10 @proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8
  65. 65. @molly_struve
  66. 66. @molly_struve module Octopus class Proxy attr_accessor :proxy_config delegate :current_shard, :current_shard=, :current_slave_group, :current_slave_group=, :shard_names, :shards_for_group, :shards, :sharded, :config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false end end
  67. 67. @molly_struve
  68. 68. Process in Bulk Hash Cache Active Record Cache
  69. 69. @molly_struve
  70. 70. @molly_struve User.where(:id => user_ids).each do |user| # Lots of user processing end
  71. 71. @molly_struve
  72. 72. @molly_struve (pry)> User.where(:id => []) User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0 => []
  73. 73. @molly_struve
  74. 74. @molly_struve return unless user_ids.any? User.where(:id => user_ids).each do |user| # Lots of user processing end
  75. 75. @molly_struve (pry)> Benchmark.realtime do > 10_000.times { User.where(:id => []) } > end => 0.5508159045130014 (pry)> Benchmark.realtime do > 10_000.times do > next unless ids.any? > User.where(:id => []) > end > end => 0.0006368421018123627
  76. 76. @molly_struve (pry)> Benchmark.realtime do > 10_000.times { User.where(:id => []) } > end => 0.5508159045130014
  77. 77. @molly_struve User.where(:id => user_ids).each do |user| # Lots of user processing end
  78. 78. @molly_struve User.where(:id => user_ids).each do |user| # Lots of user processing end users = User.where(:id => user_ids).active.short.single
  79. 79. @molly_struve .none
  80. 80. @molly_struve (pry)> User.where(:id => []).active.tall.single User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND `users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1 => [] (pry)> User.none.active.tall.single => []
  81. 81. @molly_struve
  82. 82. @molly_struve
  83. 83. @molly_struve pry(main)> Rails.logger.level = 0 $ redis-cli monitor > commands-redis-2018-10-01.txt pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)
  84. 84. @molly_struve
  85. 85. @molly_struve
  86. 86. @molly_struve
  87. 87. @molly_struve (pry)> Report.zero_assets.count => 10805 (pry)> Report.active.count => 25842 (pry)> Report.average_asset_count => 1657 Investigating Existing Reports
  88. 88. @molly_struve
  89. 89. @molly_struve
  90. 90. @molly_struve return if report.asset_count.zero?
  91. 91. @molly_struve
  92. 92. @molly_struve
  93. 93. Process in Bulk Hash Cache Database Guards Active Record Cache
  94. 94. @molly_struve Resque Workers Redis
  95. 95. @molly_struve 45 workers 45 workers 45 workers
  96. 96. @molly_struve 70 workers 70 workers 70 workers
  97. 97. @molly_struve
  98. 98. @molly_struve
  99. 99. @molly_struve
  100. 100. @molly_struve 48 MB 16 MB
  101. 101. @molly_struve 70 workers 100k 200k
  102. 102. @molly_struve
  103. 103. @molly_struve
  104. 104. @molly_struve Resque Throttling
  105. 105. @molly_struve 100k 200k
  106. 106. @molly_struve 48MB 16MB
  107. 107. @molly_struve
  108. 108. Process in Bulk Hash Cache Database Guards Remove Datastore Hits Active Record Cache
  109. 109. @molly_struve
  110. 110. @molly_struve
  111. 111. @molly_struve
  112. 112. @molly_struve https://www.linkedin.com/in/mollystruve/ https://github.com/mstruve @molly_struve molly.struve@gmail.com

×