Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cache is King - RubyHACK 2019

2,250 views

Published on

Sometimes your fastest queries can cause the most problems. I'll take you beyond the slow query optimization and instead zero in on the performance impacts surrounding the quantity of your datastore hits.. Using real world examples dealing with datastores such as Elasticsearch, MySQL, and Redis, I will demonstrate how many fast queries can wreak just as much havoc as a few big slow ones. With each example I will make use of the simple tools available in Ruby to decrease and eliminate the need for these fast and seemingly innocuous datastore hits.

Published in: Engineering
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Cache is King - RubyHACK 2019

  1. 1. @molly_struve Cache is King Get the Most Bang for Your Buck From Ruby
  2. 2. @molly_struve Site Reliability Engineer
  3. 3. @molly_struve
  4. 4. @molly_struve Adding Indexes Using SELECT statements Batch Processing
  5. 5. @molly_struve Elasticsearch::Transport::Errors::GatewayTimeout 504 { "statusCode": 200, "took": "100ms" }
  6. 6. @molly_struve Resque
  7. 7. @molly_struve
  8. 8. @molly_struve Demo Time!
  9. 9. @molly_struve Quantity of Datastore Hits
  10. 10. @molly_struve
  11. 11. @molly_struve The average company has... 60 thousand assets 24 million vulnerabilities?
  12. 12. @molly_struve MySQL Elasticsearch Cluster
  13. 13. @molly_struve Serialization
  14. 14. @molly_struve MySQL Elasticsearch Cluster ActiveModelSerializers
  15. 15. @molly_struve module Beehive module Serializers class Vulnerability < ActiveModel::Serializer attributes :id, :client_id, :created_at, :updated_at, :priority, :details, :notes, :asset_id, :solution_id, :owner_id, :ticket_id end end end
  16. 16. @molly_struve 200 MILLION
  17. 17. @molly_struve 11 hours and counting...
  18. 18. @molly_struve
  19. 19. @molly_struve
  20. 20. @molly_struve (1.6ms) (0.9ms) (4.1ms)(5.2ms) (5.2ms) (1.3ms) (3.1ms) (2.9ms) (2.2ms) (4.9ms) (6.0ms) (0.3ms) (1.6ms) (0.9ms) (2.2ms) (3.0ms) (2.1ms) (1.3ms) (2.1ms) (8.1ms) (1.4ms)
  21. 21. @molly_struve MySQL
  22. 22. @molly_struve Bulk Serialization
  23. 23. @molly_struve class BulkVulnerabilityCache attr_accessor :vulnerabilities, :client, :vulnerability_ids def initialize(vulns, client) self.vulnerabilities = vulns self.vulnerability_ids = vulns.map(&:id) self.client = client end # MySQL Lookups end
  24. 24. @molly_struve module Serializers class Vulnerability attr_accessor :vulnerability, :cache def initialize(vuln, bulk_cache) self.cache = bulk_cache self.vulnerability = vuln end end end self.cache = bulk_cache
  25. 25. @molly_struve class Vulnerability has_many :custom_fields end
  26. 26. @molly_struve CustomField.where(:vulnerability_id => vuln.id) cache.fetch('custom_fields', vuln.id)
  27. 27. @molly_struve The Result... (pry)> vulns = Vulnerability.limit(300); (pry)> Benchmark.realtime { vulns.each(&:serialize) } => 6.022452222998254 (pry)> Benchmark.realtime do > BulkVulnerability.new(vulns, [], client).serialize > end => 0.7267019419959979
  28. 28. @molly_struve Decrease in database hits Individual Serialization: Bulk Serialization: 2,100 7
  29. 29. @molly_struve 1k vulns 1k vulns 1k vulns Vulnerability Batches
  30. 30. @molly_struve 1k vulns 1k vulns 1k vulns Vulnerability Batches 7k 7
  31. 31. @molly_struve MySQL Queries Bulk Serialization Deployed
  32. 32. @molly_struve Bulk Serialization Deployed RDS CPU Utilization
  33. 33. @molly_struve Process in Bulk
  34. 34. @molly_struve
  35. 35. @molly_struve
  36. 36. @molly_struve Elasticsearch Cluster + Redis MySQL Vulnerabilities
  37. 37. @molly_struve Redis.get Client 1 Index Client 2 Index Client 3 & 4 Index
  38. 38. @molly_struve indexing_hashes = vulnerability_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end
  39. 39. @molly_struve indexing_hashes = vulnerability_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end
  40. 40. @molly_struve (pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”) DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234" DEBUG -- : [Redis] call_time=1.07 ms GET
  41. 41. @molly_struve
  42. 42. @molly_struve
  43. 43. @molly_struve client_indexes = Hash.new do |h, client_id| h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”) end
  44. 44. @molly_struve indexing_hashes = vuln_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{client_id}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end client_indexes[hash[:client_id]],
  45. 45. @molly_struve 1 + 1 + 1 Client 1 Client 2 Client 3 1k 1k 1k
  46. 46. @molly_struve 1000x
  47. 47. @molly_struve 65% job speed up
  48. 48. @molly_struve Local Cache
  49. 49. @molly_struve Redis
  50. 50. @molly_struve Process in Bulk Hash Cache
  51. 51. @molly_struve Sharded Databases CLIENT 1 CLIENT 2 CLIENT 3
  52. 52. @molly_struve Asset.with_shard(client_id).find(1)
  53. 53. @molly_struve { 'client_123' => 'shard_123', 'client_456' => 'shard_456', 'client_789' => 'shard_789' } Sharding Configuration
  54. 54. @molly_struve
  55. 55. @molly_struve Sharding Configuration Size 20 bytes 1kb 13kb
  56. 56. @molly_struve 285 Workers
  57. 57. @molly_struve 7.8 MB/second
  58. 58. @molly_struve ActiveRecord::Base.connection
  59. 59. @molly_struve (pry)> ActiveRecord::Base.connection => #<Octopus::Proxy:0x000055b38c697d10 @proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8
  60. 60. @molly_struve module Octopus class Proxy attr_accessor :proxy_config delegate :current_shard, :current_shard=, :current_slave_group, :current_slave_group=, :shard_names, :shards_for_group, :shards, :sharded, :config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false end end
  61. 61. @molly_struve Know your gems
  62. 62. @molly_struve Process in Bulk Framework Cache Hash Cache
  63. 63. @molly_struve Avoid making datastore hits you don’t need
  64. 64. @molly_struve User.where(:id => user_ids).each do |user| # Lots of user processing end
  65. 65. @molly_struve FALSE
  66. 66. @molly_struve (pry)> User.where(:id => []) User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0 => []
  67. 67. @molly_struve
  68. 68. @molly_struve return unless user_ids.any? User.where(:id => user_ids).each do |user| # Lots of user processing end
  69. 69. @molly_struve (pry)> Benchmark.realtime do > 10_000.times { User.where(:id => []) } > end => 0.5508159045130014 (pry)> Benchmark.realtime do > 10_000.times do > next unless ids.any? > User.where(:id => []) > end > end => 0.0006368421018123627
  70. 70. @molly_struve (pry)> Benchmark.realtime do > 10_000.times { User.where(:id => []) } > end => 0.5508159045130014 “Ruby is slow”Hitting the database is slow!
  71. 71. @molly_struve User.where(:id => user_ids).each do |user| # Lots of user processing end
  72. 72. @molly_struve User.where(:id => user_ids).each do |user| # Lots of user processing end users = User.where(:id => user_ids).active.short.single
  73. 73. @molly_struve .none
  74. 74. @molly_struve (pry)> User.where(:id => []).active.tall.single User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND `users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1 => [] (pry)> User.none.active.tall.single => [] .none in action...
  75. 75. @molly_struve
  76. 76. @molly_struve
  77. 77. @molly_struve Logging pry(main)> Rails.logger.level = 0 $ redis-cli monitor > commands-redis-2018-10-01.txt pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)
  78. 78. @molly_struve Preventing useless datastore hits
  79. 79. @molly_struve
  80. 80. @molly_struve Report Elasticsearch MySQL Redis
  81. 81. @molly_struve (pry)> Report.blank_reports.count => 10805 (pry)> Report.active.count => 25842 (pry)> Report.average_asset_count => 1657 Investigating Existing Reports
  82. 82. @molly_struve Report Elasticsearch MySQL Redis
  83. 83. @molly_struve
  84. 84. @molly_struve return if report.asset_count.zero?
  85. 85. @molly_struve 10+ hrs
  86. 86. @molly_struve 3 hrs
  87. 87. @molly_struve Process in Bulk Framework Cache Database Guards Hash Cache
  88. 88. @molly_struve Resque Workers Redis
  89. 89. @molly_struve 45 workers 45 workers 45 workers
  90. 90. @molly_struve 70 workers 70 workers 70 workers
  91. 91. @molly_struve
  92. 92. @molly_struve
  93. 93. @molly_struve
  94. 94. @molly_struve 48 MB 16 MB
  95. 95. @molly_struve Redis Requests 70 workers 100k 200k
  96. 96. @molly_struve
  97. 97. @molly_struve ?
  98. 98. @molly_struve Resque Throttling
  99. 99. @molly_struve Redis Requests 100k 200k
  100. 100. @molly_struve Redis Network Traffic 48MB 16MB
  101. 101. @molly_struve
  102. 102. @molly_struve Process in Bulk Framework Cache Database Guards Remove Datastore Hits Hash Cache
  103. 103. @molly_struve Every datastore hit COUNTS
  104. 104. @molly_struve
  105. 105. @molly_struve Questions
  106. 106. @molly_struve Contact https://www.linkedin.com/in/mollystruve/ https://github.com/mstruve @molly_struve molly.struve@gmail.com

×