Sometimes your fastest queries can cause the most problems. I will take you beyond the slow query optimization and instead zero in on the performance impacts surrounding the quantity of your datastore hits. Using real world examples dealing with datastores such as Elasticsearch, MySQL, and Redis, I will demonstrate how many fast queries can wreak just as much havoc as a few big slow ones. With each example I will make use of the simple tools available in Ruby to decrease and eliminate the need for these fast and seemingly innocuous datastore hits.

1. @molly_struve
Cache is King

2. @molly_struve
Site Reliability Engineer

3. @molly_struve

4. @molly_struve

5. @molly_struve
Demo Time!

6. @molly_struve
Quantity of Datastore Hits

7. @molly_struve

8. @molly_struve
The average company has...
60 thousand
assets
24 million
vulnerabilities?

9. @molly_struve
MySQL
Elasticsearch
Cluster

10. @molly_struve
Serialization

11. @molly_struve
MySQL
Elasticsearch
Cluster
ActiveModelSerializers

12. @molly_struve
class Vulnerability < ActiveModel::Serializer
attributes :id, :client_id, :created_at, :updated_at,
:priority, :details, :notes, :asset_id,
:solution_id, :owner_id, :ticket_id
end

13. @molly_struve
200 MILLION

14. @molly_struve
11 hours and counting...

15. @molly_struve

16. @molly_struve

17. @molly_struve
(1.6ms)
(0.9ms)
(4.1ms)(5.2ms)
(5.2ms)
(1.3ms)
(3.1ms)
(2.9ms)
(2.2ms)
(4.9ms)
(6.0ms)
(0.3ms)
(1.6ms)
(0.9ms)
(2.2ms)
(3.0ms)
(2.1ms)
(1.3ms)
(2.1ms)
(8.1ms)
(1.4ms)

18. @molly_struve
MySQL

19. @molly_struve
Bulk Serialization

20. @molly_struve
class BulkVulnerabilityCache
attr_accessor :vulnerabilities, :client, :vulnerability_ids
def initialize(vulns, client)
self.vulnerabilities = vulns
self.vulnerability_ids = vulns.map(&:id)
self.client = client
end
# MySQL Lookups
end

21. @molly_struve
module Serializers
class Vulnerability
attr_accessor :vulnerability, :cache
def initialize(vuln, bulk_cache)
self.cache = bulk_cache
self.vulnerability = vuln
end
end
end
self.cache = bulk_cache

22. @molly_struve
The Result...
(pry)> vulns = Vulnerability.limit(300);
(pry)> Benchmark.realtime { vulns.each(&:serialize) }
=> 6.022452222998254
(pry)> Benchmark.realtime do
> BulkVulnerability.new(vulns, [], client).serialize
> end
=> 0.7267019419959979

23. @molly_struve
Decrease in database hits
Individual Serialization:
Bulk Serialization:
2,100
7

24. @molly_struve
MySQL Queries
Bulk Serialization
Deployed

25. @molly_struve
Bulk Serialization
Deployed
RDS CPU Utilization

26. Process
in Bulk

27. @molly_struve

28. @molly_struve

29. @molly_struve
Elasticsearch
Cluster
+
Redis
MySQL
Vulnerabilities

30. @molly_struve
Redis.get
Client 1
Index
Client 2
Index
Client 3 & 4
Index

31. @molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

32. @molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

33. @molly_struve
(pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”)
DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234"
DEBUG -- : [Redis] call_time=1.07 ms
GET

34. @molly_struve

35. @molly_struve

36. @molly_struve
client_indexes = Hash.new do |h, client_id|
h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”)
end

37. @molly_struve
indexing_hashes = vuln_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{client_id}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end
client_indexes[hash[:client_id]],

38. @molly_struve
1 + 1 + 1
Client 1 Client 2 Client 3
1k 1k 1k

39. @molly_struve
1000x

40. @molly_struve
65% job
speed up

41. Process
in Bulk
Hash
Cache

42. @molly_struve
Sharded Databases
CLIENT 1
CLIENT 2
CLIENT 3

43. @molly_struve
Asset.using(:shard_1).find(1)

44. @molly_struve
{
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}
Sharding Configuration Hash

45. @molly_struve
{
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}
Sharding Configuration Hash
sharding_config_hash[client_id]

46. @molly_struve

47. @molly_struve
Sharding Configuration Size
20 bytes 1kb 13kb

48. @molly_struve
285 Workers

49. @molly_struve
7.8 MB/second

50. @molly_struve
ActiveRecord::Base.connection

51. @molly_struve
(pry)> ActiveRecord::Base.connection
=> #<Octopus::Proxy:0x000055b38c697d10
@proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8

52. @molly_struve

53. @molly_struve
module Octopus
class Proxy
attr_accessor :proxy_config
delegate :current_shard, :current_shard=,
:current_slave_group, :current_slave_group=,
:shard_names, :shards_for_group, :shards, :sharded,
:config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false
end
end

54. Process
in Bulk
Hash
Cache
Active
Record
Cache

55. @molly_struve
Avoid making datastore hits you don’t
need

56. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

57. @molly_struve
FALSE

58. @molly_struve
(pry)> User.where(:id => [])
User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0
=> []

59. @molly_struve

60. @molly_struve
return unless user_ids.any?
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

61. @molly_struve
(pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
(pry)> Benchmark.realtime do
> 10_000.times do
> next unless ids.any?
> User.where(:id => [])
> end
> end
=> 0.0006368421018123627

62. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

63. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end
users = User.where(:id => user_ids).active.short.single

64. @molly_struve
.none

65. @molly_struve
(pry)> User.where(:id => []).active.tall.single
User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND
`users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1
=> []
(pry)> User.none.active.tall.single
=> []
.none in action...

66. @molly_struve

67. @molly_struve

68. @molly_struve
Logging
pry(main)> Rails.logger.level = 0
$ redis-cli monitor > commands-redis-2018-10-01.txt
pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)

69. @molly_struve
Preventing useless
datastore hits

70. @molly_struve

71. @molly_struve
Report
Elasticsearch
MySQL Redis

72. @molly_struve
(pry)> Report.zero_assets.count
=> 10805
(pry)> Report.active.count
=> 25842
(pry)> Report.average_asset_count
=> 1657
Investigating Existing Reports

73. @molly_struve
Report
Elasticsearch
MySQL Redis

74. @molly_struve

75. @molly_struve
return if report.asset_count.zero?

76. @molly_struve
10+ hrs

77. @molly_struve
3 hrs

78. Process
in Bulk
Hash
Cache
Database
Guards
Active
Record
Cache

79. @molly_struve
Resque Workers
Redis

80. @molly_struve
45 workers
45 workers
45 workers

81. @molly_struve
70 workers
70 workers
70 workers

82. @molly_struve

83. @molly_struve

84. @molly_struve

85. @molly_struve
48 MB
16 MB

86. @molly_struve
Redis Requests
70 workers
100k
200k

87. @molly_struve

88. @molly_struve
?

89. @molly_struve
Resque
Throttling

90. @molly_struve
Redis Requests
100k
200k

91. @molly_struve
Redis Network Traffic
48MB
16MB

92. @molly_struve

93. Process
in Bulk
Hash
Cache
Database
Guards
Remove
Datastore
Hits
Active
Record
Cache

94. @molly_struve
Every
datastore hit
COUNTS

95. @molly_struve