1. @molly_struve

2. @molly_struve

3. @molly_struve

4. @molly_struve

5. @molly_struve

6. @molly_struve

7. @molly_struve

8. @molly_struve
Adding Indexes
Using SELECT statements
Batch Processing

9. @molly_struve
Elasticsearch::Transport::Errors::GatewayTimeout 504
{
"statusCode": 200,
"took": "100ms"
}

10. @molly_struve
Resque

11. @molly_struve

12. @molly_struve

13. @molly_struve

14. @molly_struve

15. @molly_struve
The average company has...
60 thousand
assets
24 million
vulnerabilities

16. @molly_struve
MySQL
Elasticsearch
Cluster

17. @molly_struve

18. @molly_struve
MySQL
Elasticsearch
Cluster
ActiveModelSerializers

19. @molly_struve
class Vulnerability < ActiveModel::Serializer
attributes :id, :client_id, :created_at, :updated_at,
:priority, :details, :notes, :asset_id,
:solution_id, :owner_id, :ticket_id
end

20. @molly_struve
200 MILLION

21. @molly_struve

22. @molly_struve

23. @molly_struve

24. @molly_struve
(1.6ms)
(0.9ms)
(4.1ms)(5.2ms)
(5.2ms)
(1.3ms)
(3.1ms)
(2.9ms)
(2.2ms)
(4.9ms)
(6.0ms)
(0.3ms)
(1.6ms)
(0.9ms)
(2.2ms)
(3.0ms)
(2.1ms)
(1.3ms)
(2.1ms)
(8.1ms)
(1.4ms)

25. @molly_struve
MySQL

26. @molly_struve

27. @molly_struve
class BulkVulnerabilityCache
attr_accessor :vulnerabilities, :client, :vulnerability_ids
def initialize(vulns, client)
self.vulnerabilities = vulns
self.vulnerability_ids = vulns.map(&:id)
self.client = client
end
# MySQL Lookups
end

28. @molly_struve
module Serializers
class Vulnerability
attr_accessor :vulnerability, :cache
def initialize(vuln, bulk_cache)
self.cache = bulk_cache
self.vulnerability = vuln
end
end
end
self.cache = bulk_cache

29. @molly_struve
class Vulnerability
has_many :custom_fields
end

30. @molly_struve
CustomField.where(:vulnerability_id => vuln.id)
cache.fetch('custom_fields', vuln.id)

31. @molly_struve
(pry)> vulns = Vulnerability.limit(300);
(pry)> Benchmark.realtime { vulns.each(&:serialize) }
=> 6.022452222998254
(pry)> Benchmark.realtime do
> BulkVulnerability.new(vulns, [], client).serialize
> end
=> 0.7267019419959979

32. @molly_struve
Individual Serialization:
Bulk Serialization:
2,100
7

33. @molly_struve
1k vulns 1k vulns 1k vulns

34. @molly_struve
1k vulns 1k vulns 1k vulns
7k 7

35. @molly_struve

36. @molly_struve

37. Process
in Bulk

38. @molly_struve

39. @molly_struve

40. @molly_struve
Elasticsearch
Cluster
+
Redis
MySQL
Vulnerabilities

41. @molly_struve
Redis.get
Client 1
Index
Client 2
Index
Client 3 & 4
Index

42. @molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

43. @molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

44. @molly_struve
(pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”)
DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234"
DEBUG -- : [Redis] call_time=1.07 ms
GET

45. @molly_struve

46. @molly_struve

47. @molly_struve
client_indexes = Hash.new do |h, client_id|
h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”)
end

48. @molly_struve
indexing_hashes = vuln_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{client_id}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end
client_indexes[hash[:client_id]],

49. @molly_struve
1 + 1 + 1
Client 1 Client 2 Client 3
1k 1k 1k

50. @molly_struve

51. @molly_struve
65% job
speed up

52. @molly_struve

53. @molly_struve

54. Process
in Bulk
Hash
Cache

55. @molly_struve
CLIENT 1
CLIENT 2
CLIENT 3

56. @molly_struve
Asset.using(:shard_1).find(1)

57. @molly_struve
{
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}

58. @molly_struve
{
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}
sharding_config_hash[client_id]

59. @molly_struve

60. @molly_struve
20 bytes 1kb 13kb

61. @molly_struve

62. @molly_struve

63. @molly_struve
ActiveRecord::Base.connection

64. @molly_struve
(pry)> ActiveRecord::Base.connection
=> #<Octopus::Proxy:0x000055b38c697d10
@proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8

65. @molly_struve

66. @molly_struve
module Octopus
class Proxy
attr_accessor :proxy_config
delegate :current_shard, :current_shard=,
:current_slave_group, :current_slave_group=,
:shard_names, :shards_for_group, :shards, :sharded,
:config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false
end
end

67. @molly_struve

68. Process
in Bulk
Hash
Cache
Active
Record
Cache

69. @molly_struve

70. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

71. @molly_struve

72. @molly_struve
(pry)> User.where(:id => [])
User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0
=> []

73. @molly_struve

74. @molly_struve
return unless user_ids.any?
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

75. @molly_struve
(pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
(pry)> Benchmark.realtime do
> 10_000.times do
> next unless ids.any?
> User.where(:id => [])
> end
> end
=> 0.0006368421018123627

76. @molly_struve
(pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014

77. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

78. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end
users = User.where(:id => user_ids).active.short.single

79. @molly_struve
.none

80. @molly_struve
(pry)> User.where(:id => []).active.tall.single
User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND
`users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1
=> []
(pry)> User.none.active.tall.single
=> []

81. @molly_struve

82. @molly_struve

83. @molly_struve
pry(main)> Rails.logger.level = 0
$ redis-cli monitor > commands-redis-2018-10-01.txt
pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)

84. @molly_struve

85. @molly_struve

86. @molly_struve

87. @molly_struve
(pry)> Report.zero_assets.count
=> 10805
(pry)> Report.active.count
=> 25842
(pry)> Report.average_asset_count
=> 1657
Investigating Existing Reports

88. @molly_struve

89. @molly_struve

90. @molly_struve
return if report.asset_count.zero?

91. @molly_struve

92. @molly_struve

93. Process
in Bulk
Hash
Cache
Database
Guards
Active
Record
Cache

94. @molly_struve
Resque Workers
Redis

95. @molly_struve
45 workers
45 workers
45 workers

96. @molly_struve
70 workers
70 workers
70 workers

97. @molly_struve

98. @molly_struve

99. @molly_struve

100. @molly_struve
48 MB
16 MB

101. @molly_struve
70 workers
100k
200k

102. @molly_struve

103. @molly_struve

104. @molly_struve
Resque
Throttling

105. @molly_struve
100k
200k

106. @molly_struve
48MB
16MB

107. @molly_struve

108. Process
in Bulk
Hash
Cache
Database
Guards
Remove
Datastore
Hits
Active
Record
Cache

109. @molly_struve

110. @molly_struve

111. @molly_struve

112. @molly_struve
https://www.linkedin.com/in/mollystruve/
https://github.com/mstruve
@molly_struve
molly.struve@gmail.com