The document discusses load testing and why it often fails. It recommends using a tool with the lowest barrier to entry, such as Blitz, which allows load testing on AWS with a web form or API. Blitz produces results but no reports, so the document shows how to modify Blitz's code to output JSON results for reporting purposes. It encourages integrating load testing into existing development workflows rather than treating it separately after deployment.

1. Load testing

3. Who is this dude?

4. Lindsay Holmwood

5. BULLETPROOF

6. I Run this thing

7. Have tried and failed many
times to implement effective
load testing

9. A story

10. New project

11. Replacing a high-
profile website
(new infrastructure,
new application)

12. 1000
concurrent users

13. “We need 100%
uptime”

14. Project starts

15. Requirements
change

16. Deadlines
aren’t met

17. Performance testing?
Fuck it!
We’ll do it live!

19. Performance testing?
“You have one day”

21. Problem:

22. Treated as
WAterfall-esque
QA

23. It’s a
non-functional
requirement

25. WHY
Do we load test?

26. Capacity
planning

27. “Can the site deal
with load?”

28. Defect
detection

29. “Is this code
going to cripple
the site?”

30. Optimisation

31. “What happens if
we change this
Setting?”

32. Historical
analysis

33. “Is the site
getting slower?”

34. Cache warming

35. “Prepare the
infrastructure
for Failover”

36. WHY
Do we load test?

37. A combination of
proactive &
Reactive reasons

38. A combination of
proactive &
Reactive reasons

39. It’s a
non-functional
requirement

41. Why
does load testing
FAIL?
(from a technical perspective)

42. How
not the
WHAT

43. TooL fetishism

44. Get bogged down
in details

45. how the testing
infrastructure works

46. Micro
vs
Macro

47. Business
Does not care
about your tools

48. Business
cares About:

49. “Can the site deal
with load?”

51. Recommendation:

52. Use tool with
lowest barrier
TO entry

54. Blitz

55. Runs on AWS

56. Web form

57. Browser plugins
Firefox, Chrome

58. REST API

59. API Clients
Ruby, Python, Java
Node.JS, Perl

60. Common
Command Line

61. Excellent
documentation
docs.blitz.io

62. Inexpensive
250 concurrent users
over 60 second period == free

63. SPRINTS
&
RUSHES

65. Let’s dive in!

66. Sign up @
http://blitz.io/

67. $ gem install blitz
$ blitz help
Usage: blitz <command> <options>
help - Display this help
account:about - Show information about your account
api:init - Validate and login with your API key
couch:fuzz - Auto generate blitz tests from CouchDB
curl - Run a sprint or a rush
curl:help - Show help on sprint and rushing
traceroute - Run traceroute remotely
traceroute:help - Show help on traceroute
version - Show the version of this Ruby gem
$ blitz api:init
# writes credentials to ~/.blitz/credentials

68. $ blitz curl:help
Usage: blitz curl <options> <url>
--user-agent -A <string> User-Agent to send to server
--cookie -b name=<string> Cookie to send to the server (multiple)
--data -d <string> Data to send in a PUT or POST request
--dump-header -D <file> Print the request/response headers
--referer -e <string> Referer URL
--help -h Help on command line options
--header -H <string> Custom header to pass to server
--pattern -p <s>-<e>:<d> Ramp from s to e concurrent requests in d secs
--region -r <string> california|oregon|virginia|singapore|ireland|japan
--status -s <number> Assert on the HTTP response status code
--timeout -T <ms> Wait time for both connect and responses
--user -u <user[:pass]> User and password for authentication
--request -X <string> Request method to use (GET, HEAD, PUT, etc.)
--variable -v <string> Define a variable to use
--verbose -V Print the request/response headers
--tlsv1 -1 Use TLSv1 (SSL)
--sslv2 -2 Use SSLv2 (SSL)
--sslv3 -3 Use SSLv3 (SSL)

70. $ blitz curl --region singapore --pattern 1-250:20
--timeout 10000 --verbose http://example.org/

71. user growth / time
$ blitz curl --region singapore --pattern 1-250:20
--timeout 10000 --verbose http://example.org/

72. user growth / time
for AWS lag
$ blitz curl --region singapore --pattern 1-250:20
--timeout 10000 --verbose http://example.org/

73. user growth / time
for AWS lag
$ blitz curl --region singapore --pattern 1-250:20
--timeout 10000 --verbose http://example.org/
rushing from singapore...
Time Users Response Hits Timeouts Errors Hits/s Mbps
2.5s 31 -1.000s 0 0 0
8.3s 103 1.075s 146 0 9 50.53 0.88
11.2s 140 0.406s 376 0 9 79.64 1.41
14.1s 176 0.409s 682 0 9 105.24 1.86
22.8s 0 0.417s 1552 0 9 15.57 0.27

74. --pattern 1-250:20

75. blitz calculates the
growth automatically
--pattern 1-250:20

76. Sawtooth
--pattern 1-250:20,1-250:20,1-250:20

77. TV AD / Twittersplosion
--pattern 200-5000:30,5000-400:60

79. Integrate
with your
development cycle

80. VCS
CI
Deploy automation

81. VCS
CI
Deploy automation

82. git
post-receive hook

83. GitHub web hooks

84. Doesn’t take into
account test
failures

85. Is the code
actually
deployed?

87. VCS
CI
Deploy automation

88. CRON JOB

89. JENKINS JOB

90. GREAT if you have
a CI system

91. Runs tests
but
Separate system

92. What if you’re making
infrastructure changes
during a load test?

94. VCS
CI
Deploy automation

95. Fabric

96. Capistrano

97. Run tests when
you deploy

98. Use the same
toolchain

99. Testing lives IN the app.

100. What about triggering
load tests on
infrastructure
changes?

101. MVP

102. Just get
something
working!

104. Problem:

105. No REporting
:-(

106. THIS IS IT
rushing from singapore...
Time Users Response Hits Timeouts Errors Hits/s Mbps
2.5s 31 -1.000s 0 0 0
8.3s 103 1.075s 146 0 9 50.53 0.88
11.2s 140 0.406s 376 0 9 79.64 1.41
14.1s 176 0.409s 682 0 9 105.24 1.86
22.8s 0 0.417s 1552 0 9 15.57 0.27

107. SOL ON:

108. Optimisation

109. “What happens if
we change this
Setting?”

110. Historical
analysis

111. “Is the site
getting slower?”

112. Showstopper?

113. ...

114. $ gem install blitz
# ...
rushing from singapore...
Time Users Response Hits Timeouts Errors Hits/s Mbps
2.5s 31 -1.000s 0 0 0
8.3s 103 1.075s 146 0 9 50.53 0.88
11.2s 140 0.406s 376 0 9 79.64 1.41
14.1s 176 0.409s 682 0 9 105.24 1.86
22.8s 0 0.417s 1552 0 9 15.57 0.27

115. Can we duck
punch our way
to success?

117. #!/usr/bin/env ruby
# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"
url = "http://example.org/"
command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)
curl = Blitz::Command::Curl.new
curl.rush(job)

118. # ruby load_test.rb
rushing from singapore...
Time Users Response Hits Timeouts Errors Hits/s Mbps
2.5s 31 -1.000s 0 0 0
8.3s 103 1.075s 146 0 9 50.53 0.88
11.2s 140 0.406s 376 0 9 79.64 1.41
14.1s 176 0.409s 682 0 9 105.24 1.86
22.8s 0 0.417s 1552 0 9 15.57 0.27

119. #!/usr/bin/env ruby
# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"
url = "http://example.org/"
command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)
curl = Blitz::Command::Curl.new
curl.rush(job)

120. #!/usr/bin/env ruby
# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"
url = "http://example.org/"
command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)
curl = Blitz::Command::Curl.new
curl.rush(job)

121. Blitz::Curl.parse(command)
returns
Blitz::Curl::Rush

122. #!/usr/bin/env ruby
# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"
url = "http://example.org/"
command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)
curl = Blitz::Command::Curl.new
curl.rush(job)

123. Blitz::Command::Curl#rush

124. initiates the test
+ outputs the test results
Blitz::Command::Curl#rush

125. #!/usr/bin/env ruby
# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"
url = "http://example.org/"
command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)
curl = Blitz::Command::Curl.new
curl.rush(job)
puts job.result.timeline.to_json

126. Produces muck

127. duck punch our
way to success

128. class Blitz::Curl::Rush
class Point
def to_json(*args)
{
'timestamp' => @timestamp,
'duration' => @duration,
'total' => @total,
'hits' => @hits,
'errors' => @errors,
'timeouts' => @timeouts,
'volume' => @volume,
'txbytes' => @txbytes,
'rxbytes' => @rxbytes,
'steps' => @steps,
}.to_json
end
end
# ...

129. # ...
class Step
def to_json(*args)
{
'duration' => @duration,
'connect' => @connect,
'errors' => @errors,
'timeouts' => @timeouts,
'asserts' => @asserts,
}.to_json
end
end
end

130. #!/usr/bin/env ruby
# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"
url = "http://example.org/"
command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)
curl = Blitz::Command::Curl.new
curl.rush(job)
puts job.result.timeline.to_json

131. {"start":1335694346,"finish":1335694367,"results":[{"timestamp":
2.502626,"duration":1.13912,"total":1,"hits":1,"errors":0,"timeouts":
0,"volume":2,"txbytes":258.0,"rxbytes":17810.0,"steps":[{"duration":
2.13912,"connect":0.260004,"errors":0,"timeouts":0,"asserts":0}]},
{"timestamp":5.016102,"duration":0.640047,"total":5,"hits":5,"errors":
0,"timeouts":0,"volume":4,"txbytes":1548.0,"rxbytes":90882.0,"steps":
[{"duration":1.640047,"connect":0.158944,"errors":0,"timeouts":
0,"asserts":0}]},{"timestamp":7.524594,"duration":0.639987,"total":
10,"hits":10,"errors":0,"timeouts":0,"volume":5,"txbytes":
3354.0,"rxbytes":182222.0,"steps":[{"duration":1.639987,"connect":
0.158999,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":
10.03388,"duration":0.63961,"total":20,"hits":20,"errors":0,"timeouts":
0,"volume":7,"txbytes":5418.0,"rxbytes":364902.0,"steps":[{"duration":
1.63961,"connect":0.158847,"errors":0,"timeouts":0,"asserts":0}]},
{"timestamp":12.543211,"duration":0.640421,"total":31,"hits":
31,"errors":0,"timeouts":0,"volume":9,"txbytes":9030.0,"rxbytes":
565850.0,"steps":[{"duration":1.640422,"connect":0.158767,"errors":
0,"timeouts":0,"asserts":0}]},{"timestamp":15.051705,"duration":
0.639894,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":
10,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":
1.639894,"connect":0.158794,"errors":0,"timeouts":0,"asserts":0}]},
{"timestamp":17.560136,"duration":-1.0,"total":44,"hits":44,"errors":
0,"timeouts":0,"volume":0,"txbytes":12384.0,"rxbytes":803334.0,"steps":
[{"duration":0.0,"connect":0.0,"errors":0,"timeouts":0,"asserts":0}]}]}

132. Do with as
you please

133. Here’s one I
prepared earlier..

135. ...

136. You don’t have to
set up a tool

137. But you have to
write reporting?

138. At least you
have load
testing?

139. It’s on the way..

140. http://support.blitz.io/discussions/questions/114-sharing-reports

142. Where to
from here?

143. Pull in data from
other places

144. new relic
Ganglia/collectd
Git commits

145. Github:
square/crossfilter

147. Don’t like Blitz?

148. SaaS
Load impact
Browser mob

149. Open Source
Tsung
JMeter

151. Start Using
something!

152. Low barrier of
entry is key

153. Integrate with your
existing dev tools

155. Questions?