Your SlideShare is downloading. ×
0
Load testing
Who is this dude?
Lindsay Holmwood
BULLETPROOF
I Run this thing
Have tried and failed manytimes to implement effective        load testing
A story
New project
Replacing a high- profile website (new infrastructure,    new application)
1000concurrent users
“We need 100%   uptime”
Project starts
Requirements   change
Deadlinesaren’t met
Performance testing?      Fuck it!  We’ll do it live!
Performance testing?“You have one day”
Problem:
Treated asWAterfall-esque       QA
It’s anon-functional  requirement
WHYDo we load test?
Capacityplanning
“Can the site deal   with load?”
Defectdetection
“Is this codegoing to cripple    the site?”
Optimisation
“What happens if we change this   Setting?”
Historical analysis
“Is the sitegetting slower?”
Cache warming
“Prepare theinfrastructure  for Failover”
WHYDo we load test?
A combination of  proactive &Reactive reasons
A combination of  proactive &Reactive reasons
It’s anon-functional  requirement
Why      does load testing           FAIL?(from a technical perspective)
Hownot theWHAT
TooL fetishism
Get bogged down   in details
how the testinginfrastructure works
Micro vsMacro
Business  Does not careabout your tools
Businesscares About:
“Can the site deal   with load?”
Recommendation:
Use tool withlowest barrier   TO entry
Blitz
Runs on AWS
Web form
Browser plugins   Firefox, Chrome
REST API
API ClientsRuby, Python, Java   Node.JS, Perl
CommonCommand Line
Excellentdocumentation   docs.blitz.io
Inexpensive    250 concurrent usersover 60 second period == free
SPRINTS   &RUSHES
Let’s dive in!
Sign up @ http://blitz.io/
$ gem install blitz$ blitz helpUsage: blitz <command> <options>                help - Display this help      account:about...
$ blitz curl:helpUsage: blitz curl <options> <url>--user-agent    -A   <string>      User-Agent to send to server--cookie ...
$ blitz curl --region singapore --pattern 1-250:20      --timeout 10000 --verbose http://example.org/
user growth / time$ blitz curl --region singapore --pattern 1-250:20      --timeout 10000 --verbose http://example.org/
user growth / time      for AWS lag$ blitz curl --region singapore --pattern 1-250:20      --timeout 10000 --verbose http:...
user growth / time         for AWS lag$ blitz curl --region singapore --pattern 1-250:20      --timeout 10000 --verbose ht...
--pattern 1-250:20
blitz calculates the     growth automatically--pattern 1-250:20
Sawtooth--pattern 1-250:20,1-250:20,1-250:20
TV AD / Twittersplosion --pattern 200-5000:30,5000-400:60
Integrate     with yourdevelopment cycle
VCS        CIDeploy automation
VCS        CIDeploy automation
gitpost-receive hook
GitHub web hooks
Doesn’t take into account test   failures
Is the code actually deployed?
VCS        CIDeploy automation
CRON JOB
JENKINS JOB
GREAT if you have   a CI system
Runs tests      butSeparate system
What if you’re makinginfrastructure changes during a load test?
VCS        CIDeploy automation
Fabric
Capistrano
Run tests when  you deploy
Use the same  toolchain
Testing lives IN the app.
What about triggering   load tests on  infrastructure      changes?
MVP
Just getsomething working!
Problem:
No REporting     :-(
THIS IS ITrushing from singapore...  Time   Users Response     Hits Timeouts   Errors   Hits/s Mbps  2.5s      31 -1.000s ...
SOL ON:
Optimisation
“What happens if we change this   Setting?”
Historical analysis
“Is the sitegetting slower?”
Showstopper?
...
$ gem install blitz# ...rushing from singapore...  Time   Users Response     Hits Timeouts   Errors   Hits/s Mbps  2.5s   ...
Can we duckpunch our way  to success?
#!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url       = "http://examp...
# ruby load_test.rbrushing from singapore...  Time   Users Response     Hits Timeouts   Errors   Hits/s Mbps  2.5s      31...
#!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url       = "http://examp...
#!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url       = "http://examp...
Blitz::Curl.parse(command)         returns    Blitz::Curl::Rush
#!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url       = "http://examp...
Blitz::Command::Curl#rush
initiates the test   + outputs the test resultsBlitz::Command::Curl#rush
#!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url       = "http://examp...
Produces muck
duck punch ourway to success
class Blitz::Curl::Rush  class Point    def to_json(*args)      {        timestamp => @timestamp,        duration => @dura...
# ...  class Step    def to_json(*args)      {        duration => @duration,        connect => @connect,        errors   =...
#!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url       = "http://examp...
{"start":1335694346,"finish":1335694367,"results":[{"timestamp":2.502626,"duration":1.13912,"total":1,"hits":1,"errors":0,...
Do with asyou please
Here’s one Iprepared earlier..
...
You don’t have to  set up a tool
But you have towrite reporting?
At least you have load  testing?
It’s on the way..
http://support.blitz.io/discussions/questions/114-sharing-reports
Where tofrom here?
Pull in data from  other places
new relicGanglia/collectd  Git commits
Github:square/crossfilter
Don’t like Blitz?
SaaS Load impactBrowser mob
Open Source    Tsung   JMeter
Start Using something!
Low barrier of entry is key
Integrate with your  existing dev tools
Questions?
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Load testing with Blitz
Upcoming SlideShare
Loading in...5
×

Load testing with Blitz

1,016

Published on

We all know that load testing is important, but it's all too common that it's left to the very end of a project and it's invariably the first thing that gets dropped when budgets and timeframes get cut. Furthermore, most of us don't know where or how to start implementing effective load tests, let alone how to analyse the results.

Lindsay Holmwood, Software Manager at Bulletproof Networks, will be talking about integrating performance testing into your application development + deploy cycle from the very beginning, using inexpensive and easy to use SaaS tools.

There will be a hands on demonstration of the Blitz load + performance testing tool, coupled with a brief dive into the Blitz API internals to retrieve and analyse advanced reporting information.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,016
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Load testing with Blitz"

  1. 1. Load testing
  2. 2. Who is this dude?
  3. 3. Lindsay Holmwood
  4. 4. BULLETPROOF
  5. 5. I Run this thing
  6. 6. Have tried and failed manytimes to implement effective load testing
  7. 7. A story
  8. 8. New project
  9. 9. Replacing a high- profile website (new infrastructure, new application)
  10. 10. 1000concurrent users
  11. 11. “We need 100% uptime”
  12. 12. Project starts
  13. 13. Requirements change
  14. 14. Deadlinesaren’t met
  15. 15. Performance testing? Fuck it! We’ll do it live!
  16. 16. Performance testing?“You have one day”
  17. 17. Problem:
  18. 18. Treated asWAterfall-esque QA
  19. 19. It’s anon-functional requirement
  20. 20. WHYDo we load test?
  21. 21. Capacityplanning
  22. 22. “Can the site deal with load?”
  23. 23. Defectdetection
  24. 24. “Is this codegoing to cripple the site?”
  25. 25. Optimisation
  26. 26. “What happens if we change this Setting?”
  27. 27. Historical analysis
  28. 28. “Is the sitegetting slower?”
  29. 29. Cache warming
  30. 30. “Prepare theinfrastructure for Failover”
  31. 31. WHYDo we load test?
  32. 32. A combination of proactive &Reactive reasons
  33. 33. A combination of proactive &Reactive reasons
  34. 34. It’s anon-functional requirement
  35. 35. Why does load testing FAIL?(from a technical perspective)
  36. 36. Hownot theWHAT
  37. 37. TooL fetishism
  38. 38. Get bogged down in details
  39. 39. how the testinginfrastructure works
  40. 40. Micro vsMacro
  41. 41. Business Does not careabout your tools
  42. 42. Businesscares About:
  43. 43. “Can the site deal with load?”
  44. 44. Recommendation:
  45. 45. Use tool withlowest barrier TO entry
  46. 46. Blitz
  47. 47. Runs on AWS
  48. 48. Web form
  49. 49. Browser plugins Firefox, Chrome
  50. 50. REST API
  51. 51. API ClientsRuby, Python, Java Node.JS, Perl
  52. 52. CommonCommand Line
  53. 53. Excellentdocumentation docs.blitz.io
  54. 54. Inexpensive 250 concurrent usersover 60 second period == free
  55. 55. SPRINTS &RUSHES
  56. 56. Let’s dive in!
  57. 57. Sign up @ http://blitz.io/
  58. 58. $ gem install blitz$ blitz helpUsage: blitz <command> <options> help - Display this help account:about - Show information about your account api:init - Validate and login with your API key couch:fuzz - Auto generate blitz tests from CouchDB curl - Run a sprint or a rush curl:help - Show help on sprint and rushing traceroute - Run traceroute remotely traceroute:help - Show help on traceroute version - Show the version of this Ruby gem$ blitz api:init# writes credentials to ~/.blitz/credentials
  59. 59. $ blitz curl:helpUsage: blitz curl <options> <url>--user-agent -A <string> User-Agent to send to server--cookie -b name=<string> Cookie to send to the server (multiple)--data -d <string> Data to send in a PUT or POST request--dump-header -D <file> Print the request/response headers--referer -e <string> Referer URL--help -h Help on command line options--header -H <string> Custom header to pass to server--pattern -p <s>-<e>:<d> Ramp from s to e concurrent requests in d secs--region -r <string> california|oregon|virginia|singapore|ireland|japan--status -s <number> Assert on the HTTP response status code--timeout -T <ms> Wait time for both connect and responses--user -u <user[:pass]> User and password for authentication--request -X <string> Request method to use (GET, HEAD, PUT, etc.)--variable -v <string> Define a variable to use--verbose -V Print the request/response headers--tlsv1 -1 Use TLSv1 (SSL)--sslv2 -2 Use SSLv2 (SSL)--sslv3 -3 Use SSLv3 (SSL)
  60. 60. $ blitz curl --region singapore --pattern 1-250:20 --timeout 10000 --verbose http://example.org/
  61. 61. user growth / time$ blitz curl --region singapore --pattern 1-250:20 --timeout 10000 --verbose http://example.org/
  62. 62. user growth / time for AWS lag$ blitz curl --region singapore --pattern 1-250:20 --timeout 10000 --verbose http://example.org/
  63. 63. user growth / time for AWS lag$ blitz curl --region singapore --pattern 1-250:20 --timeout 10000 --verbose http://example.org/rushing from singapore... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
  64. 64. --pattern 1-250:20
  65. 65. blitz calculates the growth automatically--pattern 1-250:20
  66. 66. Sawtooth--pattern 1-250:20,1-250:20,1-250:20
  67. 67. TV AD / Twittersplosion --pattern 200-5000:30,5000-400:60
  68. 68. Integrate with yourdevelopment cycle
  69. 69. VCS CIDeploy automation
  70. 70. VCS CIDeploy automation
  71. 71. gitpost-receive hook
  72. 72. GitHub web hooks
  73. 73. Doesn’t take into account test failures
  74. 74. Is the code actually deployed?
  75. 75. VCS CIDeploy automation
  76. 76. CRON JOB
  77. 77. JENKINS JOB
  78. 78. GREAT if you have a CI system
  79. 79. Runs tests butSeparate system
  80. 80. What if you’re makinginfrastructure changes during a load test?
  81. 81. VCS CIDeploy automation
  82. 82. Fabric
  83. 83. Capistrano
  84. 84. Run tests when you deploy
  85. 85. Use the same toolchain
  86. 86. Testing lives IN the app.
  87. 87. What about triggering load tests on infrastructure changes?
  88. 88. MVP
  89. 89. Just getsomething working!
  90. 90. Problem:
  91. 91. No REporting :-(
  92. 92. THIS IS ITrushing from singapore... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
  93. 93. SOL ON:
  94. 94. Optimisation
  95. 95. “What happens if we change this Setting?”
  96. 96. Historical analysis
  97. 97. “Is the sitegetting slower?”
  98. 98. Showstopper?
  99. 99. ...
  100. 100. $ gem install blitz# ...rushing from singapore... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
  101. 101. Can we duckpunch our way to success?
  102. 102. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
  103. 103. # ruby load_test.rbrushing from singapore... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
  104. 104. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
  105. 105. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
  106. 106. Blitz::Curl.parse(command) returns Blitz::Curl::Rush
  107. 107. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
  108. 108. Blitz::Command::Curl#rush
  109. 109. initiates the test + outputs the test resultsBlitz::Command::Curl#rush
  110. 110. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)puts job.result.timeline.to_json
  111. 111. Produces muck
  112. 112. duck punch ourway to success
  113. 113. class Blitz::Curl::Rush class Point def to_json(*args) { timestamp => @timestamp, duration => @duration, total => @total, hits => @hits, errors => @errors, timeouts => @timeouts, volume => @volume, txbytes => @txbytes, rxbytes => @rxbytes, steps => @steps, }.to_json end end # ...
  114. 114. # ... class Step def to_json(*args) { duration => @duration, connect => @connect, errors => @errors, timeouts => @timeouts, asserts => @asserts, }.to_json end endend
  115. 115. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)puts job.result.timeline.to_json
  116. 116. {"start":1335694346,"finish":1335694367,"results":[{"timestamp":2.502626,"duration":1.13912,"total":1,"hits":1,"errors":0,"timeouts":0,"volume":2,"txbytes":258.0,"rxbytes":17810.0,"steps":[{"duration":2.13912,"connect":0.260004,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":5.016102,"duration":0.640047,"total":5,"hits":5,"errors":0,"timeouts":0,"volume":4,"txbytes":1548.0,"rxbytes":90882.0,"steps":[{"duration":1.640047,"connect":0.158944,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":7.524594,"duration":0.639987,"total":10,"hits":10,"errors":0,"timeouts":0,"volume":5,"txbytes":3354.0,"rxbytes":182222.0,"steps":[{"duration":1.639987,"connect":0.158999,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":10.03388,"duration":0.63961,"total":20,"hits":20,"errors":0,"timeouts":0,"volume":7,"txbytes":5418.0,"rxbytes":364902.0,"steps":[{"duration":1.63961,"connect":0.158847,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":12.543211,"duration":0.640421,"total":31,"hits":31,"errors":0,"timeouts":0,"volume":9,"txbytes":9030.0,"rxbytes":565850.0,"steps":[{"duration":1.640422,"connect":0.158767,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":15.051705,"duration":0.639894,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":10,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":1.639894,"connect":0.158794,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":17.560136,"duration":-1.0,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":0,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":0.0,"connect":0.0,"errors":0,"timeouts":0,"asserts":0}]}]}
  117. 117. Do with asyou please
  118. 118. Here’s one Iprepared earlier..
  119. 119. ...
  120. 120. You don’t have to set up a tool
  121. 121. But you have towrite reporting?
  122. 122. At least you have load testing?
  123. 123. It’s on the way..
  124. 124. http://support.blitz.io/discussions/questions/114-sharing-reports
  125. 125. Where tofrom here?
  126. 126. Pull in data from other places
  127. 127. new relicGanglia/collectd Git commits
  128. 128. Github:square/crossfilter
  129. 129. Don’t like Blitz?
  130. 130. SaaS Load impactBrowser mob
  131. 131. Open Source Tsung JMeter
  132. 132. Start Using something!
  133. 133. Low barrier of entry is key
  134. 134. Integrate with your existing dev tools
  135. 135. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×