Load testing with Blitz

1,404 views

Published on

We all know that load testing is important, but it's all too common that it's left to the very end of a project and it's invariably the first thing that gets dropped when budgets and timeframes get cut. Furthermore, most of us don't know where or how to start implementing effective load tests, let alone how to analyse the results.

Lindsay Holmwood, Software Manager at Bulletproof Networks, will be talking about integrating performance testing into your application development + deploy cycle from the very beginning, using inexpensive and easy to use SaaS tools.

There will be a hands on demonstration of the Blitz load + performance testing tool, coupled with a brief dive into the Blitz API internals to retrieve and analyse advanced reporting information.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,404
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
18
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Load testing with Blitz

  1. 1. Load testing
  2. 2. Who is this dude?
  3. 3. Lindsay Holmwood
  4. 4. BULLETPROOF
  5. 5. I Run this thing
  6. 6. Have tried and failed manytimes to implement effective load testing
  7. 7. A story
  8. 8. New project
  9. 9. Replacing a high- profile website (new infrastructure, new application)
  10. 10. 1000concurrent users
  11. 11. “We need 100% uptime”
  12. 12. Project starts
  13. 13. Requirements change
  14. 14. Deadlinesaren’t met
  15. 15. Performance testing? Fuck it! We’ll do it live!
  16. 16. Performance testing?“You have one day”
  17. 17. Problem:
  18. 18. Treated asWAterfall-esque QA
  19. 19. It’s anon-functional requirement
  20. 20. WHYDo we load test?
  21. 21. Capacityplanning
  22. 22. “Can the site deal with load?”
  23. 23. Defectdetection
  24. 24. “Is this codegoing to cripple the site?”
  25. 25. Optimisation
  26. 26. “What happens if we change this Setting?”
  27. 27. Historical analysis
  28. 28. “Is the sitegetting slower?”
  29. 29. Cache warming
  30. 30. “Prepare theinfrastructure for Failover”
  31. 31. WHYDo we load test?
  32. 32. A combination of proactive &Reactive reasons
  33. 33. A combination of proactive &Reactive reasons
  34. 34. It’s anon-functional requirement
  35. 35. Why does load testing FAIL?(from a technical perspective)
  36. 36. Hownot theWHAT
  37. 37. TooL fetishism
  38. 38. Get bogged down in details
  39. 39. how the testinginfrastructure works
  40. 40. Micro vsMacro
  41. 41. Business Does not careabout your tools
  42. 42. Businesscares About:
  43. 43. “Can the site deal with load?”
  44. 44. Recommendation:
  45. 45. Use tool withlowest barrier TO entry
  46. 46. Blitz
  47. 47. Runs on AWS
  48. 48. Web form
  49. 49. Browser plugins Firefox, Chrome
  50. 50. REST API
  51. 51. API ClientsRuby, Python, Java Node.JS, Perl
  52. 52. CommonCommand Line
  53. 53. Excellentdocumentation docs.blitz.io
  54. 54. Inexpensive 250 concurrent usersover 60 second period == free
  55. 55. SPRINTS &RUSHES
  56. 56. Let’s dive in!
  57. 57. Sign up @ http://blitz.io/
  58. 58. $ gem install blitz$ blitz helpUsage: blitz <command> <options> help - Display this help account:about - Show information about your account api:init - Validate and login with your API key couch:fuzz - Auto generate blitz tests from CouchDB curl - Run a sprint or a rush curl:help - Show help on sprint and rushing traceroute - Run traceroute remotely traceroute:help - Show help on traceroute version - Show the version of this Ruby gem$ blitz api:init# writes credentials to ~/.blitz/credentials
  59. 59. $ blitz curl:helpUsage: blitz curl <options> <url>--user-agent -A <string> User-Agent to send to server--cookie -b name=<string> Cookie to send to the server (multiple)--data -d <string> Data to send in a PUT or POST request--dump-header -D <file> Print the request/response headers--referer -e <string> Referer URL--help -h Help on command line options--header -H <string> Custom header to pass to server--pattern -p <s>-<e>:<d> Ramp from s to e concurrent requests in d secs--region -r <string> california|oregon|virginia|singapore|ireland|japan--status -s <number> Assert on the HTTP response status code--timeout -T <ms> Wait time for both connect and responses--user -u <user[:pass]> User and password for authentication--request -X <string> Request method to use (GET, HEAD, PUT, etc.)--variable -v <string> Define a variable to use--verbose -V Print the request/response headers--tlsv1 -1 Use TLSv1 (SSL)--sslv2 -2 Use SSLv2 (SSL)--sslv3 -3 Use SSLv3 (SSL)
  60. 60. $ blitz curl --region singapore --pattern 1-250:20 --timeout 10000 --verbose http://example.org/
  61. 61. user growth / time$ blitz curl --region singapore --pattern 1-250:20 --timeout 10000 --verbose http://example.org/
  62. 62. user growth / time for AWS lag$ blitz curl --region singapore --pattern 1-250:20 --timeout 10000 --verbose http://example.org/
  63. 63. user growth / time for AWS lag$ blitz curl --region singapore --pattern 1-250:20 --timeout 10000 --verbose http://example.org/rushing from singapore... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
  64. 64. --pattern 1-250:20
  65. 65. blitz calculates the growth automatically--pattern 1-250:20
  66. 66. Sawtooth--pattern 1-250:20,1-250:20,1-250:20
  67. 67. TV AD / Twittersplosion --pattern 200-5000:30,5000-400:60
  68. 68. Integrate with yourdevelopment cycle
  69. 69. VCS CIDeploy automation
  70. 70. VCS CIDeploy automation
  71. 71. gitpost-receive hook
  72. 72. GitHub web hooks
  73. 73. Doesn’t take into account test failures
  74. 74. Is the code actually deployed?
  75. 75. VCS CIDeploy automation
  76. 76. CRON JOB
  77. 77. JENKINS JOB
  78. 78. GREAT if you have a CI system
  79. 79. Runs tests butSeparate system
  80. 80. What if you’re makinginfrastructure changes during a load test?
  81. 81. VCS CIDeploy automation
  82. 82. Fabric
  83. 83. Capistrano
  84. 84. Run tests when you deploy
  85. 85. Use the same toolchain
  86. 86. Testing lives IN the app.
  87. 87. What about triggering load tests on infrastructure changes?
  88. 88. MVP
  89. 89. Just getsomething working!
  90. 90. Problem:
  91. 91. No REporting :-(
  92. 92. THIS IS ITrushing from singapore... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
  93. 93. SOL ON:
  94. 94. Optimisation
  95. 95. “What happens if we change this Setting?”
  96. 96. Historical analysis
  97. 97. “Is the sitegetting slower?”
  98. 98. Showstopper?
  99. 99. ...
  100. 100. $ gem install blitz# ...rushing from singapore... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
  101. 101. Can we duckpunch our way to success?
  102. 102. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
  103. 103. # ruby load_test.rbrushing from singapore... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
  104. 104. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
  105. 105. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
  106. 106. Blitz::Curl.parse(command) returns Blitz::Curl::Rush
  107. 107. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
  108. 108. Blitz::Command::Curl#rush
  109. 109. initiates the test + outputs the test resultsBlitz::Command::Curl#rush
  110. 110. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)puts job.result.timeline.to_json
  111. 111. Produces muck
  112. 112. duck punch ourway to success
  113. 113. class Blitz::Curl::Rush class Point def to_json(*args) { timestamp => @timestamp, duration => @duration, total => @total, hits => @hits, errors => @errors, timeouts => @timeouts, volume => @volume, txbytes => @txbytes, rxbytes => @rxbytes, steps => @steps, }.to_json end end # ...
  114. 114. # ... class Step def to_json(*args) { duration => @duration, connect => @connect, errors => @errors, timeouts => @timeouts, asserts => @asserts, }.to_json end endend
  115. 115. #!/usr/bin/env ruby# load_test.rbrequire blitzarguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)puts job.result.timeline.to_json
  116. 116. {"start":1335694346,"finish":1335694367,"results":[{"timestamp":2.502626,"duration":1.13912,"total":1,"hits":1,"errors":0,"timeouts":0,"volume":2,"txbytes":258.0,"rxbytes":17810.0,"steps":[{"duration":2.13912,"connect":0.260004,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":5.016102,"duration":0.640047,"total":5,"hits":5,"errors":0,"timeouts":0,"volume":4,"txbytes":1548.0,"rxbytes":90882.0,"steps":[{"duration":1.640047,"connect":0.158944,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":7.524594,"duration":0.639987,"total":10,"hits":10,"errors":0,"timeouts":0,"volume":5,"txbytes":3354.0,"rxbytes":182222.0,"steps":[{"duration":1.639987,"connect":0.158999,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":10.03388,"duration":0.63961,"total":20,"hits":20,"errors":0,"timeouts":0,"volume":7,"txbytes":5418.0,"rxbytes":364902.0,"steps":[{"duration":1.63961,"connect":0.158847,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":12.543211,"duration":0.640421,"total":31,"hits":31,"errors":0,"timeouts":0,"volume":9,"txbytes":9030.0,"rxbytes":565850.0,"steps":[{"duration":1.640422,"connect":0.158767,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":15.051705,"duration":0.639894,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":10,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":1.639894,"connect":0.158794,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":17.560136,"duration":-1.0,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":0,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":0.0,"connect":0.0,"errors":0,"timeouts":0,"asserts":0}]}]}
  117. 117. Do with asyou please
  118. 118. Here’s one Iprepared earlier..
  119. 119. ...
  120. 120. You don’t have to set up a tool
  121. 121. But you have towrite reporting?
  122. 122. At least you have load testing?
  123. 123. It’s on the way..
  124. 124. http://support.blitz.io/discussions/questions/114-sharing-reports
  125. 125. Where tofrom here?
  126. 126. Pull in data from other places
  127. 127. new relicGanglia/collectd Git commits
  128. 128. Github:square/crossfilter
  129. 129. Don’t like Blitz?
  130. 130. SaaS Load impactBrowser mob
  131. 131. Open Source Tsung JMeter
  132. 132. Start Using something!
  133. 133. Low barrier of entry is key
  134. 134. Integrate with your existing dev tools
  135. 135. Questions?

×