Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Performance testing crash course (Dustin Whittle)

1,828 views

Published on

Session slides from Future Insights Live, Vegas 2015:
https://futureinsightslive.com/las-vegas-2015/

After attending the Performance Testing Crash Course you will walk away with an understanding of: How to capacity plan and load test server-side performance, instrumenting and understanding client-side performance. Automate performance testing for continuous integration and how to understand the business impact of performance.

Published in: Technology
  • Be the first to comment

Performance testing crash course (Dustin Whittle)

  1. 1. Performance Testing Crash Course Dustin Whittle / dustinwhittle.com / @dustinwhittle
  2. 2. Agenda • Why performance matters? • Case studies on business impact of performance • Tools of the trade • MultiMechanize • Bees with Machine Guns • Google PageSpeed • AppDynamics • Best practices • Server Side • Client Side • Integrating into your process • Develop -> Test -> Deploy -> Repeat • What was the performance impact the last release? • What about the impact of that configuration change or package upgrade?
  3. 3. Dustin Whittle • dustinwhittle.com • @dustinwhittle • San Francisco, California, USA • Technologist, Traveler, Pilot, Skier, Diver, Sailor, Golfer
  4. 4. What I have worked on • Developer Evangelist @ • Consultant & Trainer @ • Developer Evangelist @
  5. 5. Why does performance matter?
  6. 6. Microsoft found that Bing searches that were 2 seconds slower resulted in a 4.3% drop in revenue per user
  7. 7. When Mozilla shaved 2.2 seconds off their landing page, Firefox downloads increased 15.4%
  8. 8. Making Barack Obama’s website 60% faster increased donation conversions by 14%
  9. 9. Performance directly impacts the bottom line
  10. 10. Treat performance as a feature!
  11. 11. How  fast  is  fast  enough? § Performance  is  key  to  a  great  user  experience     -­‐ Under  100ms  is  perceived  as  reac1ng  instantaneously   -­‐ A  100ms  to  300ms  delay  is  percep1ble   -­‐ 1  second  is  about  the  limit  for  the  user's  flow  of  thought  to  stay   uninterrupted   -­‐ Users  expect  a  site  to  load  in  2  seconds   -­‐ A>er  3  seconds,  40%  will  abandon  your  site.   -­‐ 10  seconds  is  about  the  limit  for  keeping  the  user's  aAen1on   § Modern  applicaGons  spend  more  Gme  in  the  browser  than  on   the  server-­‐side
  12. 12. HealthCare.gov
  13. 13. UpGme  is  criGcal  for  enterprises  and  consumers
  14. 14. An example spring app to test performance
  15. 15. git clone https://github.com/ cloudfoundry-samples/hello-spring-cloud
  16. 16. mvn package
  17. 17. http://springone-demo.cfapps.io/
  18. 18. Tools of the trade for performance testing
  19. 19. Understand your baseline performance
  20. 20. Static vs Hello World vs Applications
  21. 21. Apache Bench
  22. 22. ab -c 1 -t 10 -k http://springone-demo.cfapps.io/
  23. 23. Benchmarking springone-demo.cfapps.io (be patient) Finished 187 requests Server Software: Apache-Coyote/1.1 Server Hostname: springone-demo.cfapps.io Server Port: 80 Document Path: / Document Length: 5217 bytes Concurrency Level: 1 Time taken for tests: 10.039 seconds Complete requests: 187 Failed requests: 0 Keep-Alive requests: 187 Total transferred: 1021768 bytes HTML transferred: 975579 bytes Requests per second: 18.63 [#/sec] (mean) Time per request: 53.687 [ms] (mean)
  24. 24. ab -c 10 -t 10 -k http://springone-demo.cfapps.io/
  25. 25. Benchmarking springone-demo.cfapps.io (be patient) Finished 659 requests Server Software: Apache-Coyote/1.1 Server Hostname: springone-demo.cfapps.io Server Port: 80 Document Path: / Document Length: 5217 bytes Concurrency Level: 10 Time taken for tests: 10.015 seconds Complete requests: 659 Failed requests: 0 Keep-Alive requests: 659 Total transferred: 3600776 bytes HTML transferred: 3438003 bytes Requests per second: 65.80 [#/sec] (mean) Time per request: 151.970 [ms] (mean)
  26. 26. Siege
  27. 27. siege -c 10 -b -t 10S http://springone-demo.cfapps.io/
  28. 28. ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 623 hits Availability: 100.00 % Elapsed time: 9.57 secs Data transferred: 3.10 MB Response time: 0.15 secs Transaction rate: 65.10 trans/ sec Throughput: 0.32 MB/sec Concurrency: 9.91 Successful transactions: 623 Failed transactions: 0 Longest transaction: 0.30 Shortest transaction: 0.10
  29. 29. Crawl the entire app to discover all urls
  30. 30. sproxy -o ./urls.txt
  31. 31. SPROXY v1.02 listening on port 9001 ...appending HTTP requests to: ./urls.txt ..default connection timeout: 120 seconds
  32. 32. wget -r -l 0 -t 1 --spider -w 1 -e "http_proxy = http:// 127.0.0.1:9001" "http://acmedemoapp.com/" sort -u -o urls.txt urls.txt
  33. 33. http://acmedemoapp.com/ http://acmedemoapp.com/about http://acmedemoapp.com/cart http://acmedemoapp.com/currency/change/EUR http://acmedemoapp.com/currency/change/GBP http://acmedemoapp.com/currency/change/USD http://acmedemoapp.com/login http://acmedemoapp.com/register/ http://acmedemoapp.com/t/brand/bookmania http://acmedemoapp.com/t/category/books http://acmedemoapp.com/t/category/mugs http://acmedemoapp.com/t/category/stickers http://acmedemoapp.com/terms-of-service
  34. 34. Benchmark traffic across all unique urls with siege
  35. 35. siege -v -c 50 -i -t 3M -f urls.txt -d 10
  36. 36. ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 623 hits Availability: 100.00 % Elapsed time: 9.57 secs Data transferred: 3.10 MB Response time: 0.15 secs Transaction rate: 65.10 trans/ sec Throughput: 0.32 MB/sec Concurrency: 9.91 Successful transactions: 623 Failed transactions: 0 Longest transaction: 0.30 Shortest transaction: 0.10
  37. 37. Multi-Mechanize is an open source framework for performance and load testing
  38. 38. pip install multi-mechanize
  39. 39. multimech-newproject demo
  40. 40. import requests class Transaction(object): def run(self): r = requests.get(‘http://acmedemoapp.com/) r.raw.read()
  41. 41. import mechanize import time class Transaction(object): def run(self): br = mechanize.Browser() br.set_handle_robots(False) start_timer = time.time() resp = br.open(‘http://acmedemoapp.com/) resp.read() latency = time.time() - start_timer self.custom_timers['homepage'] = latency start_timer = time.time() resp = br.open(‘http://acmedemoapp.com/cart') resp.read() latency = time.time() - start_timer self.custom_timers['cart'] = latency assert (resp.code == 200)
  42. 42. [global] run_time = 10 rampup = 5 results_ts_interval = 1 progress_bar = on console_logging = off
  43. 43. multimech-run demo
  44. 44. What about when you need more than one machine?
  45. 45. Who lives in the cloud?
  46. 46. Bees with Machine Guns
  47. 47. A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications)
  48. 48. pip install beeswithmachineguns
  49. 49. # ~/.boto [Credentials] aws_access_key_id=xxx aws_secret_access_key=xxx [Boto] ec2_region_name = us-west-2 ec2_region_endpoint = ec2.us-west-2.amazonaws.com
  50. 50. bees up -s 2 -g default -z us- west-2b -i ami-bc05898c -k appdynamics-dustinwhittle-aws- us-west-2 -l ec2-user
  51. 51. Connecting to the hive. Attempting to call up 2 bees. Waiting for bees to load their machine guns... . . .
  52. 52. bees report
  53. 53. Read 2 bees from the roster. Bee i-3828400c: running @ 54.212.22.176 Bee i-3928400d: running @ 50.112.6.191
  54. 54. bees attack -n 1000 -c 50 -u http://springone-demo.cfapps.io/
  55. 55. Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will fire 500 rounds, 25 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. … Offensive complete. Complete requests: 1000 Requests per second: 306.540000 [#/sec] (mean) Time per request: 163.112000 [ms] (mean) 50% response time: 151.000000 [ms] (mean) 90% response time: 192.000000 [ms] (mean) Mission Assessment: Target crushed bee offensive. The swarm is awaiting new orders.
  56. 56. bees attack -n 100000 -c 1000 -u http://springone-demo.cfapps.io/
  57. 57. Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will fire 50000 rounds, 500 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. … Offensive complete. Complete requests: 100000 Requests per second: 502.420000 [#/sec] (mean) Time per request: 360.114000 [ms] (mean) 50% response time: 451.000000 [ms] (mean) 90% response time: 402.000000 [ms] (mean) Mission Assessment: Target crushed bee offensive. The swarm is awaiting new orders.
  58. 58. Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will fire 50000 rounds, 500 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. Bee 0 is joining the swarm. Bee 1 is joining the swarm. Bee 0 is firing his machine gun. Bang bang! Bee 0 lost sight of the target (connection timed out). Bee 1 lost sight of the target (connection timed out). Offensive complete. Target timed out without fully responding to 2 bees. No bees completed the mission. Apparently your bees are peace-loving hippies. The swarm is awaiting new orders.
  59. 59. bees down
  60. 60. Read 2 bees from the roster. Connecting to the hive. Calling off the swarm. Stood down 2 bees.
  61. 61. locust.io https://github.com/locustio/locust
  62. 62. pip install locustio
  63. 63. There are many tools… •Gatling.io   •Wrk   •Vegeta   •Tsung   •…
  64. 64. What about the client side?
  65. 65. In modern web applications more latency comes from the client-side than the server-side.
  66. 66. Google PageSpeed
  67. 67. curl "https://www.googleapis.com/ pagespeedonline/v1/runPagespeed? url=http://dustinwhittle.com/&key=xxx"
  68. 68. WBench
  69. 69. gem install wbench
  70. 70. wbench http://dustinwhittle.com/
  71. 71. Automate client-side performance testing with Grunt
  72. 72. Use Bower (for dependencies), Grunt/Gulp (for automation), and Yeoman (for bootstrapping)
  73. 73. How many people understand exactly how fast their site runs in production?
  74. 74. Track performance in development and production
  75. 75. Instrument everything = code, databases, caches, queues, third party services, and infrastructure.
  76. 76. Chef / Sensu
  77. 77. http://sensuapp.org/
  78. 78. Statsd + Graphite + Grafana
  79. 79. Track performance of end users
  80. 80. webpagetest.org
  81. 81. SiteSpeed.io
  82. 82. LEARN TO HOW TO PROFILE CODE FOR PERFORMANCE
  83. 83. Load testing services from the cloud
  84. 84. Test for failures • NetFlix Simian Army + Chaos Monkey
 • What happens if you lose a caching layer? • What happens if dependencies slow down?
  85. 85. Best Practices • Treat performance as a feature • Capacity plan and load test the server-side • Optimize and performance test the client-side • Understand your starting point • Instrument everything • Monitor performance in development and production • Measure the difference of every change • Automate performance testing in your build and deployment process • Understand how failures impact performance
  86. 86. Integrate automated performance testing into continuous integration for server-side and client-side
  87. 87. Understand the performance implications of every deployment and package upgrade
  88. 88. Monitor end user experience from end to end in production
  89. 89. Questions?
  90. 90. Find these slides on SpeakerDeck https://speakerdeck.com/ dustinwhittle

×