This document provides an overview of performance testing and tools. It discusses why performance matters for business metrics and user experience. It then covers various tools for testing performance, including MultiMechanize for load testing applications, Bees with Machine Guns for distributed load testing across multiple machines, and Google PageSpeed for analyzing client-side performance. Best practices discussed include integrating performance testing into continuous integration/deployment processes and monitoring performance in production.

1. Performance Testing Crash Course
Dustin Whittle / dustinwhittle.com / @dustinwhittle

2. Agenda
• Why performance matters?
• Case studies on business impact of performance
• Tools of the trade
• MultiMechanize
• Bees with Machine Guns
• Google PageSpeed
• AppDynamics
• Best practices
• Server Side
• Client Side
• Integrating into your process
• Develop -> Test -> Deploy -> Repeat
• What was the performance impact the last release?
• What about the impact of that configuration change or package upgrade?

3. Dustin Whittle
• dustinwhittle.com
• @dustinwhittle
• San Francisco, California, USA
• Technologist, Traveler, Pilot, Skier, Diver, Sailor,
Golfer

4. What I have worked on
• Developer Evangelist @
• Consultant & Trainer @
• Developer Evangelist @

5. Why does
performance matter?

6. Microsoft found that Bing searches
that were 2 seconds slower
resulted in a 4.3% drop in revenue
per user

7. When Mozilla shaved 2.2 seconds
off their landing page, Firefox
downloads increased 15.4%

8. Making Barack Obama’s
website 60% faster increased
donation conversions by 14%

10. Performance directly
impacts the bottom line

12. Treat performance as a feature!

13. How
fast
is
fast
enough?
§ Performance
is
key
to
a
great
user
experience
-­‐ Under
100ms
is
perceived
as
reac1ng
instantaneously
-­‐ A
100ms
to
300ms
delay
is
percep1ble
-­‐ 1
second
is
about
the
limit
for
the
user's
flow
of
thought
to
stay
uninterrupted
-­‐ Users
expect
a
site
to
load
in
2
seconds
-­‐ A>er
3
seconds,
40%
will
abandon
your
site.
-­‐ 10
seconds
is
about
the
limit
for
keeping
the
user's
aAen1on
§ Modern
applicaGons
spend
more
Gme
in
the
browser
than
on
the
server-­‐side

14. HealthCare.gov

15. UpGme
is
criGcal
for
enterprises
and
consumers

16. An example spring app
to test performance

18. git clone https://github.com/
cloudfoundry-samples/hello-spring-cloud

19. mvn package

20. http://springone-demo.cfapps.io/

22. Tools of the trade for
performance testing

23. Understand your
baseline performance

24. Static
vs
Hello World
vs
Applications

25. Apache Bench

26. ab -c 1 -t 10 -k
http://springone-demo.cfapps.io/

27. Benchmarking springone-demo.cfapps.io (be patient)
Finished 187 requests
Server Software: Apache-Coyote/1.1
Server Hostname: springone-demo.cfapps.io
Server Port: 80
Document Path: /
Document Length: 5217 bytes
Concurrency Level: 1
Time taken for tests: 10.039 seconds
Complete requests: 187
Failed requests: 0
Keep-Alive requests: 187
Total transferred: 1021768 bytes
HTML transferred: 975579 bytes
Requests per second: 18.63 [#/sec] (mean)
Time per request: 53.687 [ms] (mean)

28. ab -c 10 -t 10 -k
http://springone-demo.cfapps.io/

29. Benchmarking springone-demo.cfapps.io (be patient)
Finished 659 requests
Server Software: Apache-Coyote/1.1
Server Hostname: springone-demo.cfapps.io
Server Port: 80
Document Path: /
Document Length: 5217 bytes
Concurrency Level: 10
Time taken for tests: 10.015 seconds
Complete requests: 659
Failed requests: 0
Keep-Alive requests: 659
Total transferred: 3600776 bytes
HTML transferred: 3438003 bytes
Requests per second: 65.80 [#/sec] (mean)
Time per request: 151.970 [ms] (mean)

30. Siege

31. siege -c 10 -b -t 10S http://springone-demo.cfapps.io/

32. ** Preparing 10 concurrent users for
battle.
The server is now under siege...
Lifting the server siege... done.
Transactions: 623 hits
Availability: 100.00 %
Elapsed time: 9.57 secs
Data transferred: 3.10 MB
Response time: 0.15 secs
Transaction rate: 65.10 trans/
sec
Throughput: 0.32 MB/sec
Concurrency: 9.91
Successful transactions: 623
Failed transactions: 0
Longest transaction: 0.30
Shortest transaction: 0.10

33. Crawl the entire app to
discover all urls

34. sproxy -o ./urls.txt

35. SPROXY v1.02 listening on port 9001
...appending HTTP requests to: ./urls.txt
..default connection timeout: 120 seconds

36. wget -r -l 0 -t 1 --spider -w 1 -e "http_proxy = http://
127.0.0.1:9001" "http://acmedemoapp.com/"
sort -u -o urls.txt urls.txt

37. http://acmedemoapp.com/
http://acmedemoapp.com/about
http://acmedemoapp.com/cart
http://acmedemoapp.com/currency/change/EUR
http://acmedemoapp.com/currency/change/GBP
http://acmedemoapp.com/currency/change/USD
http://acmedemoapp.com/login
http://acmedemoapp.com/register/
http://acmedemoapp.com/t/brand/bookmania
http://acmedemoapp.com/t/category/books
http://acmedemoapp.com/t/category/mugs
http://acmedemoapp.com/t/category/stickers
http://acmedemoapp.com/terms-of-service

38. Benchmark traffic across
all unique urls with siege

39. siege -v -c 50 -i -t 3M -f urls.txt -d 10

40. ** Preparing 10 concurrent users for
battle.
The server is now under siege...
Lifting the server siege... done.
Transactions: 623 hits
Availability: 100.00 %
Elapsed time: 9.57 secs
Data transferred: 3.10 MB
Response time: 0.15 secs
Transaction rate: 65.10 trans/
sec
Throughput: 0.32 MB/sec
Concurrency: 9.91
Successful transactions: 623
Failed transactions: 0
Longest transaction: 0.30
Shortest transaction: 0.10

42. Multi-Mechanize is an open
source framework for
performance and load testing

43. pip install multi-mechanize

44. multimech-newproject demo

45. import requests
class Transaction(object):
def run(self):
r = requests.get(‘http://acmedemoapp.com/)
r.raw.read()

46. import mechanize
import time
class Transaction(object):
def run(self):
br = mechanize.Browser()
br.set_handle_robots(False)
start_timer = time.time()
resp = br.open(‘http://acmedemoapp.com/)
resp.read()
latency = time.time() - start_timer
self.custom_timers['homepage'] = latency
start_timer = time.time()
resp = br.open(‘http://acmedemoapp.com/cart')
resp.read()
latency = time.time() - start_timer
self.custom_timers['cart'] = latency
assert (resp.code == 200)

47. [global]
run_time = 10
rampup = 5
results_ts_interval = 1
progress_bar = on
console_logging = off

48. multimech-run demo

50. What about when you need
more than one machine?

51. Who lives in the cloud?

52. Bees with Machine Guns

53. A utility for arming (creating)
many bees (micro EC2 instances)
to attack (load test)
targets (web applications)

54. pip install beeswithmachineguns

55. # ~/.boto
[Credentials]
aws_access_key_id=xxx
aws_secret_access_key=xxx
[Boto]
ec2_region_name = us-west-2
ec2_region_endpoint = ec2.us-west-2.amazonaws.com

56. bees up -s 2 -g default -z us-
west-2b -i ami-bc05898c -k
appdynamics-dustinwhittle-aws-
us-west-2 -l ec2-user

57. Connecting to the hive.
Attempting to call up 2 bees.
Waiting for bees to load their machine guns...
.
.
.

58. bees report

59. Read 2 bees from the roster.
Bee i-3828400c: running @ 54.212.22.176
Bee i-3928400d: running @ 50.112.6.191

60. bees attack -n 1000 -c 50 -u
http://springone-demo.cfapps.io/

61. Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 500 rounds, 25 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
…
Offensive complete.
Complete requests: 1000
Requests per second: 306.540000 [#/sec] (mean)
Time per request: 163.112000 [ms] (mean)
50% response time: 151.000000 [ms] (mean)
90% response time: 192.000000 [ms] (mean)
Mission Assessment: Target crushed bee offensive.
The swarm is awaiting new orders.

62. bees attack -n 100000 -c 1000 -u
http://springone-demo.cfapps.io/

63. Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 50000 rounds, 500 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
…
Offensive complete.
Complete requests: 100000
Requests per second: 502.420000 [#/sec] (mean)
Time per request: 360.114000 [ms] (mean)
50% response time: 451.000000 [ms] (mean)
90% response time: 402.000000 [ms] (mean)
Mission Assessment: Target crushed bee offensive.
The swarm is awaiting new orders.

64. Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 50000 rounds, 500 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Bee 0 is firing his machine gun. Bang bang!
Bee 0 lost sight of the target (connection timed out).
Bee 1 lost sight of the target (connection timed out).
Offensive complete.
Target timed out without fully responding to 2 bees.
No bees completed the mission. Apparently your bees are peace-loving hippies.
The swarm is awaiting new orders.

65. bees down

66. Read 2 bees from the roster.
Connecting to the hive.
Calling off the swarm.
Stood down 2 bees.

67. locust.io
https://github.com/locustio/locust

68. pip install locustio

72. There are many tools…
•Gatling.io
•Wrk
•Vegeta
•Tsung
•…

73. What about the client side?

74. In modern web applications
more latency comes from the
client-side than the server-side.

75. Google PageSpeed

79. curl "https://www.googleapis.com/
pagespeedonline/v1/runPagespeed?
url=http://dustinwhittle.com/&key=xxx"

80. WBench

81. gem install wbench

82. wbench http://dustinwhittle.com/

84. Automate client-side
performance testing with Grunt

85. Use Bower (for dependencies),
Grunt/Gulp (for automation),
and Yeoman (for bootstrapping)

88. How many people
understand exactly how fast
their site runs in production?

89. Track performance in
development and
production

90. Instrument everything = code,
databases, caches, queues, third
party services, and infrastructure.

92. Chef / Sensu

93. http://sensuapp.org/

94. Statsd + Graphite + Grafana

96. Track performance of
end users

99. webpagetest.org

103. SiteSpeed.io

110. LEARN TO HOW TO PROFILE
CODE FOR PERFORMANCE

111. Load testing services
from the cloud

115. Test for failures
• NetFlix Simian Army + Chaos Monkey
• What happens if you lose a caching layer?
• What happens if dependencies slow down?

116. Best Practices
• Treat performance as a feature
• Capacity plan and load test the server-side
• Optimize and performance test the client-side
• Understand your starting point
• Instrument everything
• Monitor performance in development and production
• Measure the difference of every change
• Automate performance testing in your build and deployment
process
• Understand how failures impact performance

117. Integrate automated performance
testing into continuous integration
for server-side and client-side

118. Understand the performance
implications of every deployment
and package upgrade

119. Monitor end user
experience from end to
end in production

120. Questions?

121. Find these slides on SpeakerDeck
https://speakerdeck.com/
dustinwhittle