The software development lifecycle is a pretty complex process in many organizations. However, by using monitoring tools and methodologies, you can accelerate testing and release higher quality code―the cornerstone of rapid software delivery. These tools provide immediate feedback with actionable information so you can address problems as they are detected instead of waiting until the end of a testing cycle. Earlier detection, combined with tests that are a better representation of production workloads, are key to releasing better code, faster. Jim Hirschauer shows how to use monitoring software to make a major impact during development, test, and production. He describes typical use cases for server monitoring, log monitoring, and application performance monitoring. Learn about open source testing tools including Siege, Multi-Mechanize, and Bees with Machine Guns. Understand how to use each of these tools and more in development, test, and production as well as creating a feedback loop that drives continuous improvement.

1. T14
Test Techniques
10/16/2014 1:30:00 PM
Speed Up Testing with
Monitoring Tools
Presented by:
Jim Hirschauer
AppDynamics
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Jim Hirschauer
AppDynamics
Technology evangelist for AppDynamics, Jim Hirschauer has an extensive
background in highly available, business critical, large enterprise IT operations
environments. Jim has been interested in application performance testing and
monitoring since his days as a systems administrator in a retail bank. His passion
for performance analysis led him down a path to design, implement, and manage
the cloud computing monitoring architecture for a top-ten investment bank.
During his tenure at the bank, Jim created new processes and procedures that
reduced the time spent in the test and QA cycle while simultaneously increasing
overall code release quality and dramatically improving end user experience.

3. Awesome
Ops-ing!
!
Speed up Testing Using
Monitoring Tools"
Abstract"
• In many organizations, the software development lifecycle is a pretty
tedious process. However, by using the right tools and process, you can
accelerate development and release higher quality code than ever before.
Proper monitoring tools and methodologies are the cornerstone of rapid
software delivery.These tools provide immediate feedback with actionable
information so that problems can be addressed as they are detected
instead of waiting until the end of a testing cycle. Earlier detection of
problems combined with tests that are a better representation of
production workloads are the keys to releasing better code, faster. Jim
Hirschauer shows how to use monitoring software to make a major
impact during development, test, and production. Examine typical use cases
for server monitoring, log monitoring, and application performance
monitoring, and learn about open source testing tools including Siege,
Multi-Mechanize, and Bees with Machine Guns. Understand how to use
each of these tools in development, test, and production as well as creating
a feedback loop that drives continuous improvement."
10/1/14% 2%@HirschOnAPM%

4. Jim Hirschauer"
Technology Evangelist with AppDynamics"
"
@HirschOnAPM"
"
Aerospace Engineer, Guitarist, Dad,
Snowboarder, Beer and Scotch Drinker,
Mediocre Sailor, Better Pilot"
10/1/14% 3%@HirschOnAPM%
Rumored to have designed monitoring
architectures for top Financial Services
institutions."
Also Rumored to have introduced new
test and QA processes at these very
same institutions."
"
These allegations can neither be
confirmed nor denied."
10/1/14% 4%@HirschOnAPM%

5. 1. Ops bring valuable skills to the SDLC"
"
2. “Early Ops” helps production Ops"
"
3. Ops speed up the SDLC"
10/1/14% 5%@HirschOnAPM%
Why does
performance
matter?"
10/1/14% 6%@HirschOnAPM%

6. Performance!
=!
Revenue"
10/1/14% 7%@HirschOnAPM%
Microsoft found that Bing
searches that were 2 seconds
slower resulted in a 4.3% drop in
revenue per user"
10/1/14% 8%@HirschOnAPM%

7. When Mozilla shaved 2.2 seconds
off their landing page, Firefox
downloads increased 15.4%!(60 million more downloads)!
"
10/1/14% 9%@HirschOnAPM%
Making Barack
Obama s website 60%
faster increased donation
conversions by 14%"
10/1/14% 10%@HirschOnAPM%

8. Amazon and Walmart
increased revenue 1% for every
100ms of improvement"
10/1/14% 11%@HirschOnAPM%
What was Amazon’s
revenue last year?"
10/1/14% 12%@HirschOnAPM%

9. $75 BB!
!
X 1%!
!
= $750 MM"
10/1/14% 13%@HirschOnAPM%
Performance directly
impacts the bottom
line"
10/1/14% 14%@HirschOnAPM%

10. Getting woken up in the middle of the
night sucks!!!"
10/1/14% 15%@HirschOnAPM%
Brand/Reputation Damage"
10/1/14% @HirschOnAPM% 16%

11. The SDLCVisualized"
10/1/14% @HirschOnAPM% 17%
*Blatantly%stolen%from%my%friend%and%colleague%DusGn%WhiIle%
10/1/14% 18%@HirschOnAPM%

12. HOW DO YOU DEAL WITH TEST
WINDOW OBLITERATION?"
TESTING!
BEFORE!
TEST"
10/1/14% @HirschOnAPM% 20%

13. Basic testing earlier using FOSS"
10/1/14% @HirschOnAPM% 21%
Siege"
10/1/14% 22%@HirschOnAPM%

14. siege -c 10 -b -t 10S
http://dustinwhittle.com/%
10/1/14% 23%@HirschOnAPM%
** SIEGE 2.72
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege... done.
Transactions: 263 hits
Availability: 100.00 %
Elapsed time: 9.36 secs
Data transferred: 0.35 MB
Response time: 0.35 secs
Transaction rate: 28.10 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 9.82
Successful transactions: 263
Failed transactions: 0
Longest transaction: 0.54
Shortest transaction: 0.19%
10/1/14% 24%@HirschOnAPM%

15. Crawl the entire app
to discover all urls"
10/1/14% 25%@HirschOnAPM%
sproxy -o ./urls.txt %
10/1/14% 26%@HirschOnAPM%

16. SPROXY v1.02 listening on port 9001"
...appending HTTP requests to: ./urls.txt"
...default connection timeout: 120
seconds%
10/1/14% 27%@HirschOnAPM%
wget -r -o verbose.txt -l 0 -t 1 --spider -w 1 -
e robots=on"
-e "http_proxy = http://127.0.0.1:9001""
"http://dustinwhittle.com/"%
sort -u -o urls.txt
urls.txt%
10/1/14% 28%@HirschOnAPM%

17. Benchmark traffic
across all unique urls
with siege"
10/1/14% 29%@HirschOnAPM%
siege -v -c 50 -i -t 3M -f
urls.txt -d 10%
10/1/14% 30%@HirschOnAPM%

18. 10/1/14% 31%@HirschOnAPM%
Multi-Mechanize is an open
source framework for
performance and load
testing"
10/1/14% 32%@HirschOnAPM%

19. Multi-Mechanize Benefits"
"
Run many test scripts"
"
Control workload pattern (ramping)"
"
Visual output"
10/1/14% 34%@HirschOnAPM%

20. What about when
you need more than
one test machine?"
10/1/14% 35%@HirschOnAPM%
Bees with Machine Guns"
10/1/14% 36%@HirschOnAPM%

21. A utility for arming (creating) !
many bees (micro EC2
instances)!
to attack (load test)!
targets (web applications)"
10/1/14% 37%@HirschOnAPM%
pip install beeswithmachineguns"
10/1/14% 38%@HirschOnAPM%

22. # ~/.boto!
!
!
[Credentials]!
!
aws_access_key_id=xxx!
aws_secret_access_key=xxx!
!
!
[Boto]!
!
ec2_region_name = us-west-2!
ec2_region_endpoint = ec2.us-west-2.amazonaws.com"
10/1/14% 39%@HirschOnAPM%
bees up -s 2 -g default -z us-
west-2b -i ami-bc05898c -k
appdynamics-dustinwhittle-
aws-us-west-2 -l ec2-user"
10/1/14% 40%@HirschOnAPM%

23. Connecting to the hive.
Attempting to call up 2 bees.
Waiting for bees to load their machine
guns...
.
.
.
.
Bee i-3828400c is ready for the attack.
Bee i-3928400d is ready for the attack.
The swarm has assembled 2 bees."
10/1/14% 41%@HirschOnAPM%
bees report"
10/1/14% 42%@HirschOnAPM%

24. Read 2 bees from the roster.
Bee i-3828400c: running @ 54.212.22.176
Bee i-3928400d: running @ 50.112.6.191%
10/1/14% 43%@HirschOnAPM%
bees attack -n 1000 -c 50 -
u
http://dustinwhittle.com/"
10/1/14% 44%@HirschOnAPM%

25. Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 50000 rounds, 125 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Bee 0 is firing his machine gun. Bang bang!
Bee 1 is firing his machine gun. Bang bang!
Bee 1 is out of ammo.
Bee 0 is out of ammo.
Offensive complete.
Complete requests: 100000
Requests per second: 1067.110000 [#/sec] (mean)
Time per request: 278.348000 [ms] (mean)
50% response time: 47.500000 [ms] (mean)
90% response time: 114.000000 [ms] (mean)
Mission Assessment: Target crushed bee offensive.
The swarm is awaiting new orders.%
10/1/14% 45%@HirschOnAPM%
bees down"
10/1/14% 46%@HirschOnAPM%

26. BWMG - Disclaimer"
10/1/14% 47%@HirschOnAPM%
A SHORT BACKGROUND
STORY"
10/1/14% 48%@HirschOnAPM%

27. Creating new words using “ing”"
The present participle refers to things that are
still happening.To make the present participle,
the ending -ing is added to the infinitive ('I am
asking her a question')."
Source: http://www.oxforddictionaries.com/us/words/verb-tenses-adding-ed-and-ing"
10/1/14% 49%@HirschOnAPM%
Sportsing!!!"
10/1/14% 50%@HirschOnAPM%

28. AND NOW BACK TO OUR
REGULARLY SCHEDULED
PRESENTATION"
10/1/14% 51%@HirschOnAPM%

29. Ops Offer-ings "
• Distributed troubleshooting experience"
• Extensive infrastructure knowledge"
• Monitoring tool expertise"
• General IT Awesome-ing!!!"
10/1/14% 53%@HirschOnAPM%
Process ING"
10/1/14% 54%@HirschOnAPM%

30. Instrument-ing"
To equip with instruments
especially for measuring and
recording data."
Source: http://www.merriam-webster.com/dictionary/instrument"
"
10/1/14% 55%@HirschOnAPM%
Oracle%
Purchase%
Search%Flight%
Flight%Status%
Login%
Network%
CDN%
Browser(s)%
NaGve%
Mobile%
App%
Tomcat%Service%
Weblogic%Service% .NET%Service%
Cassandra%
ESB/MQ%
Apache%
JBoss%Service%
OS#
End#User# Applica1on#
Network#
Instrument Everything?"
10/1/14% @HirschOnAPM% 56%
Log#Files#
Database#

31. 10/1/14% 57%@HirschOnAPM%
percepGon%
aIenGon%
observaGon%
NoGce[ing%
Architecture problems?"
10/1/14% @HirschOnAPM% 58% 58%

32. Resource contention?"
10/1/14% @HirschOnAPM% 59% 59%
Infrastructure problems?"

33. Network problems?"
Code problems?"
10/1/14% @HirschOnAPM% 62% 62%

34. Poor performing queries?"
10/1/14% @HirschOnAPM% 63% 63%
Grok-ing"
Grok means to understand so thoroughly that
the observer becomes a part of the observed—
to merge, blend, intermarry, lose identity in
group experience. It means almost everything
that we mean by religion, philosophy, and science
—and it means as little to us (because of our
Earthling assumptions) as color means to a blind
man."
Source: http://en.wikipedia.org/wiki/Grok"
10/1/14% @HirschOnAPM% 64%

35. Source:%hIp://ihumanable.com/blog/2010/01/11/grok/%
10/1/14% 65%@HirschOnAPM%
10/1/14% 66%@HirschOnAPM%

36. Know your applications!"
10/1/14% 67%@HirschOnAPM%
Log Analytics Tools"
FOSS:"
• Logstash"
• Greylog2"
• Elastic Search + Flume"
"
Paid:"
• Splunk"
• Loggly"
• SumoLogic"
• AppDynamics (beta)"
10/1/14% 68%@HirschOnAPM%

37. Infrastructure Monitoring Tools"
FOSS:"
• Nagios"
• Groundworks"
• Ganglia"
Paid:"
• Zenoss"
• Groundworks"
• HP Patrol"
10/1/14% 69%@HirschOnAPM%
Network Monitoring Tools"
FOSS:"
• OpenNMS"
• OP5"
• NetXMS"
"
Paid:"
• PRTG"
• ExtraHop"
• Boundary"
• Solarwinds"
10/1/14% 70%@HirschOnAPM%

38. Application Monitoring Tools"
FOSS:"
• Helios"
• Hyperic"
• Jconsole"
"
Paid:"
• AppDynamics (Free LiteVersion)"
• CA Wily"
• IBM SmartCloud APM"
10/1/14% 71%@HirschOnAPM%
THE RESULT"
10/1/14% 72%@HirschOnAPM%

39. • 2-5X faster testing"
• Fewer Day 1 – Week 1 issues"
• Better production performance"
• Prioritization of development effort"
• Continuous improvement cycle"
10/1/14% 73%@HirschOnAPM%
When you get back to work (dev)…"
• Contact Ops team and ask about tools"
• Contact QA and ask about production
workload details"
• Invite monitoring ops to participate in next
load test"
• Walk monitoring ops through application
functionality before next test"
• Actively participate in next load test"

40. When you get back to work (ops)…"
• Find a problematic application"
• Determine if production workload details are
available"
• Contact Dev team and discuss assisting during
next load test"
• Contact QA and ask about production workload
details"
• Enable and/or access appropriate monitoring in
load test environment"
• Actively participate in next load test"
If you do these things, you’ll be on
the path to…"
10/1/14% 76%@HirschOnAPM%

41. ThankYou"
Shameless Plug: Free Monitoring"
http://www.appdynamics.com"
10/1/14% 77%@HirschOnAPM%