The benefits of using an APM solution while performance testing


Published on

The benefits of using an APM solution while performance testing or "why load testing without APM is like Corona without the lime...".

The deck covers a brief overview of APM, the market & major players, and 4 key benefits from using APM tools during your performance testing cycle.

Published in: Technology
1 Comment
  • The concept is great. I've seen in dozens of situations is that it's extremely difficult to generate a workload on the system(s) under test that is sufficiently similar to the production environment that problematic application behaviors are revealed in the test environment. I have had many situations where everything worked fine in test, no problems found even with instrumented applications, but the apps had problems in production, so application performance analysis had to be done in production -- not desirable, to say the least. In other cases, the performance and functional testing environments were shared, so the test environment was in constant flux and we couldn't get a fix on a problem. Any advice on how to make sure your test environment accurately replicates your production environment?
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Keeps bugs out of your applicationClear Glass bottle – stops your application “going off” once it’s deployed to productionIt’s “disinfects your performance bottlenecks” (just like lime does to the top of the bottle) It’s a cynical marketing ploy to get you buy more stuff…… Unlike almost every other beer in the U.S. market, Corona is bottled in clear glass rather than brown or green. Those who know beer recognize that light is an enemy of the brew, turning beverages exposed to it "skunky." The lime, therefore, masks the altered taste resultant from Coronas' having been exposed to light by virtue of their packaging.Corona is a mild-tasting beverage, with the inserted lime adding its only discernable note of flavor.The provision of a lime dates to the days when metal caps sometimes left circlets of rust on the rims of beer bottles. The fruit slice was used to wipe away rust stains the brew's drinker would otherwise have been putting his lips to.Lime (or lemon) is said to work to keep flies away. In an expansion of that theory, prior to the lime slice, fly spray used to combat the flying hordes adversely affected the taste of the beer.Some bright spark who works for the brewery came up with the idea of festooning bottles of Corona with wedges of lime, both in an effort to create a more visually enticing image and to provide what might otherwise be regarded as a somewhat uninspired beer with a hint of cachet, and possibly even an intriguing (if unstated) backstory. People like both ritual and mystery, after all, and the lime provides both.On a bet with a colleague about whether a single person could start a nationwide trend, a bartender came up with the notion of shoving limes into Coronas.Read more at
  •  Senge P (1990) The Fifth Discipline – The art and practice of the learning organisation, Century Business, ISBN 0-7126-56871. Open University Website, introduction to systems thinking
  • The benefits of using an APM solution while performance testing

    1. 1. DevOpsGuys The benefits of using an APM solution while performance testing Why Perf testing without APM is like Corona without the Lime… @DevOpsGuys DevOpsGuys 1
    2. 2. @DevOpsGuys DevOpsGuys 2
    3. 3. Perf Testing & APM in 30mins • • • • • Why Performance Test? What is APM anyway? Benefits of APM when Perf Testing With some case studies along the way! Q&A @DevOpsGuys DevOpsGuys 3
    4. 4. Why Performance Test? 1. Ensure that we meet our customer’s expectations of website response time – So that they convert and we make £££ 2. Ensure that we have sufficient computing capacity to meet #1 under load (load testing) 3. Understand how the system reacts when things go wrong (stress testing) @DevOpsGuys DevOpsGuys 4
    5. 5. @DevOpsGuys DevOpsGuys 5
    6. 6. APM Defined “In the fields of information technology and systems management, application performance management (APM), is the monitoring and managing of performance and availability of software applications. APM strives to detect and diagnose application performance problems to maintain an expected level of service. APM is the translation of IT metrics into business meaning (i.e. value).” @DevOpsGuys DevOpsGuys 6
    7. 7. “5 Dimensions of APM*” 1. End user experience monitoring - (Active and passive) 2. Application runtime architecture discovery and modeling 3. User-defined transaction profiling (also called business transaction management) 4. Application component monitoring 5. Reporting & Application data analytics *According to Gartner  @DevOpsGuys DevOpsGuys 7
    8. 8. Gartner APM Magic Quadrant 2013 2012 2011 IBM Riverbed CA Compuware Oracle Quest Opnet Microsoft BM Dell SL C New Relic HP AppDynamics OpTier Inetco AppNeta NetScout Precise Progress Managed Engine eG Innovations Network Instruments Idera Visual networks ASG Correlsense Nastel Coradient Aternity Knoa Info vista OpTier Arctures Technologies @DevOpsGuys DevOpsGuys
    9. 9. Never leave home without it… BENEFITS OF LOAD TESTING WITH APM TOOLS @DevOpsGuys DevOpsGuys 9
    10. 10. #1 See the Big Picture (aka Systems Thinking) @DevOpsGuys DevOpsGuys 10
    11. 11. Systems Thinking 101 “Systems thinking is a framework for seeing interrelationships rather than things, for seeing patterns rather than static snapshots. It is a set of general principles spanning fields as diverse as physical and social sciences, engineering and management.” Peter Senge @DevOpsGuys DevOpsGuys 11
    12. 12. Command & Control vs Systems Thinking @DevOpsGuys DevOpsGuys 12
    13. 13. “any improvement not made at the constraint is an illusion” CASE STUDY #1 2 WEEKS WASTED… @DevOpsGuys DevOpsGuys 13
    14. 14. The Scenario E-commerce customer New website High data-start times on page load Poor load testing results Initial conversation with client 2 week delay while hosting provider scaled out the hardware… • Problem still not fixed… • • • • • • @DevOpsGuys DevOpsGuys 14
    15. 15. Waterfall Diagram of page load times What’s the root cause of the ~1sec Data Start? @DevOpsGuys DevOpsGuys 15
    16. 16. @DevOpsGuys DevOpsGuys 16
    17. 17. #2 Drill down to the Detail (Code-level) @DevOpsGuys DevOpsGuys 17
    18. 18. Find the slow code! @DevOpsGuys DevOpsGuys 18
    19. 19. #3 Faster Iteration = Better Value @DevOpsGuys DevOpsGuys 19
    20. 20. Maximise Testing Value Deploy Fix @DevOpsGuys Test Analyse DevOpsGuys 20
    21. 21. 9am to 9pm. 12hrs. 56x faster! CASE STUDY #1 56X FASTER IN A DAY… @DevOpsGuys DevOpsGuys 21
    22. 22. The Scenario • • • • • New website pre-launch testing Customer Portal (SaaS solution) Start Load testing 9am 4 “Test & Fix” cycles in the day Final testing completed at 9pm @DevOpsGuys DevOpsGuys 22
    23. 23. The End Results = 56x Throughput (Req/sec) 16/min 59/min 56x faster Initial Testing Final Result Average Response 3024ms 54ms 3.5x throughput Before 3024ms @DevOpsGuys DevOpsGuys After 54ms 23
    24. 24. Nice flat line! Before After @DevOpsGuys DevOpsGuys 24
    25. 25. #4 Team Work “The Enemy is Performance, not each other…” @DevOpsGuys DevOpsGuys 25
    26. 26. The Blame Game… Ops @DevOpsGuys QA DevOpsGuys Dev 26
    27. 27. The Fix-It Game… Ops Dev QA @DevOpsGuys DevOpsGuys 27
    28. 28. @DevOpsGuys DevOpsGuys 28
    29. 29. Summary 1. 2. 3. 4. See the Big Picture (systems thinking) Drill down to the details Faster Iteration = Better Value Stop the “Blame Game” – aka “There’s your Problem!” @DevOpsGuys DevOpsGuys 29
    30. 30. DevOpsGuys Thank You Workshop Session @ 1345 End-to-end testing & APM demo operations | availability | performance | scalability | migration | strategy @DevOpsGuys DevOpsGuys 30
    31. 31. What We Do DevOpsGuys provide development & operations support (DevOps) services for your online applications. You run your business. We run your website. It’s as simple as that. @DevOpsGuys DevOpsGuys 31
    32. 32. Our service portfolio • 24 x 7 Application-Centric web operations • DevOps Transformation & Training • Continuous Delivery implementation • Infrastructure Automation & “Infrastructure as Code” • Application Performance Management • AppDynamics Implementation & Consultancy @DevOpsGuys • Web Performance consulting • Web monitoring • Load Testing • Test Automation and scripting • JetBrains Product Implementation & Consultancy • Octopus Deploy implementation DevOpsGuys 32
    33. 33. The Leadership Team James Smith (@TheDevMgr) Steve Thair (@TheOpsMgr) • 16 years of software delivery experience. • Proven track record • 25 years of IT Operations experience. • Proven track record – – – – – – – Pizza Hut KFC Dell RAC Aviva Fujitsu Siemens Whitbread @DevOpsGuys – – – – – – – DevOpsGuys Vodafone Credit Suisse Group IBM BNP Paribas Zurich Financial Services Totaljobs Group TSL Education 33
    34. 34. Contact Us Visit us at Call our team on +44 (0)20 8798 3079 Email our engineers at @DevOpsGuys DevOpsGuys 34