Is the current model of load testing broken ukcmg - steve thair


Published on

Does the current load testing paradigm make sense in a Agile Web 2.0 AJAX World?

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Thanks for coming to my talk… I know it must have been hard to tear yourself away from all about “Workload License charges in IBM System Z” so I appreciate your trust! We have lot of really interesting stuff to talk about…You will get a lot of food for thought and I will confess right now that I don’t know all the answers to this as yet, but hopefully we might find some out along the way!
  • I run a monthly Meetup group on Web Performance… and back in December 2011 we had a presentation from the Performance team at Betfair about performance testing in their continuous integration environment.
  • And they presented an environment like this…
  • So I am like what, huh, that’s that? You separated load injection from performance measurement? Doesn’t that sort of destroy half the value proposition of the all those expensive load testing tools? Why did you do that?How are you measuring it then? And then all sorts of other issues came out of that…
  • So I had all these questions in my head… but before I get to that I need to address Alex Podelko’s objection raised in a comment on my blog…
  • True, I didn’t…. So say hello to my little friend… the Straw Man!
  • How do you get “results” when the testing never “stops”…
  • APM tools offer more insight…. I mean, that’s what they are designed to do so it’s hardly a surprise. So increasing in my load testing I search
  • Some RUM tools are even free – like Google SiteSpeedSo why do I need expensive load tools if I all I am doing is measuring load?
  • WebSockets is a new HTML 5 API protocol for bi-directional real-time communication between browser (client) and server.But the key here is that there isn’t a nice request/response “round-trip” any more – the very thing that most of the current generation of test tools rely on (especially the HTTP 1.1 level protocol tools like JmeterAnd in a HTTP 2.0 world it gets even worse… because you have HTTP channel multiplexing…
  • Websites are getting more and more complex… especially as we add in new functionality like AJAX and HTML5 I have been playing with a solution
  • I created a proof of concept
  • DB2 was the reason I moved into local area networking…
  • Is the current model of load testing broken ukcmg - steve thair

    1. 1. Is the current model of loadtesting broken? Steve Thair Seriti Consulting @TheOpsMgr
    2. 2. 2 Some background….  My User Group – Dec 2011 - Seriti Consulting, 2011
    3. 3. Delivery Alert Deviation from Team normal? Ready Fail Fail to for Build meet Test? SLA’s? Deploy Measurements Environment Continuous End User Perf Trends Load Response Injection For Each Data Service/API Point Usage response Profile Production logs System Perf Details monitors CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
    4. 4. 4 The Killer Comment… “We’ve had to look at separating load injection from performance measurement”. - Andrew Harding, Betfair /12/9/is-the-current-model-of- loadperformance-testing-broken.html(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
    5. 5. So why are you paying all 5 that money for that What are key expensive brand-name reasons behind load testing tool then? that decision? Surely there are open source or cheap cloud tools if all you want is load? What other issues might we have with What are you “traditional” loadusing to measure testing models Performance in a Web 2.0 then? world? @TheOpsMgr #ukcmg #webperf
    6. 6. 6Defining the “current model”“Stephen didn’t explicitly definewhat the current model ofload/performance testing is…”  Alex Podelko @TheOpsMgr #ukcmg #webperf
    7. 7. 7 “Current Model Straw Man”(1) Waterfall Development Cycle(2) Load Tool compiles the report(3) Reporting at the end of the test(4) Request / Response paradigm(5) Well-defined customer journeys(c) Seriti Consulting, 2011
    8. 8. 8Testing in a CI/Agile world Performance testing initiated every time code is committed to the CM repository (e.g. SVN, GIT etc) Betfair issue was that their system was complex with many layers of caching etc so it took longer to “warm up” the environment (to achieve a steady performance state) than they had between check-ins… So they needed “continuous injection” to keep the environment constantly warm… The test tool never “stopped” to “compile the report”… So any tool that “reported at the end” wasn’t as useful… @TheOpsMgr #ukcmg #webperf
    9. 9. 9Test tools in a APM & RUM world “traditional test tools” generally have the ability to deploy agents to gather metrics from the target environment But the depth of analysis and correlation falls well below that of modern Application Performance Management tools e.g. AppDynamics APM tools offer a deeper insight and better event correlation across tiers… And they are getting (much) cheaper… @TheOpsMgr #ukcmg #webperf
    10. 10. APM Measurement @TheOpsMgr #ukcmg #webperf
    11. 11. Google Real Time Analytics @TheOpsMgr #ukcmg #webperf
    12. 12. 12WebSockets HTTP 1.1  Start Timer  GET /index.html  Response 200 OK  Stop Timer WebSockets  Start Timer  Socket “upgrade”  N many frames…  Stop??? Bi-directional socket channel @TheOpsMgr #ukcmg #webperf
    13. 13. 13HTTP 2.0 HTTP 1.1  Sequential  ordered HTTP 2.0 (& SPDY)  Multiplexed over a single connection  Responses returned out of sequence…  Hard to time! @TheOpsMgr #ukcmg #webperf
    14. 14. 14Website Complexity  263,000 entry points… @TheOpsMgr #ukcmg #webperf
    15. 15. 15Visitor Flow How many paths thru a website? Classic script-driven approaches can’t hope to address the complexity Network and log file replay solutions? @TheOpsMgr #ukcmg #webperf
    16. 16. 16A PCAP Solution to Replay? POC solution based on Cloudmeter Pion + custom scripts  Read a PCAP (network capture)  Identify the HTTP traffic  Filter it (based on your requirements)  Parameterise it (query strings, POST parameters etc)  Randomise inputs from SQL, CSV etc  Replay it against a test environment  i.e. change the base URL  Amplify & rate throttle req/sec as required @TheOpsMgr #ukcmg #webperf
    17. 17. 17 Personal Opinion! YMMV…So what’s the Answer? Use the cheapest method to generate load that you can find… Move away from scripting-based approaches towards using real-user session replay (if possible!) Generate load continuously Measure continuously  using APM & RUM type tools – FOSS or Commercial  Look for changes in histograms, averages, standard dev etc Protocol and Framework aware instrumentation  AFAIK this currently doesn’t exist… @TheOpsMgr #ukcmg #webperf
    18. 18. 18 @LDNWebPerf User Group!  Join our London Web Performance Meetup   Next Wednesday 17th Oct – 7pm – Central London  Follow us on Twitter @LDNWebPerf  #LDNWebPerf & #WebPerf(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
    19. 19. 19 About Me  21yrs IT experience.  Started with www in 1998 (IIS3! Site Server 3!).  Web Architect @ BNP Paribas, CSFB etc  Web Operations Manager for,  Professional Services Manager @  Seriti Consulting – specialising in web operations, management and Performance   m:+44 7971 815 940  Twitter:  Blog:  LinkedIn:  Skype: seriti-steve(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
    20. 20. 20 Questions?(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf