• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Is the current model of load testing broken   ukcmg - steve thair
 

Is the current model of load testing broken ukcmg - steve thair

on

  • 966 views

Does the current load testing paradigm make sense in a Agile Web 2.0 AJAX World?

Does the current load testing paradigm make sense in a Agile Web 2.0 AJAX World?

Statistics

Views

Total Views
966
Views on SlideShare
966
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Thanks for coming to my talk… I know it must have been hard to tear yourself away from all about “Workload License charges in IBM System Z” so I appreciate your trust! We have lot of really interesting stuff to talk about…You will get a lot of food for thought and I will confess right now that I don’t know all the answers to this as yet, but hopefully we might find some out along the way!
  • I run a monthly Meetup group on Web Performance… and back in December 2011 we had a presentation from the Performance team at Betfair about performance testing in their continuous integration environment.
  • And they presented an environment like this…
  • So I am like what, huh, that’s that? You separated load injection from performance measurement? Doesn’t that sort of destroy half the value proposition of the all those expensive load testing tools? Why did you do that?How are you measuring it then? And then all sorts of other issues came out of that…
  • So I had all these questions in my head… but before I get to that I need to address Alex Podelko’s objection raised in a comment on my blog…
  • True, I didn’t…. So say hello to my little friend… the Straw Man!
  • How do you get “results” when the testing never “stops”…
  • APM tools offer more insight…. I mean, that’s what they are designed to do so it’s hardly a surprise. So increasing in my load testing I search
  • Some RUM tools are even free – like Google SiteSpeedSo why do I need expensive load tools if I all I am doing is measuring load?
  • WebSockets is a new HTML 5 API protocol for bi-directional real-time communication between browser (client) and server.But the key here is that there isn’t a nice request/response “round-trip” any more – the very thing that most of the current generation of test tools rely on (especially the HTTP 1.1 level protocol tools like JmeterAnd in a HTTP 2.0 world it gets even worse… because you have HTTP channel multiplexing…
  • Websites are getting more and more complex… especially as we add in new functionality like AJAX and HTML5 I have been playing with a solution
  • I created a proof of concept
  • DB2 was the reason I moved into local area networking…

Is the current model of load testing broken   ukcmg - steve thair Is the current model of load testing broken ukcmg - steve thair Presentation Transcript

  • Is the current model of loadtesting broken? Steve Thair Seriti Consulting @TheOpsMgr
  • 2 Some background….  My User Group – Dec 2011 - http://www.meetup.com/London-Web-Performance-Group/(c) Seriti Consulting, 2011
  • Delivery Alert Deviation from Team normal? Ready Fail Fail to for Build meet Test? SLA’s? Deploy Measurements Environment Continuous End User Perf Trends Load Response Injection For Each Data Service/API Point Usage response Profile Production logs System Perf Details monitorshttp://www.slideshare.net/sthair/continuous-integration-a-performance-engineers-journey CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
  • 4 The Killer Comment… “We’ve had to look at separating load injection from performance measurement”. - Andrew Harding, Betfair http://www.seriticonsulting.com/blog/2011 /12/9/is-the-current-model-of- loadperformance-testing-broken.html(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
  • So why are you paying all 5 that money for that What are key expensive brand-name reasons behind load testing tool then? that decision? Surely there are open source or cheap cloud tools if all you want is load? What other issues might we have with What are you “traditional” loadusing to measure testing models Performance in a Web 2.0 then? world? @TheOpsMgr #ukcmg #webperf
  • 6Defining the “current model”“Stephen didn’t explicitly definewhat the current model ofload/performance testing is…”  Alex Podelkohttp://applicationperformanceengineeringhub.com/is-the-current-model-of-loadperformance-testing-broken/ @TheOpsMgr #ukcmg #webperf
  • 7 “Current Model Straw Man”(1) Waterfall Development Cycle(2) Load Tool compiles the report(3) Reporting at the end of the test(4) Request / Response paradigm(5) Well-defined customer journeys(c) Seriti Consulting, 2011
  • 8Testing in a CI/Agile world Performance testing initiated every time code is committed to the CM repository (e.g. SVN, GIT etc) Betfair issue was that their system was complex with many layers of caching etc so it took longer to “warm up” the environment (to achieve a steady performance state) than they had between check-ins… So they needed “continuous injection” to keep the environment constantly warm… The test tool never “stopped” to “compile the report”… So any tool that “reported at the end” wasn’t as useful… @TheOpsMgr #ukcmg #webperf
  • 9Test tools in a APM & RUM world “traditional test tools” generally have the ability to deploy agents to gather metrics from the target environment But the depth of analysis and correlation falls well below that of modern Application Performance Management tools e.g. AppDynamics APM tools offer a deeper insight and better event correlation across tiers… And they are getting (much) cheaper… @TheOpsMgr #ukcmg #webperf
  • APM Measurement @TheOpsMgr #ukcmg #webperf
  • Google Real Time Analytics @TheOpsMgr #ukcmg #webperf
  • 12WebSockets HTTP 1.1  Start Timer  GET /index.html  Response 200 OK  Stop Timer WebSockets  Start Timer  Socket “upgrade”  N many frames…  Stop??? Bi-directional socket channel @TheOpsMgr #ukcmg #webperf
  • 13HTTP 2.0 HTTP 1.1  Sequential  ordered HTTP 2.0 (& SPDY)  Multiplexed over a single connection  Responses returned out of sequence…  Hard to time!http://stackoverflow.com/questions/10480122/difference-between-http-pipeling-and-http-multiplexing-with-spdy @TheOpsMgr #ukcmg #webperf
  • 14Website Complexity  263,000 entry points… @TheOpsMgr #ukcmg #webperf
  • 15Visitor Flow How many paths thru a website? Classic script-driven approaches can’t hope to address the complexity Network and log file replay solutions? @TheOpsMgr #ukcmg #webperf
  • 16A PCAP Solution to Replay? POC solution based on Cloudmeter Pion + custom scripts  Read a PCAP (network capture)  Identify the HTTP traffic  Filter it (based on your requirements)  Parameterise it (query strings, POST parameters etc)  Randomise inputs from SQL, CSV etc  Replay it against a test environment  i.e. change the base URL  Amplify & rate throttle req/sec as required http://www.cloudmeter.com/pion/data-processing.php @TheOpsMgr #ukcmg #webperf
  • 17 Personal Opinion! YMMV…So what’s the Answer? Use the cheapest method to generate load that you can find… Move away from scripting-based approaches towards using real-user session replay (if possible!) Generate load continuously Measure continuously  using APM & RUM type tools – FOSS or Commercial  Look for changes in histograms, averages, standard dev etc Protocol and Framework aware instrumentation  AFAIK this currently doesn’t exist… @TheOpsMgr #ukcmg #webperf
  • 18 @LDNWebPerf User Group!  Join our London Web Performance Meetup  http://www.meetup.com/London-Web-Performance-Group/  Next Wednesday 17th Oct – 7pm – Central London  Follow us on Twitter @LDNWebPerf  #LDNWebPerf & #WebPerf(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
  • 19 About Me  21yrs IT experience.  Started with www in 1998 (IIS3! Site Server 3!).  Web Architect @ BNP Paribas, CSFB etc  Web Operations Manager for www.totaljobs.com, www.tes.co.uk  Professional Services Manager @ www.siteconfidence.com  Seriti Consulting – specialising in web operations, management and Performance  e:stephen.thair@seriticonsulting.com  m:+44 7971 815 940  Twitter: http://twitter.com/TheOpsMgr  Blog: http://www.seriticonsulting.com/blog/  LinkedIn: http://uk.linkedin.com/in/stephenthair  Skype: seriti-steve(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
  • 20 Questions?(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf