Thanks for coming to my talk… I know it must have been hard to tear yourself away from all about “Workload License charges in IBM System Z” so I appreciate your trust! We have lot of really interesting stuff to talk about…You will get a lot of food for thought and I will confess right now that I don’t know all the answers to this as yet, but hopefully we might find some out along the way!
I run a monthly Meetup group on Web Performance… and back in December 2011 we had a presentation from the Performance team at Betfair about performance testing in their continuous integration environment.
And they presented an environment like this…
So I am like what, huh, that’s that? You separated load injection from performance measurement? Doesn’t that sort of destroy half the value proposition of the all those expensive load testing tools? Why did you do that?How are you measuring it then? And then all sorts of other issues came out of that…
So I had all these questions in my head… but before I get to that I need to address Alex Podelko’s objection raised in a comment on my blog…
True, I didn’t…. So say hello to my little friend… the Straw Man!
How do you get “results” when the testing never “stops”…
APM tools offer more insight…. I mean, that’s what they are designed to do so it’s hardly a surprise. So increasing in my load testing I search
Some RUM tools are even free – like Google SiteSpeedSo why do I need expensive load tools if I all I am doing is measuring load?
WebSockets is a new HTML 5 API protocol for bi-directional real-time communication between browser (client) and server.But the key here is that there isn’t a nice request/response “round-trip” any more – the very thing that most of the current generation of test tools rely on (especially the HTTP 1.1 level protocol tools like JmeterAnd in a HTTP 2.0 world it gets even worse… because you have HTTP channel multiplexing…
Websites are getting more and more complex… especially as we add in new functionality like AJAX and HTML5 I have been playing with a solution
I created a proof of concept
DB2 was the reason I moved into local area networking…
Is the current model of load testing broken ukcmg - steve thair
Is the current model of loadtesting broken? Steve Thair Seriti Consulting @TheOpsMgr
2 Some background…. My User Group – Dec 2011 - http://www.meetup.com/London-Web-Performance-Group/(c) Seriti Consulting, 2011
4 The Killer Comment… “We’ve had to look at separating load injection from performance measurement”. - Andrew Harding, Betfair http://www.seriticonsulting.com/blog/2011 /12/9/is-the-current-model-of- loadperformance-testing-broken.html(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
So why are you paying all 5 that money for that What are key expensive brand-name reasons behind load testing tool then? that decision? Surely there are open source or cheap cloud tools if all you want is load? What other issues might we have with What are you “traditional” loadusing to measure testing models Performance in a Web 2.0 then? world? @TheOpsMgr #ukcmg #webperf
6Defining the “current model”“Stephen didn’t explicitly definewhat the current model ofload/performance testing is…” Alex Podelkohttp://applicationperformanceengineeringhub.com/is-the-current-model-of-loadperformance-testing-broken/ @TheOpsMgr #ukcmg #webperf
7 “Current Model Straw Man”(1) Waterfall Development Cycle(2) Load Tool compiles the report(3) Reporting at the end of the test(4) Request / Response paradigm(5) Well-defined customer journeys(c) Seriti Consulting, 2011
8Testing in a CI/Agile world Performance testing initiated every time code is committed to the CM repository (e.g. SVN, GIT etc) Betfair issue was that their system was complex with many layers of caching etc so it took longer to “warm up” the environment (to achieve a steady performance state) than they had between check-ins… So they needed “continuous injection” to keep the environment constantly warm… The test tool never “stopped” to “compile the report”… So any tool that “reported at the end” wasn’t as useful… @TheOpsMgr #ukcmg #webperf
9Test tools in a APM & RUM world “traditional test tools” generally have the ability to deploy agents to gather metrics from the target environment But the depth of analysis and correlation falls well below that of modern Application Performance Management tools e.g. AppDynamics APM tools offer a deeper insight and better event correlation across tiers… And they are getting (much) cheaper… @TheOpsMgr #ukcmg #webperf
Google Real Time Analytics @TheOpsMgr #ukcmg #webperf
12WebSockets HTTP 1.1 Start Timer GET /index.html Response 200 OK Stop Timer WebSockets Start Timer Socket “upgrade” N many frames… Stop??? Bi-directional socket channel @TheOpsMgr #ukcmg #webperf
13HTTP 2.0 HTTP 1.1 Sequential ordered HTTP 2.0 (& SPDY) Multiplexed over a single connection Responses returned out of sequence… Hard to time!http://stackoverflow.com/questions/10480122/difference-between-http-pipeling-and-http-multiplexing-with-spdy @TheOpsMgr #ukcmg #webperf
15Visitor Flow How many paths thru a website? Classic script-driven approaches can’t hope to address the complexity Network and log file replay solutions? @TheOpsMgr #ukcmg #webperf
16A PCAP Solution to Replay? POC solution based on Cloudmeter Pion + custom scripts Read a PCAP (network capture) Identify the HTTP traffic Filter it (based on your requirements) Parameterise it (query strings, POST parameters etc) Randomise inputs from SQL, CSV etc Replay it against a test environment i.e. change the base URL Amplify & rate throttle req/sec as required http://www.cloudmeter.com/pion/data-processing.php @TheOpsMgr #ukcmg #webperf
17 Personal Opinion! YMMV…So what’s the Answer? Use the cheapest method to generate load that you can find… Move away from scripting-based approaches towards using real-user session replay (if possible!) Generate load continuously Measure continuously using APM & RUM type tools – FOSS or Commercial Look for changes in histograms, averages, standard dev etc Protocol and Framework aware instrumentation AFAIK this currently doesn’t exist… @TheOpsMgr #ukcmg #webperf
18 @LDNWebPerf User Group! Join our London Web Performance Meetup http://www.meetup.com/London-Web-Performance-Group/ Next Wednesday 17th Oct – 7pm – Central London Follow us on Twitter @LDNWebPerf #LDNWebPerf & #WebPerf(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf
19 About Me 21yrs IT experience. Started with www in 1998 (IIS3! Site Server 3!). Web Architect @ BNP Paribas, CSFB etc Web Operations Manager for www.totaljobs.com, www.tes.co.uk Professional Services Manager @ www.siteconfidence.com Seriti Consulting – specialising in web operations, management and Performance e:firstname.lastname@example.org m:+44 7971 815 940 Twitter: http://twitter.com/TheOpsMgr Blog: http://www.seriticonsulting.com/blog/ LinkedIn: http://uk.linkedin.com/in/stephenthair Skype: seriti-steve(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf