Test Expo 2009 Site Confidence & Seriti Consulting Load Test Case Study

Agenda  Introductions  Before you start  Designing the test  On the Day…  Analysing the results and taking action  The vendor bit…  Questions? 2 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved

Why do you want to load test?  New website still under development?  Existing website about to head into a business peak period?  New marketing campaign about to launch?  The Met Office is predicting snow... 6 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved

“Load test” vs. “Performance Test”  Performance vs Load Test?  This affects: How you will do it What you will do during the test Who needs to be involved What information you record! Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 7

Load vs. Performance?  Load Test = quot;Black Boxquot; I only care if the whole system can handle the load If it can't I probably don't have a lot of clues as to why not...  Performance Test = quot;White Boxquot; Looking inside the App and Infrastructure architecture to identify the key elements Designing tests to stress each element Monitoring those key elements during the test Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 8

Test In-house or use 3rd party?  Project triangle decision Time Money Scope  Resources required... Testing software e.g. LoadRunner, Selenium, Site Confidence People to script it Hardware & network to generate the load Accessibility of test environments (or use Prod) Reporting interface to get the right stats & graphs People to interpret the results... 9 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved

Decide on your critical user journeys  Home page  Search pages  Details pages  Upload/download  Checkout/e-commerce  SEO Crawl – Google can hammer sites! Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 12

Set your targets  How many transactions/sec?  How many concurrent users?  What is the mix of the user journeys? Don't underestimate the impact of concurrence...  What's your quot;safety marginquot; We used 40% Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 13

TIP - Web Analytics is your BFF  Leverage your site metrics! Google Analytics, WebTrends or Omniture etc Use the information to identify your peak loads across different user journeys It is very common for people to test their sites at very unrealistic rates... Or focus on the wrong parts… • Don’t forget deep linking! Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 14

TIP – Management Speak..  The Management tends to ask “how many users will the site support”…  How many users do what, how often? Users on playing online poker are different to users reading “The Register”…  I find it best to talk about “page impressions” since that is a metrics they are probably familiar with by now… Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 15

Concurrent users & “think time”  Take the number of page impression per hour  Divide by 3600  That gives you the page impressions (quot;transactionsquot;) per second target  Then take the number of quot;concurrent usersquot; and divide by the quot;think timequot; (dwell time) to give you transactions a second  1200 users with a 60s think time = 20 transactions /sec  600 users with 30s think time = 20 transactions /sec  300 users with 15s think time = 20 transactions /sec  100 users with a 5s think time = 20 transactions /sec Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 16

Design the test… Scripts Profiles Scenarios  Scripts - the actual user journey (“clicks”)  Profiles - how many, how fast, how long etc  Scenarios - combining profiles to match the traffic patterns on your site… The “mix” of user journeys at any one time Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 17

3 Quick Tips…  Baseline your network first  Do a “GET” for a 100K JPEG and see how many you get before you saturate your network  Do “single profile” scenarios first  Work from simple to complex  Test each element before you try combinations  Think about “keyhole” tests to isolate specific application functions  Rather than simulate the “click by click” of the user just “POST” (form submit) directly to the site  This enables you to focus on the app performance rather than the UI elements (you need to do both though!) Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 21

One more tip…  DON’T FORGET YOUR BATCH PROCESSES  How does the system perform during… • Backups • Re-indexes • ETL activities • Any other batch process that stresses the system…  Many a site dies when these processes run  And if you are truly 24x7 online – when is your window? Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 22

Plan out the timetable!  Make the most of your testing window…  Plan carefully but BE REALISTIC!  Allow a lot of time for review between each scenario run...  You WILL change your plans on the day... Be flexible!  Get the right people – Operations, QA, Network, Database etc Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 24

On the day...  DON’T - Suddenly remember that you forgot to tell your hosting provider...  And watch their IDS system think it was a DDOS attack and drop all your packets...  DO – keep an eye on the time (and ensure that time is sync’s across the systems)  You will want to correlate events!  DO – use a meaningful name for each run and add comments  So you know how many users are added, over what ramp up period, and run for how long at a glance  DO – think about cleaning up after you!  If you have been running against Prod you might have created a lot of dummy transactions (and your web logs will be HUGE!) Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 25

Lots of questions…  Did you hit your targets?  Where there any odd peaks/troughs?  Was “mixed scenario” performance different?  Was performance consistent across test runs?  Did your releases make things better (or worse)?  What was the Max/Mean/Min values  And what was the variance? 2 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 7

Taking action…  Use the performance stat (if you gathered any) to isolate the weak links Upgrade/re-factor & re-test  Use the (bad) stats to get more budget to fix the problem…  Use the (good) stats to get more a bonus because you did such a good job… 3 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 1

Contact Details – Site Confidence  Talk to Site Confidence on their stand at the TestExpo after the presentation or if you would like a copy of the presentation  Email James – James.Bavington@SiteConfidence.com  Visit their website - www.siteconfidence.com 3 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 5

Contact Details – Seriti Consulting  If you have any questions around  web hosting (IIS, Cloud)  Performance tools (Appmetrics, Avicode)  Operations & ITIL (service-now.com)  Configuration Management (Configuresoft ECM)  Near-shore development, functional test (Endava)  Email me – Stephen.Thair@SeritiConsulting.com  Connect with me on LinkedIn  http://www.linkedin.com/in/stephenthair 3 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 6

http://www.seriticonsulting.com	185
http://theopsmgr.seriticonsulting.com	5
http://www.slideshare.net	1
url_unknown	1

Test Expo 2009 Site Confidence & Seriti Consulting Load Test Case Study

by Stephen Thair

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 192

Statistics