Successfully reported this slideshow.
SERITI
   CONSULTING




Load Testing Case Study


    Some hints and tips on load-testing your website...




           ...
Agenda


 Introductions
 Before you start
 Designing the test
 On the Day…
 Analysing the results and taking action
...
SERITI
    CONSULTING




Before you start


    Working out what to test and how much to test it




                    ...
Why do you want to load test?


 New website still under development?
 Existing website about to head into a
  business ...
“Load test” vs. “Performance Test”



 Performance vs Load Test?

 This affects:
  How you will do it
  What you will ...
Load vs. Performance?

 Load Test = quot;Black Boxquot;
  I only care if the whole system can handle the
   load
  If i...
Test In-house or use 3rd party?

 Project triangle decision
 Time
 Money
 Scope

 Resources required...
 Testing sof...
SERITI
   CONSULTING




Designing the test


    Working out what to test and how much to test it




                   ...
Decide on your critical user journeys



 Home page

 Search pages

 Details pages

 Upload/download

 Checkout/e-com...
Set your targets

 How many transactions/sec?

 How many concurrent users?

 What is the mix of the user journeys?
  D...
TIP - Web Analytics is your BFF

 Leverage your site metrics!
  Google Analytics, WebTrends or Omniture etc
  Use the i...
TIP – Management Speak..

 The Management tends to ask “how many
  users will the site support”…

 How many users do wha...
Concurrent users & “think time”

 Take the number of page impression per hour
 Divide by 3600
 That gives you the page ...
Design the test…


                     Scripts                                Profiles                    Scenarios


 S...
What does a Profile look like?




       Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved ...
3 Quick Tips…

 Baseline your network first
   Do a “GET” for a 100K JPEG and see how many you get before you
    satura...
One more tip…

 DON’T FORGET YOUR BATCH PROCESSES
  How does the system perform during…
   •   Backups
   •   Re-indexes...
On the day – running the test!

                        Who does what, where & when?




                                 ...
Plan out the timetable!

 Make the most of your testing window…

 Plan carefully but BE REALISTIC!

 Allow a lot of tim...
On the day...

 DON’T - Suddenly remember that you forgot to tell your hosting
  provider...
   And watch their IDS syst...
SERITI
    CONSULTING




Analysing the results and taking action

          Data -> Information -> Knowledge -> Action


...
Lots of questions…

 Did you hit your targets?

 Where there any odd peaks/troughs?

 Was “mixed scenario” performance ...
Before and After a Network upgrade…




     Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserv...
Lower & longer is better




Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved   29
Faster response, with more users…




       Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserv...
Taking action…

 Use the performance stat (if you gathered any)
  to isolate the weak links
 Upgrade/re-factor & re-test...
SERITI
   CONSULTING




The vendor bit…



                                       Over to James… 




                  ...
Site Confidence Clients…




       Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved   34
Contact Details – Site Confidence

 Talk to Site Confidence on their stand at the TestExpo after the
  presentation or if...
Contact Details – Seriti Consulting

 If you have any questions around
   web hosting (IIS, Cloud)
   Performance tools...
Any Questions?




                                                                                        3
      Commerc...
Upcoming SlideShare
Loading in …5
×

Test Expo 2009 Site Confidence & Seriti Consulting Load Test Case Study

1,786 views

Published on

Presentation given at the 2009 TestExpo in London www.testexpo.co.uk

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Test Expo 2009 Site Confidence & Seriti Consulting Load Test Case Study

  1. 1. SERITI CONSULTING Load Testing Case Study Some hints and tips on load-testing your website... 1 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
  2. 2. Agenda  Introductions  Before you start  Designing the test  On the Day…  Analysing the results and taking action  The vendor bit…  Questions? 2 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
  3. 3. SERITI CONSULTING Before you start Working out what to test and how much to test it 5 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
  4. 4. Why do you want to load test?  New website still under development?  Existing website about to head into a business peak period?  New marketing campaign about to launch?  The Met Office is predicting snow... 6 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
  5. 5. “Load test” vs. “Performance Test”  Performance vs Load Test?  This affects: How you will do it What you will do during the test Who needs to be involved What information you record! Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 7
  6. 6. Load vs. Performance?  Load Test = quot;Black Boxquot; I only care if the whole system can handle the load If it can't I probably don't have a lot of clues as to why not...  Performance Test = quot;White Boxquot; Looking inside the App and Infrastructure architecture to identify the key elements Designing tests to stress each element Monitoring those key elements during the test Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 8
  7. 7. Test In-house or use 3rd party?  Project triangle decision Time Money Scope  Resources required... Testing software e.g. LoadRunner, Selenium, Site Confidence People to script it Hardware & network to generate the load Accessibility of test environments (or use Prod) Reporting interface to get the right stats & graphs People to interpret the results... 9 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
  8. 8. SERITI CONSULTING Designing the test Working out what to test and how much to test it 1 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 0
  9. 9. Decide on your critical user journeys  Home page  Search pages  Details pages  Upload/download  Checkout/e-commerce  SEO Crawl – Google can hammer sites! Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 12
  10. 10. Set your targets  How many transactions/sec?  How many concurrent users?  What is the mix of the user journeys? Don't underestimate the impact of concurrence...  What's your quot;safety marginquot; We used 40% Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 13
  11. 11. TIP - Web Analytics is your BFF  Leverage your site metrics! Google Analytics, WebTrends or Omniture etc Use the information to identify your peak loads across different user journeys It is very common for people to test their sites at very unrealistic rates... Or focus on the wrong parts… • Don’t forget deep linking! Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 14
  12. 12. TIP – Management Speak..  The Management tends to ask “how many users will the site support”…  How many users do what, how often? Users on playing online poker are different to users reading “The Register”…  I find it best to talk about “page impressions” since that is a metrics they are probably familiar with by now… Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 15
  13. 13. Concurrent users & “think time”  Take the number of page impression per hour  Divide by 3600  That gives you the page impressions (quot;transactionsquot;) per second target  Then take the number of quot;concurrent usersquot; and divide by the quot;think timequot; (dwell time) to give you transactions a second  1200 users with a 60s think time = 20 transactions /sec  600 users with 30s think time = 20 transactions /sec  300 users with 15s think time = 20 transactions /sec  100 users with a 5s think time = 20 transactions /sec Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 16
  14. 14. Design the test… Scripts Profiles Scenarios  Scripts - the actual user journey (“clicks”)  Profiles - how many, how fast, how long etc  Scenarios - combining profiles to match the traffic patterns on your site… The “mix” of user journeys at any one time Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 17
  15. 15. What does a Profile look like? Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 19
  16. 16. 3 Quick Tips…  Baseline your network first  Do a “GET” for a 100K JPEG and see how many you get before you saturate your network  Do “single profile” scenarios first  Work from simple to complex  Test each element before you try combinations  Think about “keyhole” tests to isolate specific application functions  Rather than simulate the “click by click” of the user just “POST” (form submit) directly to the site  This enables you to focus on the app performance rather than the UI elements (you need to do both though!) Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 21
  17. 17. One more tip…  DON’T FORGET YOUR BATCH PROCESSES  How does the system perform during… • Backups • Re-indexes • ETL activities • Any other batch process that stresses the system…  Many a site dies when these processes run  And if you are truly 24x7 online – when is your window? Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 22
  18. 18. On the day – running the test! Who does what, where & when? 23 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
  19. 19. Plan out the timetable!  Make the most of your testing window…  Plan carefully but BE REALISTIC!  Allow a lot of time for review between each scenario run...  You WILL change your plans on the day... Be flexible!  Get the right people – Operations, QA, Network, Database etc Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 24
  20. 20. On the day...  DON’T - Suddenly remember that you forgot to tell your hosting provider...  And watch their IDS system think it was a DDOS attack and drop all your packets...  DO – keep an eye on the time (and ensure that time is sync’s across the systems)  You will want to correlate events!  DO – use a meaningful name for each run and add comments  So you know how many users are added, over what ramp up period, and run for how long at a glance  DO – think about cleaning up after you!  If you have been running against Prod you might have created a lot of dummy transactions (and your web logs will be HUGE!) Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 25
  21. 21. SERITI CONSULTING Analysing the results and taking action Data -> Information -> Knowledge -> Action 26 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
  22. 22. Lots of questions…  Did you hit your targets?  Where there any odd peaks/troughs?  Was “mixed scenario” performance different?  Was performance consistent across test runs?  Did your releases make things better (or worse)?  What was the Max/Mean/Min values  And what was the variance? 2 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 7
  23. 23. Before and After a Network upgrade… Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 28
  24. 24. Lower & longer is better Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 29
  25. 25. Faster response, with more users… Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 30
  26. 26. Taking action…  Use the performance stat (if you gathered any) to isolate the weak links Upgrade/re-factor & re-test  Use the (bad) stats to get more budget to fix the problem…  Use the (good) stats to get more a bonus because you did such a good job… 3 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 1
  27. 27. SERITI CONSULTING The vendor bit… Over to James…  3 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 2
  28. 28. Site Confidence Clients… Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 34
  29. 29. Contact Details – Site Confidence  Talk to Site Confidence on their stand at the TestExpo after the presentation or if you would like a copy of the presentation  Email James – James.Bavington@SiteConfidence.com  Visit their website - www.siteconfidence.com 3 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 5
  30. 30. Contact Details – Seriti Consulting  If you have any questions around  web hosting (IIS, Cloud)  Performance tools (Appmetrics, Avicode)  Operations & ITIL (service-now.com)  Configuration Management (Configuresoft ECM)  Near-shore development, functional test (Endava)  Email me – Stephen.Thair@SeritiConsulting.com  Connect with me on LinkedIn  http://www.linkedin.com/in/stephenthair 3 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 6
  31. 31. Any Questions? 3 Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 7

×