Test Expo 2009 Site Confidence & Seriti Consulting Load Test Case Study

SERITI
CONSULTING

Load Testing Case Study

Some hints and tips on load-testing your website...

1
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved

Agenda

 Introductions
 Before you start
 Designing the test
 On the Day…
 Analysing the results and taking action
 The vendor bit…
 Questions?
2

SERITI
CONSULTING

Before you start

Working out what to test and how much to test it

5

Why do you want to load test?

 New website still under development?
 Existing website about to head into a
business peak period?
 New marketing campaign about to
launch?
 The Met Office is predicting snow...

6

“Load test” vs. “Performance Test”

 Performance vs Load Test?

 This affects:
How you will do it
What you will do during the test
Who needs to be involved
What information you record!

Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved 7

Load vs. Performance?

 Load Test = quot;Black Boxquot;
I only care if the whole system can handle the
load
If it can't I probably don't have a lot of clues as to
why not...

 Performance Test = quot;White Boxquot;
Looking inside the App and Infrastructure
architecture to identify the key elements
Designing tests to stress each element
Monitoring those key elements during the test


Test In-house or use 3rd party?

 Project triangle decision
Time
Money
Scope

 Resources required...
Testing software e.g. LoadRunner, Selenium, Site Confidence
People to script it
Hardware & network to generate the load
Accessibility of test environments (or use Prod)
Reporting interface to get the right stats & graphs
People to interpret the results...

9

SERITI
CONSULTING

Designing the test

Working out what to test and how much to test it

1
0

Decide on your critical user journeys

 Home page

 Search pages

 Details pages

 Upload/download

 Checkout/e-commerce

 SEO Crawl – Google can hammer sites!


Set your targets

 How many transactions/sec?

 How many concurrent users?

 What is the mix of the user journeys?
Don't underestimate the impact of
concurrence...

 What's your quot;safety marginquot;
We used 40%


TIP - Web Analytics is your BFF

 Leverage your site metrics!
Google Analytics, WebTrends or Omniture etc
Use the information to identify your peak
loads across different user journeys
It is very common for people to test their sites
at very unrealistic rates... Or focus on the
wrong parts…
• Don’t forget deep linking!


TIP – Management Speak..

 The Management tends to ask “how many
users will the site support”…

 How many users do what, how often?
Users on playing online poker are different to users reading “The
Register”…

 I find it best to talk about “page impressions”
since that is a metrics they are probably familiar
with by now…


Concurrent users & “think time”

 Take the number of page impression per hour
 Divide by 3600
 That gives you the page impressions (quot;transactionsquot;) per
second target
 Then take the number of quot;concurrent usersquot; and divide
by the quot;think timequot; (dwell time) to give you transactions
a second
 1200 users with a 60s think time = 20 transactions /sec

 600 users with 30s think time = 20 transactions /sec

 300 users with 15s think time = 20 transactions /sec

 100 users with a 5s think time = 20 transactions /sec


Design the test…

Scripts Profiles Scenarios

 Scripts - the actual user journey (“clicks”)
 Profiles - how many, how fast, how long etc
 Scenarios - combining profiles to match the
traffic patterns on your site…
The “mix” of user journeys at any one time


What does a Profile look like?


3 Quick Tips…

 Baseline your network first
 Do a “GET” for a 100K JPEG and see how many you get before you
saturate your network

 Do “single profile” scenarios first
 Work from simple to complex
 Test each element before you try combinations

 Think about “keyhole” tests to isolate specific
application functions
 Rather than simulate the “click by click” of the user just “POST” (form
submit) directly to the site
 This enables you to focus on the app performance rather than the UI
elements (you need to do both though!)


One more tip…

 DON’T FORGET YOUR BATCH PROCESSES
 How does the system perform during…
• Backups
• Re-indexes
• ETL activities
• Any other batch process that stresses the system…

 Many a site dies when these processes run

 And if you are truly 24x7 online – when is your window?


On the day – running the test!

Who does what, where & when?

23

Plan out the timetable!

 Make the most of your testing window…

 Plan carefully but BE REALISTIC!

 Allow a lot of time for review between each
scenario run...

 You WILL change your plans on the day...
Be flexible!

 Get the right people – Operations, QA,
Network, Database etc


On the day...

 DON’T - Suddenly remember that you forgot to tell your hosting
provider...
 And watch their IDS system think it was a DDOS attack and drop all your
packets...

 DO – keep an eye on the time (and ensure that time is sync’s
across the systems)
 You will want to correlate events!

 DO – use a meaningful name for each run and add comments
 So you know how many users are added, over what ramp up period,
and run for how long at a glance

 DO – think about cleaning up after you!
 If you have been running against Prod you might have created a lot of
dummy transactions (and your web logs will be HUGE!)


SERITI
CONSULTING

Analysing the results and taking action

Data -> Information -> Knowledge -> Action

26

Lots of questions…

 Did you hit your targets?

 Where there any odd peaks/troughs?

 Was “mixed scenario” performance different?

 Was performance consistent across test runs?
 Did your releases make things better (or worse)?

 What was the Max/Mean/Min values
 And what was the variance?

2
7

Before and After a Network upgrade…


Lower & longer is better


Faster response, with more users…


Taking action…

 Use the performance stat (if you gathered any)
to isolate the weak links
Upgrade/re-factor & re-test

 Use the (bad) stats to get more budget to fix
the problem…

 Use the (good) stats to get more a bonus
because you did such a good job…

3
1

SERITI
CONSULTING

The vendor bit…

Over to James… 

3
2

Site Confidence Clients…


Contact Details – Site Confidence

 Talk to Site Confidence on their stand at the TestExpo after the
presentation or if you would like a copy of the presentation

 Email James – James.Bavington@SiteConfidence.com

 Visit their website - www.siteconfidence.com

3
5

Contact Details – Seriti Consulting

 If you have any questions around
 web hosting (IIS, Cloud)
 Performance tools (Appmetrics, Avicode)
 Operations & ITIL (service-now.com)
 Configuration Management (Configuresoft ECM)
 Near-shore development, functional test (Endava)

 Email me – Stephen.Thair@SeritiConsulting.com

 Connect with me on LinkedIn
 http://www.linkedin.com/in/stephenthair

3
6

Any Questions?

3
7

Test Expo 2009 Site Confidence & Seriti Consulting Load Test Case Study

Recommended

Recommended

More Related Content

Similar to Test Expo 2009 Site Confidence & Seriti Consulting Load Test Case Study

Similar to Test Expo 2009 Site Confidence & Seriti Consulting Load Test Case Study (20)

More from Stephen Thair

More from Stephen Thair (16)

Recently uploaded

Recently uploaded (20)