Are You Ready?
A New Approach for Ensuring Your Web Applications
Perform Under Load
Matt Hintze - Director of Web Experience Services, Universal Mind
Colin Mason - Product Manager Gomez
Manager,

Web Experience Practice
The Universal Mind Web Experience Practice
Core team of Allaire, Macromedia, and Adobe experts
Over 10 years of load testing experience.
We Work with customers to resolve load related incidents.
We Work with customers to prevent load related incidents.
We Focus on maximizing the end user experience.

Load Testing should answer these questions
How many servers do we need?
Can the network infrastructure support the expected traffic?
Will the site stay up during long periods of stress?
Are response times acceptable?
Does our fail-over solution work?
Are we ready to go live?

Why Load Test?
You can t improve what you can’t measure
can’t can t
Why i l d t ti
Wh is load testing paramount to the business ?
t t th b i
Ensures that web applications deliver superior customer experience to
drive greater on-line revenue.
Revenue risked when critical web pages are underperforming
Reduce web application operating costs.
Maintain peak web application performance and availability to protect
brand reputation.
Change M
Ch Management.
t

Why Load Test?
You can t improve what you can’t measure
can’t can t
Why is load testing paramount to the business (cont) ?
Service level compliance of vendors, partners and internal customers.
The Oprah Effect - Avoid being blind sided by load related catastrophic
failures.
failures
Stress test to size server and network infrastructure requirements.
Identify and eliminate bottlenecks.
bottlenecks
Tune application code, server settings, and validate CDN performance.
Validate capacity and performance expectations especially for traffic
expectations,
spikes from campaigns.

Load testing methodology
3 focus areas of load testing
Application Tuning
Identify code bottlenecks and scalability problems
Server Tuning
Tune server threading, CPU utilization, and memory
threading utilization
Endurance testing
Ensure site performance and availability over long
periods of usage and stress.

Load testing methodology
(cont)
The Tuning Process
Tuning Cycle
Change one thing at a time.
Load Test and measure performance.
Document results.
Repeat until the optimal performance is identified (sweet
spot).
spot)
Modify code or server configs
Stress T t
St Test
Document Performance Results
Do it again

Real World Examples & Experiences.
The cost of not load testing
Preparation for marketing campaigns
p g pg

Gomez Profile
▪ First SaaS Web Application
▪ Headquarters
Experience Management S it
E i M t Suite
▪ Lexington, MA
▪ Web App Experience management across
▪ Global Offices the application lifecycle
pp y
▪ UK ▪ 80,000+ internet experience measurement
points in 162 countries
▪ Germany
▪ Switzerland
▪ Over 2,500 Customers Worldwide
▪ China
▪ 12 out of 20 most visited US Websites
utilize Gomez
▪ 250+ Employees
9

Web Application Delivery
In A Web 2.0 World
▪ Web applications
pp
increasingly
assembled at
browser level
▪ 3rd party
performance from
ads, videos, shopping
cart vary widely
under load
▪ Content Delivery
Network k
configuration and
effectiveness impact
end-user
end user experience
10

Critical To Test Across Entire
Web Application Delivery Chain
11

Load Testing 2.0 –
The Next Generation
▪ Load Testing 2.0 - evolution of performance
testing for a Web 2.0 world
t ti f W b20 ld
▪ Self-service testing gives you control to test as needed
▪ Test both internal and external web application
components to ensure transactions perform under load
▪ T t cloud-based applications and services
Test l d b d li ti d i
▪ SaaS based Load Testing 2.0 solutions enable on-
demand, scalable testing without associated hardware
investment and maintenance costs
▪ Test the full web application delivery chain all the way
to your end users’ desktops
d ’d k
12

Firewall Bottleneck
Company
• Online Sporting Goods Retailer
Situation
• As part of a major system upgrade a number
of key infrastructure pieces were upgraded
• Following a Best Practices Methodology the
testing began with a simple test designed to
validate the new firewall
• First Step was to benchmark the current
firewall
• Second Step repeat the test on the new
firewall
13

Firewall Bottleneck
Test 1: Old Firewall
• Users ramp up, and the firewall
supported more than 2500
transactions per second.
• This was more than the approx
2000 transactions per second
seen in peak live traffic.
Test 2: New Firewall
•U
Users ramp up, b t th
but the
transaction rate bottlenecks at
1000 per second.
• Response times for users
climb.
14

Firewall Bottleneck
▪ Cause/Resolution
▪ The new firewall was only licensed for 1000 transactions
per second.
▪R l
Replace th license with the enterprise license they had
the li ith th t i li th hd
purchased.
▪ Business Impact
▪ This was a simple mistake, but not uncommon, and if it
was not caught their next peak day customers would
have been negatively impacted by
▪ Higher response times
▪ Connection errors
15

Third-Party Bottleneck
Company
• Online Retailer
Situation
• The application consists of a typical
e-Commerce
e Commerce type application
• Several 3rd Parties are now involved
in serving up key content
16

Third-Party Issues
The load increases
throughout the test.
The transaction rate
increases and then
falls off as response
times climb.
ti li b
Errors are seen, all
on a 3rd party
object.
bj t
17

Third-Party Bottleneck
▪ Cause/Resolution
▪ The 3rd party hardware was insufficient for overall
demands on the application
▪B
Based on SLAs the 3rd party had to improve their
d SLA th 3 d t h dt i th i
performance in order to get paid
▪ All aspects of the user experience delivery chain must
be tested
▪ Business Impact
▪ Customers negatively impacted by
▪ Higher response times
▪ Time-out errors
18

Real User Bottleneck
Company
• Regional Online News Source
Situation
Sit ti
• Began testing for the 2008 election season
• Looking to validate performance of overall
application, but in particular for users in 2
key geographies
• Testing started with the homepage
19

The False Positive
Increase and hold load and not exceed
There was only 1
response times of 4 seconds and Success
pg
page error and 11
Rate of 99%
errors total out of
60000+ transactions.
Page response
times stayed under
4 seconds, outside
of one brief blip
p
By traditional test standards the test passed
passed.
20

The False Positive
Real User Experience
Last Mile data showing substantial number of
measurements greater than 4 seconds
21

The False Positive
Primary Geographies
Key geographies for this customer are
New York and Pennsylvania.
The response
time met the 4
second average
goal.
By these standards the test failed. Availability was
Less than 99%.
99%
22

The False Positive
User Experience Drill Down
• Last Mile Users exceeding the 4-
second goal – 29%
d l
• Overall availability was 96.3%
• Availability for Last Mile Users
exceeding 4 seconds – 87%
• Last Mile Users reporting 10+
second response times – 8%
By these standards the test failed.
B th t d d th t t f il d
23

False Positive
▪ Cause/Resolution
▪ The key action item was for the customer to focus on
improving the response times of the page template from
which the homepage was designed
▪ Focusing on cutting 1.5-2 seconds off the response time
for the page would move another 16% of the Last Mile
users into the 4 second response time range
▪ Business Impact
▪ Customers negatively impacted by
▪ Higher response times
▪ Availability issues
24

Ensure Web Experiences
Scale Under Load
▪ Deploy applications and
infrastructure with confidence
▪ Combine high volume
application load testing with
broad geographic experience
testing
▪ Ensure on-demand testing
flexibility
▪ Detect failures in user
experience not caught by
traditional load testing

Find Out How Gomez Can Help You
Ensuring great experiences in a Web 2.0 Some of Our Customers
world requires a new approach to
application load testing
testing.
Load Testing 2.0
• Self-service
• T t outside-in, with drill d
Tests t id i ith d ill down t web
to b
application components
• SaaS with no investment or maintenance
costs and rapid payback
• Full desktop browser testing across
globally distributed geographies
Learn M
L More:
Load Testing 2.0 Whitepaper
http://www.gomez.com/registration/white_paper.
php?id=41
Speak To A Gomez Solutions Expert 26
Toll Free: 877.372.6732