You might think that with web applications in the cloud, that load testing from the cloud provides all the testing you need. You might think that testing from the cloud can tell you if your website can handle peak traffic loads, driven by marketing campaigns, or seasonal events. Unfortunately you may be wrong. In the Web 2.0 world; applications are combined on the fly inside the browser, from third-party and shared services both in the cloud and from behind the firewall. Imad Mouline, CTO of Gomez will tell you — the cloud is not the answer. Join Imad Mouline, on Wednesday September 8th for this provocative session around today’s highly complex, distributed Web applications and how to test them. Imad is a veteran of software architecture, research & development and an expert in Web application development, testing and performance management. In this session, Mouline will discuss: * The evolution architecture and structure of Web applications * The current state of load testing approaches and how they apply to a variety of architectures * How existing and emerging testing techniques are applied to different types of applications * The future architecture of Web applications and what it means to the future of testing

1. Why Load Testing your Website from the Cloud
Doesn’t Work
Imad Mouline - CTO, Compuware Gomez

2. Customer Expectations Are Rising For Web & Mobile Applications
47% of consumers expect a Web 58% of mobile phone
page to load in 2 seconds or less users expect websites to
load almost as quickly or
faster on their mobile
phone than their PC

3. The Impact of Web Performance on Page Abandonment
10
Performance
improvement 5
(seconds) 6 secs 8 secs
2 secs 4 secs
0
-5 -8%
-10
-25%
-15
Percentage -33%
change in -20 -38%
page
abandonment -25
-30
-35
-40
Source: Gomez 2010 Study of 500 Million End-User Interactions Across 200+ Web Sites

4. The Browser Is Becoming THE Integration Platform
Number of hosts accessed directly by the browser,
per user transaction, averaged across 3,000 companies 8.87

5. The Challenge Of Delivering Quality Web Experiences
Systems …user is
management NOT happy
tools: “OK”
The Web Application Delivery Chain
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
• Inconsistent geo performance • Poorly
Web • Bad performance under load • Network peering performing
Servers • Blocking content delivery problems JavaScript
• Incorrect geo-targeted content • Bandwidth • Inconsistent
Mobile throttling CSS rendering
Components • Inconsistent • Browser/device
• Configuration • Network peering connectivity incompatibility
App errors
Servers problems • Page size too
• Application • Outages Internet big
DB design issues Major • Network resource • Conflicting
Servers • Code defects shortage HTML tag
ISP • Faulty content support
• Insufficient
Storage infrastructure transcoding • Too many
• SMS routing / objects
Mainframe latency issues • Content not
• Configuration issues optimized for
Network • Oversubscribed POP Content Delivery Mobile
Networks Carrier device
• Poor routing optimization • Low cache hit
• Low cache hit rate rate

6. Pinpointing Problems and Optimizing Performance
The Web Application Delivery Chain
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Is it my
Mobile
Components Is it an Is it a Is it a
App
Servers
data ISP or the 3rd party
Internet
browser or
DB center? Internet?
Major provider? device?
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

7. Test Across the Entire Web Application Delivery Chain
The Web Application Delivery Chain
Load Testing 2.0 (Gomez)
Load Testing 1.5
Load Testing 1.0
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Mobile
Components
App
Servers Internet
DB Major
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

8. Load Testing 1.0 Works… Some of the Time
Company: Online presence for a popular TV show
• Following episodes of the TV show the web site sees high traffic spikes
• Goal was to achieve 1500 logins per minute
• Load tested DB to improve performance in anticipation of another traffic spike
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Mobile
Components
App
Servers Internet
DB Major
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

9. Application Bottleneck Causes Response Time Issue
• As users were added, the
response time of step 3 (the
login) climbed immediately
• The test bottlenecked at 160
logins per minute (Goal 1500)
• But quickly dropped off as
users received server errors
• New login query was not
optimized and was
bottlenecking the database
servers’ CPUs

10. Application Bottleneck – Re-test
•After tuning- application performance
improved.
•New Bottleneck occurred 1300 logins
per minute.
•Bandwidth limit reached at 90 Mbps
Summary:
•Problem found inside firewall
•Fixes made for application issue
•Retest shows second issue-bandwidth
1.0 1.5 2.0
•First test
•Second test
1.0 1.5 2.0

11. Load Testing from the cloud misses the point… end users matter
Company: Online Gaming Site
Testing a new rollout in support of a new sports season
• Support anticipated traffic increases
• Load tested something using cloud and Last Mile to validate
performance for real users in new geographies.
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Mobile
Components
App
Servers Internet
DB Major
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

12. View from the Cloud
• First 20 minutes Cloud testing
shows acceptable performance
• After 2500 users Response time
climbs Availability drops Error
rate climbs

13. View from the Last Mile
• Last Mile shows
different story
• Availability is terrible
even at minimal load
for real users
Summary:
Cloud-only testing may give
misleading availability data
Cloud starts with 100%
availability
Less than 25% for the Last
Mile
1.0 1.5 2.0

14. Difference in browsers
• Varied commercial browsers
show different response
times
• Response time change as a
function of load is
dependent on the browser.
Summary:
Understanding response
time under load requires
the use of multiple
browsers
1.0 1.5 2.0

15. Load testing from the cloud ignores real locations
Company: Regional Online News Source
• Began testing for the election season
• Goal was to validate overall performance focusing in 2 key regions
3rd Party/ Browsers and
Load Cloud Services Local ISP devices Users
Balancers
Web
Servers
Mobile
Components
App Internet
Servers
Major
DB ISP
Servers
Storage
Mainframe
Content Delivery Mobile
Network Networks Carrier

16. No Performance Issues Detected From Data-Center
Increase and hold load and not exceed response
times of 4 seconds and Success Rate of 99%
There was only 1 page error and 11
errors total out of 60000+ transactions
Page response times stayed
under 4 seconds, outside of
one brief blip
1.0 or 1.5 load testing shows tests passed

17. Last Mile Case Study: Primary Geographies
Key geographies for this customer are
New York and Pennsylvania.
The response time
never met the 4
second average goal
Summary: Availability
was Less
Last Mile shows goal not than 99%
reached
Cloud can’t detect the end
user issue
1.0 1.5 2.0

18. Don’t trust third parties to perform…test the whole WADC
Company: Online Retailer
• Several 3rd Parties now involved in serving up key content
• Goal was to validate performance of entire application
3rd Party/ Browsers and
Load Cloud Services Local ISP devices Users
Balancers
Web
Servers
Mobile
Components
App Internet
Servers
Major
DB ISP
Servers
Storage
Mainframe
Content Delivery Mobile
Network Networks Carrier

19. Understand 3rd Party Performance Under Load
Load Test Response Time Per Page During the course of a
Load Test, Page 3 is
identified as the culprit
Page Level Drill Down shows performance by Host
Summary: Host & Object Level
Trending
3rd party vendor did not • As the Load test
have capacity
progressed, the 3rd
Only component to fail party search tool
contributed to over
90% of the response
1.0 1.5 2.0 time

20. The Internet is global – where your customers are matters
Company: International Hotel chain
• New reservations system rollout
• New global server load balancing rolled out across multiple data
centers
• Validate that system works globally
3rd Party/ Browsers and
Load Cloud Services Local ISP devices Users
Balancers
Web
Servers
Mobile
Components
App Internet
Servers
Major
DB ISP
Servers
Storage
Mainframe
Content Delivery Mobile
Network Networks Carrier

21. Major Hotel Reservation System unavailable in 4 countries
0% availability in UK, Germany, Japan
99%+ availability in US, Canada, France
Summary:
Internal U.S. test looked good
Distributed testing fails in key
locations.
1.0 1.5 2.0
?

22. Load Testing 2.0 shows you what your customer sees
Company: eRetailer fashion
• 100% virtual store
• Daily sales spike driving 90% of revenue stream
3rd Party/ Browsers and
Load Cloud Services Local ISP devices Users
Balancers
Web
Servers
Mobile
Components
App Internet
Servers
Major
DB ISP
Servers
Storage
Mainframe
Content Delivery Mobile
Network Networks Carrier

23. Load Testing with multiple browsers shows discrepancies
Availability vastly different between browsers

24. Comparison of Performance across the country - Firefox
Using Firefox browser – shows 100% availability for website
Wide variations in response time based on geography

25. Comparison of Performance across the country – IE
IE Browser : shows under 12 percent availability
Availability and performance tied to geography

26. Page Element Downloads: IE Versus Firefox- Order Varies
Explanation Summary:
•Third party ad provider Only real locations can show
modifying the DOM what really happens.
•Depending on the load order
of the third party the java 1.0 1.5 2.0
script in the ad would
overwrite the DOM but only
on IE

27. Gomez Load Testing: Most Accurate Load Test for User Experience
Load Test 1.0 Load Test 1.5 Load Test 2.0
HTTP : Behind the HTTP : Data Centers Browser : Data Real World
Firewall Centers Desktops
Last Mile
Traditional
Client/ Datacenter Testing
Server Test
Only Gomez Spans
Accuracy of End-User Incomplete Incomplete Indicative Most Accurate
Response Time
Accuracy of Invalid Indicative Indicative Most Accurate
Application Availability
Ability to drive large Yes-requires Best Better Good
load volume substantial
hardware
Understand CDN No Misleading Misleading Most
Impact Accurate
Understand 3rd Party No Minimal Some Most
(ads, feeds, etc…) Accurate
Realistic object No No Yes Yes
download Static Only
Visibility behind the Best Good Good Good
firewall

28. Questions
Gomez Customers Enjoy
Measurable Benefits
Increased conversions 10%
Reduced homepage load time from
11.3 seconds to 3.4 seconds
Improved page load times 23%
Saved 50%+ in staff and fees
Reduced seven-step
transaction time by 50%
Reduced downtime 45%
Achieved under 3 second
response time and 99%+
availability
Validated decision to consolidate
three data centers
For more information visit Gomez.com or contact us at +1 781.778.2700