The document discusses how the architecture of web applications is evolving to become more distributed and complex. It notes that web applications now involve many composite components across different servers, locations, and clouds. It also discusses how content is becoming more dynamic and interdependent, and how processing is increasingly being pushed to the client side. The document argues that this new architecture requires a revolution in load testing to properly evaluate performance across the entire delivery chain from data centers to end users.

1. The Evolution of the Architecture of Web
Applications Requires a Load Testing Revolution
Imad Mouline – CTO – Compuware APM
@imadmouline

2. Where The Data Comes From
Synthetic monitoring and load testing scripts from 3,000 enterprises,
across broad number of verticals
 Number of distinct scripts ranges from 45K to 68K, depending on data
mining exercise
 150+ backbone / data center / cloud testing locations, and thousands of
Last Mile / desktop testing locations
Real user monitoring measurements from 200+ sites
 Users from around the world
 Measurement samples range from 117 million to 526 million pages / user
interactions, depending on data mining exercise

3. Observation 1
Web applications are becoming more
composite

4. The Client Is Becoming THE Integration Platform
* Source: Gomez 2010

5. By The Numbers – February 2010
Number of hosts accessed directly by the browser, per user
transaction, averaged across 3,000 companies
Measurement Number of hosts per
city user transaction
Hong Kong 5.51
Beijing 6.91
London 7.46
New York 8.17
Frankfurt 8.66

6. By The Numbers – June 2010
Number of hosts accessed directly by the browser, per user
transaction, averaged across 3,000 companies
Measurement Number of hosts per
city user transaction
Hong Kong 7.56
Beijing 8.57
London 8.59
New York 8.85
Frankfurt 8.87

7. By The Numbers – September 2010
Number of hosts accessed directly by the browser, per user
transaction, averaged across 3,000 companies
Measurement Number of hosts per
city user transaction
Hong Kong 6.82
Beijing 8.87
London 7.95
New York 9.82
Frankfurt 8.71
Paris 10.12
Stockholm 10.48
Helsinki 12.71

8. By The Numbers – November 2010
Number of hosts accessed directly by the browser, per user
transaction, averaged across 3,000 companies
Measurement city Number of hosts per user
transaction
Hong Kong 5.50
Paris 6.27
Amsterdam 6.90
London 7.25
Frankfurt 7.45
Beijing 9.10
Stockholm 9.61
New York 10.50
Helsinki 11.57

9. Observation 1a
Enterprises Are Adopting the Cloud
(with or without their knowledge)

10. Web Applications Are Moving To The Cloud – June 2010
Percentage of web app transactions that include at least one
object hosted on Amazon EC2
Amazon EC2 Region Percentage
EC2 Asia Pacific - Singapore 0.002
EC2 US West - Northern California 0.659
EC2 EU - Ireland 2.733
EC2 US East - Northern Virginia 16.194
TOTAL 19.588

11. Enterprises ARE Adopting Cloud Computing – Nov 2010
Percentage of web app transactions that include at least one
object hosted on Amazon EC2
Amazon EC2 Region Percentage
EC2 Asia Pacific - Singapore 0.151
EC2 EU - Ireland 1.578
EC2 US West - Northern California 2.066
EC2 US East - Northern Virginia 24.144
TOTAL 27.938

12. Observation 2
Content is becoming increasingly
dynamic and distributed

13. Geographic Distribution Of Content Sources
How many cities does content come from to form the average
transaction?
Distribution of host cities by test Distribution of host cities by test
(measured from single location) (measured from multiple locations)
>30
21-30 >30
21-30 8%
4% 2%
7%
1 1
11-20 27%
1
11% 2-5
34% 11-20
6-10 12% 6-10
13% 11-20
6-10 21-30
15% 2-5 >30
31%
2-5
36%
Source: Gomez Active Backbone Monitoring
Sample of 12,000 production monitoring scripts
Multiple runs over 24 hours

14. Observation 3
Content is becoming increasingly
inter-dependent

15. Browser Impact on Response Time
Response times differences across Firefox and IE for a 6-step
transaction
Internet Explorer 7
Firefox 3.5

16. Major Differences In IE And Firefox Waterfall Charts
IE 7 Waterfall Chart Firefox 3.5 Waterfall Chart
Connection 3
Connection 1 Connection 2
Connection 1
Connection 6
Connection 5
Connection 4
Connection 2
Connection 7
Connection 8
Connection 9
Connection 3 Connection 10
Connection 5
Connection 4
Connection 12
Connection 6
Connection 11
Connection 7

17. Performance & Availability Issues Can Be Browser Specific
Issue is 3rd party content
blocking on IE only
Performance issue impacting
Internet Explorer 7
Internet Explorer 7
Firefox 3.5

18. Load Testing with HTTP Playback: Testing from NYC vs. Atlanta
Some of the measurements
are different.
The load order is the
same using HTTP- From
NY or Atlanta.

19. Load Testing with IE Playback: Testing from NYC vs. Atlanta
The load order is
different between
Atlanta and NY

20. Observation 4
Processing is increasingly being
pushed to the client

21. Significant Performance Differences Across Browsers
Load Time Perceived Render
7
6
5
4
3
2
1
0
Source: Gomez Real-User Monitoring (October 2010)
Real users around the world 526 million page measurements
Broadband connections only 200+ sites

22. Browsers Are Evolving To Support Heavier Client-Side Code
HTML5 support
 Application cache canvas,
audio, video, local storage,
geo-location, web workers
etc.
CSS3 Support
 Webfonts, animations,
gradients, shadows, etc.
Performance improvements
 Faster JavaScript processing
 Parallel download of JS scripts
 More parallel connections
 Resource pre-fetching
 Multi-threading in JS
Key Trend - more and more client-side processing

23. RIA Frameworks Adoption
25.18 % of transactions surveyed depend on at least one of these
frameworks
Percentage of transactions that leverage framework
14.00%
12.00%
10.00%
8.00%
6.00%
4.00%
2.00%
0.00%
Source: Gomez Active Backbone Monitoring
~ 3,000 enterprises
48K+ distinct transactions active at least once during a 1 hr time period

24. Observation 5
Mobile Users Are Becoming Less
Patient

25. Web & Mobile Site Performance Impacts User Behavior
Abandonment Rate Across 200+ Sites / 177+ Million Page Views
over 1 week / All Browsers vs. iPhone Safari
30
25
Abandonment Rate (%)
20 Abandonment Rate -
All Browsers
15
Abandonment Rate -
iPhone Safari
10
5
0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Page Load Time Band (sec.)
Source: Gomez real user monitoring

26. Significant Performance Differences Across Browsers & Devices
Load Time Perceived Render
12
10
8
6
4
2
0
Source: Gomez Real-User Monitoring (October 2010)
Real users around the world 526 million page measurements
Broadband & wireless connections only 200+ sites

27. Recap of Observations
1. Web applications are getting more composite
 The browser is becoming the integration platform
of choice
 Cloud-hosted app components are mainstream
2. Content is becoming increasingly dynamic
3. Content is becoming increasingly inter-dependent
4. Processing is moving to the client
5. Mobile users are becoming less patient

28. The Traditional View of Web Application Delivery
Systems …user is
management happy
tools: “OK”
Load Users
Balancers
Web
Servers
Mobile
Components
App
Servers
DB Web application
Servers
Storage
Mainframe
Network
Traditional zone
of control

29. The Reality of Web Application Delivery
Traditional Approach
Slow response time …user is
NOT happy
DB Network Server Geographic disparities
Load Users
Balancers
Transactions fail
4 sec’s
Web
Servers
22 sec’s
Mobile
Components
App
Servers
Faulty display or operation
DB
Servers
Storage
Mainframe
Network
Traditional zone
Traditional zone
of control
of control

30. The Challenge of Managing Application Performance
The Application Delivery Chain
Customers
Local Browsers
Data Center ISP
3rd Party/
Virtual/physical environment Cloud Services
DB App Web Load
Mainframe Servers Servers Servers Balancers
Major
Storage Network ISP
Content
Delivery
Networks
Web Mobile WAN Virtual
Services Components Optimization Desktops
Mobile
Carrier
Devices
Employees Employees
(via WAN)

31. The Challenge of Managing Application Performance
Traditional Approach
…user is
The Application Delivery Chain NOT happy
DB Network Server
Customers
Local Browsers
Data Center • Inconsistent geo performance ISP
• Bad performance under load • Network peering
3rd Party/ • Poorly
Virtual/physical environment delivery
• Blocking content problems
Cloud Services performing
DB • Incorrect geo-targeted Load
App Web content • Bandwidth JavaScript
Mainframe Servers Servers Servers Balancers throttling • Browser/device
• Inconsistent incompatibility
• Configuration • Network peering connectivity • Page size
errors problems too big
• Application • Outages Major • Too many
Storage Network
design issues ISP • Network resource objects
• Code defects shortage
Content • Low cache
• Insufficient • Delivery
Faulty content hit rate
infrastructureWAN Networks
transcoding
Web Mobile Virtual
Services Components Optimization Desktops • SMS routing /
Mobile
latency issues
• Configuration issues Carrier
• Oversubscribed POP Devices
Employees
• Poor routing optimization Employees
(via WAN)
• Low cache hit rate

32. Simplification of the Problem is Key
The Application Delivery Chain
Customers
Local Browsers
Data Center ISP
3rd Party/
Virtual/physical environment Cloud Services
DB App Web Load
Mainframe Servers Servers Servers Balancers
Major
Storage Network ISP
Content
Delivery
Networks
Web Mobile WAN Virtual
Services Components Optimization Desktops
Mobile
Carrier
Devices
Employees Employees
(via WAN)

33. Test Across the Entire Web Application Delivery Chain
The Application Delivery Chain
Load Testing 2.0
Load Testing 1.5
Load Testing 1.0 Customers
Local Browsers
Data Center ISP
3rd Party/
Virtual/physical environment Cloud Services
DB App Web Load
Mainframe Servers Servers Servers Balancers
Major
Storage Network ISP
Content
Delivery
Networks
Web Mobile WAN Virtual
Services Components Optimization Desktops
Mobile
Carrier
Devices
Employees Employees
(via WAN)

34. Load Testing 1.0 Works For Specific Situations
Company: Online presence for a popular TV show
• Following episodes of the TV show the web site sees high traffic spikes
• Goal was to achieve 1500 logins per minute
• Load tested DB to improve performance in anticipation of another traffic spike
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Mobile
Components
App
Servers Internet
DB Major
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

35. Application Bottleneck Causes Response Time Issue
• As users were added, the
response time of step 3 (the
login) climbed immediately
• The test bottlenecked at 160
logins per minute (Goal 1500)
• But quickly dropped off as
users received server errors
• New login query was not
optimized and was
bottlenecking the
database servers’ CPUs

36. Application Bottleneck – Re-test
•After tuning- application performance
improved.
•New Bottleneck occurred 1300 logins
per minute.
•Bandwidth limit reached at 90 Mbps
Summary:
•Problem found inside firewall
•Fixes made for application issue
•Retest shows second issue-bandwidth
1.0 1.5 2.0
•First test  
  
•Second test
1.0 1.5 2.0
 
 

37. Load Testing from the Cloud misses end-user perspective
Company: Online Gaming Site
Testing a new rollout in support of a new sports season
• Support anticipated traffic increases
• Load tested system using cloud and Last Mile to validate performance
for real users in new geographies.
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Mobile
Components
App
Servers Internet
DB Major
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

38. View from the Cloud
• First 20 minutes Cloud testing
shows acceptable performance
• After 2500 users, Response time
climbs, Availability drops, Error
rate climbs

39. View from the Last Mile
• Last Mile shows
different story
• Availability is terrible
even at minimal load
for real users
Summary:
Cloud-only testing may give
misleading availability data
Cloud starts with 100%
availability
Less than 25% for the Last Mile
1.0 1.5 2.0


40. Load testing 1.0 and 1.5 miss regional issues
Company: Regional Online News Source
• Began testing for the election season
• Goal was to validate overall performance focusing in 2 key regions
3rd Party/ Browsers and
Load Cloud Services Local ISP devices Users
Balancers
Web
Servers
Mobile
Components
App Internet
Servers
Major
DB ISP
Servers
Storage
Mainframe
Content Delivery Mobile
Network Networks Carrier

41. No Performance Issues Detected From Data Center
Increase and hold load and not exceed response
times of 4 seconds and Success Rate of 99%
There was only 1 page error and 11
errors total out of 60000+ transactions
Page response times stayed
under 4 seconds, outside of
one brief blip
1.0 or 1.5 load testing shows tests passed

42. Last Mile Case Study: Primary Geographies
Key geographies for this customer are
New York and Pennsylvania.
The response time
never met the 4
second average goal
Summary: Availability
was Less
Last Mile shows goal not than 99%
reached
Cloud can’t detect the end
user issue
1.0 1.5 2.0


43. The Internet is global – where your customers are matters
Company: International Hotel chain
• New reservations system rollout
• New global server load balancing rolled out across multiple data
centers
• Validate that system works globally
3rd Party/ Browsers and
Load Cloud Services Local ISP devices Users
Balancers
Web
Servers
Mobile
Components
App Internet
Servers
Major
DB ISP
Servers
Storage
Mainframe
Content Delivery Mobile
Network Networks Carrier

44. Major Hotel Reservation System unavailable in 4 countries
0% availability in UK, Germany, Japan
99%+ availability in US, Canada, France
Summary:
Internal U.S. test looked good
Distributed testing fails in key
locations.
1.0 1.5 2.0
? 

45. Load Testing 2.0 shows you what your customer will see
Company: eRetailer fashion
• 100% virtual store
• Daily sales spike driving 90% of revenue stream
3rd Party/ Browsers and
Load Cloud Services Local ISP devices Users
Balancers
Web
Servers
Mobile
Components
App Internet
Servers
Major
DB ISP
Servers
Storage
Mainframe
Content Delivery Mobile
Network Networks Carrier

46. Load Testing with multiple browsers shows discrepancies
Availability vastly different between browsers

47. Comparison of Performance across the country - Firefox
Using Firefox browser – shows 100% availability for website
Wide variations in response time based on geography

48. Comparison of Performance across the country – IE
IE Browser : shows under 12 percent availability
Availability and performance tied to geography

49. Page Element Downloads: IE Versus Firefox- Order Varies
Root Cause: Summary:
•Third party ad provider Only full browsers in target
modifying the DOM locations can show what really
happens.
•Depending on the load order
of the third party the JavaScript 1.0 1.5 2.0
in the ad would overwrite the
DOM but only on IE 

50. Load Testing Approaches : Which one is best for you?
Load Test 1.0 Load Test 1.5 Load Test 2.0
HTTP : Behind the HTTP : Data Centers Browser : Data Real World Desktops
Firewall Centers Last Mile
Traditional
Client/ Datacenter Testing
Server Test
Accuracy of End-User Incomplete Incomplete Indicative Most Accurate
Response Time
Accuracy of Invalid Indicative Indicative Most Accurate
Application Availability
Ability to drive large Yes-requires Best Better Good
load volume substantial
hardware
Understand CDN No Misleading Misleading Most
Impact Accurate
Understand 3rd Party No Minimal Some Most
(ads, feeds, etc…) Accurate
Realistic object No No Yes Yes
download Static Only
Visibility behind the Best Good Good Good
firewall

51. So what? Now what?
Broaden the definition of your web application
Test early, test often
Know your end-users, and test from their
perspective
Pay attention to all 4 buckets:
 Data Center, Internet, 3rd Parties, Client
Look for the breaking point of the end-user
experience, not just the breaking point of the
application infrastructure

52. See you at CMG'11
Dec 5th - 9th, 2011
Washington, DC