Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Evolution of the Architecture of WebApplications Requires a Load Testing Revolution               Imad Mouline – CTO –...
Where The Data Comes From Synthetic monitoring and load testing scripts from 3,000 enterprises, across broad number of ver...
Observation 1  Web applications are becoming more              composite
The Client Is Becoming THE Integration Platform * Source: Gomez 2010
By The Numbers – February 2010 Number of hosts accessed directly by the browser, per user transaction, averaged across 3,0...
By The Numbers – June 2010 Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 c...
By The Numbers – September 2010 Number of hosts accessed directly by the browser, per user transaction, averaged across 3,...
By The Numbers – November 2010 Number of hosts accessed directly by the browser, per user transaction, averaged across 3,0...
Observation 1a    Enterprises Are Adopting the Cloud            (with or without their knowledge)
Web Applications Are Moving To The Cloud – June 2010   Percentage of web app transactions that include at least one       ...
Enterprises ARE Adopting Cloud Computing – Nov 2010   Percentage of web app transactions that include at least one        ...
Observation 2     Content is becoming increasingly        dynamic and distributed
Geographic Distribution Of Content Sources     How many cities does content come from to form the average     transaction?...
Observation 3     Content is becoming increasingly            inter-dependent
Browser Impact on Response TimeResponse times differences across Firefox and IE for a 6-steptransaction                   ...
Major Differences In IE And Firefox Waterfall ChartsIE 7 Waterfall Chart              Firefox 3.5 Waterfall Chart         ...
Performance & Availability Issues Can Be Browser Specific                                    Issue is 3rd party content   ...
Load Testing with HTTP Playback: Testing from NYC vs. Atlanta                                       Some of the measuremen...
Load Testing with IE Playback: Testing from NYC vs. Atlanta                   The load order is                   differen...
Observation 4       Processing is increasingly being            pushed to the client
Significant Performance Differences Across Browsers                                     Load Time     Perceived Render   7...
Browsers Are Evolving To Support Heavier Client-Side CodeHTML5 support   Application cache canvas,    audio, video, local...
RIA Frameworks Adoption     25.18 % of transactions surveyed depend on at least one of these     frameworks               ...
Observation 5      Mobile Users Are Becoming Less                 Patient
Web & Mobile Site Performance Impacts User Behavior                                    Abandonment Rate Across 200+ Sites ...
Significant Performance Differences Across Browsers & Devices                                           Load Time     Perc...
Recap of Observations1. Web applications are getting more composite    The browser is becoming the integration platform  ...
The Traditional View of Web Application Delivery    Systems                                         …user ismanagement    ...
The Reality of Web Application Delivery    Traditional Approach                             Slow response time            ...
The Challenge of Managing Application Performance                                          The Application Delivery Chain ...
The Challenge of Managing Application Performance       Traditional Approach                                              ...
Simplification of the Problem is Key                                          The Application Delivery Chain              ...
Test Across the Entire Web Application Delivery Chain                                          The Application Delivery Ch...
Load Testing 1.0 Works For Specific Situations    Company: Online presence for a popular TV show          • Following epis...
Application Bottleneck Causes Response Time Issue                            • As users were added, the                   ...
Application Bottleneck – Re-test                                   •After tuning- application performance                 ...
Load Testing from the Cloud misses end-user perspective  Company: Online Gaming Site  Testing a new rollout in support of ...
View from the Cloud                      • First 20 minutes Cloud testing                          shows acceptable perfor...
View from the Last Mile                                  • Last Mile shows                                      different ...
Load testing 1.0 and 1.5 miss regional issues   Company: Regional Online News Source    • Began testing for the election s...
No Performance Issues Detected From Data Center Increase and hold load and not exceed response times of 4 seconds and Succ...
Last Mile Case Study: Primary Geographies    Key geographies for this customer are    New York and Pennsylvania.          ...
The Internet is global – where your customers are mattersCompany: International Hotel chain • New reservations system roll...
Major Hotel Reservation System unavailable in 4 countries        0% availability in UK, Germany, Japan        99%+ availab...
Load Testing 2.0 shows you what your customer will see  Company: eRetailer fashion   • 100% virtual store   • Daily sales ...
Load Testing with multiple browsers shows discrepanciesAvailability vastly different between browsers
Comparison of Performance across the country - FirefoxUsing Firefox browser – shows 100% availability for websiteWide vari...
Comparison of Performance across the country – IE            IE Browser : shows under 12 percent availability            A...
Page Element Downloads: IE Versus Firefox- Order VariesRoot Cause:                         Summary:•Third party ad provide...
Load Testing Approaches : Which one is best for you?                               Load Test 1.0                Load Test ...
So what? Now what? Broaden the definition of your web application Test early, test often Know your end-users, and test fro...
See you at CMG11Dec 5th - 9th, 2011 Washington, DC
Upcoming SlideShare
Loading in …5
×

The Evolution of the Architecture of Web Applications Requires a Load Testing Revolution

2,514 views

Published on

Long-held best practices of n-tier application architecture are being challenged. Old design patterns are new again. This session discusses the architecture evolution of Web applications, load testing approaches, and the future.

Published in: Business
  • Be the first to comment

  • Be the first to like this

The Evolution of the Architecture of Web Applications Requires a Load Testing Revolution

  1. 1. The Evolution of the Architecture of WebApplications Requires a Load Testing Revolution Imad Mouline – CTO – Compuware APM @imadmouline
  2. 2. Where The Data Comes From Synthetic monitoring and load testing scripts from 3,000 enterprises, across broad number of verticals  Number of distinct scripts ranges from 45K to 68K, depending on data mining exercise  150+ backbone / data center / cloud testing locations, and thousands of Last Mile / desktop testing locations Real user monitoring measurements from 200+ sites  Users from around the world  Measurement samples range from 117 million to 526 million pages / user interactions, depending on data mining exercise
  3. 3. Observation 1 Web applications are becoming more composite
  4. 4. The Client Is Becoming THE Integration Platform * Source: Gomez 2010
  5. 5. By The Numbers – February 2010 Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 companies Measurement Number of hosts per city user transaction Hong Kong 5.51 Beijing 6.91 London 7.46 New York 8.17 Frankfurt 8.66
  6. 6. By The Numbers – June 2010 Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 companies Measurement Number of hosts per city user transaction Hong Kong 7.56 Beijing 8.57 London 8.59 New York 8.85 Frankfurt 8.87
  7. 7. By The Numbers – September 2010 Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 companies Measurement Number of hosts per city user transaction Hong Kong 6.82 Beijing 8.87 London 7.95 New York 9.82 Frankfurt 8.71 Paris 10.12 Stockholm 10.48 Helsinki 12.71
  8. 8. By The Numbers – November 2010 Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 companies Measurement city Number of hosts per user transaction Hong Kong 5.50 Paris 6.27 Amsterdam 6.90 London 7.25 Frankfurt 7.45 Beijing 9.10 Stockholm 9.61 New York 10.50 Helsinki 11.57
  9. 9. Observation 1a Enterprises Are Adopting the Cloud (with or without their knowledge)
  10. 10. Web Applications Are Moving To The Cloud – June 2010 Percentage of web app transactions that include at least one object hosted on Amazon EC2Amazon EC2 Region PercentageEC2 Asia Pacific - Singapore 0.002EC2 US West - Northern California 0.659EC2 EU - Ireland 2.733EC2 US East - Northern Virginia 16.194 TOTAL 19.588
  11. 11. Enterprises ARE Adopting Cloud Computing – Nov 2010 Percentage of web app transactions that include at least one object hosted on Amazon EC2Amazon EC2 Region PercentageEC2 Asia Pacific - Singapore 0.151EC2 EU - Ireland 1.578EC2 US West - Northern California 2.066EC2 US East - Northern Virginia 24.144 TOTAL 27.938
  12. 12. Observation 2 Content is becoming increasingly dynamic and distributed
  13. 13. Geographic Distribution Of Content Sources How many cities does content come from to form the average transaction? Distribution of host cities by test Distribution of host cities by test (measured from single location) (measured from multiple locations) >30 21-30 >30 21-30 8% 4% 2% 7% 1 1 11-20 27% 1 11% 2-5 34% 11-20 6-10 12% 6-10 13% 11-20 6-10 21-30 15% 2-5 >30 31% 2-5 36%Source: Gomez Active Backbone Monitoring Sample of 12,000 production monitoring scripts Multiple runs over 24 hours
  14. 14. Observation 3 Content is becoming increasingly inter-dependent
  15. 15. Browser Impact on Response TimeResponse times differences across Firefox and IE for a 6-steptransaction Internet Explorer 7 Firefox 3.5
  16. 16. Major Differences In IE And Firefox Waterfall ChartsIE 7 Waterfall Chart Firefox 3.5 Waterfall Chart Connection 3 Connection 1 Connection 2 Connection 1 Connection 6 Connection 5 Connection 4 Connection 2 Connection 7 Connection 8 Connection 9 Connection 3 Connection 10 Connection 5 Connection 4 Connection 12 Connection 6 Connection 11 Connection 7
  17. 17. Performance & Availability Issues Can Be Browser Specific Issue is 3rd party content blocking on IE onlyPerformance issue impacting Internet Explorer 7 Internet Explorer 7 Firefox 3.5
  18. 18. Load Testing with HTTP Playback: Testing from NYC vs. Atlanta Some of the measurements are different. The load order is the same using HTTP- From NY or Atlanta.
  19. 19. Load Testing with IE Playback: Testing from NYC vs. Atlanta The load order is different between Atlanta and NY
  20. 20. Observation 4 Processing is increasingly being pushed to the client
  21. 21. Significant Performance Differences Across Browsers Load Time Perceived Render 7 6 5 4 3 2 1 0Source: Gomez Real-User Monitoring (October 2010) Real users around the world 526 million page measurements Broadband connections only 200+ sites
  22. 22. Browsers Are Evolving To Support Heavier Client-Side CodeHTML5 support  Application cache canvas, audio, video, local storage, geo-location, web workers etc.CSS3 Support  Webfonts, animations, gradients, shadows, etc.Performance improvements  Faster JavaScript processing  Parallel download of JS scripts  More parallel connections  Resource pre-fetching  Multi-threading in JS Key Trend - more and more client-side processing
  23. 23. RIA Frameworks Adoption 25.18 % of transactions surveyed depend on at least one of these frameworks Percentage of transactions that leverage framework 14.00% 12.00% 10.00% 8.00% 6.00% 4.00% 2.00% 0.00%Source: Gomez Active Backbone Monitoring ~ 3,000 enterprises 48K+ distinct transactions active at least once during a 1 hr time period
  24. 24. Observation 5 Mobile Users Are Becoming Less Patient
  25. 25. Web & Mobile Site Performance Impacts User Behavior Abandonment Rate Across 200+ Sites / 177+ Million Page Views over 1 week / All Browsers vs. iPhone Safari 30 25Abandonment Rate (%) 20 Abandonment Rate - All Browsers 15 Abandonment Rate - iPhone Safari 10 5 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Page Load Time Band (sec.)Source: Gomez real user monitoring
  26. 26. Significant Performance Differences Across Browsers & Devices Load Time Perceived Render 12 10 8 6 4 2 0Source: Gomez Real-User Monitoring (October 2010) Real users around the world 526 million page measurements Broadband & wireless connections only 200+ sites
  27. 27. Recap of Observations1. Web applications are getting more composite  The browser is becoming the integration platform of choice  Cloud-hosted app components are mainstream2. Content is becoming increasingly dynamic3. Content is becoming increasingly inter-dependent4. Processing is moving to the client5. Mobile users are becoming less patient
  28. 28. The Traditional View of Web Application Delivery Systems …user ismanagement happy tools: “OK”Load UsersBalancersWebServersMobileComponentsAppServersDB Web applicationServersStorageMainframeNetwork Traditional zone of control
  29. 29. The Reality of Web Application Delivery Traditional Approach Slow response time …user is NOT happy DB Network Server Geographic disparitiesLoad UsersBalancers Transactions fail 4 sec’sWebServers 22 sec’sMobileComponentsAppServers Faulty display or operationDBServersStorageMainframeNetwork Traditional zone Traditional zone of control of control
  30. 30. The Challenge of Managing Application Performance The Application Delivery Chain Customers Local Browsers Data Center ISP 3rd Party/ Virtual/physical environment Cloud Services DB App Web LoadMainframe Servers Servers Servers Balancers Major Storage Network ISP Content Delivery Networks Web Mobile WAN VirtualServices Components Optimization Desktops Mobile Carrier Devices Employees Employees (via WAN)
  31. 31. The Challenge of Managing Application Performance Traditional Approach …user is The Application Delivery Chain NOT happy DB Network Server Customers Local Browsers Data Center • Inconsistent geo performance ISP • Bad performance under load • Network peering 3rd Party/ • Poorly Virtual/physical environment delivery • Blocking content problems Cloud Services performing DB • Incorrect geo-targeted Load App Web content • Bandwidth JavaScriptMainframe Servers Servers Servers Balancers throttling • Browser/device • Inconsistent incompatibility • Configuration • Network peering connectivity • Page size errors problems too big • Application • Outages Major • Too many Storage Network design issues ISP • Network resource objects • Code defects shortage Content • Low cache • Insufficient • Delivery Faulty content hit rate infrastructureWAN Networks transcoding Web Mobile VirtualServices Components Optimization Desktops • SMS routing / Mobile latency issues • Configuration issues Carrier • Oversubscribed POP Devices Employees • Poor routing optimization Employees (via WAN) • Low cache hit rate
  32. 32. Simplification of the Problem is Key The Application Delivery Chain Customers Local Browsers Data Center ISP 3rd Party/ Virtual/physical environment Cloud Services DB App Web LoadMainframe Servers Servers Servers Balancers Major Storage Network ISP Content Delivery Networks Web Mobile WAN VirtualServices Components Optimization Desktops Mobile Carrier Devices Employees Employees (via WAN)
  33. 33. Test Across the Entire Web Application Delivery Chain The Application Delivery Chain Load Testing 2.0 Load Testing 1.5 Load Testing 1.0 Customers Local Browsers Data Center ISP 3rd Party/ Virtual/physical environment Cloud Services DB App Web LoadMainframe Servers Servers Servers Balancers Major Storage Network ISP Content Delivery Networks Web Mobile WAN VirtualServices Components Optimization Desktops Mobile Carrier Devices Employees Employees (via WAN)
  34. 34. Load Testing 1.0 Works For Specific Situations Company: Online presence for a popular TV show • Following episodes of the TV show the web site sees high traffic spikes • Goal was to achieve 1500 logins per minute • Load tested DB to improve performance in anticipation of another traffic spike 3rd Party/ BrowsersLoad Cloud Services Local ISP and devices UsersBalancersWebServersMobileComponentsAppServers InternetDB MajorServers ISPStorageMainframeNetwork Content Delivery Mobile Networks Carrier
  35. 35. Application Bottleneck Causes Response Time Issue • As users were added, the response time of step 3 (the login) climbed immediately • The test bottlenecked at 160 logins per minute (Goal 1500) • But quickly dropped off as users received server errors • New login query was not optimized and was bottlenecking the database servers’ CPUs
  36. 36. Application Bottleneck – Re-test •After tuning- application performance improved. •New Bottleneck occurred 1300 logins per minute. •Bandwidth limit reached at 90 Mbps Summary: •Problem found inside firewall •Fixes made for application issue •Retest shows second issue-bandwidth 1.0 1.5 2.0 •First test      •Second test 1.0 1.5 2.0    
  37. 37. Load Testing from the Cloud misses end-user perspective Company: Online Gaming Site Testing a new rollout in support of a new sports season • Support anticipated traffic increases • Load tested system using cloud and Last Mile to validate performance for real users in new geographies. 3rd Party/ Browsers Load Cloud Services Local ISP and devices Users Balancers Web Servers Mobile Components App Servers Internet DB Major Servers ISP Storage Mainframe Network Content Delivery Mobile Networks Carrier
  38. 38. View from the Cloud • First 20 minutes Cloud testing shows acceptable performance • After 2500 users, Response time climbs, Availability drops, Error rate climbs
  39. 39. View from the Last Mile • Last Mile shows different story • Availability is terrible even at minimal load for real users Summary: Cloud-only testing may give misleading availability data Cloud starts with 100% availability Less than 25% for the Last Mile 1.0 1.5 2.0 
  40. 40. Load testing 1.0 and 1.5 miss regional issues Company: Regional Online News Source • Began testing for the election season • Goal was to validate overall performance focusing in 2 key regions 3rd Party/ Browsers and Load Cloud Services Local ISP devices Users Balancers Web Servers Mobile Components App Internet Servers Major DB ISP Servers Storage Mainframe Content Delivery Mobile Network Networks Carrier
  41. 41. No Performance Issues Detected From Data Center Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99% There was only 1 page error and 11 errors total out of 60000+ transactions Page response times stayed under 4 seconds, outside of one brief blip 1.0 or 1.5 load testing shows tests passed
  42. 42. Last Mile Case Study: Primary Geographies Key geographies for this customer are New York and Pennsylvania. The response time never met the 4 second average goal Summary: Availability was Less Last Mile shows goal not than 99% reached Cloud can’t detect the end user issue 1.0 1.5 2.0 
  43. 43. The Internet is global – where your customers are mattersCompany: International Hotel chain • New reservations system rollout • New global server load balancing rolled out across multiple data centers • Validate that system works globally 3rd Party/ Browsers and Load Cloud Services Local ISP devices Users Balancers Web Servers Mobile Components App Internet Servers Major DB ISP Servers Storage Mainframe Content Delivery Mobile Network Networks Carrier
  44. 44. Major Hotel Reservation System unavailable in 4 countries 0% availability in UK, Germany, Japan 99%+ availability in US, Canada, France Summary: Internal U.S. test looked good Distributed testing fails in key locations. 1.0 1.5 2.0 ? 
  45. 45. Load Testing 2.0 shows you what your customer will see Company: eRetailer fashion • 100% virtual store • Daily sales spike driving 90% of revenue stream 3rd Party/ Browsers and Load Cloud Services Local ISP devices Users Balancers Web Servers Mobile Components App Internet Servers Major DB ISP Servers Storage Mainframe Content Delivery Mobile Network Networks Carrier
  46. 46. Load Testing with multiple browsers shows discrepanciesAvailability vastly different between browsers
  47. 47. Comparison of Performance across the country - FirefoxUsing Firefox browser – shows 100% availability for websiteWide variations in response time based on geography
  48. 48. Comparison of Performance across the country – IE IE Browser : shows under 12 percent availability Availability and performance tied to geography
  49. 49. Page Element Downloads: IE Versus Firefox- Order VariesRoot Cause: Summary:•Third party ad provider Only full browsers in targetmodifying the DOM locations can show what really happens.•Depending on the load orderof the third party the JavaScript 1.0 1.5 2.0in the ad would overwrite theDOM but only on IE 
  50. 50. Load Testing Approaches : Which one is best for you? Load Test 1.0 Load Test 1.5 Load Test 2.0 HTTP : Behind the HTTP : Data Centers Browser : Data Real World Desktops Firewall Centers Last Mile Traditional Client/ Datacenter Testing Server TestAccuracy of End-User Incomplete Incomplete Indicative Most AccurateResponse TimeAccuracy of Invalid Indicative Indicative Most AccurateApplication AvailabilityAbility to drive large Yes-requires Best Better Goodload volume substantial hardwareUnderstand CDN No Misleading Misleading MostImpact AccurateUnderstand 3rd Party No Minimal Some Most(ads, feeds, etc…) AccurateRealistic object No No Yes Yesdownload Static OnlyVisibility behind the Best Good Good Goodfirewall
  51. 51. So what? Now what? Broaden the definition of your web application Test early, test often Know your end-users, and test from their perspective Pay attention to all 4 buckets:  Data Center, Internet, 3rd Parties, Client Look for the breaking point of the end-user experience, not just the breaking point of the application infrastructure
  52. 52. See you at CMG11Dec 5th - 9th, 2011 Washington, DC

×