Best Practices for Identifying Web Performance Issues Before Your Customers DoA Gomez/Dollar Thrifty Web Performance Testi...
Agenda<br />Reality Load Overview – Imad Mouline<br />Dollar Thrifty Case Study – Jim Arrowood<br />Product Demo – Emanuel...
The Challenge of Delivering Quality Web Experiences<br />Traditional testing: “OK”<br />…user is NOT happy<br />The Web Ap...
The Web Application Delivery Chain<br /><ul><li>Inconsistent geo performance
Bad performance under load
Blocking content delivery
Incorrect geo-targeted content
Network peering problems
Bandwidth throttling
Inconsistent connectivity
Poorly performing JavaScript
Browser/device incompatibility
Page size too big
Too many objects
Low cache hit rate
Configuration errors
Application design issues
Code defects
Insufficient infrastructure
Network peering problems
Outages
Network resource shortage
Faulty content transcoding
SMS routing / latency issues
Configuration issues
Oversubscribed POP
Poor routing optimization
Low cache hit rate</li></ul>The Challenge of Ensuring Quality Web Experiences<br />Traditional testing: “OK”<br />…user is...
Gomez Load Testing: On-Demand Realistic Load Testing from Browser to Data Center<br />Backbone<br />Last Mile<br />       ...
Gomez Network: The World’s Most Comprehensive Performance and Testing Network <br />Backbone<br />Virtual Test Bed<br />Go...
Reality Load XF – Overview & Demo <br /><ul><li>SaaS with no investment or maintenance costs and rapid payback
Self-service, full turnkey solution, or tailored to meet your needs with Gomez Professional Services offerings
Tests from an “Outside-in” customer point of view, with drill down to all web application components
Full desktop browser testing across globally distributed geographies</li></ul>New in July 2009<br />Simplified Test Provis...
Save and re-use test  scenarios
Last Mile UI scheduling improvements</li></ul>Expanded Load Generation <br /><ul><li>Additional Geographies for Based Load...
Upcoming SlideShare
Loading in...5
×

Best Practices for Identifying Web Performance Issues Before Your Customers Do- A Gomez/Dollar Thrifty Web Performance Testing Case Study

3,371

Published on

Dollar Thrifty Automotive Group, Inc. (DTG) customers are increasingly choosing the Internet as the primary way to rent vehicles from the Dollar Rent A Car and Thrifty Car Rental Web sites. Recently DTG undertook a significant redesign initiative for its two Web sites to optimize customer experience ahead of its busiest summer travel season and used Gomez’s web load and performance testing solution to validate their efforts. Attendees of this hands-on Webinar will see a Gomez Reality Load product demonstration and learn about the steps DTG took to validate peak performance for all internal and external components including Content Delivery Networks (CDNs), ads, analytics and ecommerce platforms, delivered across the Internet to its customers’ browsers. This Webinar will cover:

•How Dollar Thrifty geared up for their peak summer season.
•How a new style of load testing enables organizations to “walk in their customer’s shoes” and find problems before end-users find them.
•Best practices for identifying and resolving Web performance issues across the entire Web application delivery chain, inside and outside the firewall.
•Testing approaches that don’t require costly hardware or software investments.
•How to uncover geographical response time discrepancies that may surface under load.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,371
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
160
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Key themes:Find and resolve problems across the entire Web application delivery chainWorld’s largest load testing network reach means that you can test performance under load from browser to data centerFind and fix user experience and performance problems undetectable with traditional solutionsTalk trackThe answer to understanding and controlling the Web application delivery chain is to take a new view on your Web or mobile application: you need to adopt an outside-in, customer point of view. You need to “look back” at your web site and infrastructure the same way your customer does: by operating and running it “from the outside in”What does this mean?<click to animate>It means you need to test with real world load the same way your customers will. So, you need web load and performance testing to : Test across geographies, measuring web application performance at peak loads from your end-users’ perspective Find problems missed by lab-based “behind the firewall” solutionsHow does Reality Load help you do this?<click to animate>Reality Load XF’s Last Mile testing finds problems across the entire Web application delivery chain that cloud or data center-only solutions miss to accurately measure response time and find real end-user problems.With Gomez you can be fully informed about what’s going on with your web and mobile application and ensure quality experiences for your users.
  • Key themes:Gomez covers the globe with the most comprehensive testing networkWe are where your customers areTalk trackThis is a visual depiction of our global testing network.You can see where our Backbone and Last Mile testing locations are.Our Last Mile locations literally span the globe and allow you to test and monitor from any significant location in the world. And it’s growing every day.You can use these for a combination of monitoring and load testing.You can’t see the locations for the virtual test bed because it’s virtual – i.e. location independent.
  • ResolutionThe query was optimized and indexed to achieve greater performance with less CPU utilization.Business ImpactHad the application been launched with this query in place the login process would have bottlenecked at less than 10% of the anticipated traffic, resulting in:Higher response timesServer errors
  • All aspects of the user experience delivery chain must be testedBusiness ImpactCustomers negatively impacted by Higher response timesTime-out errors
  • Best Practices for Identifying Web Performance Issues Before Your Customers Do- A Gomez/Dollar Thrifty Web Performance Testing Case Study

    1. 1. Best Practices for Identifying Web Performance Issues Before Your Customers DoA Gomez/Dollar Thrifty Web Performance Testing Case Study: <br />Imad Mouline, CTO, Gomez<br />Emanuel Daniele, Sr. Solution Engineer, Gomez<br />Jim Arrowood, DTAG, Director, Web Development and Architecture<br />
    2. 2. Agenda<br />Reality Load Overview – Imad Mouline<br />Dollar Thrifty Case Study – Jim Arrowood<br />Product Demo – Emanuel Daniele<br />
    3. 3. The Challenge of Delivering Quality Web Experiences<br />Traditional testing: “OK”<br />…user is NOT happy<br />The Web Application Delivery Chain<br />3rd Party/Cloud Services<br />Browsers and devices<br />Users<br />Users<br />Local ISP<br />Load Balancing<br />Load Balancing<br />Web Servers<br />Web Servers<br />App Servers<br />App Servers<br />Internet<br />DB Servers<br />DB Servers<br />MajorISP<br />Mobile Carrier<br />Storage<br />Storage<br />Mobile Components<br />Mobile Components<br />Content DeliveryNetworks<br />Traditional zone of control<br />
    4. 4. The Web Application Delivery Chain<br /><ul><li>Inconsistent geo performance
    5. 5. Bad performance under load
    6. 6. Blocking content delivery
    7. 7. Incorrect geo-targeted content
    8. 8. Network peering problems
    9. 9. Bandwidth throttling
    10. 10. Inconsistent connectivity
    11. 11. Poorly performing JavaScript
    12. 12. Browser/device incompatibility
    13. 13. Page size too big
    14. 14. Too many objects
    15. 15. Low cache hit rate
    16. 16. Configuration errors
    17. 17. Application design issues
    18. 18. Code defects
    19. 19. Insufficient infrastructure
    20. 20. Network peering problems
    21. 21. Outages
    22. 22. Network resource shortage
    23. 23. Faulty content transcoding
    24. 24. SMS routing / latency issues
    25. 25. Configuration issues
    26. 26. Oversubscribed POP
    27. 27. Poor routing optimization
    28. 28. Low cache hit rate</li></ul>The Challenge of Ensuring Quality Web Experiences<br />Traditional testing: “OK”<br />…user is NOT happy<br />Traditional zone of control<br />Zone of customer expectation<br />Zone of customer expectation<br />
    29. 29. Gomez Load Testing: On-Demand Realistic Load Testing from Browser to Data Center<br />Backbone<br />Last Mile<br /> Real-world load<br /> Find user experience breaking points<br /> Accurately measure response time<br />High volume load (HTTP, Browser)<br />Find infrastructure breaking points<br /> Define capacity headroom<br />100,000+ consumer- grade desktops<br />168+ countries<br />2,500+ ISPs<br />Major mobile carriers around the globe<br />100+ commercial-grade nodes & data centers<br />
    30. 30. Gomez Network: The World’s Most Comprehensive Performance and Testing Network <br />Backbone<br />Virtual Test Bed<br />Gomez Last Mile<br />Web Performance Management and Load Testing 100+ locations<br />Cross-Browser Testing 500+ browser/OS combo’s<br />5,000+ supported devices<br />Web Performance Management and Load Testing 100,000+ locations<br />
    31. 31. Reality Load XF – Overview & Demo <br /><ul><li>SaaS with no investment or maintenance costs and rapid payback
    32. 32. Self-service, full turnkey solution, or tailored to meet your needs with Gomez Professional Services offerings
    33. 33. Tests from an “Outside-in” customer point of view, with drill down to all web application components
    34. 34. Full desktop browser testing across globally distributed geographies</li></ul>New in July 2009<br />Simplified Test Provisioning<br /><ul><li>More easily schedule complex test scenarios
    35. 35. Save and re-use test scenarios
    36. 36. Last Mile UI scheduling improvements</li></ul>Expanded Load Generation <br /><ul><li>Additional Geographies for Based Load high volume testing
    37. 37. More Last Mile peer populations</li></ul>Enhanced Reporting<br /><ul><li>Detailed reporting for complex scenarios
    38. 38. Filter results by load origin or time
    39. 39. Streamlined data and chart exports</li></li></ul><li>Dollar Thrifty Case Study<br />Jim Arrowood, DTAG<br />Director -Web Development and Architecture<br />8<br />
    40. 40. Who is Dollar Thrifty?<br /><ul><li>Driven by the mission “Value Every Time,” the Company’s brands, Dollar Rent A Car and Thrifty Car Rental, serve value-conscious travelers in over 70 countries.
    41. 41. Dollar and Thrifty have over 600 corporate and franchised locations in the United States and Canada, operating in virtually all of the top U.S. and Canadian airport markets.
    42. 42. The Company’s approximately 6,400 employees are located mainly in North America, but global service capabilities exist through an expanding international franchise network. </li></ul>9<br />
    43. 43. Dollar and Thrifty.com<br />10<br />
    44. 44. The Dollar Thrifty Environment<br />11<br />
    45. 45. The Problem<br /><ul><li>In a highly, horizontally scaled environment, development, test, and staging environments rarely 100% match production
    46. 46. Dollar Thrifty significantly redesigned both of our websites in 2008 leaving question as to how the sites will perform in the peak of the 2009 summer
    47. 47. Internal load tests using traditional methods in test and staging environments did not provide the needed confidence</li></ul>12<br />
    48. 48. The Gomez Relationship<br />Dollar Thrifty has utilized the Gomez ActiveXF platform for several years<br />Synthetic Tests of Key Business Processes<br />Monitors for response time (Home Page Load & Reservation Process)<br />Monitors for successful execution (Home Page & Reservation Process)<br />13<br />
    49. 49. Gomez Reality Load<br />In Early 2009, Dollar Thrifty teamed with Gomez to launch the first external load test of the company’s infrastructure and key software platforms utilizing Gomez Reality Load<br />14<br />
    50. 50. The Goals<br />Regardless of how much load, where is our weakest point?<br />Can we handle our previous year’s peak +25% traffic?<br />15<br />
    51. 51. Identify Core Business Processes<br />How do the vast majority of the consumers interact with the site?<br />Use 80-20 rule<br />Shop for Rates<br />Make Reservation<br />Modify Reservation<br />Cancel Reservation<br />16<br />
    52. 52. Distribute Core Business Processes<br />By percentage, how do consumers interact with the site?<br />Shop for Rates – 70%*<br />Make Reservation – 15%*<br />Modify Reservation – 10%*<br />Cancel Reservation – 5%*<br />*Hypothetical<br />17<br />
    53. 53. Forecast Visitors<br />Based on historical web analytic data and the goal of the test, compute the intended visitor capacity of the site within an hour<br />10,000 visitors per hour at peak historically*<br />If we intend to support peak +25% then our visitor forecast is 12,500 visitors<br />*Hypothetical<br />18<br />
    54. 54. Identify the Needed Scripts<br />Throughput Tests – Where are we weakest?<br />One script per core business process with no “think-time”<br />Capacity / Load Tests – How much load can we legitimately handle?<br />One script per core business process with “think-time”<br />19<br />
    55. 55. Develop Needed Scripts<br /><ul><li>Gomez Script Recorder allows the client to create of the script, inject look-up data, and replay the script
    56. 56. RealityLoad utilizes the same scripting engine as ActiveXF
    57. 57. The base of our scripts existed as we already had them built to execute in the ActiveXF platform
    58. 58. Added logic to pull from look-up data
    59. 59. Added think time
    60. 60. Once, complete the tests are loaded into the Gomez portal</li></ul>20<br />
    61. 61. Configure Tests<br />Using the Gomez portal, tests are configured and scheduled for execution<br />21<br />
    62. 62. Execute<br /><ul><li>Execute tests at off peak time
    63. 63. We chose 2 AM to 4 AM CST
    64. 64. Establish war room with key system owners
    65. 65. Establish bridge line with key infrastructure and monitoring personnel
    66. 66. As tests are executing ensure all internal monitoring tools in all tiers of the application are capturing as much detail as possible without skewing results
    67. 67. CPU, Memory, Disk IO, Network Throughput, etc.</li></ul>22<br />
    68. 68. Analyze Results<br />Each system owner and infrastructure owner collects key metrics and findings and submits to the test leader<br />23<br />
    69. 69. Discuss Key Findings<br /><ul><li>Thrifty.com front-end servers ran hot while the middle-tier was cold and overpowered
    70. 70. Dollar.com front-end servers ran cool while the middle-tier was running on target
    71. 71. Discovered excessive calls for validation to legacy reservation system
    72. 72. Thrifty.com middle-tier had sticky sessions enabled causing load balancing to become less ineffective</li></ul>24<br />
    73. 73. Make Corrective Actions & Retest<br /><ul><li>Mitigate the discovered issues and risks
    74. 74. We were able make a simple configuration change to resolve the “sticky sessions” issue
    75. 75. We were able to move Thrifty middle-tier boxes into the Thrifty.com front-end and Dollar.com middle-tier to better distribute our load with no additional expense
    76. 76. When time and budget allows, retest to ensure mitigating actions resolve the issue and ensure no additional bottlenecks have appeared</li></ul>25<br />
    77. 77. Demo<br />Emanuel Daniele <br />Gomez Reality Load Demo<br />26<br />
    78. 78. Wrap Up<br />Questions & Answers <br />Check back on QA Forums<br />To find out more about Reality Load:<br />cmason@gomez.com<br />mgil@gomez.com<br />Product Information<br />http://www.gomez.com/products-solutions/products/load-testing/<br />2 Minute Explainer<br />http://www.gomez.com/resources/video-library/gomez-reality-load-testing-two-minute-explainer/<br />
    79. 79. Ensuring Performance Of Login Process<br />Company: Online presence for a popular TV show<br /><ul><li>Following episodes of the TV show the web site sees high traffic spikes
    80. 80. Goal was to achieve 1500 logins per minute
    81. 81. Load tested DB to improve performance in anticipation of another traffic spike</li></ul>3rd Party/Cloud Services<br />Browsers and devices<br />Local ISP<br />Users<br />Load Balancing<br />Load Balancing<br />Web Servers<br />Web Servers<br />1<br />App Servers<br />App Servers<br />Internet<br />DB Servers<br />DB Servers<br />MajorISP<br />Mobile Carrier<br />Storage<br />Storage<br />Mobile Components<br />Mobile Components<br />Content DeliveryNetworks<br />
    82. 82. Application Bottleneck Causes Immediate Response Time Issue<br /><ul><li>As users were added, the response time of step 3 (the login) climbed immediately
    83. 83. The test bottlenecked at 160 logins per minute (Goal 1500)
    84. 84. But quickly dropped off as users received server errors
    85. 85. New login query was not optimized and was bottlenecking the database servers’ CPUs</li></li></ul><li>Application Bottleneck – Re-test<br /><ul><li>Following the tuning effort the application performance was improved.
    86. 86. This time the bottleneck was at 1300 logins per minute.
    87. 87. A bandwidth limit was reached at just under 90 Mbps, resulting in an overall slowdown as users were added.
    88. 88. This highlights:
    89. 89. The importance of re-testing following each change.
    90. 90. The fact that applications often have many bottlenecks, that can only be uncovered one at a time.</li></ul>30<br />
    91. 91. Ensuring Performance of All 3rd Party Components<br />Company: Online Retailer<br /><ul><li>Several 3rd Parties now involved in serving up key content
    92. 92. Goal was to validate performance of entire application</li></ul>3rd Party/Cloud Services<br />Browsers and devices<br />Local ISP<br />Users<br />Load Balancing<br />Load Balancing<br />2<br />Web Servers<br />Web Servers<br />App Servers<br />App Servers<br />Internet<br />DB Servers<br />DB Servers<br />MajorISP<br />Mobile Carrier<br />Storage<br />Storage<br />Mobile Components<br />Mobile Components<br />Content DeliveryNetworks<br />
    93. 93. Response Times Rise Due To 3rd Party Object Error <br />The transaction rate increases and then falls off as response times climb<br />The load increases throughout the test<br />Errors are seen, all on a 3rd party object<br /><ul><li>3rd party hardware was insufficient for overall demands on application
    94. 94. Based on SLAs 3rd party had to improve performance to get paid</li></li></ul><li>Ensuring Performance in Key Markets<br />Company: Regional Online News Source<br /><ul><li>Began testing for the 2008 election season
    95. 95. Goal was to validate overall performance focusing in 2 key regions</li></ul>3rd Party/Cloud Services<br />Browsers and devices<br />Local ISP<br />Users<br />Load Balancing<br />Load Balancing<br />Web Servers<br />Web Servers<br />3<br />App Servers<br />App Servers<br />Internet<br />DB Servers<br />DB Servers<br />MajorISP<br />Mobile Carrier<br />Storage<br />Storage<br />Mobile Components<br />Mobile Components<br />Content DeliveryNetworks<br />
    96. 96. No Performance Issues Detected From Data-Center<br />Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%<br />There was only 1 page error and 11 errors total out of 60000+ transactions<br />Page response times stayed under 4 seconds, outside of one brief blip<br />By traditional test standards the test passed<br />
    97. 97. Performance Issues Detected From Real User Desktops<br />Key geographies for this customer are New York and Pennsylvania<br />Last Mile data showing substantial number of measurements greater than 4 seconds <br />
    98. 98. Last Mile Case Study: Primary Geographies<br />Key geographies for this customer are New York and Pennsylvania.<br />The response time never met the 4 second average goal.<br />By these standards the test failed.<br />Availability was Less than 99%.<br />36<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×