Best Practices for Identifying Web Performance Issues Before Your Customers Do- A Gomez/Dollar Thrifty Web Performance Testing Case Study

  • 3,051 views
Uploaded on

Dollar Thrifty Automotive Group, Inc. (DTG) customers are increasingly choosing the Internet as the primary way to rent vehicles from the Dollar Rent A Car and Thrifty Car Rental Web sites. Recently …

Dollar Thrifty Automotive Group, Inc. (DTG) customers are increasingly choosing the Internet as the primary way to rent vehicles from the Dollar Rent A Car and Thrifty Car Rental Web sites. Recently DTG undertook a significant redesign initiative for its two Web sites to optimize customer experience ahead of its busiest summer travel season and used Gomez’s web load and performance testing solution to validate their efforts. Attendees of this hands-on Webinar will see a Gomez Reality Load product demonstration and learn about the steps DTG took to validate peak performance for all internal and external components including Content Delivery Networks (CDNs), ads, analytics and ecommerce platforms, delivered across the Internet to its customers’ browsers. This Webinar will cover:

•How Dollar Thrifty geared up for their peak summer season.
•How a new style of load testing enables organizations to “walk in their customer’s shoes” and find problems before end-users find them.
•Best practices for identifying and resolving Web performance issues across the entire Web application delivery chain, inside and outside the firewall.
•Testing approaches that don’t require costly hardware or software investments.
•How to uncover geographical response time discrepancies that may surface under load.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,051
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
147
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Key themes:Find and resolve problems across the entire Web application delivery chainWorld’s largest load testing network reach means that you can test performance under load from browser to data centerFind and fix user experience and performance problems undetectable with traditional solutionsTalk trackThe answer to understanding and controlling the Web application delivery chain is to take a new view on your Web or mobile application: you need to adopt an outside-in, customer point of view. You need to “look back” at your web site and infrastructure the same way your customer does: by operating and running it “from the outside in”What does this mean?<click to animate>It means you need to test with real world load the same way your customers will. So, you need web load and performance testing to : Test across geographies, measuring web application performance at peak loads from your end-users’ perspective Find problems missed by lab-based “behind the firewall” solutionsHow does Reality Load help you do this?<click to animate>Reality Load XF’s Last Mile testing finds problems across the entire Web application delivery chain that cloud or data center-only solutions miss to accurately measure response time and find real end-user problems.With Gomez you can be fully informed about what’s going on with your web and mobile application and ensure quality experiences for your users.
  • Key themes:Gomez covers the globe with the most comprehensive testing networkWe are where your customers areTalk trackThis is a visual depiction of our global testing network.You can see where our Backbone and Last Mile testing locations are.Our Last Mile locations literally span the globe and allow you to test and monitor from any significant location in the world. And it’s growing every day.You can use these for a combination of monitoring and load testing.You can’t see the locations for the virtual test bed because it’s virtual – i.e. location independent.
  • ResolutionThe query was optimized and indexed to achieve greater performance with less CPU utilization.Business ImpactHad the application been launched with this query in place the login process would have bottlenecked at less than 10% of the anticipated traffic, resulting in:Higher response timesServer errors
  • All aspects of the user experience delivery chain must be testedBusiness ImpactCustomers negatively impacted by Higher response timesTime-out errors

Transcript

  • 1. Best Practices for Identifying Web Performance Issues Before Your Customers DoA Gomez/Dollar Thrifty Web Performance Testing Case Study:
    Imad Mouline, CTO, Gomez
    Emanuel Daniele, Sr. Solution Engineer, Gomez
    Jim Arrowood, DTAG, Director, Web Development and Architecture
  • 2. Agenda
    Reality Load Overview – Imad Mouline
    Dollar Thrifty Case Study – Jim Arrowood
    Product Demo – Emanuel Daniele
  • 3. The Challenge of Delivering Quality Web Experiences
    Traditional testing: “OK”
    …user is NOT happy
    The Web Application Delivery Chain
    3rd Party/Cloud Services
    Browsers and devices
    Users
    Users
    Local ISP
    Load Balancing
    Load Balancing
    Web Servers
    Web Servers
    App Servers
    App Servers
    Internet
    DB Servers
    DB Servers
    MajorISP
    Mobile Carrier
    Storage
    Storage
    Mobile Components
    Mobile Components
    Content DeliveryNetworks
    Traditional zone of control
  • 4. The Web Application Delivery Chain
    • Inconsistent geo performance
    • 5. Bad performance under load
    • 6. Blocking content delivery
    • 7. Incorrect geo-targeted content
    • 8. Network peering problems
    • 9. Bandwidth throttling
    • 10. Inconsistent connectivity
    • 11. Poorly performing JavaScript
    • 12. Browser/device incompatibility
    • 13. Page size too big
    • 14. Too many objects
    • 15. Low cache hit rate
    • 16. Configuration errors
    • 17. Application design issues
    • 18. Code defects
    • 19. Insufficient infrastructure
    • 20. Network peering problems
    • 21. Outages
    • 22. Network resource shortage
    • 23. Faulty content transcoding
    • 24. SMS routing / latency issues
    • 25. Configuration issues
    • 26. Oversubscribed POP
    • 27. Poor routing optimization
    • 28. Low cache hit rate
    The Challenge of Ensuring Quality Web Experiences
    Traditional testing: “OK”
    …user is NOT happy
    Traditional zone of control
    Zone of customer expectation
    Zone of customer expectation
  • 29. Gomez Load Testing: On-Demand Realistic Load Testing from Browser to Data Center
    Backbone
    Last Mile
    Real-world load
    Find user experience breaking points
    Accurately measure response time
    High volume load (HTTP, Browser)
    Find infrastructure breaking points
    Define capacity headroom
    100,000+ consumer- grade desktops
    168+ countries
    2,500+ ISPs
    Major mobile carriers around the globe
    100+ commercial-grade nodes & data centers
  • 30. Gomez Network: The World’s Most Comprehensive Performance and Testing Network
    Backbone
    Virtual Test Bed
    Gomez Last Mile
    Web Performance Management and Load Testing 100+ locations
    Cross-Browser Testing 500+ browser/OS combo’s
    5,000+ supported devices
    Web Performance Management and Load Testing 100,000+ locations
  • 31. Reality Load XF – Overview & Demo
    • SaaS with no investment or maintenance costs and rapid payback
    • 32. Self-service, full turnkey solution, or tailored to meet your needs with Gomez Professional Services offerings
    • 33. Tests from an “Outside-in” customer point of view, with drill down to all web application components
    • 34. Full desktop browser testing across globally distributed geographies
    New in July 2009
    Simplified Test Provisioning
    • More easily schedule complex test scenarios
    • 35. Save and re-use test scenarios
    • 36. Last Mile UI scheduling improvements
    Expanded Load Generation
    • Additional Geographies for Based Load high volume testing
    • 37. More Last Mile peer populations
    Enhanced Reporting
    • Detailed reporting for complex scenarios
    • 38. Filter results by load origin or time
    • 39. Streamlined data and chart exports
  • Dollar Thrifty Case Study
    Jim Arrowood, DTAG
    Director -Web Development and Architecture
    8
  • 40. Who is Dollar Thrifty?
    • Driven by the mission “Value Every Time,” the Company’s brands, Dollar Rent A Car and Thrifty Car Rental, serve value-conscious travelers in over 70 countries.
    • 41. Dollar and Thrifty have over 600 corporate and franchised locations in the United States and Canada, operating in virtually all of the top U.S. and Canadian airport markets.
    • 42. The Company’s approximately 6,400 employees are located mainly in North America, but global service capabilities exist through an expanding international franchise network.
    9
  • 43. Dollar and Thrifty.com
    10
  • 44. The Dollar Thrifty Environment
    11
  • 45. The Problem
    • In a highly, horizontally scaled environment, development, test, and staging environments rarely 100% match production
    • 46. Dollar Thrifty significantly redesigned both of our websites in 2008 leaving question as to how the sites will perform in the peak of the 2009 summer
    • 47. Internal load tests using traditional methods in test and staging environments did not provide the needed confidence
    12
  • 48. The Gomez Relationship
    Dollar Thrifty has utilized the Gomez ActiveXF platform for several years
    Synthetic Tests of Key Business Processes
    Monitors for response time (Home Page Load & Reservation Process)
    Monitors for successful execution (Home Page & Reservation Process)
    13
  • 49. Gomez Reality Load
    In Early 2009, Dollar Thrifty teamed with Gomez to launch the first external load test of the company’s infrastructure and key software platforms utilizing Gomez Reality Load
    14
  • 50. The Goals
    Regardless of how much load, where is our weakest point?
    Can we handle our previous year’s peak +25% traffic?
    15
  • 51. Identify Core Business Processes
    How do the vast majority of the consumers interact with the site?
    Use 80-20 rule
    Shop for Rates
    Make Reservation
    Modify Reservation
    Cancel Reservation
    16
  • 52. Distribute Core Business Processes
    By percentage, how do consumers interact with the site?
    Shop for Rates – 70%*
    Make Reservation – 15%*
    Modify Reservation – 10%*
    Cancel Reservation – 5%*
    *Hypothetical
    17
  • 53. Forecast Visitors
    Based on historical web analytic data and the goal of the test, compute the intended visitor capacity of the site within an hour
    10,000 visitors per hour at peak historically*
    If we intend to support peak +25% then our visitor forecast is 12,500 visitors
    *Hypothetical
    18
  • 54. Identify the Needed Scripts
    Throughput Tests – Where are we weakest?
    One script per core business process with no “think-time”
    Capacity / Load Tests – How much load can we legitimately handle?
    One script per core business process with “think-time”
    19
  • 55. Develop Needed Scripts
    • Gomez Script Recorder allows the client to create of the script, inject look-up data, and replay the script
    • 56. RealityLoad utilizes the same scripting engine as ActiveXF
    • 57. The base of our scripts existed as we already had them built to execute in the ActiveXF platform
    • 58. Added logic to pull from look-up data
    • 59. Added think time
    • 60. Once, complete the tests are loaded into the Gomez portal
    20
  • 61. Configure Tests
    Using the Gomez portal, tests are configured and scheduled for execution
    21
  • 62. Execute
    • Execute tests at off peak time
    • 63. We chose 2 AM to 4 AM CST
    • 64. Establish war room with key system owners
    • 65. Establish bridge line with key infrastructure and monitoring personnel
    • 66. As tests are executing ensure all internal monitoring tools in all tiers of the application are capturing as much detail as possible without skewing results
    • 67. CPU, Memory, Disk IO, Network Throughput, etc.
    22
  • 68. Analyze Results
    Each system owner and infrastructure owner collects key metrics and findings and submits to the test leader
    23
  • 69. Discuss Key Findings
    • Thrifty.com front-end servers ran hot while the middle-tier was cold and overpowered
    • 70. Dollar.com front-end servers ran cool while the middle-tier was running on target
    • 71. Discovered excessive calls for validation to legacy reservation system
    • 72. Thrifty.com middle-tier had sticky sessions enabled causing load balancing to become less ineffective
    24
  • 73. Make Corrective Actions & Retest
    • Mitigate the discovered issues and risks
    • 74. We were able make a simple configuration change to resolve the “sticky sessions” issue
    • 75. We were able to move Thrifty middle-tier boxes into the Thrifty.com front-end and Dollar.com middle-tier to better distribute our load with no additional expense
    • 76. When time and budget allows, retest to ensure mitigating actions resolve the issue and ensure no additional bottlenecks have appeared
    25
  • 77. Demo
    Emanuel Daniele
    Gomez Reality Load Demo
    26
  • 78. Wrap Up
    Questions & Answers
    Check back on QA Forums
    To find out more about Reality Load:
    cmason@gomez.com
    mgil@gomez.com
    Product Information
    http://www.gomez.com/products-solutions/products/load-testing/
    2 Minute Explainer
    http://www.gomez.com/resources/video-library/gomez-reality-load-testing-two-minute-explainer/
  • 79. Ensuring Performance Of Login Process
    Company: Online presence for a popular TV show
    • Following episodes of the TV show the web site sees high traffic spikes
    • 80. Goal was to achieve 1500 logins per minute
    • 81. Load tested DB to improve performance in anticipation of another traffic spike
    3rd Party/Cloud Services
    Browsers and devices
    Local ISP
    Users
    Load Balancing
    Load Balancing
    Web Servers
    Web Servers
    1
    App Servers
    App Servers
    Internet
    DB Servers
    DB Servers
    MajorISP
    Mobile Carrier
    Storage
    Storage
    Mobile Components
    Mobile Components
    Content DeliveryNetworks
  • 82. Application Bottleneck Causes Immediate Response Time Issue
    • As users were added, the response time of step 3 (the login) climbed immediately
    • 83. The test bottlenecked at 160 logins per minute (Goal 1500)
    • 84. But quickly dropped off as users received server errors
    • 85. New login query was not optimized and was bottlenecking the database servers’ CPUs
  • Application Bottleneck – Re-test
    • Following the tuning effort the application performance was improved.
    • 86. This time the bottleneck was at 1300 logins per minute.
    • 87. A bandwidth limit was reached at just under 90 Mbps, resulting in an overall slowdown as users were added.
    • 88. This highlights:
    • 89. The importance of re-testing following each change.
    • 90. The fact that applications often have many bottlenecks, that can only be uncovered one at a time.
    30
  • 91. Ensuring Performance of All 3rd Party Components
    Company: Online Retailer
    • Several 3rd Parties now involved in serving up key content
    • 92. Goal was to validate performance of entire application
    3rd Party/Cloud Services
    Browsers and devices
    Local ISP
    Users
    Load Balancing
    Load Balancing
    2
    Web Servers
    Web Servers
    App Servers
    App Servers
    Internet
    DB Servers
    DB Servers
    MajorISP
    Mobile Carrier
    Storage
    Storage
    Mobile Components
    Mobile Components
    Content DeliveryNetworks
  • 93. Response Times Rise Due To 3rd Party Object Error
    The transaction rate increases and then falls off as response times climb
    The load increases throughout the test
    Errors are seen, all on a 3rd party object
    • 3rd party hardware was insufficient for overall demands on application
    • 94. Based on SLAs 3rd party had to improve performance to get paid
  • Ensuring Performance in Key Markets
    Company: Regional Online News Source
    • Began testing for the 2008 election season
    • 95. Goal was to validate overall performance focusing in 2 key regions
    3rd Party/Cloud Services
    Browsers and devices
    Local ISP
    Users
    Load Balancing
    Load Balancing
    Web Servers
    Web Servers
    3
    App Servers
    App Servers
    Internet
    DB Servers
    DB Servers
    MajorISP
    Mobile Carrier
    Storage
    Storage
    Mobile Components
    Mobile Components
    Content DeliveryNetworks
  • 96. No Performance Issues Detected From Data-Center
    Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%
    There was only 1 page error and 11 errors total out of 60000+ transactions
    Page response times stayed under 4 seconds, outside of one brief blip
    By traditional test standards the test passed
  • 97. Performance Issues Detected From Real User Desktops
    Key geographies for this customer are New York and Pennsylvania
    Last Mile data showing substantial number of measurements greater than 4 seconds
  • 98. Last Mile Case Study: Primary Geographies
    Key geographies for this customer are New York and Pennsylvania.
    The response time never met the 4 second average goal.
    By these standards the test failed.
    Availability was Less than 99%.
    36