Your SlideShare is downloading. ×
Best Practices for Identifying Web Performance Issues Before Your Customers Do- A Gomez/Dollar Thrifty Web Performance Testing Case Study
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Best Practices for Identifying Web Performance Issues Before Your Customers Do- A Gomez/Dollar Thrifty Web Performance Testing Case Study

3,179
views

Published on

Dollar Thrifty Automotive Group, Inc. (DTG) customers are increasingly choosing the Internet as the primary way to rent vehicles from the Dollar Rent A Car and Thrifty Car Rental Web sites. Recently …

Dollar Thrifty Automotive Group, Inc. (DTG) customers are increasingly choosing the Internet as the primary way to rent vehicles from the Dollar Rent A Car and Thrifty Car Rental Web sites. Recently DTG undertook a significant redesign initiative for its two Web sites to optimize customer experience ahead of its busiest summer travel season and used Gomez’s web load and performance testing solution to validate their efforts. Attendees of this hands-on Webinar will see a Gomez Reality Load product demonstration and learn about the steps DTG took to validate peak performance for all internal and external components including Content Delivery Networks (CDNs), ads, analytics and ecommerce platforms, delivered across the Internet to its customers’ browsers. This Webinar will cover:

•How Dollar Thrifty geared up for their peak summer season.
•How a new style of load testing enables organizations to “walk in their customer’s shoes” and find problems before end-users find them.
•Best practices for identifying and resolving Web performance issues across the entire Web application delivery chain, inside and outside the firewall.
•Testing approaches that don’t require costly hardware or software investments.
•How to uncover geographical response time discrepancies that may surface under load.

Published in: Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,179
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
157
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Key themes:Find and resolve problems across the entire Web application delivery chainWorld’s largest load testing network reach means that you can test performance under load from browser to data centerFind and fix user experience and performance problems undetectable with traditional solutionsTalk trackThe answer to understanding and controlling the Web application delivery chain is to take a new view on your Web or mobile application: you need to adopt an outside-in, customer point of view. You need to “look back” at your web site and infrastructure the same way your customer does: by operating and running it “from the outside in”What does this mean?<click to animate>It means you need to test with real world load the same way your customers will. So, you need web load and performance testing to : Test across geographies, measuring web application performance at peak loads from your end-users’ perspective Find problems missed by lab-based “behind the firewall” solutionsHow does Reality Load help you do this?<click to animate>Reality Load XF’s Last Mile testing finds problems across the entire Web application delivery chain that cloud or data center-only solutions miss to accurately measure response time and find real end-user problems.With Gomez you can be fully informed about what’s going on with your web and mobile application and ensure quality experiences for your users.
  • Key themes:Gomez covers the globe with the most comprehensive testing networkWe are where your customers areTalk trackThis is a visual depiction of our global testing network.You can see where our Backbone and Last Mile testing locations are.Our Last Mile locations literally span the globe and allow you to test and monitor from any significant location in the world. And it’s growing every day.You can use these for a combination of monitoring and load testing.You can’t see the locations for the virtual test bed because it’s virtual – i.e. location independent.
  • ResolutionThe query was optimized and indexed to achieve greater performance with less CPU utilization.Business ImpactHad the application been launched with this query in place the login process would have bottlenecked at less than 10% of the anticipated traffic, resulting in:Higher response timesServer errors
  • All aspects of the user experience delivery chain must be testedBusiness ImpactCustomers negatively impacted by Higher response timesTime-out errors
  • Transcript

    • 1. Best Practices for Identifying Web Performance Issues Before Your Customers DoA Gomez/Dollar Thrifty Web Performance Testing Case Study:
      Imad Mouline, CTO, Gomez
      Emanuel Daniele, Sr. Solution Engineer, Gomez
      Jim Arrowood, DTAG, Director, Web Development and Architecture
    • 2. Agenda
      Reality Load Overview – Imad Mouline
      Dollar Thrifty Case Study – Jim Arrowood
      Product Demo – Emanuel Daniele
    • 3. The Challenge of Delivering Quality Web Experiences
      Traditional testing: “OK”
      …user is NOT happy
      The Web Application Delivery Chain
      3rd Party/Cloud Services
      Browsers and devices
      Users
      Users
      Local ISP
      Load Balancing
      Load Balancing
      Web Servers
      Web Servers
      App Servers
      App Servers
      Internet
      DB Servers
      DB Servers
      MajorISP
      Mobile Carrier
      Storage
      Storage
      Mobile Components
      Mobile Components
      Content DeliveryNetworks
      Traditional zone of control
    • 4. The Web Application Delivery Chain
      • Inconsistent geo performance
      • 5. Bad performance under load
      • 6. Blocking content delivery
      • 7. Incorrect geo-targeted content
      • 8. Network peering problems
      • 9. Bandwidth throttling
      • 10. Inconsistent connectivity
      • 11. Poorly performing JavaScript
      • 12. Browser/device incompatibility
      • 13. Page size too big
      • 14. Too many objects
      • 15. Low cache hit rate
      • 16. Configuration errors
      • 17. Application design issues
      • 18. Code defects
      • 19. Insufficient infrastructure
      • 20. Network peering problems
      • 21. Outages
      • 22. Network resource shortage
      • 23. Faulty content transcoding
      • 24. SMS routing / latency issues
      • 25. Configuration issues
      • 26. Oversubscribed POP
      • 27. Poor routing optimization
      • 28. Low cache hit rate
      The Challenge of Ensuring Quality Web Experiences
      Traditional testing: “OK”
      …user is NOT happy
      Traditional zone of control
      Zone of customer expectation
      Zone of customer expectation
    • 29. Gomez Load Testing: On-Demand Realistic Load Testing from Browser to Data Center
      Backbone
      Last Mile
      Real-world load
      Find user experience breaking points
      Accurately measure response time
      High volume load (HTTP, Browser)
      Find infrastructure breaking points
      Define capacity headroom
      100,000+ consumer- grade desktops
      168+ countries
      2,500+ ISPs
      Major mobile carriers around the globe
      100+ commercial-grade nodes & data centers
    • 30. Gomez Network: The World’s Most Comprehensive Performance and Testing Network
      Backbone
      Virtual Test Bed
      Gomez Last Mile
      Web Performance Management and Load Testing 100+ locations
      Cross-Browser Testing 500+ browser/OS combo’s
      5,000+ supported devices
      Web Performance Management and Load Testing 100,000+ locations
    • 31. Reality Load XF – Overview & Demo
      • SaaS with no investment or maintenance costs and rapid payback
      • 32. Self-service, full turnkey solution, or tailored to meet your needs with Gomez Professional Services offerings
      • 33. Tests from an “Outside-in” customer point of view, with drill down to all web application components
      • 34. Full desktop browser testing across globally distributed geographies
      New in July 2009
      Simplified Test Provisioning
      • More easily schedule complex test scenarios
      • 35. Save and re-use test scenarios
      • 36. Last Mile UI scheduling improvements
      Expanded Load Generation
      • Additional Geographies for Based Load high volume testing
      • 37. More Last Mile peer populations
      Enhanced Reporting
      • Detailed reporting for complex scenarios
      • 38. Filter results by load origin or time
      • 39. Streamlined data and chart exports
    • Dollar Thrifty Case Study
      Jim Arrowood, DTAG
      Director -Web Development and Architecture
      8
    • 40. Who is Dollar Thrifty?
      • Driven by the mission “Value Every Time,” the Company’s brands, Dollar Rent A Car and Thrifty Car Rental, serve value-conscious travelers in over 70 countries.
      • 41. Dollar and Thrifty have over 600 corporate and franchised locations in the United States and Canada, operating in virtually all of the top U.S. and Canadian airport markets.
      • 42. The Company’s approximately 6,400 employees are located mainly in North America, but global service capabilities exist through an expanding international franchise network.
      9
    • 43. Dollar and Thrifty.com
      10
    • 44. The Dollar Thrifty Environment
      11
    • 45. The Problem
      • In a highly, horizontally scaled environment, development, test, and staging environments rarely 100% match production
      • 46. Dollar Thrifty significantly redesigned both of our websites in 2008 leaving question as to how the sites will perform in the peak of the 2009 summer
      • 47. Internal load tests using traditional methods in test and staging environments did not provide the needed confidence
      12
    • 48. The Gomez Relationship
      Dollar Thrifty has utilized the Gomez ActiveXF platform for several years
      Synthetic Tests of Key Business Processes
      Monitors for response time (Home Page Load & Reservation Process)
      Monitors for successful execution (Home Page & Reservation Process)
      13
    • 49. Gomez Reality Load
      In Early 2009, Dollar Thrifty teamed with Gomez to launch the first external load test of the company’s infrastructure and key software platforms utilizing Gomez Reality Load
      14
    • 50. The Goals
      Regardless of how much load, where is our weakest point?
      Can we handle our previous year’s peak +25% traffic?
      15
    • 51. Identify Core Business Processes
      How do the vast majority of the consumers interact with the site?
      Use 80-20 rule
      Shop for Rates
      Make Reservation
      Modify Reservation
      Cancel Reservation
      16
    • 52. Distribute Core Business Processes
      By percentage, how do consumers interact with the site?
      Shop for Rates – 70%*
      Make Reservation – 15%*
      Modify Reservation – 10%*
      Cancel Reservation – 5%*
      *Hypothetical
      17
    • 53. Forecast Visitors
      Based on historical web analytic data and the goal of the test, compute the intended visitor capacity of the site within an hour
      10,000 visitors per hour at peak historically*
      If we intend to support peak +25% then our visitor forecast is 12,500 visitors
      *Hypothetical
      18
    • 54. Identify the Needed Scripts
      Throughput Tests – Where are we weakest?
      One script per core business process with no “think-time”
      Capacity / Load Tests – How much load can we legitimately handle?
      One script per core business process with “think-time”
      19
    • 55. Develop Needed Scripts
      • Gomez Script Recorder allows the client to create of the script, inject look-up data, and replay the script
      • 56. RealityLoad utilizes the same scripting engine as ActiveXF
      • 57. The base of our scripts existed as we already had them built to execute in the ActiveXF platform
      • 58. Added logic to pull from look-up data
      • 59. Added think time
      • 60. Once, complete the tests are loaded into the Gomez portal
      20
    • 61. Configure Tests
      Using the Gomez portal, tests are configured and scheduled for execution
      21
    • 62. Execute
      • Execute tests at off peak time
      • 63. We chose 2 AM to 4 AM CST
      • 64. Establish war room with key system owners
      • 65. Establish bridge line with key infrastructure and monitoring personnel
      • 66. As tests are executing ensure all internal monitoring tools in all tiers of the application are capturing as much detail as possible without skewing results
      • 67. CPU, Memory, Disk IO, Network Throughput, etc.
      22
    • 68. Analyze Results
      Each system owner and infrastructure owner collects key metrics and findings and submits to the test leader
      23
    • 69. Discuss Key Findings
      • Thrifty.com front-end servers ran hot while the middle-tier was cold and overpowered
      • 70. Dollar.com front-end servers ran cool while the middle-tier was running on target
      • 71. Discovered excessive calls for validation to legacy reservation system
      • 72. Thrifty.com middle-tier had sticky sessions enabled causing load balancing to become less ineffective
      24
    • 73. Make Corrective Actions & Retest
      • Mitigate the discovered issues and risks
      • 74. We were able make a simple configuration change to resolve the “sticky sessions” issue
      • 75. We were able to move Thrifty middle-tier boxes into the Thrifty.com front-end and Dollar.com middle-tier to better distribute our load with no additional expense
      • 76. When time and budget allows, retest to ensure mitigating actions resolve the issue and ensure no additional bottlenecks have appeared
      25
    • 77. Demo
      Emanuel Daniele
      Gomez Reality Load Demo
      26
    • 78. Wrap Up
      Questions & Answers
      Check back on QA Forums
      To find out more about Reality Load:
      cmason@gomez.com
      mgil@gomez.com
      Product Information
      http://www.gomez.com/products-solutions/products/load-testing/
      2 Minute Explainer
      http://www.gomez.com/resources/video-library/gomez-reality-load-testing-two-minute-explainer/
    • 79. Ensuring Performance Of Login Process
      Company: Online presence for a popular TV show
      • Following episodes of the TV show the web site sees high traffic spikes
      • 80. Goal was to achieve 1500 logins per minute
      • 81. Load tested DB to improve performance in anticipation of another traffic spike
      3rd Party/Cloud Services
      Browsers and devices
      Local ISP
      Users
      Load Balancing
      Load Balancing
      Web Servers
      Web Servers
      1
      App Servers
      App Servers
      Internet
      DB Servers
      DB Servers
      MajorISP
      Mobile Carrier
      Storage
      Storage
      Mobile Components
      Mobile Components
      Content DeliveryNetworks
    • 82. Application Bottleneck Causes Immediate Response Time Issue
      • As users were added, the response time of step 3 (the login) climbed immediately
      • 83. The test bottlenecked at 160 logins per minute (Goal 1500)
      • 84. But quickly dropped off as users received server errors
      • 85. New login query was not optimized and was bottlenecking the database servers’ CPUs
    • Application Bottleneck – Re-test
      • Following the tuning effort the application performance was improved.
      • 86. This time the bottleneck was at 1300 logins per minute.
      • 87. A bandwidth limit was reached at just under 90 Mbps, resulting in an overall slowdown as users were added.
      • 88. This highlights:
      • 89. The importance of re-testing following each change.
      • 90. The fact that applications often have many bottlenecks, that can only be uncovered one at a time.
      30
    • 91. Ensuring Performance of All 3rd Party Components
      Company: Online Retailer
      • Several 3rd Parties now involved in serving up key content
      • 92. Goal was to validate performance of entire application
      3rd Party/Cloud Services
      Browsers and devices
      Local ISP
      Users
      Load Balancing
      Load Balancing
      2
      Web Servers
      Web Servers
      App Servers
      App Servers
      Internet
      DB Servers
      DB Servers
      MajorISP
      Mobile Carrier
      Storage
      Storage
      Mobile Components
      Mobile Components
      Content DeliveryNetworks
    • 93. Response Times Rise Due To 3rd Party Object Error
      The transaction rate increases and then falls off as response times climb
      The load increases throughout the test
      Errors are seen, all on a 3rd party object
      • 3rd party hardware was insufficient for overall demands on application
      • 94. Based on SLAs 3rd party had to improve performance to get paid
    • Ensuring Performance in Key Markets
      Company: Regional Online News Source
      • Began testing for the 2008 election season
      • 95. Goal was to validate overall performance focusing in 2 key regions
      3rd Party/Cloud Services
      Browsers and devices
      Local ISP
      Users
      Load Balancing
      Load Balancing
      Web Servers
      Web Servers
      3
      App Servers
      App Servers
      Internet
      DB Servers
      DB Servers
      MajorISP
      Mobile Carrier
      Storage
      Storage
      Mobile Components
      Mobile Components
      Content DeliveryNetworks
    • 96. No Performance Issues Detected From Data-Center
      Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%
      There was only 1 page error and 11 errors total out of 60000+ transactions
      Page response times stayed under 4 seconds, outside of one brief blip
      By traditional test standards the test passed
    • 97. Performance Issues Detected From Real User Desktops
      Key geographies for this customer are New York and Pennsylvania
      Last Mile data showing substantial number of measurements greater than 4 seconds
    • 98. Last Mile Case Study: Primary Geographies
      Key geographies for this customer are New York and Pennsylvania.
      The response time never met the 4 second average goal.
      By these standards the test failed.
      Availability was Less than 99%.
      36