Best Practices for Identifying Web Performance Issues Before Your Customers Do- A Gomez/Dollar Thrifty Web Performance Testing Case Study

Best Practices for Identifying Web Performance Issues Before Your Customers DoA Gomez/Dollar Thrifty Web Performance Testing Case Study:
Imad Mouline, CTO, Gomez
Emanuel Daniele, Sr. Solution Engineer, Gomez
Jim Arrowood, DTAG, Director, Web Development and Architecture

Agenda
Reality Load Overview – Imad Mouline
Dollar Thrifty Case Study – Jim Arrowood
Product Demo – Emanuel Daniele

The Challenge of Delivering Quality Web Experiences
Traditional testing: “OK”
…user is NOT happy
The Web Application Delivery Chain
3rd Party/Cloud Services
Browsers and devices
Users
Users
Local ISP
Load Balancing
Load Balancing
Web Servers
Web Servers
App Servers
App Servers
Internet
DB Servers
DB Servers
MajorISP
Mobile Carrier
Storage
Storage
Mobile Components
Mobile Components
Content DeliveryNetworks
Traditional zone of control

The Web Application Delivery Chain

Inconsistent geo performance
Bad performance under load
Blocking content delivery
Incorrect geo-targeted content
Network peering problems
Bandwidth throttling
Inconsistent connectivity
Poorly performing JavaScript
Browser/device incompatibility
Page size too big
Too many objects
Low cache hit rate
Configuration errors
Application design issues
Code defects
Insufficient infrastructure
Network peering problems
Outages
Network resource shortage
Faulty content transcoding
SMS routing / latency issues
Configuration issues
Oversubscribed POP
Poor routing optimization
Low cache hit rate

The Challenge of Ensuring Quality Web Experiences
Traditional testing: “OK”
…user is NOT happy
Traditional zone of control
Zone of customer expectation
Zone of customer expectation

Gomez Load Testing: On-Demand Realistic Load Testing from Browser to Data Center
Backbone
Last Mile
Real-world load
Find user experience breaking points
Accurately measure response time
High volume load (HTTP, Browser)
Find infrastructure breaking points
Define capacity headroom
100,000+ consumer- grade desktops
168+ countries
2,500+ ISPs
Major mobile carriers around the globe
100+ commercial-grade nodes & data centers

Gomez Network: The World’s Most Comprehensive Performance and Testing Network
Backbone
Virtual Test Bed
Gomez Last Mile
Web Performance Management and Load Testing 100+ locations
Cross-Browser Testing 500+ browser/OS combo’s
5,000+ supported devices
Web Performance Management and Load Testing 100,000+ locations

Reality Load XF – Overview & Demo

SaaS with no investment or maintenance costs and rapid payback
Self-service, full turnkey solution, or tailored to meet your needs with Gomez Professional Services offerings
Tests from an “Outside-in” customer point of view, with drill down to all web application components
Full desktop browser testing across globally distributed geographies

New in July 2009
Simplified Test Provisioning

More easily schedule complex test scenarios
Save and re-use test scenarios
Last Mile UI scheduling improvements

Expanded Load Generation

Additional Geographies for Based Load high volume testing
More Last Mile peer populations

Enhanced Reporting

Detailed reporting for complex scenarios
Filter results by load origin or time
Streamlined data and chart exports

Dollar Thrifty Case Study
Jim Arrowood, DTAG
Director -Web Development and Architecture
8

Who is Dollar Thrifty?

Driven by the mission “Value Every Time,” the Company’s brands, Dollar Rent A Car and Thrifty Car Rental, serve value-conscious travelers in over 70 countries.
Dollar and Thrifty have over 600 corporate and franchised locations in the United States and Canada, operating in virtually all of the top U.S. and Canadian airport markets.
The Company’s approximately 6,400 employees are located mainly in North America, but global service capabilities exist through an expanding international franchise network.

9

Dollar and Thrifty.com
10

The Dollar Thrifty Environment
11

The Problem

In a highly, horizontally scaled environment, development, test, and staging environments rarely 100% match production
Dollar Thrifty significantly redesigned both of our websites in 2008 leaving question as to how the sites will perform in the peak of the 2009 summer
Internal load tests using traditional methods in test and staging environments did not provide the needed confidence

12

The Gomez Relationship
Dollar Thrifty has utilized the Gomez ActiveXF platform for several years
Synthetic Tests of Key Business Processes
Monitors for response time (Home Page Load & Reservation Process)
Monitors for successful execution (Home Page & Reservation Process)
13

Gomez Reality Load
In Early 2009, Dollar Thrifty teamed with Gomez to launch the first external load test of the company’s infrastructure and key software platforms utilizing Gomez Reality Load
14

The Goals
Regardless of how much load, where is our weakest point?
Can we handle our previous year’s peak +25% traffic?
15

Identify Core Business Processes
How do the vast majority of the consumers interact with the site?
Use 80-20 rule
Shop for Rates
Make Reservation
Modify Reservation
Cancel Reservation
16

Distribute Core Business Processes
By percentage, how do consumers interact with the site?
Shop for Rates – 70%*
Make Reservation – 15%*
Modify Reservation – 10%*
Cancel Reservation – 5%*
*Hypothetical
17

Forecast Visitors
Based on historical web analytic data and the goal of the test, compute the intended visitor capacity of the site within an hour
10,000 visitors per hour at peak historically*
If we intend to support peak +25% then our visitor forecast is 12,500 visitors
*Hypothetical
18

Identify the Needed Scripts
Throughput Tests – Where are we weakest?
One script per core business process with no “think-time”
Capacity / Load Tests – How much load can we legitimately handle?
One script per core business process with “think-time”
19

Develop Needed Scripts

Gomez Script Recorder allows the client to create of the script, inject look-up data, and replay the script
RealityLoad utilizes the same scripting engine as ActiveXF
The base of our scripts existed as we already had them built to execute in the ActiveXF platform
Added logic to pull from look-up data
Added think time
Once, complete the tests are loaded into the Gomez portal

20

Configure Tests
Using the Gomez portal, tests are configured and scheduled for execution
21

Execute

Execute tests at off peak time
We chose 2 AM to 4 AM CST
Establish war room with key system owners
Establish bridge line with key infrastructure and monitoring personnel
As tests are executing ensure all internal monitoring tools in all tiers of the application are capturing as much detail as possible without skewing results
CPU, Memory, Disk IO, Network Throughput, etc.

22

Analyze Results
Each system owner and infrastructure owner collects key metrics and findings and submits to the test leader
23

Discuss Key Findings

Thrifty.com front-end servers ran hot while the middle-tier was cold and overpowered
Dollar.com front-end servers ran cool while the middle-tier was running on target
Discovered excessive calls for validation to legacy reservation system
Thrifty.com middle-tier had sticky sessions enabled causing load balancing to become less ineffective

24

Make Corrective Actions & Retest

Mitigate the discovered issues and risks
We were able make a simple configuration change to resolve the “sticky sessions” issue
We were able to move Thrifty middle-tier boxes into the Thrifty.com front-end and Dollar.com middle-tier to better distribute our load with no additional expense
When time and budget allows, retest to ensure mitigating actions resolve the issue and ensure no additional bottlenecks have appeared

25

Demo
Emanuel Daniele
Gomez Reality Load Demo
26

Wrap Up
Questions & Answers
Check back on QA Forums
To find out more about Reality Load:
cmason@gomez.com
mgil@gomez.com
Product Information
http://www.gomez.com/products-solutions/products/load-testing/
2 Minute Explainer
http://www.gomez.com/resources/video-library/gomez-reality-load-testing-two-minute-explainer/

Ensuring Performance Of Login Process
Company: Online presence for a popular TV show

Following episodes of the TV show the web site sees high traffic spikes
Goal was to achieve 1500 logins per minute
Load tested DB to improve performance in anticipation of another traffic spike

Local ISP
Users
Load Balancing
Load Balancing
Web Servers
Web Servers
1
App Servers
App Servers
Internet
DB Servers
DB Servers
MajorISP
Mobile Carrier
Storage
Storage
Mobile Components
Mobile Components

Application Bottleneck Causes Immediate Response Time Issue

As users were added, the response time of step 3 (the login) climbed immediately
The test bottlenecked at 160 logins per minute (Goal 1500)
But quickly dropped off as users received server errors
New login query was not optimized and was bottlenecking the database servers’ CPUs

Application Bottleneck – Re-test

Following the tuning effort the application performance was improved.
This time the bottleneck was at 1300 logins per minute.
A bandwidth limit was reached at just under 90 Mbps, resulting in an overall slowdown as users were added.
This highlights:
The importance of re-testing following each change.
The fact that applications often have many bottlenecks, that can only be uncovered one at a time.

30

Ensuring Performance of All 3rd Party Components
Company: Online Retailer

Several 3rd Parties now involved in serving up key content
Goal was to validate performance of entire application

Local ISP
Users
Load Balancing
Load Balancing
2
Web Servers
Web Servers
App Servers
App Servers
Internet
DB Servers
DB Servers
MajorISP
Mobile Carrier
Storage
Storage
Mobile Components
Mobile Components

Response Times Rise Due To 3rd Party Object Error
The transaction rate increases and then falls off as response times climb
The load increases throughout the test
Errors are seen, all on a 3rd party object

3rd party hardware was insufficient for overall demands on application
Based on SLAs 3rd party had to improve performance to get paid

Ensuring Performance in Key Markets
Company: Regional Online News Source

Began testing for the 2008 election season
Goal was to validate overall performance focusing in 2 key regions

Local ISP
Users
Load Balancing
Load Balancing
Web Servers
Web Servers
3
App Servers
App Servers
Internet
DB Servers
DB Servers
MajorISP
Mobile Carrier
Storage
Storage
Mobile Components
Mobile Components

No Performance Issues Detected From Data-Center
Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%
There was only 1 page error and 11 errors total out of 60000+ transactions
Page response times stayed under 4 seconds, outside of one brief blip
By traditional test standards the test passed

Performance Issues Detected From Real User Desktops
Key geographies for this customer are New York and Pennsylvania
Last Mile data showing substantial number of measurements greater than 4 seconds

Last Mile Case Study: Primary Geographies
Key geographies for this customer are New York and Pennsylvania.
The response time never met the 4 second average goal.
By these standards the test failed.
Availability was Less than 99%.
36