Best Practices
Protect your Online Revenue through Web
Performance Testing
Melinda Carol-Ballou - Application Life-Cycle Management
pp y g
& IT Executive Strategies, IDC
Imad Mouline - CTO Gomez
CTO,

Optimizing Application Quality and
Performance to Drive the Business
Melinda Ballou
Application Life-Cycle Management
IT Executive Strategies
E ti St t i
September 2009
Copyright 2009 IDC. Reproduction is forbidden unless authorized. All rights reserved.

Agenda
Understanding the drivers and challenges for taking a
life-cycle approach to application quality and performance
Coordinating quality and performance from
requirements building through development and
deployment
Establishing process maturity and organizational
strategies for successful application deployments
© 2009 IDC 3

Disruptive Trends Driving Automated
Software Quality Adoption (ASQ)
• Evolving, flexible development paradigm with services creation will
increasingly demand technology and business collaboration and drive for
service quality within and outside the firewall – strong agile emergence
• Changing approaches to application testing as application packages
evolve to new architectures
• End-user experience and business impact challenges of emerging web
technologies (rich Internet), virtualization, cloud computing and Web 2.0
• Global economic competition plus local compliance initiatives across
geographies demand quality, adaptability and rigor
• Complex sourcing/off-shoring necessitate strong teaming and effective
testing and quality management; limited infrastructure creates barriers
• License and delivery mechanisms with SaaS for new models, choices,
customer support, speed adoption, limit complexity (
pp , p p , p y (human and systems)
y )
© 2009 IDC 4

“Quality Gap”: High Cost of Failure
Poor Quality = Increased Business Risk
y
($$$$$)
Lost Revenue Lost Customers Increased Costs
Damaged Brand Lost P d ti it
L t Productivity Lower P fit
L Profits
© 2009 IDC 5

Two Goals of Effective IT/Business
Alignment
Innovation:
Maximize Upside
Through Technology-
Enabled Business
Processes
New Business
Value
Reduced
Exposure
Compliance:
Minimize Downside
Through Risk
g
Management
© 2009 IDC 6

Key IT Challenges: Business-
Criticality
Today’s applications are high-visibility, and carry a
high cost of failure:
cost-of-failure:
Customer “self-serve”
Supplier/channel integration
Internal applications that automate critical business
processes
“Network effect”: failure in one applicationcan mean
loss of service in others.
“Outside-in” Testing is Increasingly Important Due
to Web Application Delivery Chain Complexity
© 2009 IDC 7

Performance Problems: a Leading
Cause of Failure for New Applications
Q. When your software fails to satisfy users, the top 3 reasons are:
Flawed specification
p
Under-estimated resources /time required
Poor performance /capacity/response time
Failure of app. to evolve with business needs
Inability to integrate with other software
Dissatisfaction with user interface
Data quality issues
Flawed implementation of a correct spec
Quality of code /lack of testing
Inefficient or incorrect configuration
0 5 10 15 20 25 30 35
Source: IDC n=467
(% of respondents)
© 2009 IDC 8

IT Business Challenges: Silos, Gaps
The need for a quality life-cycle is key since G2000
organizations are split across interdependent groups:
Business/users stakeholders
Developers
QA professionals
Operational staff
Increased complexity for both technology and source
(offshore, etc.) drives need for coordination
Must extend the Quality life cycle across phases and
life-cycle
groups
© 2009 IDC 9

Rise of Global Teams and Businesses
European
UI Team Dev/QA/
Business Unit
Support APAC
Dev/QA/ Business Unit
Support
US Business
Content
Sponsors
Content
Business
Analysts Dev/QA/
Support
© 2009 IDC 10

Coordinating across the Life-Cycle
• Coordinating requirements, testing, and operational
performance are key across core emerging technologies
• Slow response times for key business areas are problematic
• Organizations should target quality life-cycle approaches
through requirements, unit t t system i t
th h i t it test, t integration and pre-
ti d
deployment and performance testing and change management
• Leveraging test automation for synthetic transaction
monitoring enables users to manage user experience with
applications
• As business requirements change, a cogent life-cycle
approach enables adaptive testing and monitoring for flexible
responses
© 2009 IDC 11

Establishing Process and Org Strategies
Engage Upfront Business input with requirements and QA
and change activities via organizational and process strategy
“Customer advocates” – attuned to business and IT
– Design a test plan based on how real users interact with
your applications
• Where they do it
– LA, Edmonton, London, Bangkok, NY, etc
• What they do
– Key tasks, transactions, and business processes
• How they do it
– Browsers galore, mobile devices
• When they do it (peak times, spikes and how frequently)
Better estimate effort better coverage
Early requirements/test/design helps reveal design p
y q g p g problems –
managed change key
© 2009 IDC 12

Establishing Process and Org Strategies
Involve LoB & Ops in projects earlier
Refine test plans
– Identify key geographies and third-party components to test
– Help define third-party SLAs, and overall “business SLAs”
Coordinate testing of external components and
services delivered from outside the datacenter
Anticipate deployment and configuration issues
Leverage tools, tests and skill sets across the life-cycle
to ensure functionality, availability, and performance
y y p
goals are met
© 2009 IDC 13

Application Quality – Design and Delivery
Quality Evaluation Matrix
•Easy to Use •Easy to Use
•Unpredictable
U di t bl •Behaves as
B h
Behavior Designed Quality of
Design
Customer
Experience
•Difficult to Use •Difficult to Use
•Unpredictable •Behaves as
Behavior
Beha ior Designed
D i d
Delivery
© 2009 IDC 14

App Deployment & Support: Old View
DEV
Define
SLAs, Test/
Provision Monitor Support
Tune
Issues
Result:
Little input into specifications or development
Little leverage between development/ops of
testing/monitoring investments
Testing/tuning LATE in the cycle!
© 2009 IDC 15

Closing the Loop: Leverage Skills &
Tools for Agile, Iterative Approach
•Shared goals,
g ,
•Shared metrics,
•Share tests, Design
tools,and
tools and skills Define
•Shared systems
Develop
Support
Deploy & Test &
Monitor Tune
© 2009 IDC 16

Establish a Common Language
“Sales says “Call Center’s “Network statistics look great.”
customers can t
can’t swamped with “All servers are available.”
All available
find product in the Order Status
database.” inquiries.” “All databases are available.”
“99% of our monitored apps are OK”
“Order “We’re not “Service levels are within limits.”
Service limits.
volume is meeting our first
down.” call resolution
metric today.”
Business IT
Bridge the communication gap!
Establish a common language!
© 2009 IDC 17

Bridging the Customer Experience Gap
“Is the end-to-end
Business
Management
M
business process
working?”
Customer End-to-End Usage Scenarios
g
Experience “Can customers
Management do what they
Web, Voice
Web Voice, Web Services, email FAX
Services email, FAX… want to do?”
do?
Application “Can I proactively
Management fix things before
they impact the
Server, Network & business?”
Element Management
© 2009 IDC 18

Closing a ‘Customer Experience’ Gap
1. Establish a common language between IT and the
business, designate a liaison
2. Agree on business process measurement priorities
3. Implement testing/management for end-to-end
p g g
business processes – from the customer/end-user
perspective
4. Eliminate silos and close the loop between
development and operations
5. S
Share goals, metrics, f
feedback with the entire team:
make Customer Experience Management part of the
culture
© 2009 IDC 19

Summary
•Coordinate a Quality Life-Cycle approach from requirements
through to operations to enable adaptive businesses
•Schisms between business, development, testers and
operations must be addressed
•IT groups and the business must build a common language,
common metrics, and common tools and practices
•Focusing on the customer experience will help IT achieve this
along with core incentives to collaboration
•Making “customer experience” the culture is key to help bridge
the gap and to enable successful collaboration across groups
© 2009 IDC 20

Best Practices: Protecting your Online Revenue
BP #1: Establish a Common Language
between IT and the business
BP #2: Make Customer Experience
Management part of the culture
BP #3: Implement testing &
management for end‐to‐end business
processes from the customer/end
processes – from the customer/end‐
user perspective

BP #1: Establish a Common Language Between IT & the Business:
Customer Satisfaction Index
Apdex (Application Quickly isolate and prioritize
Performance Index)
P f I d ) opportunities for performance
ii f f
Single measurement of improvement based on real
user satisfaction with customer satisfaction
Web performance
W b f Why is customer satisfaction
h f
Based on response time decreasing?
thresholds What do users have in
common? ?
How do I compare against
Satisfied Tolerating Frustrated my competitors?
http://www.apdex.org 2007

Customer Satisfaction Index: An Example
A l di if i d
As load increases customer satisfaction decreases
Customer
satisfaction
decreases
Load
increases

BP #2: Make Customer Experience Management Part of your
Culture ‐ Design your Web site for a Multi‐browser World
Will it work across all browsers?
ill i k ll b ?
Functions correctly
Does not function correctly = brand damage

Make Customer Experience Management Part of your
Culture ‐ Test & Monitor your Key Transactions
Further analysis shows sequence 0 of However sequence 1, the
the transaction - the login page - is authentication step, is
unaffected impacted b a major
i t d by j
performance issue

Make Customer Experience Management Part of your Culture
– Ensure that your Applications Will Scale Under Load
Obama family exclusively
wears leading fashion brand
for inauguration ceremony
Leading fashion brand’s
web application crashes
leading to short and long
term loss of revenue and
brand equity

BP #3: Implement Testing/Management for End‐to‐end
Business Process from the Customer/End‐user Perspective
Behind‐the‐ …user is
firewall testing NOT happy
tools: “OK”
Users
Load
Balancing
Web
Servers
App
Servers The traditional view of Web Application Delivery
The traditional view of Web Application Delivery
DB
Servers
Storage
Mobile
Components
Traditional zone
of control

Critical to Test Across Entire Web Application Delivery Chain
Behind‐the‐ …user is
firewall testing NOT happy
tools: “OK”
The Web Application Delivery Chain
3rd Party/ Browsers
Cloud Services Local ISP and Devices Users
Load • Inconsistent geo performance • Network peering • Poorly
Balancing • Bad performance under load
Bad performance under load p
problems p
performing g
• Blocking content delivery • Bandwidth throttling JavaScript
Web • Inconsistent • Browser/device
Servers • Incorrect geo‐targeted content
connectivity incompatibility
• Configuration • Page size
App • Network peering too big
Servers errors problems
• Application •TToo many
DB design issues • Outages Internet objects
• Network resource • Low cache
Servers • Code defects Major shortage hit rate
• Insufficient ISP • Configuration issues • Faulty content
Storage infrastructure transcoding
• Oversubscribed POP
Mobile • Poor routing optimization SMS routing /
• SMS i /
Components • Low cache hit rate latency issues
Content Delivery Mobile
Networks Carrier
Traditional zone Zone of customer expectation
of control

Load Testing 2.0
Performance Testing For A Web 2.0 World
Self‐service testing gives you
control to test as needed
Test internal & external web
T ti t l& t l b
application components to ensure
transactions perform under load
Test cloud‐based applications &
Test cloud‐based applications &
services
SaaS based Load Testing 2.0
solutions enable on‐demand, ,
scalable testing without associated
hardware investment &
maintenance costs
Combine hi h volume
C bi high l Test the full web application
delivery chain all the way to your
application load testing end users’ desktops
with broad geographic
experience testing from
the outside-in

Gomez Reality Load XF:
On‐Demand Realistic Load Testing from Browser to Data Center
Backbone Last Mile
Last Mile
High volume load (HTTP, Browser) Real‐world load
Find infrastructure breaking points Find user experience breaking points
Define capacity headroom Accurately measure response time
3rd Party/ Browsers
Cloud Services Local ISP and Devices Users
Load 100,000+
Balancing consumer‐
grade
d
Web desktops
Servers
168+
App 100+ countries
Servers commercial‐
grade
DB nodes & Internet 2,500+
data ISPs
Servers Major centers
ISP
Storage Major
mobile
Mobile carriers
around
Components the globe
Content Delivery Mobile
Networks Carrier

Ensuring Performance of All 3rd Party Components
Company: Online Retailer
• Several 3rd Parties now involved in serving up key content
• G l was to validate performance of entire application
Goal t lid t f f ti li ti
3rd Party/
y/ Browsers
Cloud Services Local ISP and devices Users
Load *
Balancing
Web
Servers
App
Servers
DB Internet
Servers Major
ISP
Storage
Mobile
Components
Content Delivery Mobile
Networks Carrier

Response Times Rise Due To 3rd Party Object Error
The load increases The transaction rate
throughout the test increases and then
falls off as response
times climb
Errors are seen, all
on a 3rd party object
• 3 d party h d
3rd t hardware was i
insufficient f overall d
ffi i t for ll demands on application
d li ti
• Based on SLAs 3rd party had to improve performance to get paid

Ensuring Performance in Key Markets
Company: Regional Online News Source
• Began testing for the 2008 election season
• G l was to validate overall performance f
Goal t lid t ll f focusing i 2 k regions
i in key i
3rd Party/
y/ Browsers
Cloud Services Local ISP and devices Users
Load
Balancing
*
Web
Servers
App
Servers
DB Internet
Servers Major
ISP
Storage
Mobile
Components
Content Delivery Mobile
Networks Carrier

No Performance Issues Detected From Data-Center
Increase and hold load and not exceed
response times of 4 seconds and Success There was only
Rate of 99% 1 page error and
11 errors total
out of 60000+
Page response transactions
times stayed under
4 seconds, outside
of one brief blip
By traditional test standards the test passed

Performance Issues Detected From Real User
Desktops
Key geographies for this customer are
New York and Pennsylvania
Last Mile data showing substantial number of
measurements greater than 4 seconds

The Gomez Platform: Web Application Experience Management
Self‐
Self‐Service SaaS Portal
Web Web Load Web Web
Cross‐Browser and Performance Performance Performance
Testing Testing Management Business Analysis
• Compatibility High volume
• High‐volume 20
Real user monitoring
• Real‐user monitoring Web performance
• Web performance
• Functionality backbone load • Last Mile monitoring analytics
• Performance • Real‐world • Backbone monitoring • Benchmarks
Last Mile load • Business dashboard
Web 2.0, RIA, Streaming, Mobile
Web 2 0 RIA Streaming Mobile
Dashboards, Recording & Alerting & APIs & Education &
Metrics & Analytics Provisioning Diagnostics Data Feeds Best Practices
World’s Most Comprehensive Testing Network
168+ countries 2,500+ ISPs
500+ 100+ 5,000+ 100,000+
combos of commercial nodes supported consumer‐grade
browsers & O/S and data centers mobile devices desktops

Gomez Customers Enjoy Measurable Benefits
Increased conversions 10%
Reduced homepage load time from
11.3 seconds to 3 4 seconds
11 3 d t 3.4 d
Improved page load times 23%
Saved 50%+ in staff and fees
Reduced seven‐step
transaction time by 50%
y
Reduced downtime 45%
Achieved under 3 second
Achieved under 3 second
response time and 99%+
availability
Validated decision to consolidate
Validated decision to consolidate
three data centers

Appendix

Ensuring Performance Of Login Process
Company: Online presence for a popular TV show
• Following episodes of the TV show the web site sees high traffic spikes
• Goal was to achieve 1500 logins per minute
• Load tested DB to improve performance in anticipation of another traffic spike
3rd Party/
y/ Browsers
Cloud Services Local ISP and devices Users
Load
Balancing 1
Web
Servers
App
Servers
DB Internet
Servers Major
ISP
Storage
Mobile
Components
Content Delivery Mobile
Networks Carrier

Application Bottleneck Causes Immediate Response
Time Issue
• As users were added the
added,
response time of step 3 (the
login) climbed immediately
• The test bottlenecked at 160
logins per minute (Goal 1500)
• But quickly dropped off as
users received server errors
• New login query was not
optimized and was
bottlenecking the database
servers’ CPU
’ CPUs