Best Practices
Protect your Online Revenue through Web
Performance Testing


  Melinda Carol-Ballou - Application Life-Cyc...
Optimizing Application Quality and
 Performance to Drive the Business


Melinda Ballou
Application Life-Cycle Management
I...
Agenda


          Understanding the drivers and challenges for taking a
      life-cycle approach to application quality ...
Disruptive Trends Driving Automated
Software Quality Adoption (ASQ)
• Evolving, flexible development paradigm with service...
“Quality Gap”: High Cost of Failure

       Poor Quality = Increased Business Risk
                  y

        ($$$$$)
  ...
Two Goals of Effective IT/Business
Alignment

             Innovation:
       Maximize Upside
     Through Technology-
   ...
Key IT Challenges: Business-
Criticality

       Today’s applications are high-visibility, and carry a
     high cost of f...
Performance Problems: a Leading
Cause of Failure for New Applications
     Q. When your software fails to satisfy users, t...
IT Business Challenges: Silos, Gaps
  The need for a quality life-cycle is key since G2000
 organizations are split across...
Rise of Global Teams and Businesses


                                    European
      UI Team    Dev/QA/
              ...
Coordinating across the Life-Cycle

• Coordinating requirements, testing, and operational
performance are key across core ...
Establishing Process and Org Strategies
 Engage Upfront Business input with requirements and QA
and change activities via ...
Establishing Process and Org Strategies

     Involve LoB & Ops in projects earlier
             Refine test plans
       ...
Application Quality – Design and Delivery

                   Quality Evaluation Matrix

                  •Easy to Use   ...
App Deployment & Support: Old View


 DEV

                          Define
                          SLAs,           Test...
Closing the Loop: Leverage Skills &
             Tools for Agile, Iterative Approach

 •Shared goals,
          g     ,
 •...
Establish a Common Language

    “Sales says       “Call Center’s              “Network statistics look great.”
  customer...
Bridging the Customer Experience Gap

                                                           “Is the end-to-end
   Bus...
Closing a ‘Customer Experience’ Gap

    1. Establish a common language between IT and the
       business, designate a li...
Summary

•Coordinate a Quality Life-Cycle approach from requirements
through to operations to enable adaptive businesses
•...
Best Practices: Protecting your Online Revenue

                       BP #1: Establish a Common Language
                ...
BP #1: Establish a Common Language Between IT & the Business:
Customer Satisfaction Index 



  Apdex (Application        ...
Customer Satisfaction Index: An Example

  A l di                        if i d
  As load increases customer satisfaction ...
BP #2: Make Customer Experience Management Part of your 
Culture ‐ Design your Web site for a Multi‐browser World
 Will it...
Make Customer Experience Management Part of your 
Culture ‐ Test & Monitor your Key Transactions


  Further analysis show...
Make Customer Experience Management Part of your Culture 
– Ensure that your Applications Will Scale Under Load

         ...
BP #3: Implement Testing/Management  for End‐to‐end 
  Business Process from the Customer/End‐user Perspective 
    Behind...
Critical to Test Across Entire Web Application Delivery Chain

    Behind‐the‐                                            ...
Load Testing 2.0
 Performance Testing For A Web 2.0 World

                                Self‐service testing gives you ...
Gomez Reality Load XF: 
On‐Demand Realistic Load Testing from Browser to Data Center
                                  Bac...
Ensuring Performance of All 3rd Party Components

     Company: Online Retailer
      • Several 3rd Parties now involved i...
Response Times Rise Due To 3rd Party Object Error

The load increases                              The transaction rate
th...
Ensuring Performance in Key Markets

     Company: Regional Online News Source
      • Began testing for the 2008 election...
No Performance Issues Detected From Data-Center

Increase and hold load and not exceed
response times of 4 seconds and Suc...
Performance Issues Detected From Real User
Desktops


Key geographies for this customer are
     New York and Pennsylvania...
The Gomez Platform: Web Application Experience Management

                                   Self‐
                      ...
Gomez Customers Enjoy Measurable Benefits

Increased conversions 10% 

Reduced homepage load time from 
11.3 seconds to 3 ...
Appendix
Ensuring Performance Of Login Process

     Company: Online presence for a popular TV show
          • Following episodes ...
Application Bottleneck Causes Immediate Response
Time Issue



                            • As users were added the
     ...
Upcoming SlideShare
Loading in …5
×

IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web Performance Testing

1,534 views

Published on

Did you know that 85% of users complain about slow response time? Poor web application performance can directly impact your bottom line

The success of your critical eBusiness initiatives depends on your ability to deliver quality web experiences. Unfortunately, 65% of applications are not properly load tested prior to launch, resulting in lost revenue, increased support costs and brand damage. So how can you ensure success when launching new applications, adding features, deploying new infrastructure, rolling out marketing campaigns, or preparing for seasonal spikes like the holiday shopping season?

Join us as our guest speaker, Melinda Ballou, IDC’s Program Director for Application Life-Cycle Management research discusses challenges, drivers and best practices for effective web performance testing and quality life-cycle management for today’s rich and complex applications. Additional topics that Imad Mouline, Gomez’s CTO will cover in this session are:

Best practices for ensuring the success of critical eBusiness initiatives
The end-user experience and business impact of emerging web technologies like Rich Internet Applications, virtualization, cloud computing and Web 2.0
A new approach for web performance and load testing that’s easy to use, delivered on-demand, and enables you to find and fix problems before they impact customers
Who Should Watch: Line of Business and eCommerce Managers, Interactive Marketing, Brand Managers, Project Managers and IT Operations Executives.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,534
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
73
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web Performance Testing

  1. 1. Best Practices Protect your Online Revenue through Web Performance Testing Melinda Carol-Ballou - Application Life-Cycle Management pp y g & IT Executive Strategies, IDC Imad Mouline - CTO Gomez CTO,
  2. 2. Optimizing Application Quality and Performance to Drive the Business Melinda Ballou Application Life-Cycle Management IT Executive Strategies E ti St t i September 2009 Copyright 2009 IDC. Reproduction is forbidden unless authorized. All rights reserved.
  3. 3. Agenda Understanding the drivers and challenges for taking a life-cycle approach to application quality and performance Coordinating quality and performance from requirements building through development and deployment Establishing process maturity and organizational strategies for successful application deployments © 2009 IDC 3
  4. 4. Disruptive Trends Driving Automated Software Quality Adoption (ASQ) • Evolving, flexible development paradigm with services creation will increasingly demand technology and business collaboration and drive for service quality within and outside the firewall – strong agile emergence • Changing approaches to application testing as application packages evolve to new architectures • End-user experience and business impact challenges of emerging web technologies (rich Internet), virtualization, cloud computing and Web 2.0 • Global economic competition plus local compliance initiatives across geographies demand quality, adaptability and rigor • Complex sourcing/off-shoring necessitate strong teaming and effective testing and quality management; limited infrastructure creates barriers • License and delivery mechanisms with SaaS for new models, choices, customer support, speed adoption, limit complexity ( pp , p p , p y (human and systems) y ) © 2009 IDC 4
  5. 5. “Quality Gap”: High Cost of Failure Poor Quality = Increased Business Risk y ($$$$$) Lost Revenue Lost Customers Increased Costs Damaged Brand Lost P d ti it L t Productivity Lower P fit L Profits © 2009 IDC 5
  6. 6. Two Goals of Effective IT/Business Alignment Innovation: Maximize Upside Through Technology- Enabled Business Processes New Business Value Reduced Exposure Compliance: Minimize Downside Through Risk g Management © 2009 IDC 6
  7. 7. Key IT Challenges: Business- Criticality Today’s applications are high-visibility, and carry a high cost of failure: cost-of-failure: Customer “self-serve” Supplier/channel integration Internal applications that automate critical business processes “Network effect”: failure in one applicationcan mean loss of service in others. “Outside-in” Testing is Increasingly Important Due to Web Application Delivery Chain Complexity © 2009 IDC 7
  8. 8. Performance Problems: a Leading Cause of Failure for New Applications Q. When your software fails to satisfy users, the top 3 reasons are: Flawed specification p Under-estimated resources /time required Poor performance /capacity/response time Failure of app. to evolve with business needs Inability to integrate with other software Dissatisfaction with user interface Data quality issues Flawed implementation of a correct spec Quality of code /lack of testing Inefficient or incorrect configuration 0 5 10 15 20 25 30 35 Source: IDC n=467 (% of respondents) © 2009 IDC 8
  9. 9. IT Business Challenges: Silos, Gaps The need for a quality life-cycle is key since G2000 organizations are split across interdependent groups: Business/users stakeholders Developers QA professionals Operational staff Increased complexity for both technology and source (offshore, etc.) drives need for coordination Must extend the Quality life cycle across phases and life-cycle groups © 2009 IDC 9
  10. 10. Rise of Global Teams and Businesses European UI Team Dev/QA/ Business Unit Support APAC Dev/QA/ Business Unit Support US Business Content Sponsors Content Business Analysts Dev/QA/ Support © 2009 IDC 10
  11. 11. Coordinating across the Life-Cycle • Coordinating requirements, testing, and operational performance are key across core emerging technologies • Slow response times for key business areas are problematic • Organizations should target quality life-cycle approaches through requirements, unit t t system i t th h i t it test, t integration and pre- ti d deployment and performance testing and change management • Leveraging test automation for synthetic transaction monitoring enables users to manage user experience with applications • As business requirements change, a cogent life-cycle approach enables adaptive testing and monitoring for flexible responses © 2009 IDC 11
  12. 12. Establishing Process and Org Strategies Engage Upfront Business input with requirements and QA and change activities via organizational and process strategy “Customer advocates” – attuned to business and IT – Design a test plan based on how real users interact with your applications • Where they do it – LA, Edmonton, London, Bangkok, NY, etc • What they do – Key tasks, transactions, and business processes • How they do it – Browsers galore, mobile devices • When they do it (peak times, spikes and how frequently) Better estimate effort better coverage Early requirements/test/design helps reveal design p y q g p g problems – managed change key © 2009 IDC 12
  13. 13. Establishing Process and Org Strategies Involve LoB & Ops in projects earlier Refine test plans – Identify key geographies and third-party components to test – Help define third-party SLAs, and overall “business SLAs” Coordinate testing of external components and services delivered from outside the datacenter Anticipate deployment and configuration issues Leverage tools, tests and skill sets across the life-cycle to ensure functionality, availability, and performance y y p goals are met © 2009 IDC 13
  14. 14. Application Quality – Design and Delivery Quality Evaluation Matrix •Easy to Use •Easy to Use •Unpredictable U di t bl •Behaves as B h Behavior Designed Quality of Design Customer Experience •Difficult to Use •Difficult to Use •Unpredictable •Behaves as Behavior Beha ior Designed D i d Delivery © 2009 IDC 14
  15. 15. App Deployment & Support: Old View DEV Define SLAs, Test/ Provision Monitor Support Tune Issues Result: Little input into specifications or development Little leverage between development/ops of testing/monitoring investments Testing/tuning LATE in the cycle! © 2009 IDC 15
  16. 16. Closing the Loop: Leverage Skills & Tools for Agile, Iterative Approach •Shared goals, g , •Shared metrics, •Share tests, Design tools,and tools and skills Define •Shared systems Develop Support Deploy & Test & Monitor Tune © 2009 IDC 16
  17. 17. Establish a Common Language “Sales says “Call Center’s “Network statistics look great.” customers can t can’t swamped with “All servers are available.” All available find product in the Order Status database.” inquiries.” “All databases are available.” “99% of our monitored apps are OK” “Order “We’re not “Service levels are within limits.” Service limits. volume is meeting our first down.” call resolution metric today.” Business IT Bridge the communication gap! Establish a common language! © 2009 IDC 17
  18. 18. Bridging the Customer Experience Gap “Is the end-to-end Business Management M business process working?” Customer End-to-End Usage Scenarios g Experience “Can customers Management do what they Web, Voice Web Voice, Web Services, email FAX Services email, FAX… want to do?” do? Application “Can I proactively Management fix things before they impact the Server, Network & business?” Element Management © 2009 IDC 18
  19. 19. Closing a ‘Customer Experience’ Gap 1. Establish a common language between IT and the business, designate a liaison 2. Agree on business process measurement priorities 3. Implement testing/management for end-to-end p g g business processes – from the customer/end-user perspective 4. Eliminate silos and close the loop between development and operations 5. S Share goals, metrics, f feedback with the entire team: make Customer Experience Management part of the culture © 2009 IDC 19
  20. 20. Summary •Coordinate a Quality Life-Cycle approach from requirements through to operations to enable adaptive businesses •Schisms between business, development, testers and operations must be addressed •IT groups and the business must build a common language, common metrics, and common tools and practices •Focusing on the customer experience will help IT achieve this along with core incentives to collaboration •Making “customer experience” the culture is key to help bridge the gap and to enable successful collaboration across groups © 2009 IDC 20
  21. 21. Best Practices: Protecting your Online Revenue BP #1: Establish a Common Language between IT and the business BP #2: Make Customer Experience  Management part of the culture BP #3: Implement testing &  management for end‐to‐end business  processes  from the customer/end processes – from the customer/end‐ user perspective
  22. 22. BP #1: Establish a Common Language Between IT & the Business: Customer Satisfaction Index  Apdex (Application  Quickly isolate and prioritize  Performance Index) P f I d ) opportunities for performance  ii f f Single measurement of  improvement based on real  user satisfaction with  customer satisfaction Web performance   W b f Why is customer satisfaction  h f Based on response time   decreasing?  thresholds What do users have in  common?  ? How do I compare against  Satisfied Tolerating Frustrated my competitors? http://www.apdex.org 2007
  23. 23. Customer Satisfaction Index: An Example A l di if i d As load increases customer satisfaction decreases Customer satisfaction decreases Load increases
  24. 24. BP #2: Make Customer Experience Management Part of your  Culture ‐ Design your Web site for a Multi‐browser World Will it work across all browsers? ill i k ll b ? Functions correctly Does not function correctly = brand damage
  25. 25. Make Customer Experience Management Part of your  Culture ‐ Test & Monitor your Key Transactions Further analysis shows sequence 0 of However sequence 1, the the transaction - the login page - is authentication step, is unaffected impacted b a major i t d by j performance issue
  26. 26. Make Customer Experience Management Part of your Culture  – Ensure that your Applications Will Scale Under Load Obama family exclusively wears leading fashion brand for inauguration ceremony Leading fashion brand’s web application crashes leading to short and long term loss of revenue and brand equity
  27. 27. BP #3: Implement Testing/Management  for End‐to‐end  Business Process from the Customer/End‐user Perspective  Behind‐the‐ …user is  firewall testing  NOT happy tools: “OK” Users Load  Balancing Web  Servers App  Servers The traditional view of Web Application Delivery The traditional view of Web Application Delivery DB  Servers Storage Mobile  Components Traditional zone  of control
  28. 28. Critical to Test Across Entire Web Application Delivery Chain Behind‐the‐ …user is  firewall testing  NOT happy tools: “OK” The Web Application Delivery Chain 3rd Party/ Browsers  Cloud Services Local ISP and Devices Users Load  • Inconsistent geo performance • Network peering  • Poorly  Balancing • Bad performance under load Bad performance under load p problems p performing g • Blocking content delivery • Bandwidth throttling JavaScript Web  • Inconsistent  • Browser/device  Servers • Incorrect geo‐targeted content connectivity incompatibility • Configuration  • Page size  App  • Network peering  too big Servers errors problems • Application  •TToo many  DB  design issues • Outages Internet objects • Network resource  • Low cache  Servers • Code defects Major shortage  hit rate • Insufficient  ISP • Configuration issues • Faulty content  Storage infrastructure transcoding  • Oversubscribed POP Mobile  • Poor routing optimization SMS routing /  • SMS i / Components • Low cache hit rate latency issues  Content Delivery Mobile  Networks Carrier Traditional zone  Zone of customer expectation of control
  29. 29. Load Testing 2.0 Performance Testing For A Web 2.0 World Self‐service testing gives you  control to test as needed Test internal & external web  T ti t l& t l b application components to ensure  transactions perform under load Test cloud‐based applications &  Test cloud‐based applications & services SaaS based Load Testing 2.0  solutions enable on‐demand,  , scalable testing without associated  hardware investment &  maintenance costs Combine hi h volume C bi high l Test the full web application  delivery chain all the way to your  application load testing end users’ desktops with broad geographic experience testing from the outside-in
  30. 30. Gomez Reality Load XF:  On‐Demand Realistic Load Testing from Browser to Data Center Backbone Last Mile Last Mile High volume load (HTTP, Browser) Real‐world load Find infrastructure breaking points Find user experience breaking points Define capacity headroom Accurately measure response time 3rd Party/ Browsers  Cloud Services Local ISP and Devices Users Load  100,000+  Balancing consumer‐ grade  d Web  desktops Servers 168+  App  100+  countries Servers commercial‐ grade  DB  nodes & Internet 2,500+  data  ISPs Servers Major centers ISP Storage Major  mobile  Mobile  carriers  around  Components the globe Content Delivery Mobile  Networks Carrier
  31. 31. Ensuring Performance of All 3rd Party Components Company: Online Retailer • Several 3rd Parties now involved in serving up key content • G l was to validate performance of entire application Goal t lid t f f ti li ti 3rd Party/ y/ Browsers  Cloud Services Local ISP and devices Users Load  * Balancing Web  Servers App  Servers DB  Internet Servers Major ISP Storage Mobile  Components Content Delivery Mobile  Networks Carrier
  32. 32. Response Times Rise Due To 3rd Party Object Error The load increases The transaction rate throughout the test increases and then falls off as response times climb Errors are seen, all on a 3rd party object • 3 d party h d 3rd t hardware was i insufficient f overall d ffi i t for ll demands on application d li ti • Based on SLAs 3rd party had to improve performance to get paid
  33. 33. Ensuring Performance in Key Markets Company: Regional Online News Source • Began testing for the 2008 election season • G l was to validate overall performance f Goal t lid t ll f focusing i 2 k regions i in key i 3rd Party/ y/ Browsers  Cloud Services Local ISP and devices Users Load  Balancing * Web  Servers App  Servers DB  Internet Servers Major ISP Storage Mobile  Components Content Delivery Mobile  Networks Carrier
  34. 34. No Performance Issues Detected From Data-Center Increase and hold load and not exceed response times of 4 seconds and Success There was only Rate of 99% 1 page error and 11 errors total out of 60000+ Page response transactions times stayed under 4 seconds, outside of one brief blip By traditional test standards the test passed
  35. 35. Performance Issues Detected From Real User Desktops Key geographies for this customer are New York and Pennsylvania Last Mile data showing substantial number of measurements greater than 4 seconds
  36. 36. The Gomez Platform: Web Application Experience Management Self‐ Self‐Service SaaS Portal Web  Web Load  Web  Web  Cross‐Browser  and Performance  Performance  Performance Testing Testing Management Business Analysis • Compatibility High volume  • High‐volume 20 Real user monitoring • Real‐user monitoring Web performance  • Web performance • Functionality backbone load • Last Mile monitoring analytics • Performance • Real‐world  • Backbone monitoring • Benchmarks Last Mile load • Business dashboard Web 2.0, RIA, Streaming, Mobile  Web 2 0 RIA Streaming Mobile Dashboards,  Recording &  Alerting &  APIs & Education & Metrics & Analytics Provisioning Diagnostics Data Feeds  Best Practices World’s Most Comprehensive Testing Network 168+ countries    2,500+ ISPs 500+  100+  5,000+  100,000+  combos of  commercial nodes  supported  consumer‐grade  browsers & O/S and data centers mobile devices desktops
  37. 37. Gomez Customers Enjoy Measurable Benefits Increased conversions 10%  Reduced homepage load time from  11.3 seconds to 3 4 seconds 11 3 d t 3.4 d Improved page load times 23% Saved 50%+ in staff and fees Reduced seven‐step  transaction time by 50% y Reduced downtime 45%  Achieved under 3 second  Achieved under 3 second response time and 99%+  availability Validated decision to consolidate  Validated decision to consolidate three data centers
  38. 38. Appendix
  39. 39. Ensuring Performance Of Login Process Company: Online presence for a popular TV show • Following episodes of the TV show the web site sees high traffic spikes • Goal was to achieve 1500 logins per minute • Load tested DB to improve performance in anticipation of another traffic spike 3rd Party/ y/ Browsers  Cloud Services Local ISP and devices Users Load  Balancing 1 Web  Servers App  Servers DB  Internet Servers Major ISP Storage Mobile  Components Content Delivery Mobile  Networks Carrier
  40. 40. Application Bottleneck Causes Immediate Response Time Issue • As users were added the added, response time of step 3 (the login) climbed immediately • The test bottlenecked at 160 logins per minute (Goal 1500) • But quickly dropped off as users received server errors • New login query was not optimized and was bottlenecking the database servers’ CPU ’ CPUs

×