Testing Tools Landscape


Published on

Quality must move beyond the purview of just the testing organization and must become an integrated part of the entire software development life cycle (SDLC) to reduce schedule-killing rework, improve user satisfaction, and reduce the risks of untested nonfunctional requirements such as security and performance. Vendors have taken note and look to provide tools that support every role in the organization, considerably broadening the testing tools landscape.

Join Margo Visitacion of Forrester and learn:

* Don't lose before you get into the game
* Why load testing can make the difference
* How planning performance testing today can help budget planning tomorrow
* How to develop your test game plan

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Delete second Name/Title if not needed, but be sure to keep the Month Day, Year information. For a teleconference, after the date, add a period and “Call in at XX:55 p.m. Eastern time” (change the time to five minutes prior to the start of the teleconference).
  • Don’t lose before you get into the gameWhy load testing can make the differencePlanning performance testing today can help budget planning tomorrowDeveloping your test game plan
  • Source: http://www.flickr.com/photos/elzey/3985788016/
  • Talk about rich internet apps, more media, very appealing, but
  • Don’t lose before you get into the gameWhy load testing can make the differencePlanning performance testing today can help budget planning tomorrowDeveloping your test game plan
  • Dial
  • Don’t lose before you get into the gameWhy load testing can make the differencePlanning performance testing today can help budget planning tomorrowDeveloping your test game plan
  • Don’t lose before you get into the gameWhy load testing can make the differencePlanning performance testing today can help budget planning tomorrowDeveloping your test game plan
  • Don’t lose before you get into the gameWhy load testing can make the differencePlanning performance testing today can help budget planning tomorrowDeveloping your test game plan
  • Not only are the usage of social and mobile increasing, but customer expectations about the performance are extremely high.
  • Matt tie into how they do it and when they do it… normal and peak usage!
  • When we talk about applications now, we are increasing talking about applications that are not isolated, and consist of a number of components delivered by a Number of hosts accessed directly by the browser, and assembled at the client end. This necessitates a different approach to testing and Load testing specifically.
  • Common Mistake:$ cost of marketing campaigns associated with driving peak traffic – driving priorityBusiness Objectives and KPIsvs the IT team’s ability to deliver.Impact on Business:Lost sales due to the site not being availableLessons Learned (case example):Nationwide – $1.5 Mill marketing to MicrositeBOA – Used microsite to generate leads – drove traffic to find breaking point (What can they support?)Informing Business of what the site can handle before they drive traffic to it
  • Common Mistake:Organizations think that performance management/testing is only an IT problemBusiness team is not asked to (or able to) provide key web metrics to inform the load test planBusiness team is not informing (or being asked to inform) IT of performance objectives to allow for business goals to be metBusiness team not informing IT of business/traffic growth projections for “next year”IT is not open to involving business inputIT has to be honest about their ability to deliver what they have to deliver with what they haveImpact on Business:“Successful” load test; unsuccessful peak traffic eventLessons Learned (case example):Retail Brand Alliance – regular testing with business involvement allows for confidence, sales success, and more effective planning for next year.???
  • Q & a session
  • Testing Tools Landscape

    1. 1.
    2. 2. Testing Tools - How Teams Make It Work<br />Margo Visitacion, Vice President<br />February 17, 2011<br />
    3. 3. Don’t lose before you get into the game……<br />
    4. 4. Agenda<br />
    5. 5. If you don’t plan your quality and build the right team, you lose before you get into the game…….<br />
    6. 6. More business is transacted online….<br />
    7. 7. But problems increase your risk and costs …<br />Performance:<br />It only takes a moment to lose customers<br />74% of surveyed customers shop online….<br />…. Yet 54% abandon shopping carts, a continuing plague for online vendors vendors<br />Security<br />Online banking: the future is now<br />90.5 million Americans now bank online…..<br />….Yet less than half of customers feel secure about their institutions’ security<br />Scalability<br />70% of auto policies renew in January<br />The global Internet population is now 1.6B and expects to grow another 42% by 2014<br />Do you have the right team in place?<br />
    8. 8. In 2010 more money was directed to toward growth and expansion – don’t let poor performance shift dollars away from innovation<br />Poor quality diverts focus<br />
    9. 9. Agenda<br />
    10. 10. Siloed approach to testing won’t let you answer these questions<br />
    11. 11. Companies are planning to spend more on securing their customer’s loyalty<br />
    12. 12. How will you change your behaviors to improve your approach to testing??<br />How will you “Change your ways”?<br />Challenges in testing are growing <br />Performance, Data privacy, compliance<br />Testing as usual won’t work today<br />Early testing is imperative<br />Teams look different because testing approaches have to change<br /> IT budgets look better, but must bring value<br />Companies want greater flexibility for their $$$<br />Automation MUST bring value for the money invested<br /> Testing is ripe for SaaS<br />21% are piloting SaaS for IT lifecycle components<br />26% are planning to adopt within the next year<br />
    13. 13. Taking a team approach<br />
    14. 14. Testing can begin from day one to prevent defects<br />Post <br />Mortem<br />Deployment<br />Business determinesobjectives, sets requirements<br />End to end testing<br />UAT<br />Business stakeholders, BAs & test team set quality and performance requirements for functional and non functional requirements<br />Development &testing continues<br />Continuous build and integrationContinuous testingmanual and automated<br />Static and automated performance testing starts<br />Design and developmentbased upon requirements<br />Set test planning &<br />Automation strategy<br />
    15. 15. Use automation to keep up with change<br /> Capital investments are scrutinized<br /> Infrastructure and environments are challenged<br /> Ramp up time takes too long<br /> Usability is a requirement<br /> Quick time to value is a necessity<br /> Performance testing is cyclical but non negotiable<br />Testing is an enabler<br />Environment<br />sans<br />constraints<br />
    16. 16. Agenda<br />
    17. 17. Costs: A drill down<br />
    18. 18. Why? Investing in quality can reduce risk, cost and raise value<br />
    19. 19. Plan Today For Tomorrow’s Budget<br />
    20. 20. Agenda<br />
    21. 21. To win the game, the business needs to be involved<br />Requirements<br />Userstories<br /> “Situational realities”<br />Risks/opportunities<br />
    22. 22. Bringing the team together needs a repeatable, measurable process<br />
    23. 23. Agenda<br />
    24. 24. Recommendations<br />Don’t leave performance testing to the end. <br />Problems will grow as business options expand <br />Anticipate the unexpected<br />Life is what happens while you’re busy making other plans<br />Leverage automation and timing<br />Flexible tools to performance-test, “just in time”<br />
    25. 25. Thank you<br />Margo Visitacion<br />+1 609.314.6745<br />mvisitacion@forrester.com<br />www.forrester.com<br />
    26. 26. Testing Tools Landscape- Load Testing<br />Matt Poepsel, VP of Performance Strategies, Compuware Gomez<br />
    27. 27. Customer Expectations Are Rising For Web & Mobile Applications<br />47% of consumers expect a Web page to load in 2 seconds or less <br />58% of mobile phone users expect websites to load almost as quickly or faster on their mobile phone than their PC<br />
    28. 28. The Challenge Of Delivering Quality Web Experiences<br />Systems management tools: “OK”<br />…user is NOT happy<br />The Web Application Delivery Chain<br />Test & monitor your site the way your customers use it:<br /><ul><li>What they do (key pages and transactions)
    29. 29. Where they do it (geographic locations)
    30. 30. How they do it (browsers and mobile devices)
    31. 31. When they do it (normal and peak usage)</li></ul>Determine the impact on their behavior and your business<br /><ul><li>Inconsistent geo performance
    32. 32. Bad performance under load
    33. 33. Blocking content delivery
    34. 34. Incorrect geo-targeted content
    35. 35. Poorly performing JavaScript
    36. 36. Inconsistent CSS rendering
    37. 37. Browser/device incompatibility
    38. 38. Page size too big
    39. 39. Conflicting HTML tag support
    40. 40. Too many objects
    41. 41. Content not optimized for device
    42. 42. Low cache hit rate
    43. 43. Network peering problems
    44. 44. Bandwidth throttling
    45. 45. Inconsistent connectivity
    46. 46. Configuration errors
    47. 47. Application design issues
    48. 48. Code defects
    49. 49. Insufficient infrastructure
    50. 50. Network peering problems
    51. 51. Outages</li></ul>“Outside-in” customer point of view<br /><ul><li>Network resource shortage
    52. 52. Faulty content transcoding
    53. 53. SMS routing / latency issues
    54. 54. Configuration issues
    55. 55. Oversubscribed POP
    56. 56. Poor routing optimization
    57. 57. Low cache hit rate</li></li></ul><li>Applications are no longer what you build inside your firewall<br />10.5<br />Number of hosts accessed directly by the browser, <br />per user transaction, averaged across 3,000 companies<br />
    58. 58. Who Owns End-User Quality?<br />Line of Business Managers<br />Product Managers<br />Marketers<br />Interface Designers<br />Application Developers<br />Architects<br />Quality Assurance Teams<br />Infrastructure/Operations Teams<br />Help Desk Staff<br />Quality is a Collective Function.<br />Everybody owns a piece of the puzzle.<br />Avoid at all costs:<br />Individual Success, Collective Failure<br />
    59. 59. How can we work together?<br />
    60. 60. Situational Assessment and Improvement<br />Provide this 5-minute assessment to each of the End-User Quality contributors for your application:<br />The results provide a baseline. Share the results across the team.<br />Identify areas of improvement and execute a plan to get better.<br />Re-administer the assessment after 90 days.<br />On a scale of 1 (poor) to 10 (excellent), <br />how well are we are doing in these areas:<br />Shared Goals<br />Shared Metrics<br />Communications<br />Solving problems as a team<br />Celebrating as a team<br />
    61. 61. Questions<br />Increased conversions 10% <br />Gomez Customers Enjoy Measurable Benefits<br />Reduced homepage load time from 11.3 seconds to 3.4 seconds<br />Improved page load times 23%<br />Saved 50%+ in staff and fees<br />Reduced seven-step transaction time by 50%<br />Reduced downtime 45% <br />Achieved under 3 second response time and 99%+ availability<br />Validated decision to consolidate three data centers<br />For more information visit Gomez.comorcontact us at +1 781.778.2700<br />