Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Survey Says: Testers Spend Their Time Doing...

433 views

Published on

How can testers contribute more to the success of their project and their company? How can they focus on asking the right questions, improving test planning and design, and finding defects so the business releases a quality product―even though there’s always one more fire to extinguish or one more request to fulfill? There aren’t enough hours in the day to do it all. Join Al Wagner as he reveals recent survey results showing where testers actually spend their time and where testers think their time would be better spent. Compare your own experience with what 250 test professionals from around the world reported. You may be surprised how prevalent testing challenges really are. Learn what techniques and technologies are available to help today’s test professionals execute what they were actually hired to do—test software. Return to your organization with an increased understanding of how other testers are dealing with their testing bottlenecks and what activities your peers view as the best use of their valuable time.

Published in: Software
  • Be the first to comment

  • Be the first to like this

The Survey Says: Testers Spend Their Time Doing...

  1. 1. 9/11/2014 1 © 2013 IBM Corporation @Al_Wagner IBM Rational Testers spend their time doing… The 1 © 2013 IBM Corporation Session Abstract How can testers contribute more to the success of their project and their company? How can they focus on asking the right questions, improving test planning and design, and finding defects so the business releases a quality product―even though there’s always one more fire to extinguish or one more request to fulfill? There aren’t enough hours in the day to do it all. Join Al Wagner as he reveals recent survey results showing where testers actually spend their time and where testers think their time would be better spent. Compare your own experience with what 250 test professionals from around the world reported. You may be surprised how prevalent testing challenges really are. Learn what techniques and technologies are available to help today’s test professionals execute what they were actually hired to do—test software. Return to your organization with an increased understanding of how other testers are dealing with their testing bottlenecks and what activities your peers view as the best use of their valuable time.
  2. 2. 9/11/2014 2 2 © 2013 IBM Corporation Why run a survey? Aside from talking to directly to customers, we were…  watching forums and blogs  reading papers  following social media channels  listening to the Analysts What we found was a small sampling of data harvested from anecdotes and discussions between small groups. We wanted to hear directly from the Testers! We wanted to hear from a lot of Testers! 3 © 2013 IBM Corporation Agenda – Survey Says  Setting the stage  Challenges testers face  Help is available  Putting it in context
  3. 3. 9/11/2014 3 4 © 2013 IBM Corporation Who responded? 250 Testing Professionals from around the world! Testing Experience Testing Roles Quality Managers, Test Managers, and Test Leads with more than 6 years experience! 5 © 2013 IBM Corporation What and how are they testing? 250 Testing Professionals dedicated to improving software quality Types of applications being tested Development methodology followed 43% cite Scrum as their main development methodology. 56% of organizations are using some variation of iterative development.
  4. 4. 9/11/2014 4 6 © 2013 IBM Corporation What tools and techniques are being used? 61% of Testers surveyed use documents & spreadsheets to manage software quality and test execution. Is this effective? You decide! 7 © 2013 IBM Corporation Agenda  Setting the stage  Challenges testers face  Help is available  Putting it in context
  5. 5. 9/11/2014 5 8 © 2013 IBM Corporation What’s The Rush? Better – Cheaper - Faster In The Past • Organizations looked to find a balance between speed, cost & risk. • Development teams were not as agile as they are today; testing never seemed to have enough time. Today • Organizations are asking teams to release higher quality software sooner. • Development teams are becoming more and more agile; testing still never seems to have enough time. 9 © 2013 IBM Corporation Change in Release Frequency How have Release Cycles changed? …over last 3 years 32% of Test Professionals surveyed have seen an increase in release frequency over the past 3 years. 34% of those who expressed no change were already releasing quarterly or better.
  6. 6. 9/11/2014 6 10 © 2013 IBM Corporation What is the ratio of Testers to Programmers on a delivery team in your organization? 11 © 2013 IBM Corporation Testing takes as long as it takes! Really? Testing will take longer when you are not “actually” testing. How Much of Your 40-Hour Workweek Is Spent on Unplanned, Non-Testing Activities? Most Disruptive non-Testing Activities! 35% of Testers are spending at least ½ of their week not testing!
  7. 7. 9/11/2014 7 12 © 2013 IBM Corporation How Confident Are You in Your Product's Quality? 44% of Test Professionals surveyed have little to no confidence in the quality of the software being released. 13 © 2013 IBM Corporation Where and How Often Do You Find Defects? 40% of Testers shared that defects are frequently found in production! Shouldn’t it be a priority to find defects sooner?
  8. 8. 9/11/2014 8 14 © 2013 IBM Corporation What change do you feel will make the biggest difference in your organization? 15 © 2013 IBM Corporation Agenda  Setting the stage  Challenges testers face  Help is available  Putting it in context
  9. 9. 9/11/2014 9 16 © 2013 IBM Corporation Testers want to spend more time on… 1 2 3 4 5 6 7 8 We asked 250 Testers where they would like to spend less time and where they felt their time would be better spent? 17 © 2013 IBM Corporation 1 2 3 4 5 6 7 8 Survey Says! I want to Spend MORE time
  10. 10. 9/11/2014 10 18 © 2013 IBM Corporation Things to remember when automating tests… Not everything should be automated! Not everything can be automated! Test automation is not a silver bullet! 19 © 2013 IBM Corporation Are you testing what’s under the surface? Build and Deploy Software components built by developers. Software being checked by Testers
  11. 11. 9/11/2014 11 20 © 2013 IBM Corporation Test Automation Are we automating the right tests? Are we automating the tests rights? Traditional Test Automation * * http://www.agilecoachjournal.com/index.php/2014-01-28/testing-2/the-agile-testing-pyramid/ User Interface tests Service tests Unit tests 21 © 2013 IBM Corporation Prioritizing your test automation effort Focus on… Manual / Exploratory Testing pushing tests lower Unit Tests Tests written before code (TDD); instant feedback to the developer on code being written; defects isolated to specific line(s) of code; run as part of build automation process Integration Tests Tests message exchange, behavior, and performance between integrated and dependent components/systems; defects isolated to specific application component, continuously executed as part of deployment automation; lower effort to maintain as application changes User Interface tests Checks functionality via the user interface without understanding the application architecture (black box); most fragile of automated test types; lacks ability to isolate defect; most time consuming to create and maintain; dependent on user interface being available and stable
  12. 12. 9/11/2014 12 22 © 2013 IBM Corporation Unattended test execution Teams need to automate tests at every layer 23 © 2013 IBM Corporation  Separate what the test accomplishes from how it is executed –Keywords abstract the details of how to execute a logical unit of function –Business scenarios compose keywords into a flow  Better leverage the skills in your organization –Subject matter experts can focus on the business scenarios independent of implementation –Test automation engineers can focus on automating  Reduce test (scenario and automation) maintenance –Keyword behavior isn’t reproduced in multiple tests; update fewer places when application changes –Reduce test cases because overlap becomes apparent when abstraction is raised to keyword level Durable, maintainable, resilient to change Testers want maintainable test automation
  13. 13. 9/11/2014 13 24 © 2013 IBM Corporation  A business process may be available through multiple channels – The goal is the same across channels – The flow is similar across channels – The UI gestures differ depending on channel  A business process may span multiple channels – Different steps in the flow occur on different channels – Consider applications taking advantage of mobile capabilities such as location and near field communication  Multichannel testing is about effectively creating, managing, and executing tests of these kinds of processes Validating business flows across multiple channels Effectively creating, managing and executing tests Keywords make supporting channels easy! 25 © 2013 IBM Corporation Testers want to spend LESS time on… 1 2 3 4 5 6 7 We asked 250 Testers where they would like to spend less time and where they felt their time would be better spent?
  14. 14. 9/11/2014 14 26 © 2013 IBM Corporation 1 2 3 4 5 6 7 8 Survey Says! I want to Spend LESS time 27 © 2013 IBM Corporation Standing up Test Environments is a key inhibitor Applications are getting more complex • Increasing expense of standing up complex environments • Delays caused by lack of system availability • Third-party services can be especially costly • Testers spend more time setting up tests than testing 30-50% The average amount of time testing teams spend on setting up test environments, instead of testing $5-30 million The typical investment to build a single test lab for a Fortune 500 company. Most have dozens …
  15. 15. 9/11/2014 15 28 © 2013 IBM Corporation How do you mitigate delays? Beware… these approaches are risky and potentially damaging to the business. 29 © 2013 IBM Corporation Solving the Test Environment dilemma Making production-like test environments a reality Today’s test environments must be:  Realistic and complete (production-like)  Inexpensive to stand up  Available on demand Agile test environments require four enabling capabilities  Test Data Management  Service Virtualization  Deployment Automation  Orchestration
  16. 16. 9/11/2014 16 30 © 2013 IBM Corporation Time Spent on Test Data Management (more than one response allowed) 64% creating test data manually! 34% cloning production and manually cleansing! 31 © 2013 IBM Corporation Refresh test data speeding testing and application delivery Reduce Risk Mask sensitive information for compliance and protection Reduce Cost Automate creation of realistic “right sized” test data to reduce the size of test environments • Understand what test data is needed for test cases • Create “right-sized” test data by subsetting • Ensure masked data is contextually appropriate to the data it replaced so as not to impede testing • Easily refresh & maintain test data by developers and testers • Automate test result comparisons to identify hidden errors • Support custom & packaged ERP applications in heterogeneous environments Improve application quality and delivery Agile testing needs continuous access to test data Speed Delivery
  17. 17. 9/11/2014 17 32 © 2013 IBM Corporation Dealing With Unavailable Software & Services (more than one response allowed) Survey says… “Top 2 tactics”: • Manual creation and maintenance of mocks • Expensive; lack of scalability; takes time away from testing • Defer testing until dependent software is available • Adds risk; decreases test coverage; potential project slip 33 © 2013 IBM Corporation What is Service Virtualization? Agile teams need to make the unavailable available for testing  Virtual components simulate the behavior of a service or application during testing – System dependencies are a key challenge in setting up test environments – Unavailable/inaccessible services slow down the testing process – Costly 3rd party access fees can significantly impact testing availability  Virtual components run on commodity hardware, private cloud, public cloud  Each developer and tester can easily have their own test environment  Developers and testers can continue to use current testing procedures and tools  Streamline creation of production-like environments and negative testing Heterogeneous Environments Public Cloud Private Cloud Data Warehouse Mainframe Enterprise Service Bus Directory Identity File systems Collaboration App Under TestRouting Service Third-party Services Portals Content Providers EJB Shared Services Archives Business Partners Messaging Services Databases Mainframe applications App Under Test Third-party Services Packaged apps, messaging services, etc. Virtual Components
  18. 18. 9/11/2014 18 34 © 2013 IBM Corporation Deployment Automation Agile teams need to accelerate application delivery across the lifecycle  Automated, consistent deployments and rollbacks of applications  Orchestration of changes across servers, tiers and components  Configuration and security differences across environments  Clear visibility: what is deployed where and who changed what  Integrated with middleware, provisioning and service virtualization Key Features: • Multi-Tier Application Models • Easy Process Designer • Track what is where: Inventory • Scalable Distributed Automation • Quality Gates and Approvals • Integrations Replace Custom Scripting Deployment Automation 35 © 2013 IBM Corporation Agenda  Setting the stage  Challenges testers face  Help is available  Putting it in context
  19. 19. 9/11/2014 19 36 © 2013 IBM Corporation Dev-Test-Ops (aka. DevOps) Enterprise capability for continuous software delivery that enables organizations to seize market opportunities and reduce time-to-customer feedback Accelerate software delivery – for faster time to value Balance speed, cost, quality and risk – for increased capacity to innovate Reduce time to customer feedback – for improved customer experience Continuous Customer Feedback & Optimization Collaborative Development Continuous Release and Deployment Continuous Monitoring Continuous Business Planning Continuous Testing Operate Develop/ Test Deploy Steer DevOps Continuous Feedback 37 © 2013 IBM Corporation All the world’s a “test” stage and all Testers merely players; They have their exits and their entrances; The Continuous Delivery Pipeline Development Testing Staging Production Business Owners Customers From ideation to implementation… • everyone contributes to quality! • everyone contributes to a positive end user experience! • everyone contributes to the success of the business!
  20. 20. 9/11/2014 20 38 © 2013 IBM Corporation Continuous Testing: • helps extend agile and lean practices across the delivery lifecycle • removes traditional testing bottlenecks such as unavailable test environments to increase efficiency • enables teams to test earlier and with greater coverage at lower cost • provides project teams with continuous feedback on software quality to reduce business risk A comprehensive Continuous Testing solution covers quality management, functional UI and integration test automation, performance testing, and service virtualization. Organizations need Continuous Testing Accelerating the delivery of higher quality software Resolve today’s testing challenges with confidence 39 © 2013 IBM Corporation Orchestration and automation for the deployment of applications, middleware configuration, and database changes into development, test and production environments. Databases Internal Messages Third-party Services virtual components Simultaneously test across multiple test stages Dev QA Service Virtualization Combining continuous testing withContinuous Testing enables Continuous Delivery Repeatable, & traceable test and deployment automation Test Automation Test Environments Dynamic Infrastructure  Virtualize unavailable yet dependent software & systems  Continuously test in production-like env. Quality Management  Manage quality across the lifecycle Deployment Automation  Deploy private, secure right sized test data Test Data Management  Deploy just built applications Continuous Delivery Orchestrate the delivery pipeline integrated with
  21. 21. 9/11/2014 21 40 © 2013 IBM Corporation Top 6 take aways 1 2 3 4 5 Not every test can be automated. Not every test should be automated. At least 1/3 of the delivery lifecycle is testing. Testers do matter! Test what matters earlier. Test it continuously! Deploy what you have. Virtualize the rest! 6 Everyone contributes to quality! You can’t test quality in. Continuous Testing enables Continuous Delivery 41 © 2013 IBM Corporation The Download your copy today! ibm.biz/TestingSurveyResults How do you stack up to what the others are saying?
  22. 22. 9/11/2014 22 42 ibm.biz/continuoustesting 43

×