Psqt east risk testing


Published on

Risk driven testing tutorial, Summer 2003, PSQT East

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Psqt east risk testing

  1. 1. Risk Driven Software Testing All The Software Processes Prioritize Document & Analyze Scope Tests Define Strategies Gather DesignRequirements Develop Tests Schedule Manage Resources Test and Report Allocate Resources Fix Implement Risk Driven Software Testing 1 PSQT East Tutorial v.1.0 ©TQ2003
  2. 2. Getting StartedIntroductionsLogisticsSetting up the Parking LotObjectives for the Course Risk Driven Software Testing 2 PSQT East Tutorial v.1.0 ©TQ2003
  3. 3. Materials in this TutorialReview materials in student notebook • job aids at front of notebook • agendas • slides • exercise materials • referencesEstablish parking lot on a flip chart page for topics tohandle later Risk Driven Software Testing 3 PSQT East Tutorial v.1.0 ©TQ2003
  4. 4. Logistics Starting and ending times Breaks Exercise Teams TimekeeperPagers and Location of support facilitiesPhones -- Off Number for messages: Emergency procedures: Attendance at all sessions is expected Risk Driven Software Testing 4 PSQT East Tutorial v.1.0 ©TQ2003
  5. 5. Exercise: Identifying Risky Projects Using the exercise booklet, and working in teams of three, complete the exercise named “Identifying Risky Projects” Expected time: 20 minutes Risk Driven Software Testing 5 PSQT East Tutorial v.1.0 ©TQ2003
  6. 6. Debriefing: Identifying Risky Projects What steps happened in your mind for you to identify risk? What types of projects you find risky? How does this affect the way you test? Risk Driven Software Testing 6 PSQT East Tutorial v.1.0 ©TQ2003
  7. 7. Course ObjectivesIdentify testing project risks and refine the plan to includemitigation and contingency activities • State testing goals and objectives • Identify risks with regard to product and project characteristics • State testing activities with acceptance criteria • Select a testing lifecycle to match the products risks and the projects schedule Risk Driven Software Testing 7 PSQT East Tutorial v.1.0 ©TQ2003
  8. 8. Common Testing ProblemsProblems we are used to see happen: • we ship before we have finished testing • our customers find all our defects • we don’t have the testers when we need them • testing is ad-hoc and results are irreproducible • we don’t test for the critical success factors • we don’t test versus requirements from customer • there are no acceptance criteria for any phase and deciding when the product is ready becomes a point of contention • ...Good risk management processes will help! Risk Driven Software Testing 8 PSQT East Tutorial v.1.0 ©TQ2003
  9. 9. Risk Management Prevents Problems Problems that risk management can prevent • We anticipated lateness but accepted sending out an unfinished product • We have too many defects to fix too late in the cycle • We have too many tests to run in a short period of time • We have tested for the simple defects and the customer gets to test for the “killer” bugs • … We know something can happen but we hope it does not happen. Risk Driven Software Testing 9 PSQT East Tutorial v.1.0 ©TQ2003
  10. 10. Start by Reviewing Project ParametersProject vision • Are objectives clear and understood? • Are objectives of customer and provider known?Project scope, milestones and deliverables • Are features agreed to? • Are commitments agreed to and documented?Roles and responsibilities • Are roles and responsibilities clear to all involved? • Are all roles staffed? • Is sponsorship and accountability clear?Use knowledge of specific project roles to identify risks Risk Driven Software Testing 10 PSQT East Tutorial v.1.0 ©TQ2003
  11. 11. Consider Lessons LearnedFor projects like this in the past • What risks were identified and handled well? • Are there lessons from those to be applied in this project? • What risks were identified, but became problems later? • What should be done differently in this project?Are there other experiences of the organization thatindicate lessons to be applied here?Is there information from the industry as a whole thatcan be applied?Risk factor tables include some such lessons Risk Driven Software Testing 11 PSQT East Tutorial v.1.0 ©TQ2003
  12. 12. Sources and Consequences of Risk Categories of Sources Project Consequences Mission and goals Cost overruns Decision drivers testing concerns Schedule slips Organization management Inadequate functionality Customer / end user Canceled projects Budget / cost Sudden personnel changes Schedule Customer dissatisfaction Project characteristics Loss of company image Development process Demoralized staff Development environment Poor product performance Personnel and relationships Legal proceedings Operational environment New technology Risk Driven Software Testing 12 PSQT East Tutorial v.1.0 ©TQ2003
  13. 13. Critical Success FactorsA product’s critical success factors (CSFs) are relatedto: • business needs for its development • coverage of all its intended user constituencies • fitness for use in the intended context • lack of dissatisfiers • competitive advantage – price? – delighters? Risk Driven Software Testing 13 PSQT East Tutorial v.1.0 ©TQ2003
  14. 14. CSF: Business Needs (Why?)Typical business reasons for new systems or changes: • cost reduction – reducing time in the field reduces costs • increased efficiency of work or resource – a better interface makes one person capable of doing the job of two • developing new markets – using a “smart card” allows very small business to get into the credit card market • improved resource management – electronic data management (EDM) systems allow insurance claims to be processed in hundreds of different locations, depending on work load What Is The Customer Going To Get Out Of The Use Of The Product? Risk Driven Software Testing 14 PSQT East Tutorial v.1.0 ©TQ2003
  15. 15. CSF: User Constituencies (Who?)Answer the questions: • given the business needs, what goals will the product help achieve? • who will have to use the product (different user constituencies) to achieve those goals?At least three levels of need: • need to increase the bottom line – typical user constituency: upper management • need to gather supervisory data – typical user constituency: middle management • need for an efficient interface – typical user constituency: end user • other? Risk Driven Software Testing 15 PSQT East Tutorial v.1.0 ©TQ2003
  16. 16. CSF: Fitness for Use (How?)“Fitness for use” • the product may meet all stated requirements but not support solving a real need or problem for a given user constituency • testing features does not cover the workflowsTest in the context of the jobTest the product as if you would have to perform the jobyourselfUse your expertise in testing to extend the possibilitiesof use cases Risk Driven Software Testing 16 PSQT East Tutorial v.1.0 ©TQ2003
  17. 17. Exercise: Critical Success FactorsUsing the exercise booklet, and working in teams ofthree, do the exercise named Product Critical SuccessFactors in the exercise booklet.Expected time: 15 minutes Risk Driven Software Testing 17 PSQT East Tutorial v.1.0 ©TQ2003
  18. 18. Debriefing: Critical Success Factors Can testing activities or techniques bring quality to the product? Has focusing on the Critical Success Factors helped you identify potential problems? Risk Driven Software Testing 18 PSQT East Tutorial v.1.0 ©TQ2003
  19. 19. Characteristics of Good Risk StatementsState specific conditions, easy to detectState specific consequences, which can be measuredAre good enough to proceed to planning • good risk statements nearly write the action planAddress risks, not project tasks in plan that are not yetdoneAddress potential problems, not current problems • Risks are possible future events • Handle current problems another way Write several risk statements as a class Risk Driven Software Testing 19 PSQT East Tutorial v.1.0 ©TQ2003
  20. 20. Exercise: Product RiskUsing the exercise booklet, and working in teams ofthree, do the exercise named Product Risk in the exercisebooklet.Expected time: 20 minutes Risk Driven Software Testing 20 PSQT East Tutorial v.1.0 ©TQ2003
  21. 21. Debriefing: Product RisksAre different testing activities or techniques applicableto different risks?How does testing help reduce project risks? Risk Driven Software Testing 21 PSQT East Tutorial v.1.0 ©TQ2003
  22. 22. High-Level Testing GoalsEffectiveness: • Defect detection – percentage of total found in some time frame – severity found after testing • Other?Efficiency: • Resource usage – time – people – machines • Reporting – time spent reproducing the defect by the developers • Other? Risk Driven Software Testing 22 PSQT East Tutorial v.1.0 ©TQ2003
  23. 23. Setting GoalsBudgeting Resources: • You have $200 • You want to: – Go to the ball game – Treat your significant other to his/her favorite meal – Buy a new set of tires – Pay off your car insurance – Join a health club – Raise your daughter’s allowance – And so many more things!How do you prioritize these? Risk Driven Software Testing 23 PSQT East Tutorial v.1.0 ©TQ2003
  24. 24. Exercise: High LevelTesting GoalsUsing the exercise booklet, and working in teams ofthree, do Activity 3 of the exercise named High LevelTesting Goals in the exercise booklet.Expected time: 20 minutes Risk Driven Software Testing 24 PSQT East Tutorial v.1.0 ©TQ2003
  25. 25. Debriefing: High Level Testing Goals What is the benefit of stating testing goals? What is the impact of testing goals on the project’s plan? What is the impact of the project’s plan on the selection of testing goals? Risk Driven Software Testing 25 PSQT East Tutorial v.1.0 ©TQ2003
  26. 26. Acceptance CriteriaFor each test selected, define: • Environment tests would have had to run on • Regression suites covered by the tests • Functionality covered by the tests • Performance testing goals reached • Volume Testing goals achieved • Reliability Testing (when applicable) • Usability Testing (when applicable)How can we tell that the work product is safe forreleasing it to the user? Risk Driven Software Testing 26 PSQT East Tutorial v.1.0 ©TQ2003
  27. 27. System Acceptance TestSystem Testing Phase Report • Reports on – tests performed • performance • volume • stress • regression • functional • others? ... – tests results • remaining deficiencies – test coverage • percentage of a given unit Risk Driven Software Testing 27 PSQT East Tutorial v.1.0 ©TQ2003
  28. 28. System Acceptance CriteriaProbably only main deployment environment • but NOT the development environment onlyUsually all regression suites for the entire tests • usually automatedMost, if not all, functionality • focus on functionality not tested or changed after unit codePerformance Testing, Volume Testing • goals MUST be met, no excuses • deployment environment used in testsReliability Testing • if applicable, statistically controlledUsability Testing • if critical, or if it leaves development organization Risk Driven Software Testing 28 PSQT East Tutorial v.1.0 ©TQ2003
  29. 29. User Acceptance TestPurpose is to detect remaining defects • focus might changeDifferent things to different people • another level of system acceptance • emphatically focused on usability • performed by users exclusively • performed by user proxies, exclusively • performed by the IV&V of the organization • other? ...Which is yours? Risk Driven Software Testing 29 PSQT East Tutorial v.1.0 ©TQ2003
  30. 30. User Acceptance CriteriaSome general rules • Cover all deployment environments • Only do regression if regression suites are different – or significant fixes and / or changes have happened after system acceptance • Functionality focus should be on usual rather than exceptional • Performance Testing should focus on throughput • Volume Testing might be skipped – unless significant fixes and / or changes have happened after system acceptance • Reliability Testing – only when thoroughly planned from day one • Usability Testing – probably the most emphatic effort goes here Risk Driven Software Testing 30 PSQT East Tutorial v.1.0 ©TQ2003
  31. 31. Exercise: User Acceptance CriteriaUsing the exercise booklet, and working in teams ofthree, do the exercise named User Acceptance Criteria inthe exercise booklet.Expected time: 30 minutes Risk Driven Software Testing 31 PSQT East Tutorial v.1.0 ©TQ2003
  32. 32. Debriefing: User Acceptance Criteria How do the business goals influence the testing acceptance criteria? Which are some of the unavoidable testing acceptance criteria? Are there any good reasons to change some testing acceptance criteria after the test has been executed? Risk Driven Software Testing 32 PSQT East Tutorial v.1.0 ©TQ2003
  33. 33. Risks from the ProjectRisks from the project come from:Project Plan Schedule Constraints • testing might be cut short if development is late • testing might have all its tasks in the critical pathModel being followed by the project • Simple Waterfall, • Parallel Waterfall, • Evolutionary, • Prototyping, • SpiralDesign ArchitectureResources • missing critical skills • budgetary shortcomings Risk Driven Software Testing 33 PSQT East Tutorial v.1.0 ©TQ2003
  34. 34. Design ArchitectureFigure out what the Architecture is going to be • Ask the designer • Request information on the design elements earlyIf not traditional, plan accordingly • Use scenarios profusely • Try having testers join teams early • Use testers insight of design to develop test cases • Push for updated documentation • Push for consistency reviews across documents – Requirements traceability – Formal reviews Risk Driven Software Testing 34 PSQT East Tutorial v.1.0 ©TQ2003
  35. 35. Testing DeliverablesPlanning Assets • Test PlanTesting Assets • Testing procedures • Testing suites • Testing templatesReporting Assets • Individual tests results • Test statistics • Acceptance report Risk Driven Software Testing 35 PSQT East Tutorial v.1.0 ©TQ2003
  36. 36. Exercise: Project’s RisksUsing the exercise booklet, and working in teams ofthree, do the exercise named Project’s Risks in theexercise booklet.Expected time: 30 minutes Risk Driven Software Testing 36 PSQT East Tutorial v.1.0 ©TQ2003
  37. 37. Debriefing: Project RiskHow does the project schedule influence the testingproject’s schedule?What other areas of a project should be explored forrisk? Risk Driven Software Testing 37 PSQT East Tutorial v.1.0 ©TQ2003
  38. 38. Strategy ProblemsProblems that faulty test strategies can cause • we spend too much time on just one phase • we place all our effort at the end of the project • we ignore regressions • we test in the wrong configuration • we receive work products that are unfit to test • we get into endless arguments about product fitness to ship • ...Good testing strategies help! Risk Driven Software Testing 38 PSQT East Tutorial v.1.0 ©TQ2003
  39. 39. Selecting a StrategyDecide on • how much testing is required • of what type • when will it happen • who will do it • how will it happen, e.g.: – Will integration be top-down or bottom-up? – Will clear box be manual or automatic? Risk Driven Software Testing 39 PSQT East Tutorial v.1.0 ©TQ2003
  40. 40. Defining a StrategyReview and rework the business goalsReview project variables and rework the Testing Phases • Consider cost, time to market, personnel, product life expectancy, users, constraintsEmphasize the tasks that tie with the goals • try to probe weak areas of the projectPoints to ponder: • regression (when, how much, which) • coverage (how much, what type, when) • deadlines (slipping deadlines, schedule compression) • integration (how, when, by whom) • assignment of responsibility (developers, testers, users) Risk Driven Software Testing 40 PSQT East Tutorial v.1.0 ©TQ2003
  41. 41. Test Strategies (1)Analyze business case • where is the payoff? – think in terms of customer satisfaction – identify particular functionality or killer faultsProbe the quality of the product with regard to it Risk Driven Software Testing 41 PSQT East Tutorial v.1.0 ©TQ2003
  42. 42. Test Strategies (2)Analyze users • by frequency of use – sporadic, heavy, etc. • by organizational level – general managers, middle managers, end users, etc. • by their knowledge of the software – experts, newcomers, etc.Build a profile of system usage • sketch scenarios • assign probabilities for each scenario – e.g.., one out of eight times the plan will be run unchangedUse this data to design the test cases to optimize testingcoverage of most frequent paths Risk Driven Software Testing 42 PSQT East Tutorial v.1.0 ©TQ2003
  43. 43. Test Strategies (3)Analyze time to market • are you going to be pressed for time? – => focus on existing functionality – => test system rather than component – => test typical rather than exceptionalDecide which tests can be run within the timeconstraints • If some of the fundamental tests cannot be run, move tests forward Risk Driven Software Testing 43 PSQT East Tutorial v.1.0 ©TQ2003
  44. 44. Test Strategies (4)Analyze cost • how many staff/hours can you pay?Figure out if the tests you have so far selected fit withinthe budgetIf so, if you have room for more…Analyze constraints – performance – precision – volume • find critical quantities • analyze or set stress testing…then include tests for them Risk Driven Software Testing 44 PSQT East Tutorial v.1.0 ©TQ2003
  45. 45. Test Strategies (5)Analyze personnel • who can you count on – reinforce testing of the products of the project’s weaker links • which of your documents are going to be weak for lack of experts – requirements or specs <=> functional tests – high-level design <=> integration – modules <=> module testingAnalyze product life expectancy • find the payoff of documented procedures, test case suites, results, etc.. – don’t overspend in a product that has a short life expectancy – don’t under spend in a product that will be around long Risk Driven Software Testing 45 PSQT East Tutorial v.1.0 ©TQ2003
  46. 46. Example Strategy (1)Business goal • Keep and expand customer baseInternal translation to project • Make sure that the user finds the entire current version’s functionality works as usualTesting strategy • Test old before new, in every phase Note: Underlying assumption is that you will not have time to do it all Risk Driven Software Testing 46 PSQT East Tutorial v.1.0 ©TQ2003
  47. 47. Example Strategy (2)Business goal Exceed customer’s expectations of product qualityInternal translation to project • Fewer than 10% of total bugs caught by the userTesting strategy • Move functional test suites to developers before and during unit code and testing Note: Underlying assumption is that you will not have time to do it all Risk Driven Software Testing 47 PSQT East Tutorial v.1.0 ©TQ2003
  48. 48. Limiting the Scope of the Testing EffortSome things cannot be tested • quality • user-friendliness • timeliness •…Some things you might not want to test • regression on a new or relatively small enhancement • performance • stress •…Some things you might not have the ability to test • reliability (e.g. MTTF) • availability (e.g. (MTTF/(MTTF+MTTR))) •… Risk Driven Software Testing 48 PSQT East Tutorial v.1.0 ©TQ2003
  49. 49. Constraints on the LifecycleReview the phases against the Project’s constraints • Can you accommodate the project plan schedule constraints? • Is the model being followed by the project – Simple Waterfall, – Parallel Waterfall, – Evolutionary, – Prototyping, – Spiral allowing for the testing tasks you’ve set? – e.g. might not have integration testing • Can the tests selected be adjusted to the design architecture? – three tiers might bring a whole set of problems • Are the goals compatible with the project’s shortcomings? Risk Driven Software Testing 49 PSQT East Tutorial v.1.0 ©TQ2003
  50. 50. Exercise: Testing LifecycleUsing the exercise booklet, and working in teams ofthree, do Activity 1 of the exercise named TestingLifecycle in the exercise booklet.Expected time: 30 minutes Risk Driven Software Testing 50 PSQT East Tutorial v.1.0 ©TQ2003
  51. 51. Debriefing: Testing LifecycleHow do the business goals influence the selection oftesting lifecycle?Which are the unavoidable testing deliverables?Are there any good reasons to avoid some testingphases? Risk Driven Software Testing 51 PSQT East Tutorial v.1.0 ©TQ2003
  52. 52. Identifying Test TasksReview the V-ModelPick and choose the adequate phases to meet the goalsYou have twenty days to visit all of EuropeYou may never be able to go backHow do you budget your time?Testing is always under-budgeted and over-committed Risk Driven Software Testing 52 PSQT East Tutorial v.1.0 ©TQ2003
  53. 53. Risk Action PlanningDeal with high exposure risks first • Research: Do we know enough about this risk? • Accept: Can we live with it and do nothing about it? • Manage: Can we take action? • Avoid: Should we cancel the project or change the approach?Balance the threat of the risk against the effort to avert it • How great is the threat? • How much does it cost to avert? Risk Driven Software Testing 53 PSQT East Tutorial v.1.0 ©TQ2003
  54. 54. Risk Contingency PlansDevise contingency plans for • High exposure risks, in case the mitigation strategy fails • Any risk for which there is no possible mitigation actionSpecify risk measures and trigger values • Measures of time, resources to handle risk • Measures of risk impact • Trigger values that tell it is time to use contingency approach nowAgree with customer and management at project starthow contingency plans will be funded and handled Risk Driven Software Testing 54 PSQT East Tutorial v.1.0 ©TQ2003
  55. 55. Example Contingency TriggersFor risks leading to schedule slips • Latest date to allow you to use alternative platforms • Latest date to select another vendorFor risks requiring additional effort or time • Latest date to have time to locate the resources • Greatest amount of penalty or fine to incur • Greatest amount of investment available for overrunLimit for extra cost to the customerLimit for learning time Risk Driven Software Testing 55 PSQT East Tutorial v.1.0 ©TQ2003
  56. 56. Example of Action and ContingencyRisk Statement • Since the project team is already behind schedule by two full weeks, we might not have time to cover all the high yield tests before the cutover date and the quality of the product will be seriously imperiled.Risk Action • Provide one of our test engineers as a testing consultant to the development team, so they can test and fix before they send the product to the Testing Team.Contingency Plan • If by the first of April, we do not have an integration test version of the product, drop the web testing suites and focus on regression so our customer can use a significantly improved current version for six months without a Web interface. Risk Driven Software Testing 56 PSQT East Tutorial v.1.0 ©TQ2003
  57. 57. Example Contingency for LatenessPlan your testing activities in passes • First Pass (mandatory): – test all modules/components – use most frequently used scenarios – use few test cases – use selected error cases • Second Pass (supplementary): – test only modules with fixes, do first pass on components – cover most scenarios – use test cases covering all data equivalence classes – include test cases for “bad values” • Third Pass (complementary): – test only modules with fixes from second pass – cover all test suite – go for 100% clear case coverage Risk Driven Software Testing 57 PSQT East Tutorial v.1.0 ©TQ2003
  58. 58. Exercise: Managing RiskUsing the exercise booklet, and working in teams ofthree, do the exercise named Managing Risk in theexercise booklet.Expected time: 30 minutes Risk Driven Software Testing 58 PSQT East Tutorial v.1.0 ©TQ2003
  59. 59. Debriefing: Managing Riskthis will be an exercise that for every risk identified inthe exercises make them think an action plan and/or acontingency plan. Risk Driven Software Testing 59 PSQT East Tutorial v.1.0 ©TQ2003
  60. 60. SummaryKey Activities in Defining the Testing Strategy • identify test tasks and their goals • rank test tasks by their goals • define the limits of the testing effort • detail testing tasks • define testing tasks entry criteria • define testing tasks acceptance criteriaOutputs to the process include • individual test task goals • phase acceptance criteria • detailed testing project tasks • all in the updated test plan Risk Driven Software Testing 60 PSQT East Tutorial v.1.0 ©TQ2003
  61. 61. Next Steps Review Priorities and ExpectationsDevelop Action List Set Completion Dates Review Results Risk Driven Software Testing 61 PSQT East Tutorial v.1.0 ©TQ2003
  62. 62. Course ObjectivesIdentify testing project risks and refine the plan to includemitigation and contingency activities • State testing goals and objectives • Identify risks with regard to product and project characteristics • State testing activities with acceptance criteria • Select a testing lifecycle to match the products risks and the projects schedule Risk Driven Software Testing 62 PSQT East Tutorial v.1.0 ©TQ2003
  63. 63. And now… let’s hear from you!Thank you very much for your participation!Please complete a course evaluation form • very useful in improving the course • most helpful if very honestPlease send us your comments as you work with thismaterial • email your instructor - contact info in student guide • send us comments on what works and what doesn’t so we can improve the course Risk Driven Software Testing 63 PSQT East Tutorial v.1.0 ©TQ2003
  64. 64. Related Courses from TeraQuestProject Launch WorkshopProject Planning and TrackingTesting Management WorkshopQuality Assurance WorkshopConfiguration Management WorkshopPeer Reviews and Inspections Risk Driven Software Testing 64 PSQT East Tutorial v.1.0 ©TQ2003
  65. 65. Information about TeraQuestTeraQuest Metrics, Inc. Process assessmentP.O. Box 200490 Process improvementAustin, Texas 78720-0490 SPI and KPA trainingUSA Risk management Software measurement1-512-219-9152 (phone)1-512-219-0587 (fax)TeraQuest Web site: http://www.teraquest.comEmail questions to: Risk Driven Software Testing 65 PSQT East Tutorial v.1.0 ©TQ2003