Your SlideShare is downloading. ×

Risk and Testing (2003)

315

Published on

Testing Masterclass for Electromind (Steve Allott) Mar 2003, London. Slideset was for discussions, not a linear presentation, then was part-extended (work-in-progress) after session.

Testing Masterclass for Electromind (Steve Allott) Mar 2003, London. Slideset was for discussions, not a linear presentation, then was part-extended (work-in-progress) after session.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
315
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Risk and Testing extended after session Presentation to Testing Master Class 27/28 March 2003 Neil Thompson www.TiSCL.com 23 Oast House Crescent NeilT@TiSCL.comThompson Farnham, Surrey Direct phone 07000 NeilThinformation GU9 0NP (634584)Systems England (UK) Direct fax 07000 NeilTFConsulting Limited Phone & fax 01252 726900 (634583)Some slides included with permission from and Paul Gerrard © Thompson information Systems Consulting Limited
  • 2. Agenda • 1. What testers mean by risk • 2. “Traditional” use of risk in testing • 3. More recent contributions to thinking • 4. Risk-Based Testing: Paul Gerrard (and I) • 5. Next steps in RBT: – end-to-end risk data model; – automation • 6. Refinements and ideas for future © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 1 of 51 information 2003 Systems Consulting Limited
  • 3. 1. What testers mean by risk• Risk that software in live use will fail: – software: could be Commercial Off The Shelf; packages such as ERP; bespoke project; integrated programmes of multiple systems...; industry-wide supply chains; any systems product• Could include risk that later stages (higher levels) of testing will be excessively disrupted by failures• Chain of risks: Error: Fault: Failure: RISK mistake made by human RISK something wrong in a product RISK deviation of product from its (eg spec-writing, (interim eg spec, expected* delivery or service program-coding) final eg executable software) (doesn’t do what it should, or does what it shouldn’t) not all errors * “expected” may be as in spec, not all faults result in faults or spec may be wrong result in failures (verification & validation) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 2 of 51 information 2003 Systems Consulting Limited
  • 4. Extra links (4,5) in the chain of risks? • Steve Allott: – distinguish faults in specifications from faults in the resulting program code Link 2: Link 3: Link 4: RISK Link 1: RISK something RISK something RISK deviation of product from its mistake made by human expected delivery or service wrong in wrong in a spec program (doesn’t do what it should, code or does what it shouldn’t)• Ed Kit (1995 book, Software Testing in the Real World): – “error” reserved for its scientific use (but not always very useful in software testing?)  Error Mistake:  (undefined):  Fault:  Failure:RISK a human action that RISK incorrect results RISK an incorrect step, RISK an incorrect result produces an incorrect process or data in specifications result (eg in spec- writing, program- Could use Defect for this? definition in a computer program  Error: amount by which coding) (ie executable result is incorrect software) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 2.x1 of 51 information 2003 Systems Consulting Limited
  • 5. Chain of risks could be up to 6 links? • Adding in a distinction by John Musa (1998 book, Software Reliability Engineering): – not all deviations are failures (but this is just the “anomaly” concept?) – (so the associated risks are in the testing process rather than development: that an anomaly may not be noticed, or may be misinterpreted) • A possible hybrid of all sources: (false alarm): or Change Request,  Anomaly: or testware mistake an unexpected result during testing RISK OF MIS- INTERPRETING Note: this fits its usage in inspections RISK OF MISSING  Mistake:  Defect:  Fault:  Failure:RISK a human action that RISK incorrect results RISK an incorrect step, RISK an incorrect result produces an incorrect in specifications process or data result (eg in spec- writing, program- definition in a computer program  Error: amount by which coding) RISK (ie executable result is incorrect software) Direct programming mistake © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 2.x2 of 51 information 2003 Systems Consulting Limited
  • 6. Three types of software risk Process Risk Project Risk variances in planning and resource constraints, estimation, shortfalls in external interfaces, staffing, failure to track supplier relationships, progress, lack of quality contract restrictions assurance and configuration management Primarily a management Planning and the development responsibility process are the main issues here. Product Risk lack of requirements stability, complexity, Testers are design quality, coding mainly quality, non-functional concerned with issues, test specifications. Product Risk Requirements risks are the most significant risks reported in risk assessments. © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 3 of 51 information 2003 Systems Consulting Limited
  • 7. Risk management components BAD THINGS WHICH COULD HAPPEN, AND PROBABILITY OF EACH N E W S CONSEQUENCE OF EACH BAD THING WHICH COULD HAPPEN © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 4 of 51 information 2003 Systems Consulting Limited
  • 8. Symmetric view of risk probability & consequence PROBABILITY risk EXPOSURE = probability (likelihood) x of bad thing occurring 3 6 9 consequence 2 4 6 1 2 3 CONSEQUENCE (impact) if bad thing does occur • This is how most people quantify risk • Adding gives same rank as multiplying, but less differentiation © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 5 of 51 information 2003 Systems Consulting Limited
  • 9. Risk management components BAD THINGS WHICH COULD HAPPEN, AND PROBABILITY OF EACHany other N E W Sdimensions? CONSEQUENCE OF EACH BAD THING WHICH COULD HAPPEN © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 4r of 51 information 2003 Systems Consulting Limited
  • 10. Do risks have any other dimensions? • In addition to probability and consequence... • Undetectability: – difficulty of seeing a bad thing if it does happen – eg insidious database corruption • Urgency: – advisability of looking for / preventing some bad things before other bad things – eg lack of requirements stability • Any others? © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 6 of 51 information 2003 Systems Consulting Limited
  • 11. 2. “Traditional” use of risk in testing • Few authors & trainers in software testing now miss an opportunity to link testing to “risk” • In recent years, almost a mandatory mantra • But it isn‟t new (what will be new is translating the mantra into everyday doings!) • Let‟s look at the major “traditional” authors: – Hetzel – Myers – Beizer – others © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 7 of 51 information 2003 Systems Consulting Limited
  • 12. Who wrote this and when? • “Part of the art of testing is to know when to stop testing”: – some recent visionary / pragmatist? – Myers? His eponymous 1979 book was the first on testing? – No, No and No! – Fred Gruenberger in the original testing book, Program Test Methods 1972-3, Ed. William C. Hetzel • Also in this book (which is the proceedings of first-ever testing conference)... © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 8 of 51 information 2003 Systems Consulting Limited
  • 13. Hetzel (Ed.) 1972-3• Little / nothing explicitly on risk, but: – reliability as a factor in quality; inability to cope with complexity of systems – “the probability of being faulty is great”p255 (Jean-Claude Rault, CRL France)... – “how to run the test for a given probability of error... number of random input combinations before... considered „good‟”p258; sampling as a principle of testing• Interestingly: – “sampling as a principle should decrease in importance and be replaced by hierarchical organization & logical reduction”p28 (William C. Hetzel)• Other curiosities: – ?source of Myers‟ triangle exercise p13 (ref. SYSTEM DESIGN Dr. Richard Hamming, “Computers and Society”) COMPON‟T DESIGN MODULE DESIGN – the first “V-model”? p172 Outside-in design, inside-out testing MODULE TEST COMPONENT TEST SYSTEM TEST (Allan L. Scherr, IBM Poughkeepsie NY / his colleagues) CUSTOMER USE © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 9 of 51 information 2003 Systems Consulting Limited
  • 14. Myers 1976: Software Reliability - Principles & Practices• Again, “risk” not explicit, but principles are there: – “reliability must be stated as a function of the severity of errors as well as their frequency”; “software reliability is the probability that the software will execute for a period of time without a failure, weighted by the cost to the user of each failure”; “probability that a user will not enter a particular set of inputs that leads to a failure”p7 – “if there is reason to believe that this set of test cases had a high probability of uncovering all possible errors, then the tests have established some confidence in the program‟s correctness”; “each test case used should provide should provide a maximum yield on our investment... the probability that the test case will expose a previously undetected error”p170, 176 – “if a reasonable estimate of [the number of remaining errors in a program] were available during the testing stages, it would help to determine when to stop testing”p329 – hazard function as a component of reliability modelsp330 © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 10 of 51 information 2003 Systems Consulting Limited
  • 15. Myers 1979: The Art of Software Testing• Risk is still not in the index, but more principles: – “the earlier that errors are found, the lower are the costs of correcting... and the higher is the probability of correcting the errors correctly”p18 – “what subset of all possible test cases has the highest probability of detecting the most errors”p36 – tries to base completion criteria for each phase of testing on an estimate of the number of errors originating in particular design processes, and during what testing phases these errors are likely to be detectedp124 – testing adds value by increasing reliabilityp5 – revisits / updates the reliability models outlined in his 1976 book: • those related to hardware reliability theory (reliability growth, Bayesian, Markov, tailored per program) • error seeding, statistical sampling theory • simple intuitive (parallel independent testers, historic error data) • complexity-based (composite design, code properties) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 11 of 51 information 2003 Systems Consulting Limited
  • 16. Hetzel (1984)-8: The Complete Guide to Software Testing• Risk appears only once in the index, but is prominent: – Testing principle #4p24: Testing Is Risk-Based • amount of testing depends on risk of failure, or of missing a defect; so... • use risk to decide number of cases, amount of emphasis, time & resources• Other principles appear: – testing measures software quality; want maximum confidence per unit cost via maximum probability of finding defectsp255 – objectives of Testing In The Large include:p123 • are major failures unlikely? • what level of quality is good enough? • what amount of implementation risk is acceptable? – System Testing should end when we have enough confidence that Acceptance Testing is ready to startp134 © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 12 of 51 information 2003 Systems Consulting Limited
  • 17. Beizer 1984: Software System Testing & Quality Assurance• Risk appears twice in index, but both insignificant• However, some relevant principles are to be found: – smartness in software production is ability to avoid past, present & future bugsp2 (and bwgs?) – now more than a dozen models/variations in software reliability theory: but all far from reality; and all far from providing simple, pragmatic tools that can be used to measure software developmentp292-293 – six specific criticisms: but if a theory were to overcome these then it would probably be too complicated to be practicalp293-294 – a compromise may be possible in future, but instead for now, suggest go-live when the system is considered to be useful, or at least sufficiently useful to permit the risk of failurep295 – plotting and extrapolation of S-curves to assess when this point attainedp295-304 © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 13 of 51 information 2003 Systems Consulting Limited
  • 18. Beizer (1983)-90: Software Testing Techniques• “Risk” word is indexed as though deliberate: – a couple of occurrences are insignificant, but others: • purpose of testing is not to prove anything but to reduce perceived risk [of software not working] to an acceptable value (penultimate phase of attitude) • testing not an act; is a mental discipline which results in low-risk software without much testing effort (ultimate phase of attitude) p4 • accepting principles of statistical quality control (but perhaps not yet implementing, because is not yet obvious how to, and in the case of small products, is dangerous) p6 • add test cases for transactions with high risksp135 • we risk release when confidence is high enoughp6• Other occurrences of key principles, including: – probability of failure due to hibernating bwgs* low enough to acceptp26 – importance of a bwg* depends on frequency, correction cost, [fix] installation cost & consequencesp27 *bwg: ghost, spectre, bogey, hobgoblin, spirit of the night,any imaginary (?) thing that frightens a person (Welsh) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 14 of 51 information 2003 Systems Consulting Limited
  • 19. Others• The “traditional” period could be said to cover the 1970s and 1980s. A variety of views can be found: – Edward Miller 1978, in Software Testing & Validation Techniques (IEEE Tutorial): • “except under very special situations [...], it is important to recognise that program testing, if performed systematically, can serve to guarantee the absence of bugs”p4 • and/but(?) “a program is well tested when the program tester has an adequately high level of confidence that there are no remaining “errors” that further testing would uncover”p9 (italics by Neil Thompson!) – DeMillo, McCracken, Martin & Passafiume 1987: Software Testing & Evaluation • “a technologically sound approach to testing will incorporate... evaluations of software status into overall assessments of risk associated with the development and eventual fielding of the system”p vii © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 15 of 51 information 2003 Systems Consulting Limited
  • 20. 3. More recent contributions to (risk use) thinking• Traditional basis of testing on risk (although more perceptive than some give credit for) is less than satisfactory because: – it tends to be “lip-service”, with no follow-through / practical application – if there is follow-through, it involves merely using risk analysis as part of the Testing Strategy (then that is shelved, and it‟s “heads down” from then on?)• Contributions more recently from (for example): – Ed Kit (Software Testing in the real world, 1995) – Testing Maturity Model (Illinois Institute of Technology) – Test Process Improvement® (Tim Koomen & Martin Pol) – Testing Organisation MaturityTM questionnaire (Systeme Evolutif) – Hans Schaefer‟s work – Zen and the art of Object-Oriented Risk Management (Neil Thompson) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 16x of 51 information 2003 Systems Consulting Limited
  • 21. Kit 1995: Software Testing in the real world• Error-fault-failure chain extendedp18• Clear statements on risk and risk managementp26: – test the parts of the system whose failures would have most serious consequencesp27 – frequent-use areas increase chances of failure foundp27 – focus on parts most likely to have errors in themp27 – risk is not only basis for test management decisions, is basis for everyday test practitioner decisionsp28• Risk management used in integration testp95 © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 16.x1 of 51 information 2003 Systems Consulting Limited
  • 22. Testing Maturity Model• Five levels of increasing maturity, based loosely on decades of testing evolution (eg in 1950s testing not even distinguished from debugging)• Maturity goals and process areas for the five levels do not include risk explicitly, although emphasis moves from tactical to strategic (eg fault detection to prevention): – in level 1, software released without adequate visibility of quality & risks – in level 3, test strategy is determined using risk management techniques – in level 4, software products are evaluated using quality criteria (relation to risk?) – in level 5, costs & test effectiveness are continually improved (sampling quality)• Strongly recommended are key practices & subpractices: – (not yet available?)• Little explicitly visible on risk; very process-oriented © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 17 of 51 information 2003 Systems Consulting Limited
  • 23. Test Process Improvement®• Only one entry in index: – risks and recommendations, substantiated with metrics (as part of Reporting) • risks indicated with regard to (parts of) the tested object • risks can be (eg) delays, quality shortfalls• But risks incorporated to some extent, eg: – Test Strategy (differentiation in test depth depending on risks) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 18 of 51 information 2003 Systems Consulting Limited
  • 24. Testing Organisation Maturity (TOMTM) questionnaire • Risk assessment is not only at the beginning: – when development slips, a risk assessment is conducted and a decision to squeeze, maintain or extend test time may be made – [for] tests that are descoped, the associated risks are identified & understood © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 19 of 51 information 2003 Systems Consulting Limited
  • 25. Hans Schaefer’s work• Squeeze on testing prioritise based on risk• Consider possibility of stepwise release: : – test most important functions first – look for functions which can be delayed• What is “important” in the potential release (key functions, worst problems?) – visibility (of function / characteristic) – frequency of use – possible cost of failure• Where likely to be most problems? – project history (new technology, methods, tools; numerous people, dispersed) – product measures (areas complex, changed, needing optimising, faulty before) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 20 of 51 information 2003 Systems Consulting Limited
  • 26. Zen and the Art of Object-Oriented Risk Management• Testing continues to lag development; but pessimism / delay was currently unacceptable (deregulation, Euro, Y2k)• Could use OO concepts to help testing: – encapsulate risk information with tests (for effectiveness); and/or – inherit tests to reuse (efficiency)• Basic risk management: – relationship to V-model (outline level) – detail level: test specification © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 21 of 51 information 2003 Systems Consulting Limited
  • 27. Risk management & the V-model Acceptance testing System Risks that system(s) have testing undetected defects in them Integration testing Unit testing © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 22a of 51 information 2003 Systems Consulting Limited
  • 28. Risk management & the V-model Risks that system(s) and business(es) are not right and ready for each other Acceptance testing System Risks that system(s) have testing undetected defects in them Integration testing Unit testing © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 22b of 51 information 2003 Systems Consulting Limited
  • 29. Where and how testing manages risks: first, at outline level Level Risks Service requirements Acceptance Undetected errors damage testing business System specification System Undetected errors waste user time testing & damage confidence in Acceptance testing Interfaces don‟t match Integration Undetected errors too late to fix testing Units don‟t work right Unit Undetected errors won‟t be found testing by later tests © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 23a of 51 information 2003 Systems Consulting Limited
  • 30. Where and how testing manages risks: first, at outline level Level Risks How managed Service requirements Specify user-wanted tests against URS Acceptance Undetected errors damage Script tests around user guide and user testing business & operator training materials System specification Use independent testers, functional & System Undetected errors waste user time technical, to get fresh view testing & damage confidence in Take last opportunity to do automated Acceptance testing stress testing before env‟ts re-used Interfaces don‟t match Use skills of designers before they move Integration Undetected errors too late to fix away testing Take last opportunity to exercise interfaces singly Units don‟t work right Use detailed knowledge of developers Unit Undetected errors won‟t be found before they forget testing by later tests Take last opportunity to exercise every error message © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 23b of 51 information 2003 Systems Consulting Limited
  • 31. Second, at detail level:risk management during test specification Test specification based on total magnitude of risks for all defects imaginable = ( Estimated probability of defect occurring x Estimated severity of defect ) • To help decision-making during the “squeezing of testing”, it would be useful to have recorded explicitly as part of the specification of each test: – the type of risk the set of tests is designed to minimise – any specific risks at which a particular test or tests is aimed • And this was one of the inputs to... © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 24 of 51 information 2003 Systems Consulting Limited
  • 32. 4. Risk-Based (E-Business) Testing• Advert! – Artech House, 2002 – ISBN 1-58053-314-0 – reviews amazon.com & co.uk• companion website www.riskbasedtesting.com – sample chapters – Master Test Planning template – comments from readers (reviews, corrections) With acknowledgements © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 25 of 51 information 2003 to lead author Paul Gerrard Systems Consulting Limited
  • 33. Risk-Based E-Business Testing: main themes• Can define approximately 100 “product” risks threatening a typical e- business system and its implementation and maintenance• Test objectives can be derived almost directly as “inverse” of risks• Usable reliability models are some way off (perhaps even unattainable?) so better for now to work on basis of stakeholders‟ perceptions of risk• Lists & explains techniques appropriate to each risk type• Includes information on commercial and DIY tools• Final chapters are on “making it happen”• Go-live decision-making: when benefits “now” exceed risks “now”• Written for e-business but principles are portable; extended to wider tutorial for EuroSTAR 2002; following slides summarise key points With acknowledgements © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 26 of 51 information 2003 to lead author Paul Gerrard Systems Consulting Limited
  • 34. Risks and test objectives - examples Risk Test Objective The web site fails to function To demonstrate that the application functions correctly on the user‟s client correctly on selected combinations of operating system and browser operating systems and browser version configuration. combinations. Bank statement details To demonstrate that statement details presented in the client presented in the client browser reconcile with browser do not match records back-end legacy systems. in the back-end legacy banking systems. Vulnerabilities that hackers To demonstrate through audit, scanning and could exploit exist in the web ethical hacking that there are no security site networking infrastructure. vulnerabilities in the web site networking infrastructure. © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 27 of 51 information 2003 Systems Consulting Limited
  • 35. Generic test objectives Test Objective Typical Test Stage Demonstrate component meets requirements Component Testing Demonstrate component is ready for reuse in larger Component Testing sub-system Demonstrate integrated components correctly Integration testing assembled/combined and collaborate Demonstrate system meets functional requirements Functional System Testing Demonstrate system meets non-functional requirements Non-Functional System Testing Demonstrate system meets industry regulation System or Acceptance requirements Testing Demonstrate supplier meets contractual obligations (Contract) Acceptance Testing Validate system meets business or user requirements (User) Acceptance Testing Demonstrate system, processes and people meet (User) Acceptance business requirements Testing © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 28 of 51 information 2003 Systems Consulting Limited
  • 36. Master test planning © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 28.x1 of 51 information 2003 Systems Consulting Limited
  • 37. Test Plans for each testing stage (example) eg for System Testing: Test 1 2 3 4 5 6 7 8 9 10 11 ... Risk F1 125 Test objective F1 Risk F2 10 Test objective F2 Risk F3 60 Test objective F3 Risk F4 32 Test objective F4 Risk U1 12 Test objective U1 Risk U2 25 Test objective U2 Risk U3 30 Test objective U3 Risk S1 100 Test objective S1 Risk S2 40 Test objective S2 Requirement F1 Requirement F2 Generic test objective G4 Requirement F3 Requirement N1 Generic test objective G5 Requirement N2 Test importance H L M H H H M M M L L ... © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 29 of 51 information 2003 Systems Consulting Limited
  • 38. Test Design: target execution schedule (example)ENVIRONMENTS TEAMS TESTERS Test execution days 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ... Balance & 1 Retests & regression tests transaction reporting Inter- 5 10Partition for account 3 7 11functional tests transfers 2 Payments 6 9 Direct debits End-to-end customer scenariosPartition for 4 8disruptive non-functional tests Comfortable completion date Earliest completion date © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 30 of 51 information 2003 Systems Consulting Limited
  • 39. Plan to manage risk (and scope) during test specification & execution Quality Quality Scope Cost Risk Qualitybest pair tofine-tune Time Time Cost Scope Scope Cost Time © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 31x of 51 information 2003 Systems Consulting Limited
  • 40. Risk as the “inverse” of quality Quality Scope Time Scope Time Risk © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 31.x1 of 51 information 2003 Systems Consulting Limited
  • 41. Managing risk during test specification initial scope set target go-live dateScope Time by requirements set in advance increasing decreasing risk of faults risk as introduced faults found Risk © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 31.x2 of 51 information 2003 Systems Consulting Limited
  • 42. Verification & validation as risk management methods © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 31.x3 of 51 information 2003 Systems Consulting Limited
  • 43. Risk management during test specification: “micro-risks” Test specification: Estimated Estimated RISKS TO •for all defects probability x consequence = TEST AGAINST imaginable... © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 32ax of 51 information 2003 Systems Consulting Limited
  • 44. Risk management clarifies during test execution Test specification: Estimated Estimated RISKS TO •for all defects probability x consequence = TEST AGAINST imaginable... Test execution: Consequence = •for each defect Probability detected =1 x f (urgency, importance) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 32bx of 51 information 2003 Systems Consulting Limited
  • 45. But clarity during test execution is only close-range: fog ahead! Test specification: Estimated Estimated RISKS TO •for all defects probability x consequence = TEST AGAINST imaginable... Test execution: Consequence = •for each failure Probability detected… =1 x f (urgency, { } importance) = REMAINING •for all Estimated Estimated RISKS anomalies as yet probability x consequence undiscovered... © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 32cx of 51 information 2003 Systems Consulting Limited
  • 46. Managing risk during test execution: when has risk got low enough? initial scope set target go-live dateScope Time by requirements set in advance increasing decreasing risk of faults risk as introduced faults found Risk © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 32.x1 of 51 information 2003 Systems Consulting Limited
  • 47. Pragmatic approximation to risk reduction: progress through “test+fix” # anomalies• Cumulative S-curves are good because they: Cumulative anomalies – show several things at once – facilitate extrapolation of trends Resolved – are based on acknowledged Awaiting theory and empirical data... fix Deferred # Closedtests date Target tests run Fail Actual tests run Pass date © Thompson Neil Thompson: Risk and Testing 27/28 Mar Slide 32.x2 of 51 information 2003 Systems Consulting Limited
  • 48. Cumulative S-curves: theoretical basis (potential) # failures per day Software reliability growth model depending on operational profile Software HardwareAbove curves based on Myers 1976: Software Reliability: Principles & Practices p10 date cumulative # Anomalies found More than 1 Much less than 1 anomaly per test anomaly per test Much more than 1 anomaly per test, but Tests run Late tests slower because go-live dateonly one visible at a time! awaiting difficult and/or lo-priority fixes dateEarly tests blocked byhi-impact anomalies Middle tests fast and productive © Thompson Neil Thompson: Risk and Testing 27/28 Mar Slide 32.x3 of 51 information 2003 Systems Consulting Limited
  • 49. A possible variation on thepotential software reliability growth model# failures Possibility of knock-on errors included in go-live date Software Littlewood & Verrall 1973: per day A Bayesian Reliability Growth Model for Computer Software (in IEEE Symposium) Bad maintenance Good maintenance Good testing & maintenance: Bad testing & maintenance: date convergence on stability divergence into instability failures failures TEST & TEST & FIX HACK RETEST RETEST failures failures CHECK AGAINST HAVE A EXPECTED STROKE THINK AND DIAGNOSE TALK TO FOLKS RESULTS CHIN © Thompson Neil Thompson: Risk and Testing 27/28 Mar Slide 32.x4 of 51 information 2003 Systems Consulting Limited
  • 50. Cumulative S-curves: more theory cumulative #Actually there Anomalies foundare severalreliability growthmodels, but: Tests run go-live date• the Rayleigh model ispart of hardwarereliability methodology dateand has been usedsuccessfully in # per daysoftware reliabilityduring development Pattern of fault discovery Dunn & Ullman 1982 p61and testing Anomalies found• its curve producesthe S-curve when go-live dateaccumulated Tests run date © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 32.x5 of 51 information 2003 Systems Consulting Limited
  • 51. Reliability theory more generally# failuresper “t” Software: exponential decay Myers 1976 Hardware: Poisson distribution Myers 1976Both of our software execution time Σtcurves are members • shapeof the Weibull parameter “m” # failures can take various • NB: models tend todistribution: per “t” values (only 1 & 2 use execution time• has been used for shown here): Kan rather than elapseddecades in hardware: 1995 p180 time (because removesKan 1995 p179 Rayleigh (m=2) test distortion, and uses• two single-parameter operational profile iecases applied to how often used live)software by Wagoner Exponential (m=1)in 1970s: Dunn &Ullman 1982 p318 execution time Σt © Thompson Neil Thompson: Risk and Testing 27/28 Mar Slide 32.x6 of 51 information 2003 Systems Consulting Limited
  • 52. Reliability models taxonomy (Beizer, Dunn & Ullman, + Musa) future addition to slide pack © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 32.x7 of 51 information 2003 Systems Consulting Limited
  • 53. Reliability: problems with theories (Beizer, 1984)• Main problem is getting theories to match reality• Several acknowledged shortcomings of many theories, eg: – don‟t evaluate consequence (severity) of anomalies – assume testing is like live (eg relatively few special cases) – don‟t correct properly for stress-test effects, or code enhancements – don‟t consider interactions between faults – don‟t allow for debugging getting harder over time• The science is moving on eg Wiley, Journal of Software Testing, Verification & Reliability but: – a reliability theory that satisfied all the above would be complex – would project managers use it, or would they go live anyway?• So until these are resolved, let‟s turn to empirical data... © Thompson Neil Thompson: Risk and Testing 27/28 Mar Slide 32.x8 of 51 information 2003 Systems Consulting Limited
  • 54. Cumulative S-curves: empirical dataObserved empirically that faults plot takes characteristic shape Hetzel 1988 p210 # Anomalies found Possible to use to roughly gauge test time or faults remaining Hetzel 1988 p210 Tests run go-live date date S-curve also visible in Kit 1995: Software Testing in the Real World p135 The Japanese “Project Bankruptcy” study: Abe, Sakamura & Aiso 1979, in Beizer 1984 • analysed 23 projects, including application software & system software developments • included new code, modifications to existing code, and combinations • remarkable similarity across all projects for shape of test completion curve • anomaly detection rates not significant (eg low could mean good software or bad testing) • significant were (a) length of initial slow progress, and (b) shape of anomaly detection curve... © Thompson Neil Thompson: Risk and Testing 27/28 Mar Slide 32.x9 of 51 information 2003 Systems Consulting Limited
  • 55. Project bankruptcy study,cumulative # summarised by Beizer Anomalies found (b) A secondary indicator was start of integration stage anomaly detection rate planned go-live date Derivative (looks like rate Tests run Rayleigh “bankrupt” curve) failure date success inevitable Phase I............II...............III 100%All the projects had 3 phases: Beizer 1984: Software System Testing & Quality Assurance, p297-300(a) Duration of phases was primary indicator of project success or “bankruptcy”: Ph I+II...............................72%.......................................126% success “bankrupt” failure inevitable Ph I..15%................55%................97% © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 32.x10 of 51 information 2003 Systems Consulting Limited
  • 56. S-curves & reliability: summary of references • Many of the original references are very old: – illustrates the points that (a) the earliest testers seemed to be “advanced” even at that outset, and (b) software reliability seems not to have penetrated mainstream testing 30 years on! – but: means these books & papers hard to obtain, so... • Recommended “recent” references: – Boris Beizer 1984 book: Software System Testing & Quality Assurance (ISBN 0-442-21306-9) – Stephen H. Kan 1995 book: Metrics & Models in Software Quality Engineering (ISBN 0-201-63339-6) – John Musa 1999 book: Software Reliability Engineering (ISBN 0-07-913271-5) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 32.x11 of 51 information 2003 Systems Consulting Limited
  • 57. Progress through tests • We are interested in two main aspects: – can we manage the test execution to get complete before target date? – if not, can we do it for those tests of high (and medium?) importance?# tests # tests Target tests passed Target tests run Actual tests passed High Fail Actual tests run Pass Medium date date Low © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 33 of 51 information 2003 Systems Consulting Limited
  • 58. Progress through anomaly fixing and retesting • Similarly, two main aspects: – can we manage the workflow to get anomalies fixed and retested before target date? – if not, can we do it for those of material impact?# anomalies # anomalies Cumulative anomalies Outstanding material impact Resolved Awaiting fix Deferred Closed date date © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 34x of 51 information 2003 Systems Consulting Limited
  • 59. Quantitative and qualitative risk reduction from tests and retests PROGRESS & RESIDUAL RISK UP RIGHT SIDE OF W-MODEL System Large-Scale Acceptance Testing Integration Testing Testing PROGRESS THROUGH H M Material TESTS Fail impact L Pass PROGRESS THROUGH Awaiting fix Awaiting fix Resolved Material INCIDENT Deferred impact FIXING Closed & RETESTING © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 35 of 51 information 2003 Systems Consulting Limited
  • 60. …and also into regression testing Retesting & Specification Execution regression of tests of tests testing Difficulties, What’s left for: Desired Time & Physical coverage resource constraints squeeze on “complete” second allowance for con- (environ- scripting & first run further straints ments execution run runs etc.) time ... Testing Testing Test Test Execution: Execution: Regression Strategy Plan Design Scripts First run Retest & 2nd run Testing © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 36 of 51 information 2003 Systems Consulting Limited
  • 61. Managing risk during test execution, against “fixed” scope & time initial scope set target go-live dateScope Time by requirements set in advance increasing decreasing risk of faults risk as introduced faults found Risk © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 37x of 51 information 2003 Systems Consulting Limited
  • 62. From what we can report, to what we would like to report © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 37.1x of 51 information 2003 Systems Consulting Limited
  • 63. Risk-based reporting Planned start end all risks „open‟ at the start Progress through the test plan © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 38a of 51 information 2003 Systems Consulting Limited
  • 64. Risk-based reporting Planned start today end residual risks of releasing TODAY Progress through the test plan © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 38b of 51 information 2003 Systems Consulting Limited
  • 65. Report not only risk, but also scope, over time initial scope set target go-live dateScope Time by requirements set in advance how much scope safely delivered so far? increasing decreasing risk of faults risk as introduced faults found Risk © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 38.1x of 51 information 2003 Systems Consulting Limited
  • 66. Benefit & objectives based test reporting Blocked NFR 1 Blocked func 1 Blocked func 2 Objective Objective Objective Objective Benefit Benefit Benefit Benefit Open Closed Risks Open Closed Closed Open Open Closed Project objectives, hence benefits, available for release © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 39ax of 51 information 2003 Systems Consulting Limited
  • 67. Benefit & objectives based test reporting Objective Objective Objective Objective Objective Benefit Benefit Benefit Benefit Benefit Benefit Open Closed Risks Open Closed Closed Open Open Closed Project objectives, hence benefits, available for release © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 39bx of 51 information 2003 Systems Consulting Limited
  • 68. Slippages and trade-offs: an example • If Test Review Boards recommend delay, management may demand a trade-off, “slip in a little of that descoped functionality” original first target slipscope date © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 40a of 51 information 2003 Systems Consulting Limited
  • 69. Slippages and trade-offs: an example • If Test Review Boards recommend delay, management may demand a trade-off, “slip in a little of that descoped functionality” • This adds benefits but also new risks: more delay? original first actual target slip go-livescope date © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 40b of 51 information 2003 Systems Consulting Limited
  • 70. Tolerable risk-benefit balance: another example • Even if we resist temptation to trade off slippage against scope, may still need to renegotiate the tolerable level of risk balanced against benefits original target (risk - datebenefits) original target net risk date © Thompson Neil Thompson: Risk and Testing 27/28 Mar Slide 41a of 51 information 2003 Systems Consulting Limited
  • 71. Tolerable risk-benefit balance: another example • Even if we resist temptation to trade off slippage against scope, may still need to renegotiate the tolerable level of risk balanced against benefits original actual target (risk - go-live datebenefits) “go for it” margin original target net risk date © Thompson Neil Thompson: Risk and Testing 27/28 Mar Slide 41b of 51 information 2003 Systems Consulting Limited
  • 72. 5. Next steps in Risk-Based Testing • End-to-end risk data model: – is wanted to keep risk information linked to tests – is under development by Paul Gerrard – main reason is of course... • Automation: – clerical and spreadsheets really not enough – want to keep risk to test link up-to-date through descopes, reprioritisations etc – Paul is working on an Access database for now – any vendors with risk in their data models? With acknowledgements © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 42 of 51 information 2003 to lead author Paul Gerrard Systems Consulting Limited
  • 73. 6. Refinements and ideas for future• Although almost universal, the simple multiplication of probability x consequence can be troublingly over-simple: it might descope testing for huge-impact risks which are very unlikely (avionics errors?!). So use an asymmetric view?• Some risks are from technology, and other risks are business risks, to use of system. So distinguish “cause” risks from “effect” risks?• Assessing perception of risks is a start, but can metrics give better quantification? Metrics & Fault source analysis• Reliability models a key part of testing theory in the 1970s, but still not credibly usable? Reliability engineering• Wider theoretical basis distinguishing risk from uncertainty: Decision theory © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 43 of 51 information 2003 Systems Consulting Limited
  • 74. Asymmetric view of risk probability & consequence risk EXPOSURE = • very high (red) • high (orange) PROBABILITY • high-ish (yellow) (likelihood) • moderate (pale yellow) • low (pale green) • very low (green) 3 6 9 2 5 8 1 4 7 CONSEQUENCE (impact)• This is an arbitrary scheme; others are possible, of course © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 44 of 51 information 2003 Systems Consulting Limited
  • 75. Distinguishing “cause” risk from technology & “effect” risk to business) probability • Each could be exposure symmetric or RISKS TO asymmetric BUSINESS • Could weight: (from faulty documents • business risks or software) consequence higher for Acceptance Testing • technology risks RISKS FROM higher for Unit & TECHNOLOGY (of faults being made Integration Testing in documents or software) • equal for System Testing © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 45 of 51 information 2003 Systems Consulting Limited
  • 76. But what about next time: metrics? Test specification: Estimated Estimated RISKS TO •for all defects probability x consequence = TEST AGAINST imaginable... Test execution: Consequence = •for each failure Probability detected… =1 x f (urgency, } importance) •for all { anomalies as yet undiscovered... Estimated probability x Estimated consequence = REMAINING RISKS © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 46a of 51 information 2003 Systems Consulting Limited
  • 77. How metrics can help Risks to testing progress: defects reported, fixed & retested by urgency Number Low Medium High Risks to business: defects reported, fixed & retested by importance Number Low Medium High © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 47a of 51 information 2003 Systems Consulting Limited
  • 78. But what about next time: metrics? Test specification: Estimated Estimated RISKS TO •for all defects probability x consequence = TEST AGAINST imaginable... Test execution: Consequence = •for each failure Probability detected… =1 x f (urgency, } importance) •for all undiscovered... { anomalies as yet Estimated probability x Estimated consequence = REMAINING RISKS © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 46b of 51 information 2003 Systems Consulting Limited
  • 79. How metrics can help Testing progress: Risks to testing progress: tests executed by each day of defects reported, fixed & retested by urgency testing for a level (cumulative) Number Number Time Low Medium High Testing productivity: Risks to business: defects found by each day of defects reported, fixed & retested by importance testing for a level (cumulative) Number Number Time Low Medium High © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 47b of 51 information 2003 Systems Consulting Limited
  • 80. But what about next time: metrics? Test specification: Estimated Estimated RISKS TO •for all defects probability x consequence = TEST AGAINST imaginable... Test execution: Consequence = •for each failure Probability detected… =1 x f (urgency, } importance) •for all { anomalies as yet undiscovered... Estimated probability x Estimated consequence = REMAINING RISKS © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 46c of 51 information 2003 Systems Consulting Limited
  • 81. How metrics can help Testing progress: Risks to testing progress: Fault source analysis: tests executed by each day of defects reported, fixed & retested by urgency source of faults by testing level detecting them testing for a level (cumulative) Number Require- Number ments Time System spec- Low Medium High ification Testing productivity: Risks to business: faults found by each day of defects reported, fixed & retested by importance testing for a level (cumulative) Number Design Number Time Coding Low Medium High Unit Integration System Acceptance © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 47c of 51 information 2003 Systems Consulting Limited
  • 82. Fault Source Detected early Detected as intended Detected late Unacceptably late Analysis (Rehearsal of Acceptance Tests) Requirements System specificationPhase of lifecyclein which error madewhich caused fault Design Coding Unit Integration System & Acceptance Pilot Live Testing Testing LSI Testing Testing Running Where fault detected © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 48 of 51 information 2003 Systems Consulting Limited
  • 83. Reliability engineering• A 1999 view of reliability (John D. Musa): – “division between hardware & software reliability is somewhat artificial... you may combine... to get system reliability”p35 – to model software reliability, consider: • fault introduction (development process, product characteristics) • fault removal (failure discovery, quality of repair) • environment (operational profiles) – define necessary reliability – execution time increases reliability• Other books and sources © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 49 of 51 information 2003 Systems Consulting Limited
  • 84. Decision theory • Decision theory is a body of knowledge and related analytical techniques of different degrees of formality designed to help a decision-maker choose among a set of alternatives in light of their possible consequences. Decision theory can apply to conditions of certainty, risk or uncertainty • Leads to consideration of utility value, game theory, separating information from noise, etc • Bayesian was in Myers 1976, but still being discussed now as new & exciting (because of advances in algorithms & computation?) © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 50 of 51 information 2003 Systems Consulting Limited
  • 85. Summary • Risk has been part of testing for longer than many people think: about 30 years? • Main messages today: – all testing should be based on risk – risk is difficult to calculate, so use perceptions of stakeholders, and broker consensus – “enough” testing has been done when (benefits- risks) of going live today is a positive quantity • But there still seems to be far to go before we can use it scientifically in testing on an everyday basis, and we‟re just getting to grips with the art! • Plenty of fun to come... © ThompsonNeil Thompson: Risk and Testing 27/28 Mar Slide 51x of 51 information 2003 Systems Consulting Limited

×