Risk Mitigation Trees:    Review test handovers with stakeholders     12th European Conference on Software Testing, Analys...
1. Introduction• Primary objective of presentation – to share two simple  techniques (used successfully on recent projects...
Introduction (cont’d)• Secondary objective is to generalise TRBs & RMTs into a  broad framework of decision-making:      –...
Agenda•    2. Traditional methods of controlling handovers•    3. Judging the best date for handover•    4. Co-operation: ...
2. Traditional methods of  controlling handovers• “Over the wall”:      – bugs may be passed over undetected, amplifying p...
3. Judging the best date        for handover• Risk-benefit balance:      – “Good Enough Quality” is popular principle…    ...
Judging the best date for    handover (cont’d)                                                   # bugs•       Metrics – u...
Judging the best date for   handover (cont’d)• Metrics:      – that “glide path” takes a long time to reach target, and ne...
4. Co-operation: Testing       Review Boards                                                                              ...
Testing Review Boards           (cont’d)• Imagine the variation in caveats from these different  viewpoints!              ...
Testing Review Boards           (cont’d)• Not anarchy: it should be pre-agreed who is the prime decision-maker  at each me...
Testing Review Boards           (cont’d)• Some obstacles & pitfalls:             • Ways to cope:      – too many meetings ...
5. Pragmatism: Risk            Mitigation Trees                                                                           ...
Risk Mitigation Trees              (cont’d)  specific                                                   confidence in     ...
Risk Mitigation Trees             (cont’d)• Advantages of RMTs:     – they can make a previously daunting and amorphous se...
6. Decision theory• That’s the end of the straightforward practical advice: now  for some interesting background (ideas fo...
Risk or uncertainty?     Different conditions under which decisions are made…                       Certainty             ...
Uncertainty-Based             Testing!• So all this we’ve been saying                                                     ...
Uncertainty-Based          Testing (cont’d)• “Simplifying” framework has left two core uncertainties:      – what will be ...
Types of uncertainty in          testing• “Knowledge incompleteness due to inherent deficiencies  with acquired knowledge”...
Types of uncertainty in       testing (cont’d)  Ideally we’d understand defect-fault-failure chains for all  of our existi...
Can more trees help?• Event trees & fault trees can be used to investigate defect-  fault-failure relationships, and in th...
What about               mathematics?• Decision theory advises us to use utility functions• Utility may be defined as “the...
7. Game Theory• If we can’t (yet?) use reliability theory / utility functions /  Bayesian networks to assess risk-benefit ...
Snakes & Ladders?PROJECT MANAGER:                               BUSINESS SPONSOR:↑ Will I get a reputation as a           ...
n-person Game Theory •     Game Theory: a maths-based       framework for competing                                  (i)  ...
But for now:           a game of three halves  In effect, can consider TRB as a two-competitor game having  also a fore-ga...
8. Systems theory &           cybernetics                                              Philosophy                         ...
9. Conclusions &              way forward• Until reliability models get cleverer / simpler, and more  metrics collected, w...
Potential way forward                         (cont’d): benefits                                                          ...
Potential way forward                        (cont’d): risk-costs                                                         ...
Summary                                     Systems sciences                                             Decision         ...
References &          acknowledgements• Main references:      –   Beizer, Boris: Software system testing & quality assuran...
Contact details     Neil Thompson     NeilT@TiSCL.com                                            Questions?     www.TiSCL....
Upcoming SlideShare
Loading in …5
×

Risk Mitigation Trees - Review test handovers with stakeholders (2004)

492 views

Published on

EuroSTAR conference Dec 2004, Köln.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
492
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Risk Mitigation Trees - Review test handovers with stakeholders (2004)

  1. 1. Risk Mitigation Trees: Review test handovers with stakeholders 12th European Conference on Software Testing, Analysis & Review 29 November - 03 December 2004 – Köln, Germany Management Track session T6 Neil Thompson Thompson information Systems Consulting Limited www.TiSCL.com© Thompson information Systems Consulting Limited 1 www.qualtechconferences.com
  2. 2. 1. Introduction• Primary objective of presentation – to share two simple techniques (used successfully on recent projects): – Testing Review Boards (TRBs) and – Risk Mitigation Trees (RMTs)• Used for making collaborative decisions at development- test handovers, then at go-live• Go well together, though can be used separately• TRBs are meetings of testing’s stakeholders at handovers• RMTs are a diagramming technique to split residual risk• There is also a secondary objective of this presentation…© Thompson information Systems Consulting Limited 2 www.qualtechconferences.com
  3. 3. Introduction (cont’d)• Secondary objective is to generalise TRBs & RMTs into a broad framework of decision-making: – so we can better understand the nature of the decisions – so we can improve the process – in future we may be able to benefit from better reliability theories and more user-friendly statistics; – there are relevant considerations in decision theory & game theory – process overall may be integrated by cybernetics & systems science (self-organising systems)• Most of the esoteric stuff is in the accompanying paper• This presentation is based on my experiences in large projects & programmes, but principles should be valid in other handover contexts, eg software product development© Thompson information Systems Consulting Limited 3 www.qualtechconferences.com
  4. 4. Agenda• 2. Traditional methods of controlling handovers• 3. Judging the best date for handover• 4. Co-operation: Testing Review Boards• 5. Pragmatism: Risk Mitigation Trees• 6. Decision theory• 7. Game theory• 8. Systems theory and cybernetics• 9. Conclusions and way forward© Thompson information Systems Consulting Limited 4 www.qualtechconferences.com
  5. 5. 2. Traditional methods of controlling handovers• “Over the wall”: – bugs may be passed over undetected, amplifying problems later – even if the software works well, could be insufficient collaboration – not only waterfall, could also affect iterative methods (but not truly Agile methods, which emphasise interpersonal communications)• Handover certificates: – can languish in in-trays; very often signed with caveats• Correspondence of exit & entry criteria: – only the simplest waterfall method has no overlap – entry includes own responsibilities in addition to donors – with overlap, entry criteria less demanding than exit© Thompson information Systems Consulting Limited 5 www.qualtechconferences.com
  6. 6. 3. Judging the best date for handover• Risk-benefit balance: – “Good Enough Quality” is popular principle… – but different stakeholders in testing are likely to have different views of the benefits, and especially the risks – We don’t (yet?) have an objective view of the risks, in terms of the likely bug rate after go-live, despite the long history of…• Reliability theories: – dominated the early literature on testing, but have faded – mathematics drown realism – Bayesian techniques seem most promising, but – whole subject still has “holy grail” status for practical use© Thompson information Systems Consulting Limited 6 www.qualtechconferences.com
  7. 7. Judging the best date for handover (cont’d) # bugs• Metrics – use quantitative for a while: – S-curves have empirical evidence and quasi-theoretical basis Cumulative bugs – can be applied to both progress through tests and bug-fixing… – but subject to caveats: uniform test Resolved strategy, “tyranny of numbers” Awaiting – approach target as a “glide path”… fix Deferred # Closedtests date Target tests run Fail Actual tests run Pass date© Thompson information Systems Consulting Limited 7 www.qualtechconferences.com
  8. 8. Judging the best date for handover (cont’d)• Metrics: – that “glide path” takes a long time to reach target, and never totally smooth (also, target is short of perfection to begin with) – to time the landing (handover), what will be important will be the specific nature of the shortfalls, and their impacts – so are qualitative metrics better than quantitative? – I say use both: coarse-tune with quantitative, then fine-tune with qualitative• So, we need mechanisms to interpret metrics and agree what to do: – (co-operation) Testing Review Boards to review progress and agree confidence – (pragmatism) Risk Mitigation Trees to analyse residual risk and agree viable compromises and mitigations© Thompson information Systems Consulting Limited 8 www.qualtechconferences.com
  9. 9. 4. Co-operation: Testing Review Boards 7 Live• Advantages of TRBs over sign-off forms: OAT 5 6 – forum for discussing & resolving differences of view among stakeholders UAT – force a decision on given date 3 4 – robust shared decisions System Testing 2 Integration Testing 1 Development & Unit Testing TRB attenders TRB # 1 2 3 4 5 6 7 BUSINESS SPONSOR ------- ------- ------- ------- ------- attend attend IT OPERATIONS MANAGER ------- ------- ------- ------- attend attend attend PROJECT MANAGER attend attend chair attend chair attend chair TESTING / QUALITY MANAGER chair chair facilitate chair facilitate chair facilitate BUSINESS ARCHITECTS ------- attend attend attend ------- attend attend TECHNICAL ARCHITECTS attend attend ------- attend attend ------- attend DEVELOPMENT MANAGER attend attend ------- attend ------- ------- attend© Thompson information Systems Consulting Limited 9 www.qualtechconferences.com
  10. 10. Testing Review Boards (cont’d)• Imagine the variation in caveats from these different viewpoints! “quality” (or• Some may not sign low risk) in time, or at all. TESTERS BUSINESS ARCHITECT IT OPERATIONS scope (of TEST MANAGER MANAGER USER testing done) BUSINESS SPONSOR IT PROJECT DEVELOPMENT low DIRECTOR MANAGER MANAGER speed of testing, cost timeliness of implementation© Thompson information Systems Consulting Limited 10 www.qualtechconferences.com
  11. 11. Testing Review Boards (cont’d)• Not anarchy: it should be pre-agreed who is the prime decision-maker at each meeting: the “casting vote”• That should be the recipient (varies between different TRBs)• So incorporates traditional acceptance hierarchy… (examples) 5 7 agrees to receive accepts IT OPERATIONS MANAGER BUSINESS SPONSOR recommends recommends PROJECT MANAGER PROJECT MANAGER give information & opinions give information & opinions TECHNICAL OPERATORS TEST ETC. BUSINESS USERS TEST ETC. ARCHITECTS MANAGER ARCHITECTS MANAGER entry to go-live Operational (IT) Acceptance Testing© Thompson information Systems Consulting Limited 11 www.qualtechconferences.com
  12. 12. Testing Review Boards (cont’d)• Some obstacles & pitfalls: • Ways to cope: – too many meetings already! – are any existing meetings less – may be multiple sites, perhaps important? multiple companies – audio- & video-conferencing – difficult to get aligned diary – electronic scheduling slots – deputisation allowed – input information volatile, – clerical & tool assistance right up to the meeting – take simultaneous snapshots, – timing of meeting is finely- even if analysis lags balanced; late disappointments – go-ahead with extra mitigation threaten postponement is better than postponement• I’ve seen TRBs work. But may become so popular that managers request also mini-TRBs, and even micro-TRBs!© Thompson information Systems Consulting Limited 12 www.qualtechconferences.com
  13. 13. 5. Pragmatism: Risk Mitigation Trees etc (fix & retest) lo-pri significant passed failed unresolved bugs not hi-pri hi-pri run not run coverage any coverage OK now shortfall Tests Bugs specific planned “zero-bug nirvana” problems test coverage planned coverage bug acceptance criteria handover target handover target date date© Thompson information Systems Consulting Limited 13 www.qualtechconferences.com
  14. 14. Risk Mitigation Trees (cont’d) specific confidence in benefits in decisions impacts mitigations fix coming having that function, problems in time even if faulty needed business limit delay value threat exposure exposure customers internal users Go / internal costs manual fix on no go transactions table entries workarounds fail frequency numbers affected of use potential mitigations© Thompson information Systems Consulting Limited 14 www.qualtechconferences.com
  15. 15. Risk Mitigation Trees (cont’d)• Advantages of RMTs: – they can make a previously daunting and amorphous set of test shortfalls, and bugs, seem manageable – not only seem manageable, but are manageable (“divide & conquer”) – the specific impacts of shortfalls, and ways to mitigate them, can be (perhaps surprisingly) easier to make decisions about than arbitrary numbers such as “<3 highs, <20 medium bugs tolerable” – the numbers of un-run tests and un-fixed bugs are viewed in the light of high (usually) numbers of successful tests and fixed bugs• Risk & impact mitigation: – options include pre-empt/react, avoid/mitigate, transfer – examples: • some functions need not be used for several months • some failures would be in such small numbers and so easily seen that fix-on-fail is less risky than a software upgrade© Thompson information Systems Consulting Limited 15 www.qualtechconferences.com
  16. 16. 6. Decision theory• That’s the end of the straightforward practical advice: now for some interesting background (ideas for future thought and process improvement)• Decision theory: “a body of knowledge and related analytical techniques of different degrees of formality designed to help a decision-maker choose among a set of alternatives in light of their possible consequences”• Decisions typically made at TRBs: do we go live, or delay by a week, or delay by a month, or call in the lawyers?• So can decision theory help us here?© Thompson information Systems Consulting Limited 16 www.qualtechconferences.com
  17. 17. Risk or uncertainty? Different conditions under which decisions are made… Certainty Risk UncertaintyAlternatives A B C A B C A B CConsequences a b c a1 b1 c1 a1 b1 c1 a2 b2 c2 a2 b2 c2 b3 c3 b3 c3Probability b4 b4of each p(a1), p(a2), p(b1), p(b2), p(b3), p(b4), ?, ?, ?, ?, ?, ?,consequence p(c1), p(c2), p(c3) ?, ?, ?© Thompson known unknown information Systems Consulting Limited 17 www.qualtechconferences.com
  18. 18. Uncertainty-Based Testing!• So all this we’ve been saying Simplifying about Risk-Based Testing: it TRB framework… should properly be called decision Uncertainty-Based Testing Alternatives: A - go live now• We do not know the probability B - delay 1 week of each possible consequence A B C C - delay 1 month• We don’t even have a fixed set Consequences (additive): of alternatives: could be various a1 b1 c1 1 - cost of future live bugs 2 - benefit of going live combinations of delay, overtime, a2 b2 c2 3 - cost of longer project 4 - penalty payment costs descoping etc b3 c3• But suppose we could simplify c4 (example as earlier)… ?, ?, ?, ?, ?, ?, p(c4) 100%, ?, ?, ? p(b3) & p(c3) 100%, p(a2), p(b2) & p(c2) uncertain, p(a1), p(b1) & p(c1) unknown© Thompson information Systems Consulting Limited 18 www.qualtechconferences.com
  19. 19. Uncertainty-Based Testing (cont’d)• “Simplifying” framework has left two core uncertainties: – what will be the benefits of going live at a particular date? – what will be the costs of the bugs which will affect live operations?• Decision theory offers two main approaches to uncertainty: – reduce the uncertainty to mere risk by deeming the unknown probabilities “known” by using subjective estimates from experts and/or previous experience – look at the choice criteria using Game Theory (about which, more later)• Types of uncertainty: – philosophical analysis (see paper) – related to testing…© Thompson information Systems Consulting Limited 19 www.qualtechconferences.com
  20. 20. Types of uncertainty in testing• “Knowledge incompleteness due to inherent deficiencies with acquired knowledge”: – we know what we’ve tested, but: • right things? Any low-payback waste? – we know bugs found, but: • how many faults remain? Which will trigger failures? Impact? – we know bugs fixed, but: • how good was our regression testing? Impact of knock-ons?• “Ambiguity, approximations, randomness & sampling”: – what is signal, what is noise; what trends are statistically significant, eg if high-impact bugs out of testing over last 3 weeks numbered 1, 0 then 2, what does that tell us?© Thompson information Systems Consulting Limited 20 www.qualtechconferences.com
  21. 21. Types of uncertainty in testing (cont’d) Ideally we’d understand defect-fault-failure chains for all of our existing failures, and prevent/mitigate future mistakes…  Mistake:UNCERT’Y a human action that (false alarm): produces an incorrect or Change Request, result (eg in spec- writing, program- coding)  Anomaly: or testware mistake an unexpected result RISK OF MIS-  this fits its usage Note: during testing INTERPRETING RISK RISK OF MISSING in inspections Defect:  Fault:  Failure: incorrect results RISK an incorrect step, RISK an incorrect result in specifications process or data definition in a computer program  Error: amount by which RISK (ie executable result is incorrect software) Direct programming mistake© Thompson information Systems Consulting Limited 21 www.qualtechconferences.com
  22. 22. Can more trees help?• Event trees & fault trees can be used to investigate defect- fault-failure relationships, and in theory the risk of future live bugs is the sum of a set of micro-risks (but this kind of analysis is easier after the event than before!)• Attempts have been made to use decision trees (not the same as RMTs) to assess risk (ref. Shari Lawrence Pfleeger). But: – risks have a probability distribution, not a single definite probability – quantitative risk assessment can have misleading “precision” and can greatly differ from people’s perceptions – is lo-probab hi-impact really the same as hi-probab lo-impact? – if not, how to tweak the factors?© Thompson information Systems Consulting Limited 22 www.qualtechconferences.com
  23. 23. What about mathematics?• Decision theory advises us to use utility functions• Utility may be defined as “the real or fancied ability of a good or service to satisfy a human want”• This sounds like acceptance testing: working backwards, each handover decision is part of a sequence of decisions• Each decision affects next: eg handover too soon from development, & resultant disruption may exceed time saved• Simple decision theory inadequate (deterministic, Markov)• Bayesian networks look promising, but are difficult for non- mathematicians and need much computing power© Thompson information Systems Consulting Limited 23 www.qualtechconferences.com
  24. 24. 7. Game Theory• If we can’t (yet?) use reliability theory / utility functions / Bayesian networks to assess risk-benefit balance at handovers, let’s fall back on the decision theory alternative…• Game theory: – big in social sciences, and well-established in business strategy – has already been invoked for software development as a whole: a resource-limited, co-operative, finite, goal-seeking game of invention & communication Alistair Cockburn – is a research effort to use in automatic test case generation Microsoft• The relevance to Testing Review Boards is that most or all of the participants are playing a game called Career: the project, and the testing phases, are just subgames© Thompson information Systems Consulting Limited 24 www.qualtechconferences.com
  25. 25. Snakes & Ladders?PROJECT MANAGER: BUSINESS SPONSOR:↑ Will I get a reputation as a ↑ Will this project enhance skilled risk-balancer? my career? Will I be seen↓ Will I get a reputation as a as a brave decision-taker? deadline-misser / budget- ↓ Will I lose respect? Will this buster? project damage the bottom line?DEVELOPER: USER:↑ Will this project help me get ↑ Will system make our work a promotion / interesting new more fulfilling & value- project? adding?↓ Will someone find faults in ↓ Will live failures give us my modules after go-live? extra frustrating work? Will I get recalled?© Thompson information Systems Consulting Limited 25 www.qualtechconferences.com
  26. 26. n-person Game Theory • Game Theory: a maths-based framework for competing (i) moves: α β individuals or groups to each extensive αA βA 10, 4 try to maximise their utility form βB 1, 5 βA through a series of moves COMPETITOR ALPHA αB βB 9, 9 0, 3 (sequential or simultaneous) αA 10, 4 COMPETITOR BETA βA • Competitors may or may not αB 9, 9 αA 1, 5 know the moves and utilities of βB αB 0, 3 other competitors (ii) strategies: COMPETITOR BETA • In a TRB, there are more than normal units of utility two “competitors” who may be form to each Alternative Alternative competitor βA βB arguing for different outcomes and have an imperfect view of Alternative 4 5 COMPETITOR αA 10 1 utilities ALPHA Alternative 9 3 • Suggests we need the most αB 9 0 complex and abstract part of game theory… (iii) payoffs: >2 COMPETITORS characteristic v(x)=…Material combined from Game theory – a critical introduction,Hargreaves Heap & Varoufakis (Routledge 1995); and N-person function multi-dimensional space,game theory, Rapoport (Dover 1970-2001) form may involve coalitions© Thompson information Systems Consulting Limited 26 www.qualtechconferences.com
  27. 27. But for now: a game of three halves In effect, can consider TRB as a two-competitor game having also a fore-game and an after-game… “quality” (or low risk) Mitigating actions Delay how long? TESTERS BUSINESS (as from RMT) Add any scope? ARCHITECT scope (of IT TEST USER testing done) OPERATIONS MANAGER MANAGER BUSINESS Go No-go IT PROJECT DEVELOPMENT SPONSOR COMPETITORlow cost speed of testing, MANAGER MANAGER COALITION DIRECTOR timeliness of implementation BETA A B (“doves”) Utility fore-game U = u(a1)+u(a2) or u(b1)+(b2), Alternative Alternative ALPHA view or BETA view A: go B: no-go• consider preferences Alternative COMPETITOR hi-U ok-U (A, B etc) based on A: go after-game COALITION hi-U hi-U views ALPHA Alternative illogical & ok-U• form hawk / dove (“hawks”) B: no-go impossible ok-U coalitions (real or “usual case” but can’t virtual) rubber-stamp shared-fear continue to differ, © Thompson case case need to negotiate information Systems Consulting Limited 27 www.qualtechconferences.com
  28. 28. 8. Systems theory & cybernetics Philosophy “things” “thoughts” Metaphysics Logic Epistemology Natural sciences Systems sciences Social sciences General Mathematics systems Languages thinking Systems theory Cybernetics (structure) (function) control communication TRBs & RMTs Decision Complexity Theory Theory value / utility / game Decision-maker self-organising systems etc functions functions theory Probabilities Decision Objectives Constraints Consequences Alternatives certainty / risk / uncertainty© Thompson information Systems Consulting Limited 28 www.qualtechconferences.com
  29. 29. 9. Conclusions & way forward• Until reliability models get cleverer / simpler, and more metrics collected, we can’t predict bug rates after go-live• So main alternatives for managing handovers seem still: – rigid criteria – these Testing Review Board & Risk Management Tree techniques – something even more agile• To progress to something more scientific, may need to confront the following complexity…© Thompson information Systems Consulting Limited 29 www.qualtechconferences.com
  30. 30. Potential way forward (cont’d): benefits most likely relative probability of 50% point relative probability of actual go-live being achieved this benefit earliest latestof go-live this datetotal gross benefit 1 Jan 1 Apr 1 May 31 Dec date estimated estimated min gross benefit per day most likely of go-live this date max (area under curve) 1 Jan 1 Apr 1 May© Thompson Extended from material by DeMarco & Lister, Waltzing with bears (Dorset House 2003) information Systems Consulting Limited 30 www.qualtechconferences.com
  31. 31. Potential way forward (cont’d): risk-costs most likely relative probability of 50% point relative probability of actual go-live being incurred this risk-cost earliest latestof go-live this date 1 Jan 1 Apr 1 May 31 Dec date total risk-cost estimated • even more estimated min risk-cost per day most likely of go-live this date difficult to max (area under curve) estimate 1 Jan than benefits? • much more “spiky”! 1 Apr • (note: excludes fixed costs) 1 May© Thompson Extended from material by DeMarco & Lister, Waltzing with bears (Dorset House 2003) information Systems Consulting Limited 31 www.qualtechconferences.com
  32. 32. Summary Systems sciences Decision Theory value / utility / game Risks & benefits functions functions theory forecast Risks & benefits now: 1 - bugs visible now 1 - cost of future live bugs 2 - stated target benefits 2 - benefit of going live 3 - cost of longer project 4 - penalty payment costs Testing Review Boards Usable reliability theory S-curves Risk Mitigation Trees decision Bayesian networks support© Thompson information Systems Consulting Limited 32 www.qualtechconferences.com
  33. 33. References & acknowledgements• Main references: – Beizer, Boris: Software system testing & quality assurance – Pirsig, Robert M: Lila – an enquiry into morals – Bach, James: Good enough quality etc – DeMarco & Lister: Waltzing with bears – Principia Cybernetica Web – Rapoport, Anatol: N-person game theory – Kaner, Cem: Software testing as a social science – (for others, see associated paper in conference proceedings)• Acknowledgements: – to Pat, Rob & Rupert for their very considerable input to the Risk Mitigation Trees method – to all the team on that programme and at that client© Thompson information Systems Consulting Limited 33 www.qualtechconferences.com
  34. 34. Contact details Neil Thompson NeilT@TiSCL.com Questions? www.TiSCL.com 23 Oast House Crescent Farnham, Surrey, England GU9 0NP, United Kingdom phone +44 (0)7000 NeilTh (634584) or +44 (0)7710 305907© Thompson information Systems Consulting Limited 34 www.qualtechconferences.com

×