'Best Practices' & 'Context-Driven' - Building a bridge (2003)


Published on

STAREast conference May 2003, Orlando. #1 in what I pretentiously call my Goldratt Trilogy.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

'Best Practices' & 'Context-Driven' - Building a bridge (2003)

  1. 1. “Best Practices” and “Context-Driven: Building a Bridge Neil Thompson © 2003 www.TiSCL.com Thompson +44 (0)7000 NeilTh informationTest Management track presentation (634584) SystemsSTAREast T15 Neil Thompson slide 0 of 29 Consulting Limited
  2. 2. Presentation contents • Theme: what to bridge, and how................. • Learning objectives • Instead of Best Practice: “always-good” practices • How deep is the schism, how strong the bridge supports? • Goldratt’s Theory of Constraints • The thinking tools and how to apply them • Relationship to process improvement • Conclusions & key references • Lessons learned, and way forward © Thompson information Systems T15 Consulting LimitedSTAREast Neil Thompson 1 of 29 2003
  3. 3. Theme: from conflict to integrated framework Best Unifying principles Practice “Always-Good” Practices “fossilised What Goldratt’s “formalised How thinking” “thinking tools” sloppiness” Constraints, Context- Requirements, Driven Objectives etc Thompson Expert pragmatism© information Systems with structure Consulting Limited T15 2003 STAREast Neil Thompson 2 of 29
  4. 4. Learning objectives• Understand the Best Practice v. Context-Driven debate• If your allegiance is already fixed, please un-fix at least for now: consider “always-good” principles & practices• Gain knowledge on Goldratt’s Theory of Constraints (like it or not, it’s widely used)• Appreciate the subtleties of extending its use beyond manufacturing etc• Learn enough about the thinking tools to use them• Understand enough of my examples to determine if, and how, you can use some or all of this framework © Thompson information Systems T15 Consulting LimitedSTAREast Neil Thompson 3 of 29 2003
  5. 5. Instead of Best Practice: “always-good” practices (the top-down view) Always-good Effectiveness Efficiency • “doing the right things” • “doing things right” (eg optimising speed Risk Quality management & minimising cost) management Insurance Assurance • detecting errors, • giving confidence in faults & failures fitness-for-purpose © Thompson information Systems T15 Consulting LimitedSTAREast Neil Thompson 4 of 29 2003
  6. 6. Always-good practices (bottom-up) Principles Elements What testing against, Detecting errors: the V-model QA context Giving confidence: the W-model Good enough quality Quality & risk management Risk use & test Risk management & testing prioritisation Risks by layer of V-model Tests’ priorities based on risksEffective- Appropriate stepwise Strategy, plan, design, cases, scripts, data, execution & ness refinement management procedures Structured & Test, check results, debug, fix, retest & regression-test controlled execution (problems & changes, by urgency & importance) Informed decision- Test coverage & handover criteria making Metrics (progress & problems) Role of metrics in risk management Summary: quantify residual risks & confidence Appropriate skills Business, technical & testing; roles & responsibilities; independent testingEfficiency Appropriate Glass-box, black-box etc techniques Rehearsing acceptance Appropriate tools Capture-replay, management etc Testing process Process review & improvement, eg symptom-based, overall Capability Maturity Model© Thompson information Decide targets, and improve as appropriate Systems Consulting Limited T15 2003 STAREast Neil Thompson 5 of 29
  7. 7. How deep is the schism? • CD views of BP: • BP views of CD: “fossilised thinking, “formalised sloppiness, disenfranchising, posturing, “I agree”, a disservice to testing” stating the obvious” Each side may be merely competitive, or ethically outraged • CD more outraged? • BP more willing to appease / compromise? Not trying to be controversial: is this schism good for testing? © Thompson information Systems T15 Consulting LimitedSTAREast Neil Thompson 6 of 29 2003
  8. 8. How strong are the bridge supports? This bridge is supported by “always-good” practices and Goldratt How always-good are How applicable is Goldratt these practices really? to this situation? • survived some scrutiny so far • began in manufacturing • not guaranteed; I modify them • since applied to other areas, especially slightly over time project management • but you could use your own • limited attempts on software dev’t, but... • other authors say fixed “principles”: • principles used by Agile methods this is just adding a level below • new here is applying all the thinking • deliberately omit trivial-obvious, eg tools to process context (not just use good env’ts (?) improvement) • omit if unclear message, eg “test before code” (?) • the ? marks illustrate value of the thinking tools: I may add these! © Thompson information Systems T15 Consulting LimitedSTAREast Neil Thompson 7 of 29 2003
  9. 9. Goldratt’s Theory of Constraints diagram based on those in “The Race”, E.M. Goldratt & R. Fox 1986Goal: to win the warObjective: to maximise throughput (right soldiers doing right things)Constraint on throughput: slowest marcherDrum Rope BufferCritical chain: weakest link is all we need fix, by means of...Five focussing steps: identify constraint, exploit it, subordinate all else, elevate it (i.e. strengthen so not now weakest), then... identify next constraintBut now it’s no longer simple: so need iterative tools for: what to change, what to change to, howFive thinking tools (based on sufficient causes & © Thompson information necessary conditions) Systems T15 Consulting LimitedSTAREast Neil Thompson 8 of 29 2003
  10. 10. Five logical tools for process improvement (Dettmer*, after Goldratt) What to ........... What to change to ....... change (1) (2) (3) Core problem+ Prerequisites+ CURRENT REALITY (other) Root causes Conflicts + Injections Intermediate Intermediate FUTURE Requirements + effects effects INJECTIONS REALITY Undesirable Desired Objective effects effects CURRENT CONFLICT .... How to change .... REALITY RESOLUTION (4) (5) Intermediate Needs+ objectives Specific * very slightly paraphrased here actions Obstacles Intermediate Thompson effects© PRE- information Systems Objective Consulting Limited REQUISITES Objective 2003STAREast T15 Neil Thompson 9 of 29 TRANSITION
  11. 11. The five logical tools applied to testing context, practices etc. Always- Good What “appropriate” means Context good practices in this context practices in this REALITY + Injections Root “Prerequisites” context 3 causes Intermediate FUTURE Extremes Intermediate Requirements effects REALITY effects Sub-requirements (valid & invalid Desired Effects effects Objectives assumptions) + INJECTIONS CURRENT 2a Questions to Choice categories REALITY 1 POSITIONING + Justifications consider & actions 5a Choice • Methodology 2b Intermediate categories + unhappy with CONFLICT sub-prerequisites NEEDS + Specific ( actions) RESOLUTION actions INTERACTIONS • Unsure how best to test Obstacles Actions & sub- ( conditions) 4 Intermediate objectives effects PRE- Sub-prerequisite 5b© Thompson Objectives information REQUISITES Systems Sub- Consulting Limited objectives 2003 STAREast T15 Neil Thompson 10 of 29 TRANSITION
  12. 12. 1 Context (CURRENT REALITY) Business/ App type Corporate culture org. sector Nation QUALITY / RISK FACTORS SCOPE, COST, TIME, (eg USA) Technology Legal Moral Job type & size: • project/programme constraints: constraints, eg: • bespoke/product • regulation • human safety • new/maintenance • standards • money, property • convenience Process Resources: constraints, eg: • money ( skills, environments) • quality management • time • configuration mgmt • Unsure how best Methodology • Methodology to test happy with unhappy with© Thompson information Systems Consulting Limited 2003 STAREast T15 Neil Thompson 11 of 29
  13. 13. Always-good practices 2a (CONFLICT RESOLUTION upper level) Always-good © Thompson Effectiveness information Systems Consulting Limited 2003 Efficiency Risk management Quality management Decide process targets Assess where errors originally made & improve over time Insurance Assurance Be pragmatic over quality targets Plan early, then Define & use metrics Give confidence (AT) rehearse-run, Use handover & acceptance criteria Define & detect errors (UT,IT,ST) acceptance tests V-model: what testing against W-model: quality management Use independent system & acceptance testersRisks: list & evaluate Tailor risks & priorities etc to factors Use appropriate skills mix  Refine test specifications progressively: Define & agree roles & responsibilitiesPrioritise tests based on risks  Plan based on priorities & constraints  Design flexible tests to fit Use appropriate techniques & patternsDefine & measure  Allow appropriate script format(s)test coverage  Use synthetic + lifelike data Use appropriate toolsAllow & assess for coverage changes Document execution & management procedures Optimise efficiency Distinguish problems from change requests Measure progress & problem significance Prioritise urgency & importanceQuantify residual risks & confidence Distinguish retesting from regression testing STAREast T15 Neil Thompson 12 of 29
  14. 14. What are the dimensions of formality? Next diagram will take each box from the previous diagram and assess it on a formal-informal continuum, so... In preparation for this: what do we mean by “formality”? • adherence to standards • consistency and/or proprietary • contracted-ness methods • trained-ness and • detail certification of staff • amount of • “ceremony”, eg degree documentation to which tests need to • scientific-ness be witnessed, results • degree of control audited, progress • repeatability reported ©T hompson i nformationSTAREast T15 Neil Thompson 13 of 29 • any others? S C ystems onsulting Limited 2003
  15. 15. Good practices in this context 2b (CONFLICT RESOLUTION lower level) • THEY ARE LEVELS, All systems are integrated from parts We’re too Use a lazy to think NOT STAGES • SOME SPECS ARE waterfall OUT OF DATE / IMPERFECT, We want baselines to test against BUT WE COPE V-model APPROPRIATE We want to test viewpoints of: • NEED NOT BE 1-1 CORRESP USE OF • users SPECS-LEVELS V-model: • someone expert & independent • NEED NOT BE 4 LEVELS what testing • designers • programmers Two heads are better than one against • MULTIPLE PARTIAL PASSES “Conflict” We’re doing iterative development Different levels mitigate different risks We’re doing adaptive development (no specs) • CAN USE EXPLORATORY Documentation must be minimised We have little time TESTING AGAINST CONSENSUS BASIS Don’t We’re object-oriented • V-MODEL IS IMPLICIT IN BINDER’S BOOKTesting OO systems: models, use a patterns & tools V-model V-model is MANY PEOPLE We want to discredited STAND BY V-MODEL be trendy, anyway © Thompson information Systems Consulting LimitedSTAREast T15 Neil Thompson 14 of 29 2003
  16. 16. Summary of conflict resolutionThe positioning of our always-good practice on the formal-informalscale has been prepared by:• splitting it into assumed reasons eg “documentation must be minimised”• challenging the validity of those assumption and inserting “injections” of things not originally thought of• questioning generally, which also helps understand causes & effects... Questioning some Questioning Neutral questions "positive" statements "negatively" we must always... oh, really? so what? its absolutely who says? why is that? impossible to... is that still true? are there no exceptions at all? By doing the above, we have also distilled out some basic justifications of why this practice is “always-good” © Thompson information Systems Consulting LimitedSTAREast T15 Neil Thompson 15 of 29 2003
  17. 17. 3 What “appropriate” means in this context (FUTURE REALITY) The system has: • users Our user requirements • (potentially) expert & independent testers are out of date and • designers (where significant) were vague when written • programmers We do have a good functional spec and independent testers available Our system is very simple We do need separate development & acceptance V-model with test levels • NEED NOT BE 4 LEVELS only 3 levels: We don’t need a  acceptance (v. consensus) separate  system (v. spec) integration test level  unit (informal) Our programmers hate documentation © Thompson information Systems Consulting LimitedSTAREast T15 Neil Thompson 16 of 29 2003
  18. 18. Where are we up to? Remember, can apply this partially Always- Good What “appropriate” means Context good practices in this context practices in this context 3 Overall Specifics Effectiveness Can Can stop stop FUTURE Can here REALITY here stop Efficiency here CURRENT 2a Questions to Choice categories REALITY 1 Positioning consider & actions 5a 2b CONFLICT RESOLUTION 4 PRE- 5b© Thompson information REQUISITES Systems Consulting Limited 2003 STAREast T15 Neil Thompson 17 of 29 TRANSITION
  19. 19. 4 Questions to consider (PREREQUISITES) USING ON THIS JOBBased on Kaner, Bach & Pettichord (2002) a V-model with only 3 levels:Lessons learned in software testing:  acceptance (v. consensus)a Context-Driven approach  system (v. spec)  unit (informal) What are best Under what circumstances Who is How effects on: Effectiveness would this work? most likely will • you? Efficiency to benefit? you • other stakeholders? know Tried before? What’s different What if it How What happened here, and would will worked? overcome & why? it matter? you (consider objections learn? a pilot) & sell this? What are worst effects on: What if a Under what circumstances Who is most • you? key person would this not work? likely to be • other stakeholders? disagrees? disadvantaged? © Thompson information Systems Consulting LimitedSTAREast T15 Neil Thompson 18 of 29 2003
  20. 20. 5a Choices (TRANSITION lower level) USING ON THIS JOB a V-model with For each testing level: only 3 levels: Test against one basis  acceptance (v. consensus) or multiple?  system (v. spec)  unit (informal) Each basis formally CM-baselined, or loose CM? Each basis documented, partially verbal, or If documented, what format(s)? wholly verbal? • natural language • pseudo-code • UML • ELHs, STDs, FDs etc Functional / • formal / mathematical non-functional If mixed, how manage? constituents? If verbal, how communicate and how get agreement? In what “language” and level of detail are Where non-functional, risks expressed? relationship to quality standards? © Thompson information Systems Consulting LimitedSTAREast T15 Neil Thompson 19 of 29 2003
  21. 21. 5b Choice categories & actions (TRANSITION upper level) Overall objectives, eg determine process improvement targets and improve over time Processes V-model: what testing against Use independent Communications Independence system & acceptance testers plan: • training coursesINTERACTIONS Use appropriate Techniques techniques • workshops Effectiveness objectives, • documented rg quantify residual risks Tools Use appropriate in-house & confidence for each job tools guidelines Coverage Allow & assess for coverage changes Efficiency objectives, eg optimise (within context) why? Intranet Ceremony Provide standard progress reports site Investment Define & use metrics Assess where errors originally made NEEDS CMM/TMM TPI® Symptom-based level targets targets targets eg TOM™ Thompson © information Systems Consulting LimitedSTAREast T15 Neil Thompson 20 of 29 2003
  22. 22. So why not just start with a process improvement method? • Very good question! Possible answers: – “we’re not ready for process improvement, just want a structure to apply Context-Driven” – “ we want to choose a process improvement method” – “the methods are too restrictive, we want to build our own” • Theory of Constraints answer: – need to do some analysis before we know where best to focus (or to improve processes) • Thinking tools answer: – want to look at interactions between choices... © Thompson information Systems Consulting LimitedSTAREast T15 Neil Thompson 21 of 29 2003
  23. 23. Interactions between choice categories INFLUENCES Processes Independence © Thompson information Systems Consulting Limited 2003 INFLUENCE INFLUENCES Techniques HINDERS INFLUENCE CONSTRAIN GREATLY HELP Tools PROVIDE SOME DATA HELP FOR Coverage MEASURE SOME Ceremony HELPS IMPROVES MEASURE DOCUMENTATION HELPS InvestmentSTAREast T15 Neil Thompson 22 of 29
  24. 24. Can use tables before / instead of / after diagrams Good Can © Thompson Context practices stop Can What “appropriate” information Systems here Consulting Limited stop means 2003 here Can FUTURE stop CONFLICT REALITY here RESOLUTION Principles & elements Extremes & CURRENT variables Questions to Choice categories REALITY consider & actions Can stop here Needs eg TMM Can stop here PREREQUISITESSTAREast T15 Neil Thompson 23 of 29 TRANSITION
  25. 25. Conclusions• Yes, there are differences between Context-Driven & Best Practice, but: – not as great as may first appear – each side attempts to be inclusive (but sees other as not!) – partly due to different backgrounds & industry sectors – this public-domain method can bridge the gap – need not be interpreted rigidly, or executed in full – the basic thinking is causes & effects: anyone can do it• To summarise: – the framework, in effect, deconstructs then reconstructs in your context... © Thompson information Systems Consulting LimitedSTAREast T15 Neil Thompson 24 of 29 2003
  26. 26. How many diagrams, where and when“Universal” CONFLICT RESOLUTION upper one 2a-bYour context CURRENT REALITY oneEach requirement CONFLICTto examine RESOLUTION eg a V-model eg testware lifecycle eg patterns+ techniques ... 2-6 (out of 25) lower 2b-3What “appropriate” FUTUREmeans in your context REALITY ... 2-6 or more (out of 25) 3-4Questions toconsider PRE- REQUISITES ... 2-6 4-5a (out of 25)Choices TRANSITION ... 2-6 lower 5a-b (out of 25)Choicecategories TRANSITION one& actions upperwhen © Thompson information where stop how SystemsSTAREast T15 Neil Thompson 25 of 29 Consulting Limited 2003 points many
  27. 27. Deconstruct then reconstruct: the framework is a “meta-V-model”All possible CONFLICT RESOLUTION TRANSITION Choice contexts upper upper categories & actions Your context CURRENT TRANSITION REALITY lower Choices CONFLICT Each practice PRE- Questions to RESOLUTION to examine REQUISITES consider lower FUTURE FUTURE REALITY REALITY What “appropriate” © Thompson information means in your context Systems Consulting LimitedSTAREast T15 Neil Thompson 26 of 29 2003
  28. 28. • Context-Driven: Key references – Kaner, Bach & Pettichord (2002) Lessons learned in software testing, Wiley• Best Practice: – Erik van Veenendaal et al. (2002) The Testing Practitioner, U.T. Nolthenius• My inspiration: – Jens Pas (EuroSTAR 1998) Software testing metrics – Gregory Daich (STAREast 2002) Software documentation superstitions• Theory of Constraints understanding: – Eliyahu M. Goldratt (1984 then 1992 with Jeff Cox) The Goal; (1997) Critical Chain, N. River Press• TOC overview and the thinking tools: – H. William Dettmer (1997) Goldratt’s TOC: a systems approach to cont. improv’t, ASQ• Related (but differently-specialised) thinking from Agile community: – Alistair Cockburn: A methodology per project, www.crystalmethodologies.org – Mary Poppendieck: Lean development: an agile toolkit, © Thompson www.poppendieck.com information Systems Consulting LimitedSTAREast T15 Neil Thompson 27 of 29 2003
  29. 29. Lessons learned, and way forward• Reactions so far: – “but this looks very like our “Best Practice” training course syllabus” – “this may be too controversial” – “I don’t understand what you’re on about” – “why bother to analyse statements of the obvious?” – “we should be able to come up with something more specific / detailed / quantitative / prescriptive than this”• So main lessons so far are: – it may be obvious (and excitingly new) to me, but others differ, and... – the reactions are mutually inconsistent!• Ways forward: – for me: can the other parts of Goldratt’s thinking also usefully apply to testing? – for you (and therefore also for me!): can you use this framework? © Thompson information Systems Consulting LimitedSTAREast T15 Neil Thompson 28 of 29 2003
  30. 30. • Questions please? • Thank you!Neil ThompsonNeilT@TiSCL.com • Contact details...+44 (0)7000 NeilTh (634584) www.TiSCL.com23 Oast House Crescent © 2003Farnham, Surrey, England ThompsonGU9 0NP, United Kingdom informationTest Management track presentation SystemsSTAREast T15 Neil Thompson slide 29 of 29 Consulting Limited