Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Advancing Testing Using Axioms

317 views

Published on

Доклад Paul Gerrard на конференции SQA Days-20. 24-26 ноября 2016. Минск
www.sqadays.com

Published in: Education
  • Be the first to comment

  • Be the first to like this

Advancing Testing Using Axioms

  1. 1. AdvancingTesting Using Axioms @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com
  2. 2. Helping clients transform their testing through INNOVATION, COACHING and LEADERSHIP Our CLIENTS – Want to be agile rather than follow Agile dogma – Have a pragmatic approach and are focused on delivery – Want a solution that fits, not a badly fitting suit.
  3. 3. Agenda • It all started with a thought experiment • There are no agreed principles in testing • What does an Axiom look like? • The sixteenTest Axioms • Test process improvement is a waste of time • ImprovingTesting • Conducting aTest Assessment (if I have time) • Close
  4. 4. It all started with a ‘thought experiment’
  5. 5. Are there a set of axioms that provide a framework for all test approaches?
  6. 6. What is an Axiom?
  7. 7. An uncontested principle
  8. 8. 2 + 2 = ??? Something so self-evidently true it requires no proof
  9. 9. Example Axiom: “The shortest distance between two points is a straight line”
  10. 10. We believe Axioms are INFALLIBLE (but we are prepared to be proven wrong) We KNOW that heuristics are FALLIBLE (cf the Context-Driven School)
  11. 11. “Testing as the conscience of the project” Testers tend to take principled stances
  12. 12. We all have our favourite ‘testing sayings’ but… There is no agreed set of laws so our profession is based on weak foundations.
  13. 13. We need to start with a definition of ‘test’ (American Heritage Dictionary) Test: (noun) • A procedure for critical evaluation; • A means of determining the presence, quality, or truth of something; • A trial.
  14. 14. OK – that’s the intro… Let’s play: “Hunt the Axiom”
  15. 15. Diverse principles promoted by diverse set of ‘experts’
  16. 16. Amazon offers 3557 choices
  17. 17. Differing backgrounds and experience of ‘experts’
  18. 18. Each present their own test principles
  19. 19. When some of them were written
  20. 20. Early focus on BUGS
  21. 21. Testing is an intellectually difficult activity
  22. 22. Complete testing is impossible
  23. 23. Independence of mind has value
  24. 24. Testing builds confidence …and so on…
  25. 25. Test thinking has evolved from simple beginnings
  26. 26. Testing as a lifecycle activity Model-based testing Test-obsessed Testing as exploration Digital Assurance… Agile testing Test-first, test-driven Risk-based testing Testing as learning
  27. 27. What could the Test Axioms look like?
  28. 28. From any perspective, they are self-evidently true
  29. 29. They apply to any test approach
  30. 30. They are context insensitive
  31. 31. They are not practices Practices are always context-sensitive
  32. 32. A non-Axiomatic testing approach is ‘unsound'
  33. 33. Project from HELL
  34. 34. Different approaches reflect a different emphasis on the same axioms
  35. 35. Good test approaches align with the Axioms or… The Axioms areWRONG
  36. 36. Test Axioms • Formulated as a context-neutral set of rules for testing systems • They also represent some critical thinking processes required to test any system • There are clear opportunities to advance the practice of testing using them • Testers Pocketbook: testers-pocketbook.com • Test AxiomsWebsite testaxioms.com
  37. 37. The Proposed Axioms
  38. 38. Axiom Name Each Axiom has a name and a one-line summary that states the principle.
  39. 39. Stakeholder Axiom Testing needs stakeholders
  40. 40. Test Basis Axiom Testers need sources of knowledge to select things to test
  41. 41. Test Oracle Axiom Testers need sources of knowledge to evaluate actual outcomes or behaviours
  42. 42. Fallibility Axiom Our sources of knowledge are fallible and incomplete
  43. 43. Scope Management Axiom If you don’t manage scope, you may never meet stakeholder expectations
  44. 44. Design Axiom Test design is based on models
  45. 45. Coverage Axiom Testing requires a coverage model or models
  46. 46. Delivery Axiom The usefulness of the intelligence produced by test determines the value of testing
  47. 47. Environment Axiom Test execution requires a known, controlled environment
  48. 48. Event Axiom Testing never goes as planned; evidence arrives in discrete quanta
  49. 49. Prioritisation Axiom Testing needs a mechanism for ordering tests by value
  50. 50. Execution Sequencing Axiom Run our most valuable tests first – we may not have time to run them later
  51. 51. Repeat-Test Axiom Some repeated tests are inevitable
  52. 52. Good-Enough Axiom The scope of testing and acceptance are always compromises
  53. 53. Never-Finished Axiom Testing never finishes; it stops
  54. 54. Value Axiom The value of testing is measured by the confidence of stakeholder decision making
  55. 55. Design Coverage Value Scope Prioritisation Fallibility Event Oracle Never-Finished Good-Enough Environment Basis Repeat-Test Delivery The 16 Axioms Sequencing Stakeholder
  56. 56. How to group the Axioms? • Would it be helpful to group the Axioms in some way? – To give them a structure – To relate them to the ‘real world’ • How do the Axioms relate to the test process?
  57. 57. Design Coverage Value Scope Prioritisation Fallibility Event Oracle Never-Finished Good-Enough Environment Basis Repeat-Test Delivery Grouping the Axioms Sequencing Stakeholder
  58. 58. The three axiom groups Stakeholder Value Scope Fallibility Good-Enough Delivery Repeat-Test Sequence Environment Event Never-Finished Design Basis Coverage Prioritisation Oracle
  59. 59. The three groups - summarised • Stakeholder – These are our customers – what do they want from testing? – What intelligence?When? How? How often? • Design – What do we plan to deliver? – How do we plan to deliver it? • Delivery – How do we manage the process of delivery? – How do we communicate intelligence?
  60. 60. How can we use Test Axioms? • Test Axioms are not beginners guides • They can help you to think critically about testing • They expose flaws in other people’s thinking and their arguments about testing • They generate some useful by-products • They help you to separate context from values • Interesting research areas! • First Equation of Testing,Testing Uncertainty Principle, QuantumTheory, Relativity, Exclusion Principle... • You can tell I like physics.
  61. 61. Test Process Improvement is a Waste of Time!
  62. 62. How to improve… • I want to improve my (insert any activity here) • _______ people improvement • _______ organisation improvement • _______ process improvement  Changing people (like me) and organisation (like my company) is so hard – let’s not even think about it.
  63. 63. • There are no “practice” Olympics to determine the best • There is no consensus about which practices are best, unless consensus means “people I respect also say they like it” • There are practices that are more likely to be considered good and useful than others, within a certain community and assuming a certain context • Good practice is not a matter of popularity. It’s a matter of skill and context. The delusion of ‘best practice’ Derived from “No Best Practices”, James Bach, www.satisfice.com
  64. 64. The delusion of process models (e.g. Capability Maturity Model) • Google search – “CMM” – 22,900,000 – “CMM Training” – 15,200 – “CMM improves quality” – 66 • So – lots of consultants making money out of training (and assessment)? • Does CMM improve quality?
  65. 65. Coordinate measuring machine This talk Another of my talks Another of my talks Reducing defects by 50% improves quality 2x???
  66. 66. The delusion of process models (e.g. CMM) • One of my clients … – CMM level 3 and proud of it (chaotic, hero culture) – Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping) – 40+ recommendations, only 7 adopted – they couldn’t change – How on earth did they get through the CMM 3 audit?
  67. 67. “I believe that a scientist looking at nonscientific problems is just as dumb as the next guy” “It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong” Richard P. Feynman Physics quotes…
  68. 68. “I believe that a process consultant looking at non- process problems is just as dumb as the next guy” “It doesn't matter how beautiful your process model is, it doesn't matter how smart you are. If it doesn't agree with reality, it's wrong” Me! Process quotes Using process change to fix cultural or organisational problems is not going to work Improving test in isolation is not going to work either
  69. 69. Improving Testing
  70. 70. First Equation of Testing Axioms + Context + Values + Thinking =Approach
  71. 71. Why you are where you are Context + Values + Thinking =Approach <- your values <- your context <- your thinking <- your approach
  72. 72. Making change happen <- recognise <- hard to change <- could change? <- just do some <- your approach Axioms + Context + Values + Thinking =Approach
  73. 73. Using the axioms and questions • Axioms represent the critical things to think about • Associated questions act as checklists to: – Assess your current approach – Identify gaps, inconsistencies in current approach – QA your new approach in the future • Axioms represent the WHAT • Your approach specifies HOW • Use the same questions for STRATEGY
  74. 74. Why is the equation useful? • Separation of Axioms, context, values and thinking • Tools, methodologies, certification, maturity models promote approaches without reference to your context or values • No thinking is required! • Without a unifying test theory you have no objective way of making lasting change.
  75. 75. One context, multiple approaches • Given context, practitioners can promote different approaches based on their values • Values are preferences or beliefs – Pre-planned v exploratory – Predefined v custom process – Requirements-driven v goal-based – Standard documentation v face-to-face comms. • Some contexts preclude certain practices • “No best practices”
  76. 76. Axioms allow (ensure) different approaches and expose positions • Separating axioms, context and values clarifies positions, for example: – ‘Structured’ test advocates have little (useful) to say about Agile contexts – Exploratory test advocates have little (useful) to say about contract/requirements-based acceptance • The disputes between these positions is more about values than practices in context.
  77. 77. Conducting aTest Assessment
  78. 78. Conducting a Test Assessment • We might research test documentation and records • But most information will be gathered through interviews or meetings of key staff • Here are some guidelines • Each Axiom has some suggestions for good questions.
  79. 79. Planning information gathering • Need to engage people at all levels of the organisation • Start with the most senior and work downwards • Senior managers: more focused – 30-45m • Practitioners: allow 45-60 minutes • Schedule information gathering to complete in less than two weeks.
  80. 80. Planning interviews • Three interviews per day are comfortable – More than three per day can make it hard to keep up with note-taking and assimilation. • If helpful, write a brief note that introduces you, your objective and agenda • Send to all prospective interviewees so they can prepare • Prepare a list of the questions you would like answers for – use as a checklist.
  81. 81. Conducting sessions • What problems (relevant to the scope of the assessment) do people have? • What might be potential solutions to the problem? • Do they have any questions to ask us? • Advise them of next steps and how you intend to feed back our assessment, and timetable.
  82. 82. What questions should we ask? • Axiom questions make good interview questions – Most are open questions that require more than a yes/no – Some questions are more appropriate for managers than practitioners – and vice versa • Interviewees may be nervous – They may feel that they are under investigation and their ability is in question – Assure them that there is no hidden agenda • Always treat information gathered in interviews with sensitivity and respect the privacy of the individuals concerned.
  83. 83. Testing needs stakeholders (p64) Summary: Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide Consequence if ignored or violated: There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience. Questions: – Who are they? – Whose interests do they represent? – What evidence do they want? – What do they need it for? – When do they want it? – In what format? – How often?
  84. 84. Test design is based on models (p68) Summary: Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make Consequence if ignored or violated: Tests design will be meaningless and not credible to stakeholders. Questions – Are design models available to use as test models? Are they mandatory? – What test models could be used to derive tests from theTest Basis? – Which test models will be used? – Are test models to be documented or are they purely mental models? – What are the benefits of using these models? – What simplifying assumptions do these models make? – How will these models contribute to the delivery of evidence useful to the acceptance decision makers? – How will these models combine to provide sufficient evidence without excessive duplication? – How will the number of tests derived from models be bounded?
  85. 85. Close • The Axioms exist to help you understand what’s going on • Process improvement models are not going to help you improve • If you want to improve: – UnderstandWHY you behave the way you do – Given a current culture, the environment, the context is what drives most behaviour – It’s easier to change the context than culture – Use your own thinking, not someone else’s.
  86. 86. AdvancingTesting Using Axioms @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com

×