The Axioms of Testing

2,660 views

Published on

Is it possible to define a set of axioms that provide a framework for software testing that all the variations of test approach currently being advocated align with or obey? In this respect, an axiom would be an uncontested principle; something self-evidently and so obviously true and not requiring proof. What would such test axioms look like? This paper summarises some preliminary work on defining a set of Test Axioms. Some applications of the axioms that would appear useful are suggested for future development. It is also suggested the work of practitioners and researchers is on very shaky ground unless we refine and agree these Axioms. This is a work in progress.

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
2,660
On SlideShare
0
From Embeds
0
Number of Embeds
39
Actions
Shares
0
Downloads
46
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

The Axioms of Testing

  1. 1. THE<br />AXIOMS<br />TESTING<br />OF<br />Paul Gerrard<br />
  2. 2. This talk is an experiment118 slides in 35 minutes ;-)Ever seen presentationzen.com?(and I had some fun with Google images)<br />
  3. 3.
  4. 4. Test Axioms – Do They Exist?What might they look like?But first, some video…<br />
  5. 5. Pentagon Wars<br />
  6. 6. A comedy, where the military officers lack principles except for…<br />
  7. 7. The NewTest Manager<br />
  8. 8. The book/film is based on a true story – a farceMy question for you is…<br />
  9. 9. Is there a single, agreed set of guiding principles for testing and test managers?<br />
  10. 10. More formally…<br />
  11. 11. Are there a set of axioms that provide a framework for all test approaches?<br />
  12. 12. What is an Axiom?<br />
  13. 13. An uncontested principle<br />
  14. 14. 2 + 2 = ???Something soself-evidently true it requires no proof<br />
  15. 15. A foundation for thinking and behaviour<br />
  16. 16. More formally…<br />An axiom is something that is impossible to prove to be true but could be disproved with a single experiment/example.<br />
  17. 17. Example AxiomDeclaration of independence para 2:<br />“We hold these Truths to be self-evident, that all Men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty, and the pursuit of Happiness.”<br />
  18. 18. Example Axiom 2<br />“The shortest distance between two points is a straight line”<br />
  19. 19. We believe Axioms are INFALLIBLE(but we are prepared to be proven wrong)<br />We KNOW that heuristics are FALLIBLE<br />(cf the Context-Driven School)<br />
  20. 20. Testers like principles<br />
  21. 21. “Testing as the conscience of the project”Testers tend to take principled stances<br />
  22. 22. So, it would help if we had a set of principles or axioms to refer to<br />
  23. 23. We all have our favourite ‘testing sayings’ but…<br />There is no agreed set of laws so our profession is based on weak foundations.<br />Repeat after me….<br />
  24. 24. We need to start with a definition of ‘test’<br />American Heritage Dictionary:<br />Test: (noun)<br /><ul><li>A procedure for critical evaluation;
  25. 25. A means of determining the presence, quality, or truth of something;
  26. 26. A trial. </li></li></ul><li>OK – that’s the intro…Let’s play:“Hunt the Axiom”<br />
  27. 27. The testing community has many experts and evangelists<br />
  28. 28. Many books on software testing…<br />And there appear to be 713 of them …<br />
  29. 29. Chapter 1 PrinciplesChapter 2 Approachblah blah etc…<br />
  30. 30. But… there are some differences in opinion<br />
  31. 31. Diverse principles promoted by diverse set of ‘experts’<br />
  32. 32. Beizer, Black, Craig, Jaskiel, Gerrard, Thompson, Hetzel, Kaner, Bach, Petticord, Falk, Nguyen, Kit, Perry, Pol, Teunissen, van Veenendaaland many more…<br />
  33. 33. Each present their own test principles<br />
  34. 34. What do we observe here?<br />
  35. 35. Variable,credible,some even usefullet’s look closer…<br />
  36. 36. Differing backgrounds and experience of ‘experts’<br />
  37. 37. When some of them were written<br />
  38. 38.
  39. 39. Early focus on BUGS<br />
  40. 40. Test thinking has evolved<br />
  41. 41. Testing as a lifecycle activity<br />Model-based testing<br />Test-obsessed<br />Testing as exploration<br />Business goal-based testing<br />Agile testing<br />Test-first, test-driven<br />Risk-based testing<br />Testing as learning<br />
  42. 42. Some ‘principles’ appear again and again<br />
  43. 43. Testing is an intellectually difficult activity<br />
  44. 44. Complete testing is impossible<br />
  45. 45. Independence of mind has value<br />
  46. 46. Focusing on bugs is good<br />
  47. 47. Testing builds confidence…and so on…<br />
  48. 48. But some ‘principles’ are not universally accepted<br />
  49. 49.
  50. 50. …advocated as the professional approach by some…<br />
  51. 51.
  52. 52. Dynamic, concurrent test design and execution, using heuristics and an agile mentality is promoted by some…<br />
  53. 53.
  54. 54. What could theTest Axioms look like?<br />
  55. 55. Some suggested criteria for Test Axioms…<br />
  56. 56. From any perspective, they are self-evidently true<br />
  57. 57. They apply to any test approach<br />
  58. 58. Axioms are not guidelines<br />
  59. 59. They are context insensitive<br />
  60. 60. They are not practices<br />Practices are always context-sensitive<br />
  61. 61. They are easy to understand<br />
  62. 62. Corollaries<br />
  63. 63. A non-Axiomatic testing approach is ‘unsound&apos;<br />
  64. 64. Projects that ignore the axioms might well be eCcEntRicWorse, you might be working in a …<br />
  65. 65. Project fromHELL<br />
  66. 66. …or the people doing the testing, aren’t<br />
  67. 67. Different approaches reflect a different emphasis on the same axioms<br />
  68. 68. Some people think that’s wrong<br />
  69. 69. “If you believe in axioms, you have a closed mind”<br />“Murder is wrong” is axiomatic.<br />
  70. 70. “There are no universal axioms in our field”<br />ALL the Axioms have been compared with the writings of the folk who have objected.<br />
  71. 71. “Axioms define a school of testing”<br />I believe Test Axioms exist.<br />If we can get them right ALL schools of testing MUST align with them.<br />
  72. 72. Do I want to be a School?<br />Too many schools push dogma, demand obedience to school rules and mark people for life. They put up barriers to entry and exit and require members to sing the same school song. No thanks.<br />
  73. 73. Axioms are just a high level abstraction– that’s all<br />
  74. 74. Sound test approaches align with the Axioms or…<br />
  75. 75. …the Axioms are wrong<br />
  76. 76. The Proposed Axioms<br />
  77. 77. Axiom Name<br />The Axiom<br />Axioms are one-liners that succinctly state the principle.<br />
  78. 78. Stakeholder Axiom<br />Testing needs stakeholders<br />
  79. 79. Test Basis Axiom<br />You need a source of knowledge to select things to test<br />
  80. 80. Test Oracle Axiom<br />Test needs a source of knowledge to evaluate actual behaviour<br />
  81. 81. Fallibility Axiom<br />Your sources of knowledge are fallible and incomplete<br />
  82. 82. Scope Management Axiom<br />If you don’t manage scope, you may never meet stakeholder expectations<br />
  83. 83. Design Axiom<br />Test design is based on models<br />
  84. 84. Coverage Axiom<br />Testing requires a coverage model or models<br />
  85. 85. Delivery Axiom<br />The usefulness of the intelligence produced by test determines the value of testing<br />
  86. 86. Environment Axiom<br />Test execution requires a known, controlled environment<br />
  87. 87. Event Axiom<br />Testing never goes as planned<br />
  88. 88. Prioritisation Axiom<br />The most important tests are those that uncover the best intelligence, fast<br />
  89. 89. Execution Sequencing Axiom<br />Run your most important tests first – you may not have time to run them later<br />
  90. 90. Repeat-Test Axiom<br />Repeated tests are inevitable<br />
  91. 91. Good-Enough Axiom<br />Acceptance is always a compromise<br />
  92. 92. Never-Finished Axiom<br />Testing never finishes; it stops<br />
  93. 93. Value Axiom<br />The value of intelligence is independent of who produces it<br />
  94. 94. The Seventeenth Axiom<br />Suggestions?<br />
  95. 95. Applications of Test Axioms<br />
  96. 96. Think of the Axioms as thinking tools<br />
  97. 97. Test Strategy<br />
  98. 98. Contexts of Test Strategy<br />Axioms<br />Communication<br />Early Testing<br />Risks<br />De-Duplication<br />Test<br />Strategy<br />Opportunities<br />Goals<br />Automation<br />Culture<br />Contract<br />User involvement<br />Constraints<br />Human resource<br />Artefacts<br />Skills<br />Environment<br />Process(lack of?)<br />Timescales<br />
  99. 99. First Equation of Testing<br />Axioms+ Context+ Values+ Thinking<br /> Approach<br />
  100. 100. Values<br />Come from your personal or organisational view of testing.<br />Your ‘School’ of thought perhaps.<br />
  101. 101. Stakeholder<br />Axiom-Pairsgenerate questions<br />Basis<br />Oracle<br />Fallibility<br />Scope<br />Value<br />Coverage<br />Never-Finished<br />Delivery<br />Good-Enough<br />Environment<br />Repeat-Test<br />Event<br />Design<br />Prioritisation<br />Sequencing<br />
  102. 102. Stakeholder<br />What tests do the different stakeholders value most?<br />What information do the different stakeholders want first?<br />Sequencing<br />
  103. 103. Process Assessment and Improvement<br />
  104. 104. Each Axiom triggers questions naturally<br />
  105. 105. These questions frame a more natural assessment<br />Much better than generic ‘maturity models’ that MUST BE context-dependent<br />
  106. 106. E.g. Stakeholder questions(aka stakeholder analysis)<br />Who are they?<br />What intelligence do they want?<br />What do they need it for?<br />When do they want it?<br />In what format?<br />How often?<br />Etc.<br />
  107. 107. A Framework for Tester Competencies<br />
  108. 108. Event<br />Coverage<br />Risk<br />Management<br />Never<br />Finished<br />Stakeholder<br />Axioms drive competencies - example<br /><ul><li>Managing Uncertainty
  109. 109. Facilitation, influencing and reporting
  110. 110. Expectation management
  111. 111. Crisis management</li></li></ul><li>Closing Comments<br />
  112. 112. I believe Test Axioms existWe need to define, refine, and agree them<br />
  113. 113. Axioms could be very useful<br />Will<br /><br />
  114. 114. Axioms don’t constrainIf they exist, they stimulate thinking<br />
  115. 115. They represent the set of core values and beliefs about testing<br />(...Anything?)<br />
  116. 116. Reminder:our definition of ‘test’<br /><ul><li>A procedure for critical evaluation
  117. 117. A means of determining the presence, quality, or truth of something
  118. 118. A trial. </li></li></ul><li>Did you notice anything about the Axioms?<br />The words software, system, technology, methodology, v-model, entry/exit criteria, risk – do not appear<br />
  119. 119. Axioms are generic – they HAVE to be<br />Testers test software, systems, interfaces, documents, processes, buildings, furniture, people etc…<br />(Should they apply to testing anything? Discuss!)<br />
  120. 120. The Axioms need more work<br />Refinement<br />Examples<br />Applications<br />
  121. 121. Read the Pocketbook and give me feedback – please <br />gerrardconsulting.com<br />stores.lulu.com/testers-press <br />
  122. 122. THE<br />AXIOMS<br />TESTING<br />OF<br />Thank-You!<br />

×