The document discusses how test axioms can be used to advance testing practices. It introduces 16 proposed test axioms grouped into stakeholder, design, and delivery axioms. The axioms represent critical thinking processes for testing any system. The document discusses how the axioms can help testers design test strategies, assess improvement opportunities, and define needed skills. It also proposes a "first equation of testing" that separates axioms, context, values, and thinking to allow for different valid approaches. Additionally, the concept of "quantum testing" is introduced to discuss assigning significance to tests rather than defining their value, which can only be determined by stakeholders.

1. THEAXIOMSTESTINGOFAdvancing Testing Using AxiomsPaul Gerrard

2. Paul GerrardPaul is a consultant, teacher, author, webmaster, programmer, tester, conference speaker, rowing coach and a publisher. He has conducted consulting assignments in all aspects of software testing and quality assurance, specialising in test assurance. He has presented keynote talks and tutorials at testing conferences across Europe, the USA, Australia, South Africa and occasionally won awards for them.Educated at the universities of Oxford and Imperial College London, Paul was the founding chair of the BCS ISEB Testing Certificate Scheme and a member of the Working Party that produced BS 7925 – the Component Test Standard. Currently, he is Principal of Gerrard Consulting Limited and is the host of the UK Test Management Forum.

3. AgendaAxioms – a Brief IntroductionAdvancing Testing Using AxiomsFirst Equation of TestingTest Strategy and ApproachTesting ImprovementA Skills Framework for TestersQuantum TestingClose

4. Test AxiomsFormulated as a context-neutral set of rules for testing systemsThey represent the critical thinking processes required to test any systemThere are clear opportunities to advance the practice of testing using themTesters Pocketbook: testers-pocketbook.comTest Axioms Website test-axioms.com

5. How can we use Test Axioms?Test Axioms are not beginners guides

6. They can help you to think critically about testing

7. They expose flaws in other people’s thinking and their arguments about testing

8. They generate some useful by-products

9. They help you to separate context from values

10. Interesting research areas!

11. First Equation of Testing, Testing Uncertainty Principle, Quantum Theory, Relativity, Exclusion Principle...

12. You can tell I like physics16 Proposed Axioms(in three groups)Stakeholder, (Test) Design and (Test) DeliveryYou have the little book

13. Stakeholderaxioms

14. Testing needs stakeholders (p64)Summary:Identify and engage the people or organisations that will use and benefit from the test evidence we are to provideConsequence if ignored or violated:There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.Questions:Who are they?Whose interests do they represent?What evidence do they want?What do they need it for?When do they want it?In what format?How often?

15. Design axiomsFallibility

16. Test design is based on models (p68)Summary:Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models makeConsequence if ignored or violated:Tests design will be meaningless and not credible to stakeholders.QuestionsAre design models available to use as test models? Are they mandatory?What test models could be used to derive tests from the Test Basis?Which test models will be used?Are test models to be documented or are they purely mental models?What are the benefits of using these models?What simplifying assumptions do these models make?How will these models contribute to the delivery of evidence useful to the acceptance decision makers?How will these models combine to provide sufficient evidence without excessive duplication?How will the number of tests derived from models be bounded?

17. Delivery axiomsEnvironment

18. Advancing Testing Using Axioms

19. First Equation of TestingAxioms+ Context+ Values+ Thinking =Approach

20. Why is the equation useful?Separation of Axioms, context, values and thinkingTools, methodologies, certification, maturity models promote approaches without reference to your context or valuesNo thinking is required!Without a unifying test theory you have no objective way of assessing these products.

21. One context, multiple approachesGiven context, practitioners can promote different approaches based on their valuesValuesare preferences or beliefsPre-planned v exploratoryPredefined v custom processRequirements-driven v goal-basedStandard documentation v face-to-face comms.Some contexts preclude certain practices“No best practices”

22. Axioms allow (ensure) different approaches and expose positionsSeparating axioms, context and values clarifies positions, for example:‘Structured’ (certified?) test advocates have little (useful) to say about Agile contextsExploratory test advocates have little (useful) to say about contract/requirements-based acceptanceThe disputes between these positions is more about valuesthan practices in contextIs a consultant recommendation best for the stakeholders or the consultant?

23. Test Strategy and ApproachStrategy is a thought process not a document

24. Contexts of Test StrategyAxiomsCommunicationEarly TestingRisksDe-DuplicationTestStrategyOpportunitiesGoalsAutomationCultureContractUser involvementConstraintsHuman resourceArtefactsSkillsEnvironmentProcess(lack of?)Timescales

25. IEEE 829 Test Plan OutlineTest Plan IdentifierIntroductionTest ItemsFeatures to be TestedFeatures not to be TestedApproachItem Pass/Fail CriteriaSuspension Criteria and Resumption RequirementsTest DeliverablesTesting TasksEnvironmental NeedsResponsibilitiesStaffing and Training NeedsScheduleRisks and ContingenciesApprovalsBased on IEEE Standard 829-1998

26. I’m no fan of IEEE 829Used as a strategy checklistScarily vague (don’t go there)Used as a documentation template/standardFlexible, not prescriptive, but encourages copy and edit mentality (documents that no one reads)But many many testers seek guidance onWhat to consider in a test strategyCommunicating their strategy to stakeholders and project participants

27. IEEE 829 Plan and AxiomsItems 1, 2 – AdministrationItems 3+4+5 – Scope Management, PrioritisationItem 6 – All the Axioms are relevantItems 7+8 – Good-Enough, ValueItem 9 – Stakeholder, Value, ConfidenceItem 10 – All the Axioms are RelevantItem 11 – EnvironmentItem 12 – StakeholderItem 13 – All the Axioms are RelevantItem 14 – All the Axioms are RelevantItem 15 – Fallibility, EventItem 16 – Stakeholder Axioms

28. A Better Test Strategy and PlanStakeholder ObjectivesStakeholder managementGoal and risk managementDecisions to be made and how (acceptance)How testing will provide confidence and be assessedHow scope will be determinedDesign approachSources of knowledge (bases and oracles)Sources of uncertaintyModels to be used for design and coveragePrioritisation approachDelivery approachTest sequencing policyRepeat test policiesEnvironment requirementsInformation delivery approachIncident management approachExecution and end-game approachPlan (high or low-level)ScopeTasksResponsibilitiesScheduleApprovalsRisks and contingencies

29. Testing ImprovementTest process improvement is a waste of time

32. Actuallyits 11(most were not software related)

33. The delusion of process models(e.g. CMM)Google search“CMM” – 22,300,000“CMM Training” – 48,200“CMM improves quality” – 74 (BUT really 11 – most of these have NOTHING to do with software)A Gerrard Consulting client…CMM level 3 and proud of it (chaotic, hero culture)Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping)40+ recommendations, only 7 adopted – they couldn’t changeHow on earth did they get through the CMM 3 audit?

34. “Test Process Improvement is a Waste of Time”Using process change to fix cultural or organisational problems is never going to workImproving test in isolation is never going to work eitherNeed to look at changing context rather than values…

35. Why you are where you areContext+ Values+ Thinking=Approach<- your context<- your values<- your thinking<- your approach

36. Where maturity models come fromContext+ Values+ Thinking=Approach<- someone else's<- someone else's<- someone else's<- someone else's

37. Making change happenAxioms+ Context+ Values+ Thinking=Approach<- recognise<- could change?<- hard to change<- just do some<- your approach

38. Using the axioms and questionsAxioms represent the critical things to think aboutAssociated questions act as checklists to:Assess your current approachIdentify gaps, inconsistencies in current approachQA your new approach in the futureAxioms represent the WHATYour approach specifies HOW

39. Eight stage change process (after Kotter)MissionCoalitionVisionCommunicationActionWinsConsolidationAnchoringChanges identified hereIf you must use one, this is where your ‘process model’ comes into play

40. A Skills framework for testersAxioms indicate WHAT to think about......so the Axioms point to SKILLS

41. Test design is based on models (p68)Summary:Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make.Consequence if ignored or violated:Tests design will be meaningless and not credible to stakeholders.Questions:Are design models available to use as test models? Are they mandatory?What test models could be used to derive tests from the Test Basis?Which test models will be used?Are test models to be documented or are they purely mental models?What are the benefits of using these models?What simplifying assumptions do these models make?How will these models contribute to the delivery of evidence useful to the acceptance decision makers?How will these models combine to provide sufficient evidence without excessive duplication?How will the number of tests derived from models be bounded?

42. Test design and modelling skillsA tester needs to understand:Test models and how to use themHow to select test models from fallible sources of knowledgeHow to design test models from fallible sources of knowledgeSignificance, authority and precedence of test modelsHow to use models to communicateThe limitations of test modelsFamiliarity with common modelsIs this all that current certification provides?

43. Training and certification must changeIntellectual skills and capabilities are more important than the clerical skillsNeed to re-focus on:Testing thought processes (Axioms)Testing Stakeholder relationship managementTesting as an information provision serviceGoal and risk-based testingReal-world examples, not theoryPractical, hands-on, real-world training, exercises and coaching.

44. Quantum TestingIf evidence arrives in discrete quanta......can we assign a value to it?

45. How testing builds confidenceAs tests are run, every individual test has some significanceSome tests expose failures but ultimately we want all tests to PASSWhen all tests pass – the stakeholders are happy, aren’t they?Can we measure confidence by counting tests?Not really...

46. The value of a test varies by…Coverage model:A test could cover one or hundreds of functional conditions, ten thousand program statements or tenObjective:Criticality of the business goal it examplesCriticality of the risk it informsPrecedent:The first end-to-end test pass is significantThe 100th variation of a similar test is less significantFunctional dependence:A test of shared functionality used thousands of times per hour could be much more important than a peripheral feature used once/dayStakeholder:Are customers tests more or less significant than supplier tests?Context:The same test run at different times in different environments can have different value.