The document discusses testing axioms, which are context-neutral rules for testing systems. It proposes that testing axioms can be used to advance testing practices by providing a framework for critical thinking about testing. Specifically, separating axioms, context, and values allows testers to clarify positions and approaches for different contexts. It also suggests testing axioms can help identify important skills for testers, such as understanding test models and their limitations. Finally, it explores ideas from "quantum testing" such as assigning significance to individual tests, rather than attempting to quantify their value.

1. Paul Gerrard
THE
TESTING
OF
Advancing Testing Using Axioms

2. Axioms –a Brief Introduction
Advancing Testing Using Axioms
First Equation of Testing
Test Strategy and Approach
Testing Improvement
A Skills Framework for Testers
Quantum Testing
Close

3. Formulated as a context-neutral set of rules for testing systems
They represent the critical thinking processes required to test any system
There are clear opportunities to advance the practice of testing using them
Testers Pocketbook: testers-pocketbook.com
Test Axioms Website test-axioms.com

4. •Test Axioms are not beginners guides
•They can help you to think critically about testing
•They expose flaws in other people‟s thinking and their arguments about testing
•They generate some useful by-products
•They help you to separate context from values
•Interesting research areas!
•First Equation of Testing, Testing Uncertainty Principle, Quantum Theory, Relativity, Exclusion Principle...
•You can tell I like physics

5. Stakeholder, (Test) Design and (Test) Delivery
You have the little book

6. Stakeholder
Value
Scope Mgt. Good- EnoughStakeholderaxioms

7. Summary:
Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide
Consequence if ignored or violated:
There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.
Questions:
Who are they?
Whose interests do they represent?
What evidence do they want?
What do they need it for?
When do they want it?
In what format?
How often?

8. Test Model
Test Basis
Oracle
Coverage
Prioritis-ationFallibilityDesign axioms

9. Summary:
Choose test models to derive tests that are meaningful to stakeholders. Recognise the models‟ limitations and the assumptions that the models make
Consequence if ignored or violated:
Tests design will be meaningless and not credible to stakeholders.
Questions
Are design models available to use as test models? Are they mandatory?
What test models could be used to derive tests from the Test Basis?
Which test models will be used?
Are test models to be documented or are they purely mental models?
What are the benefits of using these models?
What simplifying assumptions do these models make?
How will these models contribute to the delivery of evidence useful to the acceptance decision makers?
How will these models combine to provide sufficient evidence without excessive duplication?
How will the number of tests derived from models be bounded?

10. Confidence
Sequenc- ing
Event
Never- Finished
Repeat- Test
Environment
Delivery axioms

12. Axioms+ Context+ Values+ Thinking
=Approach

13. Separation of Axioms, context, values and thinking
Tools, methodologies, certification, maturity models promote approaches without reference to your context or values
No thinking is required!
Without a unifying test theory you have no objective way of assessing these products.

14. Given context, practitioners can promote different approaches based on their values
Valuesare preferences or beliefs
Pre-planned v exploratory
Predefined v custom process
Requirements-driven v goal-based
Standard documentation v face-to-face comms.
Some contexts precludecertain practices
“No best practices”

15. Separating axioms, context and values clarifies positions, for example:
„Structured‟ (certified?) test advocates have little (useful) to say about Agile contexts
Exploratory test advocates have little (useful) to say about contract/requirements-based acceptance
The disputes between these positions is more about valuesthan practices in context
Is a consultant recommendation best for the stakeholders or the consultant?

16. Strategy is a thought process not a document

17. Test
Strategy
Risks
GoalsConstraints
Human resource
Environment
TimescalesProcess(lack of?)
ContractCulture
Opportunities
User involvement
Automation
De- Duplication
Early Testing
Skills
Communication
Axioms
Artefacts

18. 1.Test Plan Identifier
2.Introduction
3.Test Items
4.Features to be Tested
5.Features not to be Tested
6.Approach
7.Item Pass/Fail Criteria
8.Suspension Criteria and Resumption Requirements
9.Test Deliverables
10.Testing Tasks
11.Environmental Needs
12.Responsibilities
13.Staffing and Training Needs
14.Schedule
15.Risks and Contingencies
16.Approvals
Based on IEEE Standard 829-1998

19. Used as a strategy checklist
Scarily vague (don‟t go there)
Used as a documentation template/standard
Flexible, not prescriptive, but encourages copy and edit mentality (documents that no one reads)
But many manytesters seek guidance on
What to consider in a test strategy
Communicating their strategy to stakeholders and project participants

20. Items 1, 2 –Administration
Items 3+4+5 –Scope Management, Prioritisation
Item 6 –All the Axioms are relevant
Items 7+8 –Good-Enough, Value
Item 9 –Stakeholder, Value, Confidence
Item 10 –All the Axioms are Relevant
Item 11 –Environment
Item 12 –Stakeholder
Item 13 –All the Axioms are Relevant
Item 14 –All the Axioms are Relevant
Item 15 –Fallibility, Event
Item 16 –Stakeholder Axioms

21. 1.Stakeholder Objectives
Stakeholder management
Goal and risk management
Decisions to be made and how (acceptance)
How testing will provide confidence and be assessed
How scope will be determined
2.Design approach
Sources of knowledge (bases and oracles)
Sources of uncertainty
Models to be used for design and coverage
Prioritisation approach
3.Delivery approach
Test sequencing policy
Repeat test policies
Environment requirements
Information delivery approach
Incident management approach
Execution and end-game approach
4.Plan (high or low-level)
Scope
Tasks
Responsibilities
Schedule
Approvals
Risks and contingencies

22. Test process improvement is a waste of time

25. Actually
its 11
(most were not software related)

26. Google search
“CMM” –22,300,000
“CMM Training” –48,200
“CMM improves quality” –74 (BUT really 11 –most of these have NOTHING to do with software)
A Gerrard Consulting client…
CMM level 3 and proud of it (chaotic, hero culture)
Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping)
40+ recommendations, only 7 adopted –they couldn‟t change
How on earth did they get through the CMM 3 audit?

27. Using process change to fix cultural or organisational problems is never going to work
Improving test in isolation is never going to work either
Need to look at changing context rather than values…

28. Context+ Values+ Thinking
=Approach
<-your values
<-your context
<-your thinking
<-your approach

29. Context+ Values+ Thinking
=Approach
<-someone else's
<-someone else's
<-someone else's
<-someone else's

30. Axioms+ Context+ Values+ Thinking
=Approach
<-recognise
<-hard to change
<-could change?
<-just do some
<-your approach

31. Axioms represent the critical things to think about
Associated questions act as checklists to:
Assess your current approach
Identify gaps, inconsistencies in current approach
QA your new approach in the future
Axioms represent the WHAT
Your approach specifies HOW

32. Mission
Coalition
Vision
Communication
Action
Wins
Consolidation
Anchoring
Changes identified here
If you must use one, this is where your ‘process model’ comes into play

33. Axioms indicate WHAT to think about...
...so the Axioms point to SKILLS

34. Summary:
Choose test models to derive tests that are meaningful to stakeholders. Recognise the models‟ limitations and the assumptions that the models make.
Consequence if ignored or violated:
Tests design will be meaningless and not credible to stakeholders.
Questions:
Are design models available to use as test models? Are they mandatory?
What test models couldbe used to derive tests from the Test Basis?
Which test models willbe used?
Are test models to be documented or are they purely mental models?
What are the benefits of using these models?
What simplifying assumptions do these models make?
How will these models contribute to the delivery of evidence useful to the acceptance decision makers?
How will these models combine to provide sufficient evidence without excessive duplication?
How will the number of tests derived from models be bounded?

35. A tester needs to understand:
Test models and how to use them
How to select test models from fallible sources of knowledge
How to design test models from fallible sources of knowledge
Significance, authority and precedence of test models
How to use models to communicate
The limitations of test models
Familiarity with common models
Is this all that current certification provides?

36. Intellectual skills and capabilities are more important than the clerical skills
Need to re-focus on:
Testing thought processes (Axioms)
Testing Stakeholder relationship management
Testing as an information provision service
Goal and risk-based testing
Real-world examples, not theory
Practical, hands-on, real-world training, exercises and coaching.

37. If evidence arrives in discrete quanta...
...can we assign a value to it?

38. As tests are run, every individual test has some significance
Some tests expose failures but ultimately we want all tests to PASS
When all tests pass –the stakeholders are happy, aren‟t they?
Can we measure confidence by counting test?
Not really...

39. Coverage model:
A test could cover one or hundreds of functional conditions, ten thousand program statements or ten
Objective:
Criticality of the business goal it examples
Criticality of the risk it informs
Precedent:
The first end-to-end test pass is significant
The 100thvariation of a similar test is less significant
Functional dependence:
A test of shared functionality used thousands of times per hour could be much more important than a peripheral feature used once/day
Stakeholder:
Are customers tests more or less significant than supplier tests?
Context:
The same test run at different times in different environments can have different value.

40. Only a stakeholder can assign a value to a test (but that is very hard thing to do)
A tester cannot quantify value, but can define its significance
A test is significant (to stakeholders) if it:
Can be related to a meaningful test objective
Increases coverage with respect to a meaningful test model
Is considered in an acceptance decision (at any level)
Significance is a Boolean but could be 0 or 1
The number of insignificant tests should be zero.

41. Significance can only be assessed by testers if:
Our test objectives, models, coverage goals are meaningful (to stakeholders) or
Testers are authorised to create their own objectives, measures and coverage goals or
Testers are their own stakeholder
Testers need a close, trusting relationship with their stakeholdersor authorised autonomy
E.g. exploratory testing won‟t work if stakeholders do not allow autonomy
Testers should not „go it alone‟.

42. Test coverage models and goals that generate uniform distributions of tests are inefficient and uninformative
We need more and better test models
Models that are meaningful in context
Significance varies with context and can be used to explain why
e.g. some tests aren‟t useful as regression tests
How much testing is enough?
Can never be answered by coverage alone.

43. Axioms are context-neutral rules for testing
The Equation of Testing
Separates axioms, context, values and thinking
We can have sensible conversations about process
Axioms and associated questions provide context neutral checklists for test strategy, assessment/improvement and skills
Quantum Testing separates significance from value; can it answer the question, “how much testing is enough?”

44. Thank-You!
THE
TESTING
OF
testaxioms.com
testers-pocketbook.com
gerrardconsulting.com
uktmf.com