Advancing Testing Using Axioms

  • 937 views
Uploaded on

This is the talk Paul Gerrard gave at Eurostar 2010.

This is the talk Paul Gerrard gave at Eurostar 2010.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
937
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
17
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. THE
    AXIOMS
    TESTING
    OF
    Advancing Testing Using Axioms
    Paul Gerrard
  • 2. Paul Gerrard
    Paul is a consultant, teacher, author, webmaster, programmer, tester, conference speaker, rowing coach and a publisher. He has conducted consulting assignments in all aspects of software testing and quality assurance, specialising in test assurance. He has presented keynote talks and tutorials at testing conferences across Europe, the USA, Australia, South Africa and occasionally won awards for them.
    Educated at the universities of Oxford and Imperial College London, Paul was the founding chair of the BCS ISEB Testing Certificate Scheme and a member of the Working Party that produced BS 7925 – the Component Test Standard.
    Currently, he is Principal of Gerrard Consulting Limited and is the host of the UK Test Management Forum.
  • 3. Agenda
    Axioms – a Brief Introduction
    Advancing Testing Using Axioms
    First Equation of Testing
    Test Strategy and Approach
    Testing Improvement
    A Skills Framework for Testers
    Quantum Testing
    Close
  • 4. Test Axioms
    Formulated as a context-neutral set of rules for testing systems
    They represent the critical thinking processes required to test any system
    There are clear opportunities to advance the practice of testing using them
    Testers Pocketbook: testers-pocketbook.com
    Test Axioms Website test-axioms.com
  • 5. How can we use Test Axioms?
    • Test Axioms are not beginners guides
    • 6. They can help you to think critically about testing
    • 7. They expose flaws in other people’s thinking and their arguments about testing
    • 8. They generate some useful by-products
    • 9. They help you to separate context from values
    • 10. Interesting research areas!
    • 11. First Equation of Testing, Testing Uncertainty Principle, Quantum Theory, Relativity, Exclusion Principle...
    • 12. You can tell I like physics
  • 16 Proposed Axioms(in three groups)
    Stakeholder, (Test) Design and (Test) Delivery
    You have the little book
  • 13. Stakeholder
    axioms
  • 14. Testing needs stakeholders (p64)
    Summary:
    Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide
    Consequence if ignored or violated:
    There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.
    Questions:
    Who are they?
    Whose interests do they represent?
    What evidence do they want?
    What do they need it for?
    When do they want it?
    In what format?
    How often?
  • 15. Design axioms
    Fallibility
  • 16. Test design is based on models (p68)
    Summary:
    Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make
    Consequence if ignored or violated:
    Tests design will be meaningless and not credible to stakeholders.
    Questions
    Are design models available to use as test models? Are they mandatory?
    What test models could be used to derive tests from the Test Basis?
    Which test models will be used?
    Are test models to be documented or are they purely mental models?
    What are the benefits of using these models?
    What simplifying assumptions do these models make?
    How will these models contribute to the delivery of evidence useful to the acceptance decision makers?
    How will these models combine to provide sufficient evidence without excessive duplication?
    How will the number of tests derived from models be bounded?
  • 17. Delivery axioms
    Environment
  • 18. Advancing Testing Using Axioms
  • 19. First Equation of Testing
    Axioms+ Context+ Values+ Thinking
    =Approach
  • 20. Why is the equation useful?
    Separation of Axioms, context, values and thinking
    Tools, methodologies, certification, maturity models promote approaches without reference to your context or values
    No thinking is required!
    Without a unifying test theory you have no objective way of assessing these products.
  • 21. One context, multiple approaches
    Given context, practitioners can promote different approaches based on their values
    Valuesare preferences or beliefs
    Pre-planned v exploratory
    Predefined v custom process
    Requirements-driven v goal-based
    Standard documentation v face-to-face comms.
    Some contexts preclude certain practices
    “No best practices”
  • 22. Axioms allow (ensure) different approaches and expose positions
    Separating axioms, context and values clarifies positions, for example:
    ‘Structured’ (certified?) test advocates have little (useful) to say about Agile contexts
    Exploratory test advocates have little (useful) to say about contract/requirements-based acceptance
    The disputes between these positions is more about valuesthan practices in context
    Is a consultant recommendation best for the stakeholders or the consultant?
  • 23. Test Strategy and Approach
    Strategy is a thought process not a document
  • 24. Contexts of Test Strategy
    Axioms
    Communication
    Early Testing
    Risks
    De-Duplication
    Test
    Strategy
    Opportunities
    Goals
    Automation
    Culture
    Contract
    User involvement
    Constraints
    Human resource
    Artefacts
    Skills
    Environment
    Process(lack of?)
    Timescales
  • 25. IEEE 829 Test Plan Outline
    Test Plan Identifier
    Introduction
    Test Items
    Features to be Tested
    Features not to be Tested
    Approach
    Item Pass/Fail Criteria
    Suspension Criteria and Resumption Requirements
    Test Deliverables
    Testing Tasks
    Environmental Needs
    Responsibilities
    Staffing and Training Needs
    Schedule
    Risks and Contingencies
    Approvals
    Based on IEEE Standard 829-1998
  • 26. I’m no fan of IEEE 829
    Used as a strategy checklist
    Scarily vague (don’t go there)
    Used as a documentation template/standard
    Flexible, not prescriptive, but encourages copy and edit mentality (documents that no one reads)
    But many many testers seek guidance on
    What to consider in a test strategy
    Communicating their strategy to stakeholders and project participants
  • 27. IEEE 829 Plan and Axioms
    Items 1, 2 – Administration
    Items 3+4+5 – Scope Management, Prioritisation
    Item 6 – All the Axioms are relevant
    Items 7+8 – Good-Enough, Value
    Item 9 – Stakeholder, Value, Confidence
    Item 10 – All the Axioms are Relevant
    Item 11 – Environment
    Item 12 – Stakeholder
    Item 13 – All the Axioms are Relevant
    Item 14 – All the Axioms are Relevant
    Item 15 – Fallibility, Event
    Item 16 – Stakeholder Axioms
  • 28. A Better Test Strategy and Plan
    Stakeholder Objectives
    Stakeholder management
    Goal and risk management
    Decisions to be made and how (acceptance)
    How testing will provide confidence and be assessed
    How scope will be determined
    Design approach
    Sources of knowledge (bases and oracles)
    Sources of uncertainty
    Models to be used for design and coverage
    Prioritisation approach
    Delivery approach
    Test sequencing policy
    Repeat test policies
    Environment requirements
    Information delivery approach
    Incident management approach
    Execution and end-game approach
    Plan (high or low-level)
    Scope
    Tasks
    Responsibilities
    Schedule
    Approvals
    Risks and contingencies
  • 29. Testing Improvement
    Test process improvement is a waste of time
  • 30.
  • 31.
  • 32. Actually
    its 11
    (most were not software related)
  • 33. The delusion of process models(e.g. CMM)
    Google search
    “CMM” – 22,300,000
    “CMM Training” – 48,200
    “CMM improves quality” – 74 (BUT really 11 – most of these have NOTHING to do with software)
    A Gerrard Consulting client…
    CMM level 3 and proud of it (chaotic, hero culture)
    Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping)
    40+ recommendations, only 7 adopted – they couldn’t change
    How on earth did they get through the CMM 3 audit?
  • 34. “Test Process Improvement is a Waste of Time”
    Using process change to fix cultural or organisational problems is never going to work
    Improving test in isolation is never going to work either
    Need to look at changing context rather than values…
  • 35. Why you are where you are
    Context+ Values+ Thinking
    =Approach
    <- your context
    <- your values
    <- your thinking
    <- your approach
  • 36. Where maturity models come from
    Context+ Values+ Thinking
    =Approach
    <- someone else's
    <- someone else's
    <- someone else's
    <- someone else's
  • 37. Making change happen
    Axioms+ Context+ Values+ Thinking
    =Approach
    <- recognise
    <- could change?
    <- hard to change
    <- just do some
    <- your approach
  • 38. Using the axioms and questions
    Axioms represent the critical things to think about
    Associated questions act as checklists to:
    Assess your current approach
    Identify gaps, inconsistencies in current approach
    QA your new approach in the future
    Axioms represent the WHAT
    Your approach specifies HOW
  • 39. Eight stage change process (after Kotter)
    Mission
    Coalition
    Vision
    Communication
    Action
    Wins
    Consolidation
    Anchoring
    Changes identified here
    If you must use one, this is where your ‘process model’ comes into play
  • 40. A Skills framework for testers
    Axioms indicate WHAT to think about...
    ...so the Axioms point to SKILLS
  • 41. Test design is based on models (p68)
    Summary:
    Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make.
    Consequence if ignored or violated:
    Tests design will be meaningless and not credible to stakeholders.
    Questions:
    Are design models available to use as test models? Are they mandatory?
    What test models could be used to derive tests from the Test Basis?
    Which test models will be used?
    Are test models to be documented or are they purely mental models?
    What are the benefits of using these models?
    What simplifying assumptions do these models make?
    How will these models contribute to the delivery of evidence useful to the acceptance decision makers?
    How will these models combine to provide sufficient evidence without excessive duplication?
    How will the number of tests derived from models be bounded?
  • 42. Test design and modelling skills
    A tester needs to understand:
    Test models and how to use them
    How to select test models from fallible sources of knowledge
    How to design test models from fallible sources of knowledge
    Significance, authority and precedence of test models
    How to use models to communicate
    The limitations of test models
    Familiarity with common models
    Is this all that current certification provides?
  • 43. Training and certification must change
    Intellectual skills and capabilities are more important than the clerical skills
    Need to re-focus on:
    Testing thought processes (Axioms)
    Testing Stakeholder relationship management
    Testing as an information provision service
    Goal and risk-based testing
    Real-world examples, not theory
    Practical, hands-on, real-world training, exercises and coaching.
  • 44. Quantum Testing
    If evidence arrives in discrete quanta...
    ...can we assign a value to it?
  • 45. How testing builds confidence
    As tests are run, every individual test has some significance
    Some tests expose failures but ultimately we want all tests to PASS
    When all tests pass – the stakeholders are happy, aren’t they?
    Can we measure confidence by counting tests?
    Not really...
  • 46. The value of a test varies by…
    Coverage model:
    A test could cover one or hundreds of functional conditions, ten thousand program statements or ten
    Objective:
    Criticality of the business goal it examples
    Criticality of the risk it informs
    Precedent:
    The first end-to-end test pass is significant
    The 100th variation of a similar test is less significant
    Functional dependence:
    A test of shared functionality used thousands of times per hour could be much more important than a peripheral feature used once/day
    Stakeholder:
    Are customers tests more or less significant than supplier tests?
    Context:
    The same test run at different times in different environments can have different value.
  • 47. Quantum Testing
    • Only a stakeholder can assign a value to a test (but that is very hard thing to do) – (but see relativity)
    • 48. Testers can’t quantify value, but could define significance
    • 49. A test is significant (to stakeholders) if it:
    • 50. Can be related to a meaningful test objective
    • 51. Increases coverage with respect to a meaningful test model
    • 52. Is considered in an acceptance decision (at any level)
    • 53. Significance is a Boolean but could be 0 or 1
    • 54. The number of insignificant tests should be zero.
  • Assessing significance
    Significance can only be assessed by testers if:
    Our test objectives, models, coverage goals are meaningful (to stakeholders) or
    Testers are authorised to create their own objectives, measures and coverage goals or
    Testers are their own stakeholder
    Testers need a close, trusting relationship with their stakeholdersor authorised autonomy
    E.g. exploratory testing won’t work if stakeholders do not allow autonomy
    Testers should not ‘go it alone’.
  • 55. The significance of significance
    Test coverage models and goals that generate uniform distributions of tests are inefficient and uninformative
    We need more and better test models
    Models that are meaningful in context
    Significance varies with context and can be used to explain why
    e.g. some tests aren’t useful as regression tests
    How much testing is enough?
    Can never be answered by coverage alone.
  • 56. Using significance (as booleans)
    A test objective could be deemed as met when a set of significant tests are passed
    Test objectives can often be aligned with business goals (or risks) e.g.:
    Faultless end to end or straight-through processing in an ERP system
    Customer order capture, money-in or complete financial reconciliations without failure
    Could the business put a value on these goals being achieved?
    Perhaps the business case provides this level of information?
  • 57. Our current understanding of test value measurement
    In terms of a mathematical treatment of testing we are some way off
    Compare with:
    Understanding of heat and energy in 18thcentury?
    Economics in the early 20thCentury?
    A complete theory of test value may never be achieved
    Quantum testing suggests that:
    Stakeholders judge value (relatively speaking)
    Testers judge significance.
  • 58. Close
    Axioms are context-neutral rules for testing
    The Equation of Testing
    Separates axioms, context, values and thinking
    We can have sensible conversations about process
    Axioms and associated questions provide context neutral checklists for test strategy, assessment/improvement and skills
    QT separates significance from value
    Could it tell us, “how much testing is enough”?
    Right now, we can ‘feel the heat’, but not measure temperature.
  • 59. THE
    AXIOMS
    TESTING
    OF
    testaxioms.com
    testers-pocketbook.com
    gerrardconsulting.com
    uktmf.com
    Thank-You!