Download to read offline
![The problem with Test Cases
Test case
A set of input values, execution preconditions,
expected results and execution postconditions,
developed for a particular objective or test
condition, such as to exercise a particular
program path or to verify compliance with a
specific requirement. [After IEEE 610]](https://image.slidesharecdn.com/qrlyydn0t2vyiumtjxnq-signature-a965b211d936583d824b9a247589dd80e33eecf7d17ea5e216496e7a96c7ad7d-poli-160531103925/85/Are-you-in-control-of-Testing-or-does-Testing-control-you-31-320.jpg)
Uploaded bySQALab
Are you in control of Testing, or does Testing control you?
- Mike Smith argues that software testing models often rely too heavily on test cases, which may not provide the best measures for control and risk management. - An effective measurement framework separates objectives from initiatives and uses a complex model of relationships rather than a simple hierarchy. This provides better traceability and the ability to cope with change. - Lessons can be learned across different domains of measurement and testing. An ideal testing model would incorporate concepts from performance management systems like the balanced scorecard to link testing to business outcomes. - Many factors influence what level of measures and targets are suitable for a given situation, but the most important thing is that the model supports analysis and decision making to maintain control.
More Related Content
![[HCMC STC Jan 2015] Workshop Of Context-Driven Testing In Agile](https://cdn.slidesharecdn.com/ss_thumbnails/nhatdo-vuduongworkshopcontext-driventestinginagile-150208225650-conversion-gate02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Are you in control of Testing, or does Testing control you?
Are you in control of Testing, or does Testing control you?
- 1. Are you incontrol of Testing, or does Testing control you? Mike Smith Independent Consultant
- 2.
- 3. I’ve lived througha few technology eras!
- 4. Getting the balanceright can be difficult!
- 5. Trying to stayrelevant • Tool Author – Test Process Management (T-Plan) • Businessman/Entrepreneur/Tester/Innovator • UK Representative to ISTQB • Chairman ISTQB Advanced Level Syllabus Working Group • Founder member of CAT (Certified Agile Tester) SIG • Practical Assessments (R&D Project) • Web TV & Digital Marketing in sport • Professional sports governance (in my spare time!) • Writing a book!
- 6. Presentation Background • Lackof control over projects/initiatives in IT, Business & Sport • Common faults over this broad perspective • This presentation focuses on software testing as a form of measurement • But it uses analogies and common characteristics across other perspectives • I will use some examples and my experiences to help highlight key points
- 7. Presentation Background Concerns overeffectiveness of software testing as a measurement process The subject of my book – Measurement & Testing!
- 8. My theme fora book “Effective Measurement and Testing Delivers Control of Anything”
- 9. Measurement & Testing MyAxioms • Testing is more than just Testing! • Software Testing can learn from other forms of Measurement & Testing • Other forms of Measurement & Testing can learn from Software Testing • Agile Development is helping the case for Measurement & Testing • Software Testing took a wrong turn when it became obsessed with Test Cases!
- 10. Another Testing Axiom •Theanswer to the ultimate question of Life, The Universe and Everything? Answer: “42” (Hitchhikers Guide to the Galaxy – Douglas Adams) •The answer to any question about Software Testing? Answer: “It Depends!” (Dale Perry – SQE Inc. USA)
- 11. 3 Key Words Control Measurement Testing(& Software Testing)
- 12. Control – Somedefinitions •The power to influence or direct people's behaviour or the course of events. (lead) •Determine the behaviour or supervise the running of. (lag & lead)
- 13. Control vs Management What’sthe difference? Corporate Governance Definition 1.Strategic Direction of the Business 1.Effective Monitoring and Control of the Management of the Business 2.Accountable to the Shareholders of the Business
- 14.
- 15. Being in Control– Can you answer these questions? •What are my measures of success? •How will I know when I’ve met them? •How sure will I be that I’m going to meet them at any point in time? •How do I know where I am against them at any point in time? •Can I manage change effectively? •For an IT project, how useful are Test Cases to help me keep in control?
- 16. Successful Outcomes? • Reachdestination port • Get there on schedule • All passengers & crew are safe • Passengers have not been (too) seasick! • Within budget (economical with fuel) • Obstacles avoided!
- 17. How will Iknow when I’ve met the successful outcomes? • What are the key measures & targets? • Speed? • Direction? • Position? • Quality Attributes?
- 18. Keeping the shipunder control! Key Drivers Initiatives taken based these key goal & performance drivers
- 19. Other measures tosupport the key drivers for Control
- 20. There may bemany management systems to support these measures A complex measurement & management framework is needed
- 21. Simple Hierarchical Modeldoes not support the complex relationships
- 22. What is BusinessPerformance Measurement & Management? • ‘Translating Strategy into Action’ – Kaplan & Norton • Use of Business Balanced Scorecard – Different perspectives • Drives behaviour • Measures outcomes • Links actions to strategy • Leads to predictable outcomes
- 23. Dangers of PerformanceMeasurement & Management? Measures that are badly constructed can destroy an organisation’s performance… effort may be misdirected into unproductive actions and a great deal of time and effort can be wasted. ‘Balanced Scorecard in a week – Mike Bourne, Pippa Bourne’
- 24. 5 Key principles 1.Generic application 2. Objectives, Measures & Targets, Initiatives 3. Separate “What” & “How” 4. Cascading Scorecards – One person’s ‘How’ is another person’s ‘What’ – Measures & Targets become objectives for next person 1. Lead & Lag Indicators – Goal Indicators (reactive) – Performance Indicators (predicative)
- 25. Scorecard Objectives, Measures,Targets & Initiatives InitiativesObjectives TargetsMeasures WHY WHAT HOW
- 26. Scorecards linked by cascading‘whats’ & ‘hows’ HOWWHAT HOWWHAT HOWWHAT • Could be 1:many or many:many • Simplest way is 1:1 HOW HOWWHAT HOWWHAT HOWWHAT WHAT
- 27. Summary of OutcomeDriven Measurement Framework/Model • Separate “What” & “How” • Complex Relationships in Model • Information Traceability matrix Top to Bottom • Control of Complex Organisations & Systems • Good fit with Agile Development Practices!
- 28. Now consider SoftwareTesting Model • What are the Measures & Targets that support the Software Testing Process? • Test Cases? • Test Objectives? • Test Requirements? • Test Conditions? • Testers focus on Test Cases, but are they the best Measures/Targets to drive behaviour? • Are they the best to manage risk and change?
- 29. The problem withTest Cases
- 30. Test(ing) & SoftwareTesting •The means by which the presence, quality, or genuineness of anything is determined; a means of trial (lag) •Software testing is a process of executing a program or application with the intent of finding the software bugs. (lag) It can also be stated as the process of validating and verifying that a software program or application or product meets the business and technical requirements that guided it's design and development. (lag & lead) •Testing is a risk reduction process (lead)
- 31. The problem withTest Cases Test case A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]
- 32. The problem withTest Cases • How big? • When can they be defined? • They may help me Test, but do they help me Measure? • Whose Measures are they? • Do they separate the What from the How? • History of Test Cases (‘Bottom-up’) • What else fails if my Test Case fails - traceability? • Tool problems – wrong model • What if none defined?
- 33. The problem withTest Cases
- 34. 1479 test cases:but are they “useful”? Now, let’s start with a classification tree 1479 test cases, so it must be good, right? Test specification process Documentation to agree coverage
- 35.
- 36. A Complex RelationalWorld!
- 37. A simple worldbecame a complex world!
- 38. Test Model describedby ‘methods’ people at large pan- European financial institution. Test Requirements Requirements WHY Test Design Specifications Test Case Specifications WHAT WHAT HOW • “From Requirements to Test Case Specification” in test management tool
- 39. A complex modelto support software testing May or may not Need complex model: IT DEPENDS!
- 40. Guidelines for abetter model • Need to separate What & How • Model needs Complex Relationships – many:many • Test Conditions can be Considered targets for success, organised by Measurement Attributes – Functional, Behavioural • Can and should be defined early • Ability to cope with change • Know where you are at any point in time • Delivers Control of Complex Systems • Good fit with Agile Development Practices! • Business Outcome Driven Development
- 41. What factors affectthe level of measures and targets in a measurement framework? • Risk • Size & Complexity • Dev Method • Test Level • Manageability & Maintainability • Tool usage • Ability to cope with change • Quality and level of artefacts • Regulatory requirements…....
- 42. What factors affectthe level of measures and targets in a measurement framework? IT DEPENDS!! Setting good measures and targets is difficult - it requires strong analytical skills People often neglect the –ve measures and focus on the +ve
- 43. What can theworld of Measurement learn from Software Testing? • Software Testing has been developed as a discipline for many years • Very detailed processes and techniques compared with other forms of Measurement & Testing • Software Test Techniques – Support Analysis and Design Activities • Airport Example
- 44. Other Experiences • Makinga new Financial Exchange Live • An Office Move • ISTQB Syllabus Development • Practical Assessments • Governance in Sport • Mindset change and silos!
- 45. WHAT’S MISSING? General Model Control Management& Information Systems Measurement Framework Measures & Targets Objectives Initiatives
- 46. WHAT’S MISSING? Testing Model Control Testsand Test Cases Test Requirements/ Test Conditions Measures & Targets Separated from Tests Many:Many Reqs:Tests Objectives/ Requirements Initiatives
- 47. WHAT’S MISSING? New FinancialExchange New Market Tests and Test Cases Regulator Approval SPECIFICATION Measurement Framework Retrospective Test Conditions in Complex Model
- 48. WHAT’S MISSING? ISTQB Syllabi(Presented in Prague 2011) Certificates (FL,AL,EL) Learning Objectives & Syllabus VISION / STRATEGY SPECIFICATION Business Outcomes REQUIREMENTS
- 49. WHAT’S MISSING? Sports ClubConstitution Review Need to Change Old Subject Matter Experience VISION / STRATEGY CONSTITUTION/ RULES Terms of Reference REQUIREMENTS
- 50. My ‘Axioms’ “Test DrivenEverything” “Business Outcome Driven Development” “Effective Measurement and Testing Delivers Control of Anything” “Top Down, meets Bottom Up”
- 51. References • Testing EntitiesPaper – John Kent • IEEE Paper – ‘Keystone Testing Entity’ Mike Smith & Neil Thompson • Value Flow Scorecards – Mike Smith & Neil Thompson • Balanced Scorecard in a week – Mike & Pippa Bourne • My book on Measurement & Testing – when?!
- 52. What can youtake away from this? • If you are building a pure hierarchical model to manage testing, think again • Can I optimise my model to give better control without it becoming unmanageable? • Am I relying too much on my testing tools – do I need something else? • Can I use my analytical skills to engage with the business better? • I encourage you to think about these concepts, even if you think you are in control…...
- 53. If you thinkyou’ve found a good move, look for a better one!
- 54. My Measures ofSuccess for this talk?! 1.Standing Ovation? 2.Loud Applause? 3.Polite Applause? 4.Silence? 5.I won’t be let into Russia again?!!
Editor's Notes
- #2 Is the dog wagging the tail (in control) or is the tail wagging the dog! (not in control)!! Very Successful marketing campaign 15 years ago – had a call from someone who opened it while testing on a Sunday!
- #3 INTRODUCTION SLIDES Important to set a context up front, with more detail to follow. As well as a technology background, from programming to project management, I have been involved in various independent businesses and sport since 1984. Observation and age is allowing me to conduct a retrospective – with the aim of producing a book.
- #4 My kids would have you believe I was around in the Jurassic Period – maybe in terms of Technology I have!
- #5 Trying to stay relevant, but getting the balance right! (And note relevance to getting the balance right later regarding measurement frameworks!) Still trying to come up with new ideas! Play some music! (and sing a little) Presenting to IT and Engineering undergraduates in India in February (Best question asked for a long time about being in IT for 40 years!) Presenting at the ANZTB Conference in Canberra, Australia Involved in cricket – this was few years ago on a school tour to Barbados. I’m an elected board member at Essex County Cricket Club (one of 18 professional teams in England) Trying to stay relevant! ADD AUTO TRANSITION – BETTER GRAPHIC REPRESENTATION
- #6 Semi-retiring? from Executive Business Management Still many roles and interesting projects as an independent Tool Author – Test Process Management (T-Plan) Businessman/Entrepreneur/Tester/Innovator Semi-retiring? from Executive Business Management Still many roles and interesting projects as an independent UK Representative to ISTQB Chairman ISTQB Advanced Level Syllabus Working Group Founder member of CAT (Certified Agile Tester) SIG Practical Assessments (R&D Project) Web TV & Digital Marketing in sport Professional sports administration (in my spare time!) Writing a book!
- #7 Background Warning – Conceptual, Work in Progress! Method Agnostic This is a short high level overview Have a wide perspective from Business to Technical Strong interest in Innovation and R&D (tax credits!) Have spoken and written papers over many years: Software Testing and Test Process Management Governance & Compliance Business Performance Measurement & Management Concerns over effectiveness of software testing as a measurement process/tool Have found comparing principles of Business Performance Measurement and Software Testing very useful to highlight concerns The subject of my book – Measurement & Testing!
- #8 Background Warning – Conceptual, Work in Progress! Method Agnostic This is a short high level overview Have a wide perspective from Business to Technical Strong interest in Innovation and R&D (tax credits!) Have spoken and written papers over many years: Software Testing and Test Process Management Governance & Compliance Business Performance Measurement & Management Concerns over effectiveness of software testing as a measurement process/tool Have found comparing principles of Business Performance Measurement and Software Testing very useful to highlight concerns The subject of my book – Measurement & Testing!
- #9 Important to look at what I mean by these words, since they can have several generic and contextual definitions
- #11 The Hitchhikers Guide to the Galaxy – Douglas Adams. In the radio series and the first novel, a group of hyper-intelligent pan-dimensional beings demand to learn the Answer to the Ultimate Question of Life, The Universe, and Everything from the supercomputer, Deep Thought, specially built for this purpose. It takes Deep Thought 7½ million years to compute and check the answer, which turns out to be 42. Thanks to Dale Perry (SQE – Software Quality Engineering, USA) for this observation many years ago!
- #12 Important to look at what I mean by these words, since they can have several generic and contextual definitions. May seem basic, but I’m going to explore these and help define them in the my context to help the audience understand the points I am trying to make – my axioms. Control The power to influence or direct people's behaviour or the course of events. (lead) Determine the behaviour or supervise the running of. (lag & lead) Measure(ment) Ascertain the size, amount, or degree of (something) by using an instrument or device marked in standard units (lag) Assess the importance, effect, or value of (something) (lag) A plan or course of action taken to achieve a particular purpose. (lead) Testing (and Software Testing) The means by which the presence, quality, or genuineness of anything is determined; a means of trial (lag) Software testing is a process of executing a program or application with the intent of finding the software bugs. (lag) It can also be stated as the process of validating and verifying that a software program or application or product meets the business and technical requirements that guided it's design and development. (lag) Testing is a risk reduction process (lead)
- #13 There are many more, but these two help make the points. Note the terms ‘lag’ and ‘lead’. Do people understand these terms? Further development of these later.
- #14 Many people find difficulty in distinguishing between “Control” & “Management”, but there is an important distinction. People who control things, do not necessarily manage things. This is important because we are considering the definition of “Control” and not “Management” in this presentation, although there is a clear link between them (but not just that the person in control is the “Super Manager”!. It’s useful to consider the meaning of “Governance” when analysing the meaning of “Control” and useful to consider a definition of Corporate Governance.
- #15 Careful with any analogy. Not saying that controlling a ship is the same as controlling an IT project, but can help to visualise something that is often difficult to portray using a complex IT system. Remember the questions to ask What are my measures of success? How will I know when I’ve met them? How sure will I be that I’m going to meet them at any point in time? How do I know where I am against them at any point in time? Can I manage change effectively?
- #16 What does being in control mean? My set of questions that link up the Control, Measure and Test – Jumping ahead a little bit here in mentioning this in relation to Test Cases! Ask some questions about control and the audience experience in Projects and Testing (and role profile of audience).
- #22 Key visual to demonstrate that there are potential complex relationships between the measures and targets in a measurement framework that delivers effective control.
- #23 We are now going to consider some principles of Business Performance Measurement & Management. This has been a key part of my analysis over the years to compare with IT projects and software testing as a measurement process.
- #26 Why? – What does success look like? What are my desirable outcomes? What? – How will I know when I reached my desirable outcomes? (and monitored them and coped with changes along the way) How? – What actions do I need to take in line with the measures and targets? How, driven by the what, becomes the next level Why, which in turns leads to other subordinate whats – a ‘What’/’How’ framework
- #27 Reference Mike Smith and Neil Thompson’s paper for full detail – ‘Keystone Testing Entity’.
- #29 Test Requirements now equivalent to Test Conditions in ISTQB Glossary.
- #31 I prefer to emphasise the risk reduction process - one that will find bugs during that process. Note the historic emphasis on the purpose of software testing being to find bugs. This emphasis was in the first ISEB/ISTQB Foundation Syllabus released in 1996 and highlights the fact that the syllabus was based on a Unit Test Standard, not from a Business/System Testing basis.
- #32 From the ISTQB Glossary. Far too many terms and subjectivity. What if no Test Cases? Use of ‘Test’ – defined as a set of one or more test cases
- #33 Do you use a proprietary tool to manage tests? Does it help you control? Do you find you have to build your own control system because it doesn’t deliver what you need? That is a good sign!! Rather than be a slave to the tool, you are thinking what you really need to get in control.
- #39 This approach led to huge volumes of low and no value tests based on drilling down to a very low level at requirements and assuming all tests must fit into a hierarchy within the requirements hierarchy. THIS PROJECT BECAME OUT OF CONTROL, COSTING TENS OF MILLIONS AND WAS EVENTUALLY SCRAPPED!
- #40 Can be extended and consolidated for a range of other measures for projects outside of software testing – ref Scorecards and Performance Measurement & Management. Need not be IT Depends. Explain factors affecting level and model required. You may be doing this already, or you may say its not applicable to you. IT Depends! I’ve seen a lot of projects out of control and helped them with this approach. The failed project referred to earlier threw out this model!!
- #43 It is a very difficult activity, requiring string analytical skills
- #45 Ref MoneyBall re sports mindsets!
- #47 Case study produced
- #48 Case study produced
- #53 Mindset change – silos – even change to openness in command & control systems in safety critical world
- #54 Because I like using analogies , here is another one that I think is appropriate. Mindset change is often the hardest!
- #57 Let’s now look at Measurement and some simple definitions