Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Are you in control of Testing, or does Testing control you?

10,408 views

Published on

Доклад Mike Smith на конференции SQA Days-19, 20-21 мая 2016 г., Санкт-Петербург

Published in: Education
  • Be the first to comment

Are you in control of Testing, or does Testing control you?

  1. 1. Are you in control of Testing, or does Testing control you? Mike Smith Independent Consultant
  2. 2. 40th Year in IT!
  3. 3. I’ve lived through a few technology eras!
  4. 4. Getting the balance right can be difficult!
  5. 5. Trying to stay relevant • Tool Author – Test Process Management (T-Plan) • Businessman/Entrepreneur/Tester/Innovator • UK Representative to ISTQB • Chairman ISTQB Advanced Level Syllabus Working Group • Founder member of CAT (Certified Agile Tester) SIG • Practical Assessments (R&D Project) • Web TV & Digital Marketing in sport • Professional sports governance (in my spare time!) • Writing a book!
  6. 6. Presentation Background • Lack of control over projects/initiatives in IT, Business & Sport • Common faults over this broad perspective • This presentation focuses on software testing as a form of measurement • But it uses analogies and common characteristics across other perspectives • I will use some examples and my experiences to help highlight key points
  7. 7. Presentation Background Concerns over effectiveness of software testing as a measurement process The subject of my book – Measurement & Testing!
  8. 8. My theme for a book “Effective Measurement and Testing Delivers Control of Anything”
  9. 9. Measurement & Testing My Axioms • Testing is more than just Testing! • Software Testing can learn from other forms of Measurement & Testing • Other forms of Measurement & Testing can learn from Software Testing • Agile Development is helping the case for Measurement & Testing • Software Testing took a wrong turn when it became obsessed with Test Cases!
  10. 10. Another Testing Axiom •The answer to the ultimate question of Life, The Universe and Everything? Answer: “42” (Hitchhikers Guide to the Galaxy – Douglas Adams) •The answer to any question about Software Testing? Answer: “It Depends!” (Dale Perry – SQE Inc. USA)
  11. 11. 3 Key Words Control Measurement Testing (& Software Testing)
  12. 12. Control – Some definitions •The power to influence or direct people's behaviour or the course of events. (lead) •Determine the behaviour or supervise the running of. (lag & lead)
  13. 13. Control vs Management What’s the difference? Corporate Governance Definition 1.Strategic Direction of the Business 1.Effective Monitoring and Control of the Management of the Business 2.Accountable to the Shareholders of the Business
  14. 14. Control vs Management?
  15. 15. Being in Control – Can you answer these questions? •What are my measures of success? •How will I know when I’ve met them? •How sure will I be that I’m going to meet them at any point in time? •How do I know where I am against them at any point in time? •Can I manage change effectively? •For an IT project, how useful are Test Cases to help me keep in control?
  16. 16. Successful Outcomes? • Reach destination port • Get there on schedule • All passengers & crew are safe • Passengers have not been (too) seasick! • Within budget (economical with fuel) • Obstacles avoided!
  17. 17. How will I know when I’ve met the successful outcomes? • What are the key measures & targets? • Speed? • Direction? • Position? • Quality Attributes?
  18. 18. Keeping the ship under control! Key Drivers Initiatives taken based these key goal & performance drivers
  19. 19. Other measures to support the key drivers for Control
  20. 20. There may be many management systems to support these measures A complex measurement & management framework is needed
  21. 21. Simple Hierarchical Model does not support the complex relationships
  22. 22. What is Business Performance Measurement & Management? • ‘Translating Strategy into Action’ – Kaplan & Norton • Use of Business Balanced Scorecard – Different perspectives • Drives behaviour • Measures outcomes • Links actions to strategy • Leads to predictable outcomes
  23. 23. Dangers of Performance Measurement & Management? Measures that are badly constructed can destroy an organisation’s performance… effort may be misdirected into unproductive actions and a great deal of time and effort can be wasted. ‘Balanced Scorecard in a week – Mike Bourne, Pippa Bourne’
  24. 24. 5 Key principles 1. Generic application 2. Objectives, Measures & Targets, Initiatives 3. Separate “What” & “How” 4. Cascading Scorecards – One person’s ‘How’ is another person’s ‘What’ – Measures & Targets become objectives for next person 1. Lead & Lag Indicators – Goal Indicators (reactive) – Performance Indicators (predicative)
  25. 25. Scorecard Objectives, Measures, Targets & Initiatives InitiativesObjectives TargetsMeasures WHY WHAT HOW
  26. 26. Scorecards linked by cascading ‘whats’ & ‘hows’ HOWWHAT HOWWHAT HOWWHAT • Could be 1:many or many:many • Simplest way is 1:1 HOW HOWWHAT HOWWHAT HOWWHAT WHAT
  27. 27. Summary of Outcome Driven Measurement Framework/Model • Separate “What” & “How” • Complex Relationships in Model • Information Traceability matrix Top to Bottom • Control of Complex Organisations & Systems • Good fit with Agile Development Practices!
  28. 28. Now consider Software Testing Model • What are the Measures & Targets that support the Software Testing Process? • Test Cases? • Test Objectives? • Test Requirements? • Test Conditions? • Testers focus on Test Cases, but are they the best Measures/Targets to drive behaviour? • Are they the best to manage risk and change?
  29. 29. The problem with Test Cases
  30. 30. Test(ing) & Software Testing •The means by which the presence, quality, or genuineness of anything is determined; a means of trial (lag) •Software testing is a process of executing a program or application with the intent of finding the software bugs. (lag) It can also be stated as the process of validating and verifying that a software program or application or product meets the business and technical requirements that guided it's design and development. (lag & lead) •Testing is a risk reduction process (lead)
  31. 31. The problem with Test Cases Test case A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]
  32. 32. The problem with Test Cases • How big? • When can they be defined? • They may help me Test, but do they help me Measure? • Whose Measures are they? • Do they separate the What from the How? • History of Test Cases (‘Bottom-up’) • What else fails if my Test Case fails - traceability? • Tool problems – wrong model • What if none defined?
  33. 33. The problem with Test Cases
  34. 34. 1479 test cases: but are they “useful”? Now, let’s start with a classification tree 1479 test cases, so it must be good, right? Test specification process Documentation to agree coverage
  35. 35. A Simple Hierarchical World
  36. 36. A Complex Relational World!
  37. 37. A simple world became a complex world!
  38. 38. Test Model described by ‘methods’ people at large pan- European financial institution. Test Requirements Requirements WHY Test Design Specifications Test Case Specifications WHAT WHAT HOW • “From Requirements to Test Case Specification” in test management tool
  39. 39. A complex model to support software testing May or may not Need complex model: IT DEPENDS!
  40. 40. Guidelines for a better model • Need to separate What & How • Model needs Complex Relationships – many:many • Test Conditions can be Considered targets for success, organised by Measurement Attributes – Functional, Behavioural • Can and should be defined early • Ability to cope with change • Know where you are at any point in time • Delivers Control of Complex Systems • Good fit with Agile Development Practices! • Business Outcome Driven Development
  41. 41. What factors affect the level of measures and targets in a measurement framework? • Risk • Size & Complexity • Dev Method • Test Level • Manageability & Maintainability • Tool usage • Ability to cope with change • Quality and level of artefacts • Regulatory requirements…....
  42. 42. What factors affect the level of measures and targets in a measurement framework? IT DEPENDS!! Setting good measures and targets is difficult - it requires strong analytical skills People often neglect the –ve measures and focus on the +ve
  43. 43. What can the world of Measurement learn from Software Testing? • Software Testing has been developed as a discipline for many years • Very detailed processes and techniques compared with other forms of Measurement & Testing • Software Test Techniques – Support Analysis and Design Activities • Airport Example
  44. 44. Other Experiences • Making a new Financial Exchange Live • An Office Move • ISTQB Syllabus Development • Practical Assessments • Governance in Sport • Mindset change and silos!
  45. 45. WHAT’S MISSING? General Model Control Management & Information Systems Measurement Framework Measures & Targets Objectives Initiatives
  46. 46. WHAT’S MISSING? Testing Model Control Tests and Test Cases Test Requirements/ Test Conditions Measures & Targets Separated from Tests Many:Many Reqs:Tests Objectives/ Requirements Initiatives
  47. 47. WHAT’S MISSING? New Financial Exchange New Market Tests and Test Cases Regulator Approval SPECIFICATION Measurement Framework Retrospective Test Conditions in Complex Model
  48. 48. WHAT’S MISSING? ISTQB Syllabi (Presented in Prague 2011) Certificates (FL,AL,EL) Learning Objectives & Syllabus VISION / STRATEGY SPECIFICATION Business Outcomes REQUIREMENTS
  49. 49. WHAT’S MISSING? Sports Club Constitution Review Need to Change Old Subject Matter Experience VISION / STRATEGY CONSTITUTION/ RULES Terms of Reference REQUIREMENTS
  50. 50. My ‘Axioms’ “Test Driven Everything” “Business Outcome Driven Development” “Effective Measurement and Testing Delivers Control of Anything” “Top Down, meets Bottom Up”
  51. 51. References • Testing Entities Paper – John Kent • IEEE Paper – ‘Keystone Testing Entity’ Mike Smith & Neil Thompson • Value Flow Scorecards – Mike Smith & Neil Thompson • Balanced Scorecard in a week – Mike & Pippa Bourne • My book on Measurement & Testing – when?!
  52. 52. What can you take away from this? • If you are building a pure hierarchical model to manage testing, think again • Can I optimise my model to give better control without it becoming unmanageable? • Am I relying too much on my testing tools – do I need something else? • Can I use my analytical skills to engage with the business better? • I encourage you to think about these concepts, even if you think you are in control…...
  53. 53. If you think you’ve found a good move, look for a better one!
  54. 54. My Measures of Success for this talk?! 1.Standing Ovation? 2.Loud Applause? 3.Polite Applause? 4.Silence? 5.I won’t be let into Russia again?!!

×