Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A New Model for Testing


Published on

Доклад Пола Геррарда на конференции SQA Days-20. 24-26 ноября 2016. Минск

Published in: Education
  • Login to see the comments

  • Be the first to like this

A New Model for Testing

  1. 1. A New Model for Testing @paul_gerrard Paul Gerrard
  2. 2. Helping clients transform their testing through INNOVATION, COACHING and LEADERSHIP Our CLIENTS – Want to be agile rather than follow Agile dogma – Have a pragmatic approach and are focused on delivery – Want a solution that fits, not a badly fitting suit.
  3. 3. Agenda • Innovations that Drive Change • 3 Patterns of Software Development? • A New Model for Testing • Some Consequences • Some Research • A New Test Process? • Close
  4. 4. What's happening? • Stampede to mobile computing - Digital • Big Data • Internet of (Every)Thing, Pervasive Computing • Continuous Delivery, DevOps • (Test) Analytics, Data-Driven Development • Shift-left, embedded testers, no test team • Did you really think Agile was the last word?
  5. 5. Three Innovations That Are Driving Change Being agile Shift Left Analytics (Digital/IoT/IoE and others too, of course)
  6. 6. #1 Agile doesn’t work but being agile might Agile is no longer "innovative" Challenging Agile and being agile might be Is continuous delivery for you? Have you been Agiled?
  7. 7. #2 Shift Left Developers in test? Testers in development? Automation, analytics, collaboration, experimentation
  8. 8. #3 Test Analytics Shift-Right? Analytics from production Analytics from test An opportunity for testing
  9. 9. Three Software Development Patterns? A Model
  10. 10. There are three patterns, not just two • Many ‘new’ approaches are labelled 'Agile' but – They aren’t Agile, they just work – Aimed at factory workers, not knowledge workers • Continuous-delivery approach is appropriate for rapid-delivery into online/mobile systems • CD advocates often have an Agile background, so you might think CD is Agile • Not so. Intelligent Definition and Assurance Slide 10
  11. 11. Three development patterns Intelligent Definition and Assurance Slide 11 Structured Agile Continuous Autonomous
  12. 12. Key pattern characteristics Characteristic Summary Structure Structure of the project team Pace/cadence What drives the rate of decisions making? Leadership How is the team managed/directed? Definition In what format is requirements knowledge captured? Testing How is testing (mostly) performed? Auto. Test When are automated tests created? Who leads? Measurement What/how is project measurement performed? Governance What form does governance take? Intelligent Definition and Assurance Slide 12 All models are WRONG, some are USEFUL Needless to say, I’ll be generalising quite a lot 
  13. 13. Profiles of the three patterns Characteristic Structured Agile Continuous Structure Managed team Autonomous Production Cell Pace/cadence Business decision Team decision Feedback Leadership Project Managed Guided Research Line Managed Definition Fixed spec Dynamic spec Live Specs Testing Scripted Exploratory Automated Auto. Test Retrospective Developer led Pervasive Measurement Pervasive Avoided Analytics Governance Bureaucratic Trust-based Insight-Driven Intelligent Definition and Assurance Slide 13
  14. 14. There aren't three patterns There are many
  15. 15. From high process to DIY process • In the past, high process was seen to be the technical solution to the software problem • But technical solutions cannot solve human problems • True agility means a team can adapt or select its process on-the-fly • We need to strip down our thinking to the essential aspects…
  16. 16. The old ways won't work in the future We need a New Model of Testing (free from logistics)
  17. 17. Forget Logistics (for the time being) Document or not? Automated or manual? Agile v waterfall? This business or that business? This technology v that technology?
  18. 18. ALL Testing is Exploratory We explore sources of knowledge ... ... to build test models ... ... that inform our testing.
  19. 19. Examples of models
  20. 20. Models are innate, essential, human
  21. 21. Judgement, exploring and testing Testing (the system) Our model(s) are adequate Our model(s) are not adequate Exploring (sources) Judgement Creates test models Uses test models We explore sources of knowledge to build test models that inform our testing BTW – Do Developers explore the same way? I think so.
  22. 22. Exploration process ExplorationDefinitions specs/stories People (& you) Sources Require- ments Test Models Enquiring Challenging Sources: People, documents, experience, system under test Modelling Test Models: Can be documented or mental models Predicting System under test
  23. 23. Testing process Testing System Under Test Refining Informing Applying Interpreting Test Models Revise the System More exploring Reporting
  24. 24. New ModelTesting My BBC talk: 29 page paper:
  25. 25. Some Consequences There are others
  26. 26. Relation to TDD and BDD? Is TDD really testing BDD is modelling using stories
  27. 27. Test automation from a different perspective Automation efforts fail too often Automation uses different test models
  28. 28. Capabilities Enquiring, Modelling, Predicting, Challenging Informing,Applying, Interpreting, Refining Reporting and Logging
  29. 29. • Analysis, enquiry and elicitation • Modelling • Creation of custom models, using heuristics, guesses, brainstorming, ideation, creative thinking • Custom test design techniques • Comparison of models, value, advantages, disadvantages, compromises • Identification, validation and use of oracles • Predicate logic and proof • Hypothesis and inference • Socratic method • Rapid Review and Inspection techniques • Test case design • Test models and the meaning of coverage • Testing as controlled experiment • Observation, Note taking, recording A very different skillset • Basic data analysis and statistics • Decision-making with incomplete data • Computer forensics • Fault tree analysis • Failure diagnosis • Bug advocacy, triage processes and negotiation • Meaningful software and test metrics • Visual presentation of data • Reporting and presentation skills • Understanding stakeholders • Test analytics • Risk management, risk-based testing and decision-making • Critical Thinking • Interpersonal skills • Dealing with uncertainty/fallibility
  30. 30. Testing Career Development (speculative) Foundations Technical Management Strategic Test Strategy Project Intelligence Test Assurance Exploration Forensics Interpretation Scripting/ Programming Test Automation Technical (Excel, SQL, OS utils etc) Stakeholder management Analytics & visualisation Managing uncertainty Critical Thinking ISTQB, TMMi etc... Supplier Management Test Process Management Methodology
  31. 31. Will Robots Replace Testers? Some research There is a paper at:
  32. 32. A recent study*… • Over the next two decades, 47% of jobs in the US may be under threat • It ranks 702 occupations in order of their probability of computerisation – Telemarketers: 99% likely – Recreational therapists: 0.28% likely – Computer programmers: 48% likely • Something significant is going on out there • If programmers have a 50/50 chance of being replaced by robots, we should think seriously on how the same might happen to testers. * “The future of employment: how susceptible are jobs to computerisation?” Intelligent Definition and Assurance Slide 32
  33. 33. Some systems-related occupations Intelligent Definition and Assurance Slide 33 Occupation Rank (out of 702) Probability of Computeris- ation Computer and Information Research Scientists 69 1.5% Network and Computer Systems Administrators 109 3.0% Computer and Information Systems Managers 118 3.5% Information Security Analysts, Web Developers, and Computer Network Architects 208 21% Computer Occupations, All Other 212 22% Computer Programmers 293 48% Computer Support Specialists 359 65% Computer Operators 428 78% Inspectors, Testers, Sorters, Samplers and Weighers 670 98%
  34. 34. Test Automation = Mechanical Tools What we REALLY need are THINKING TOOLS
  35. 35. The term Test Automation misleads • It misleads as a label because the whole of testing cannot be automated • The label is bad, but the scope of Test Automation is what I call ‘Applying’ in the New Model Intelligent Definition and Assurance Slide 35
  36. 36. Testers need ThinkingTools • There are ten testing activities in the New Model – Test automation tools only support one:‘Applying’ • The remaining nine activities (information gathering, analysis, modelling, challenging, test design and so on) are not well supported • All require some level of thinking and skills • Checking is possible when a system and its purpose are well understood and trusted • Test automation tools are simple in principle… … compared to the rest of the test process. Intelligent Definition and Assurance Slide 36
  37. 37. Four quadrant model of intelligent test tools Ability to Investigate AbilitytoCaptureKnowledge • Text editors, Screen Shots Models, visualisations, relationships, transformations • Note Takers • Mind Maps • UML/Case Tools Control,imagination,discernment,foresight • Pencil and paper, sketching tools Intelligent Definition and Assurance Slide 37
  38. 38. Intelligent Definition and Assurance Slide 38 TERMINATOR TESTER Not Yet!
  39. 39. Cervaya™ Tool Architecture A nine month bot journey (But it’s been a twenty year testing journey so far) management/new-model-testing-new-test-process-tool/
  40. 40. The vision thing • I want a bot partner/pair that supports exploratory testing • You know my view (model) of testing already • Can we use the explore v test paradigm in a bot that allows you to: – Explore, take notes and model – Record ideas, risks, tests, outcomes and bugs – Generate reports and documentation as a consequence, rather than requiring you write them • Codename: Cervaya ( is a holding page for now) Intelligent Definition and Assurance Slide 40
  41. 41. Schematic Intelligent Definition and Assurance Slide 41 Schema Server Schema Manager (web site) Schema Repository (web service) Cervaya Bot Client Robot Engine Cervaya Server Actions (web services) Application Web Site Speech Recognition Interface Command Line Interface Administer Schemas through the Web Download Schema Perform Robot Actions through services Application Reporting, Monitoring, Control, Management Many App Servers – one (or many) per Schema robotschemas cervaya
  42. 42. The target user is… • (Not really a freestyle exploratory tester working in a start-up or small product company) • Target users are testers working in: – Regulated, high-integrity, safety critical – High documentation, high accountability – Environments where testers are constrained by their processes – Testers struggling to apply ‘trad’ methods in an Agile, Digital, DevOps environment • Why can’t we dictate a detailed test plan for a bot to document, analyse, visualise? Intelligent Definition and Assurance Slide 42
  43. 43. The location hierarchy • Explorers create maps (or should do!) • The system model/map is a hierarchy of locations as follows: • Application – Places • Features – Forms » Fields • All other content is located with respect to some level and location in the system model. Intelligent Definition and Assurance Slide 43
  44. 44. A New Test Process
  45. 45. Exploration support • Frustration: – testers are unimaginative, working by-rote – constant pressure to cut costs • Productivity of exploratory testing is proven • Testers want to explore, but the need for control and documentation constrains them • Testers needs tools that can capture plans and tester activity in real-time • The next generation will be led by tools that support the exploration of sources of knowledge • These tools might use a “Surveying” metaphor. Intelligent Definition and Assurance Slide 45
  46. 46. Staged Process System Testing Re-Test Big, up-front Test Planning Regression Test Development Automation From Staged to Continuous Scoping exploration sessions Interactive Testing Sessions Clarifications Continuous Integration Changes Refinements Continuous Dev, Test and Delivery Bot Supported Bot Supported
  47. 47. Speech recognition, text to speech • Right now, I’m using: – Google Speech Recognition (a slow web service) – pyttsx a PythonText to Speech module (depends on pywin32-extensions for Windows) • I’m looking at Amazon Alexa as an alternative – – Alexa is the web service, Echo Dot is the latest device that captures and synthesises voice – It promises to be a lot faster (and it works better). Intelligent Definition and Assurance Slide 48
  48. 48. Real-Time test management • Once you have a shared system model: – The activity of the team can be visualised in real-time – Tester activity: visible – testers liaise when necessary – Test paths through the system map can be traced • Developers can also see progress and requests for more information in context: • Cervaya has an integration with Slack: – Model changes, additions, queries, test failures can be posted to Slack/ChatOps service. Intelligent Definition and Assurance Slide 49
  49. 49. From Staged to ContinuousTesting • Waterfall projects create, execute tests in stages • Shifted-left testers can: – Model features as they are created by the developers – Create accurate tests a low level as the functionality emerges over time – End to end tests use low-level tests as building blocks • Tests and test run histories are manage against versions (and platforms/environments soon) • The aim: to export tests as soon as practical to be automated to provide continuous feedback. Intelligent Definition and Assurance Slide 50
  50. 50. Testing Styles,Approaches “Structured/Waterfall/ Staged”Testing • Systematic • Transparent, Documented • Reviewable • Auditable • Repeatable, measurable • Automatable • Inflexible, not responsive • Obsolescent/inaccurate documentation • Prone to biases, inattention • Outdated process • Expensive, Inefficient • Unimaginative, boring “Exploratory”Testing • Agile (with a small ‘a’) • Improvised, imaginative • Flexible and responsive to change • Faster, cheaper • More effective • Personally enjoyable • Not repeatable • Not easily automated • Little or no documentation • Hard to manage • Hard to scale • Opaque • Not auditable, measurable. Intelligent Definition and Assurance Slide 51 We need to change the testing metaphor
  51. 51. Surveying and testing – the new process? • The “tester as surveyor” affects the relationship of testing to development • A new style of testing process emerges: – No more documentation created in a knowledge vacuum – Iterative, incremental knowledge capture closely aligned with the emergent delivery of functionality • Could this be an Agile test process at last? • It fits the increasingly popular Continuous Delivery, DevOps development approaches. Intelligent Definition and Assurance Slide 52
  52. 52. Close
  53. 53. New Model Testing changes what testers & developers need to learn Roles of developers/testers become blurred Perhaps they are not different disciplines after all? Why have a tester certification?
  54. 54. Do Testers Need to Learn How to Code? Some, perhaps many will have to
  55. 55. Testers don’t own testing anymore Be aTestMaster "Assurance through the lifecycle"
  56. 56. The phase after development is REWORK, not Testing Testing must align with development, not compete with it or rescue it
  57. 57. A New Model for Testing @paul_gerrard Paul Gerrard