How to Test Anything
•Download as PPTX, PDF•
1 like•2,955 views
Presented at https://www.onlinetestconf.com/program-spring-otc-2020/ Sometimes you’re asked to start testing in a context that is not ideal: you’ve only just joined the project, the test environment is broken, the product is migrating to a new stack, the developer has left, no-one seems quite sure what’s being done or why, and there is not much time. Knowing where to begin and what to focus on can be difficult and so in this talk I’ll describe how I try to meet that challenge. I’ll share a definition of testing which helps me to navigate uncertainty across contexts and decide on a starting point. I’ll catalogue tools that I use regularly such as conversation, modelling, and drawing; the rule of three, heuristics, and background knowledge; mission-setting, hypothesis generation, and comparison. I’ll show how they’ve helped me in my testing, and how I iterate over different approaches regularly to focus my testing. The takeaways from this talk will be a distillation of hard-won, hands-on experience that has given me * an expansive, iterative view of testing * a comprehensive catalogue of testing tools * the confidence to start testing anything from anywhere
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How to Test Anything
Similar to How to Test Anything (20)
More from James Thomas
Recently uploaded
Recently uploaded (20)
How to Test Anything
- 1. HOW TO TEST ANYTHING James Thomas @qahiccupps qahiccupps.blogspot.com OnlineTestConf 2020-05-20
- 2. @qahiccupps
- 3. @qahiccupps
- 4. ASSOCIATION FOR SOFTWARE TESTING Advancing understanding of the science and practice of software testing @qahiccupps
- 5. HOW TO TEST ANYTHING @qahiccupps
- 6. How to test anything! Really? MODESTY BYPASS! @qahiccupps
- 7. @qahiccupps
- 8. THE SAME TASK? Are skill and adaptability relevant? What does "same task” mean? From whose perspective? @qahiccupps
- 9. Automation of some task becomes tractable at the point where we become indifferent to the details of it. Harry Collins and Martin Kusch @qahiccupps
- 10. @qahiccupps
- 11. TESTING IS NOT Mindlessly following steps @qahiccupps
- 12. TESTING REQUIRES Intention Deliberate actions Agency and response to observation @qahiccupps
- 13. TO TEST SOMETHING An idea of what testing is Applicable testing skills To start testing @qahiccupps
- 14. HOW I TEST ANYTHING My definition of testing Some of my testing tools How I choose where to start @qahiccupps
- 15. WHAT IS TESTING? @qahiccupps WHATISTESTING? -TESTINGTOOLS - STARTINGTESTING
- 17. DEFINITIONS Executing a program with the intent of finding errors – Meyers (1976) Finds information … that informs critical decisions – Kaner, Bach, Pettichord (2002) Interact with SUT, observe behaviour, compare to expectations – Hendrickson (2013) @qahiccupps WHATISTESTING?
- 18. WHAT IS TESTING TO ME? @qahiccupps WHATISTESTING?
- 19. undirected learning cross-referencingnote-taking cataloguing correlating analysing sense-making skim-reading reasoning searching re-reading reflecting researching factoring filtering comparing modelling consistency-seeking thinking laterally thinking critically being sceptical evaluating conversation hypothesis generation uncertainty telling the testing story prioritisation identifying problems exploring scientific method engineering method heuristics directed learning @qahiccupps WHATISTESTING?
- 20. Testing is simple: you understand what is important and then you test it. Rikard Edgren, EuroSTAR 2015 @qahiccupps WHATISTESTING?
- 21. Testing a sub-sub-feature Testing a sub-feature Testing a feature Testing the model Testing the PO’s view Testing the links between features Testing another feature Testing PO’s expression of their view Testing the way I’m talking to the PO Testing whether the PO is the best person to talk to Testing the end user need Testing the feature testing Testing the reason for testing @qahiccupps WHATISTESTING?
- 23. WHAT IS TESTING TO ME? Testing is the pursuit of relevant incongruity @qahiccupps WHATISTESTING?
- 24. INCONGRUITY Not in harmony or keeping with the surroundings – Oxford Dictionaries @qahiccupps WHATISTESTING?
- 26. RELEVANT To someone who matters At a time that matters @qahiccupps WHATISTESTING?
- 27. WHAT IS TESTING TO ME? Testing is the pursuit of relevant incongruity @qahiccupps WHATISTESTING?
- 28. SO WHAT? It keeps me focussed on what matters I can tell whether I’m testing or not I can decide whether to test or not @qahiccupps WHATISTESTING?
- 29. WHAT IS TESTING? Intentional, directed, responsive About more than the software Interpretation and choices made in context @qahiccupps WHATISTESTING? -TESTINGTOOLS - STARTINGTESTING
- 30. TESTING TOOLS @qahiccupps WHATISTESTING? -TESTINGTOOLS - STARTINGTESTING
- 31. undirected learning cross-referencingnote-taking cataloguing correlating analysing sense-making skim-reading reasoning searching re-reading reflecting researching factoring filtering comparing modelling consistency-seeking thinking laterally thinking critically being sceptical evaluating conversation hypothesis generation uncertainty telling the testing story prioritisation identifying problems exploring scientific method engineering method heuristics directed learning @qahiccupps TESTINGTOOLS
- 35. MODEL Describes a (part of a) system Aids understanding Permits predictions @qahiccupps TESTINGTOOLS
- 41. Information consists of differences that make a difference. Edward R. Tufte @qahiccupps TESTINGTOOLS
- 43. COMPARED TO WHAT? Our competitor's software is fast. Fast ... compared to what? We must export to a good range of image formats. Good ... compared to what? The layout must be clean. Clean ... compared to what? @qahiccupps TESTINGTOOLS
- 44. ORACLE Something to compare to Helps decide whether there’s an interesting difference @qahiccupps TESTINGTOOLS
- 46. CRITICAL THINKING @qahiccupps I see X, the oracle says Y. Must be a bug! I see X, the oracle says Y. Is this a reasonable oracle? I see X, the oracle says Y. Did I compare correctly? TESTINGTOOLS
- 47. All models are wrong but some are useful. George Box @qahiccupps Models are oracles. Oracles are heuristic. TESTINGTOOLS
- 48. KNOW THE PROBLEM TESTING TOOLS @qahiccupps
- 49. A problem is a difference between things as desired and things as perceived. Donald C. Gause and Gerald M. Weinberg For any abstract X, X is X to some person, at some time. Michael Bolton @qahiccupps TESTINGTOOLS
- 50. WHAT AND WHEN IS REALLY THE PROBLEM HERE? The thing The perception of that thing The desires for that thing The person(s) desiring or perceiving The context(s) in which the desiring or perceiving is taking place @qahiccupps TESTINGTOOLS
- 51. Donald C. Gause and Gerald M. Weinberg If you can't think of at least three things that might be wrong with your understanding of the problem, you don't understand the problem. @qahiccupps TESTINGTOOLS
- 53. THE SCIENTIFIC AND ENGINEERING METHODS TESTING TOOLS @qahiccupps
- 55. Billy Vaughn Koen Do what you think represents best¹ practice² at the time you must decide. 1. best in context 2. practices are heuristic @qahiccupps TESTINGTOOLS
- 58. BBST Courses – Association for Software Testing @qahiccupps TESTINGTOOLS
- 59. Heuristic Test Strategy Model – James Bach @qahiccupps TESTINGTOOLS
- 62. TESTING TOOLS Named Practice Heuristic @qahiccupps WHATISTESTING? -TESTINGTOOLS - STARTINGTESTING
- 63. STARTING TESTING @qahiccupps WHATISTESTING? -TESTINGTOOLS - STARTINGTESTING
- 64. STARTING CAN BE HARD But starting testing is starting learning @qahiccupps STARTINGTESTING
- 65. Begin where you are. Keri Smith Testing is simple: you understand what is important and then you test it. Rikard Edgren, EuroSTAR 2015 @qahiccupps STARTINGTESTING
- 67. FOR EXAMPLE… Improve your technical understanding Look for areas of highest risk (of what, to who) Build rapport with your team and stakeholders @qahiccupps STARTINGTESTING
- 68. EXPLORE X USING Y TO Z Outcome focus Open approach Timebox @qahiccupps Explore the new component using the command line to look for incompatible options. Elisabeth Hendrickson STARTINGTESTING
- 69. STARTING TESTING Begin where you are Know your context Find your mission @qahiccupps WHATISTESTING? -TESTINGTOOLS - STARTINGTESTING
- 70. @qahiccupps
- 71. SOMETHING @qahiccupps WHATISTESTING? -TESTINGTOOLS - STARTINGTESTING- SOMETHING
- 72. Pairing README • I will pair on any task on any subject • I will try any format of pairing • We’ll agree a mission and debrief afterwards • If deep prep is needed, tell me … • … if not, I'll just drop into it cold @qahiccupps SOMETHINGTOTEST
- 73. HOW I TEST ANYTHING My definition of testing Some of my testing tools How I choose where to start @qahiccupps
- 74. NOW YOU GO AND TEST SOMETHING @qahiccupps
- 76. Selected Credits • Twitter logo: http://clipart-library.com/clipart/kTKoxpgBc.htm • Robot: https://sites.suffolk.edu/jhtl/category/robotics/ • Harry Collins: https://www.amazon.co.uk/Shape-Actions-MIT-Press/dp/0262526522 • Uncertainty: https://ansonanalytics.com/316-321-338-fiduciaries/confusion-311388_1280/ • Rikard Edgren: http://qahiccupps.blogspot.co.uk/2015/11/testing-is-simple-and-complicated.html • Arborosa: https://arborosa.org/2015/06/14/what-is-testing/ • George Box: https://egtheory.wordpress.com/2013/11/06/wrong-models/ • Edward R. Tufte: https://www.abebooks.co.uk/first-edition/ENVISIONING-INFORMATION-Tufte- Edward-R-Graphics/194976494/bd
- 77. Selected Credits • Me at DEWT: https://twitter.com/ard_kramer/status/1223527049593282560 • Gause and Weinberg: https://www.kobo.com/us/en/ebook/are-your-lights-on • Relative rule: https://www.developsense.com/blog/2010/09/done-the-relative-rule-and-the- unsettling-rule/ • Scientific Method: https://en.wikipedia.org/wiki/File:The_Scientific_Method_(simple).png • Billy Vaughn Koen: https://www.amazon.co.uk/Definition-Engineering-Method-Billy- Vaughn/dp/0878231013 • Keri Smith: https://www.amazon.co.uk/How-Explorer-World-Keri-Smith/dp/024195388X • Heuristic Test Strategy Model: https://www.satisfice.com/download/heuristic-test-strategy-model • Black Box Software Testing courses: https://www.associationforsoftwaretesting.org/courses/ • Columbo: https://medium.com/arscognitionis/what-columbo-can-teach-us-about-rationality- 49d21380a1e6