The Art of Skillful Investigation
•Download as PPTX, PDF•
1 like•1,282 views
Slides of the presentation that I did at the Belgium Testing Days 2014.
Recommended
Recommended
More Related Content
Similar to The Art of Skillful Investigation
Similar to The Art of Skillful Investigation (20)
Recently uploaded
Recently uploaded (20)
The Art of Skillful Investigation
- 1. The Art of Skillful Investigation Joris Meerts, 18 March 2014 The Art of Skillful Investigation Belgium Testing Days Joris Meerts, 18 March 2014 v1.0
- 2. The Art of Skillful Investigation Joris Meerts, 18 March 2014 About me • Joris Meerts • Software tester since 2007 • Capgemini • Context-driven school • www.testingreferences.com • Member of the Dutch Exploratory Workshop on Testing (DEWT) • @testingref
- 3. The Art of Skillful Investigation Joris Meerts, 18 March 2014 I want something that explains what testing IS not what it SHOULD BE
- 4. The Art of Skillful Investigation Joris Meerts, 18 March 2014 Normative theories
- 5. The Art of Skillful Investigation Joris Meerts, 18 March 2014 Thou shalt have no other gods before me. Thou shalt not make unto thee any graven image, or any likeness of any thing that is in heaven above, or that is in the earth beneath, or that is in the water under the earth. Thou shalt not bow down thyself to them, nor serve them: for I the LORD thy God am a jealous God, visiting the iniquity of the fathers upon the children unto the third and fourth generation of them that hate me; And shewing mercy unto thousands of them that love me, and keep my commandments. Thou shalt not take the name of the LORD thy God in vain; for the LORD will not hold him guiltless that taketh his name in vain. Remember the sabbath day, to keep it holy. Six days shalt thou labour, and do all thy work: But the seventh day is the sabbath of the Lord thy God: in it thou shalt not do any work, thou, nor thy son, nor thy daughter, thy manservant, nor thy maidservant, nor thy cattle, nor thy stranger that is within thy gates: For in six days the Lord made heaven and earth, the sea, and all that in them is, and rested the seventh day: wherefore the Lord blessed the sabbath day, and hallowed it. What are normative theories?
- 6. The Art of Skillful Investigation Joris Meerts, 18 March 2014 Normative theories in testing • Development of normative theories started early. • We have many of them. • IEEE 829, IEEE 1012 (V&V), TMap, ISEB, ISQTB, TMMi, ISO 29119
- 7. The Art of Skillful Investigation Joris Meerts, 18 March 2014 • Normative theories are hiding behind a ‘toolbox of practices’. • They are never truly seen as theories. • They are not challenged as theories would be challenged. Why are normative theories still around?
- 8. The Art of Skillful Investigation Joris Meerts, 18 March 2014 “Whether this method is successful or not, is not known. There are no scientific publications on the subject.”
- 9. The Art of Skillful Investigation Joris Meerts, 18 March 2014 • William Howden - Reliability of the Path Analysis Testing Strategy (1976) • Richard Hamlet - Theoretical Comparison of Testing Methods (1989) • Elaine Weyuker - The Evaluation of Program-Based Software Test Data Adequacy Criteria (1988) • Lee J. White - A Domain Strategy for Computer Program Testing (1980) • Luc Bougé - Test sets generation from algebraic specifications using logic programming (1986) • Jefferson Offut - Criteria for Generating Specification-based Tests (1999) • Richard A. Kemmerer - Testing Formal Specifications to Detect Design Errors (1985) • Robert M. Hierons - Testing from a Z specification (1997) • John Goodenough, Susan Gerhart - Toward a Theory of Test Data Selection (1975) • Thomas J. Ostrand, Elaine Weyuker - Theories of program testing and the application to revealing subdomains (1980) • Jeremy Dick, Alain Faivre - Automating the Generation and Sequencing of test cases from model-based specifications (1993) • Deepinder Sidhu, Ting-Kau Lung - Formal Methods in Protocol Testing: a Detailed Study (1989) Mind you, there is research in testing!
- 10. The Art of Skillful Investigation Joris Meerts, 18 March 2014 “A scientific discipline emerges . . . with the discovery of useful and helpful concepts. Scientific thought comprises in addition the conscious search for the useful and helpful concepts.” Edsger Wybe Dijkstra On the role of scientific thought (1974)
- 11. The Art of Skillful Investigation Joris Meerts, 18 March 2014 Descriptive theories
- 12. The Art of Skillful Investigation Joris Meerts, 18 March 2014 “Descriptive theories are needed when nothing or very little is known about the phenomenon in question.” Jaqueline Fawcett The Relationship of Theory and Research (1986)
- 13. The Art of Skillful Investigation Joris Meerts, 18 March 2014 “Anyway, I think we should stop teaching testers practices that they might or might not need. Instead we should focus more on teaching testers how to evaluate their situation” Markus Gaertner Software Tester – From New to Good to Great (2014)
- 14. The Art of Skillful Investigation Joris Meerts, 18 March 2014 Testing as investigation
- 15. The Art of Skillful Investigation Joris Meerts, 18 March 2014 Testing is the art of skillful investigation by experimentation
- 16. The Art of Skillful Investigation Joris Meerts, 18 March 2014 • Mind maps • Drawings • Relationships • Interaction How to investigate: mental models
- 17. The Art of Skillful Investigation Joris Meerts, 18 March 2014 How to investigate: comparisons (1) • Product history • Manuals • Standards • Conversations • Other products (such as ketchup) • Analogies
- 18. The Art of Skillful Investigation Joris Meerts, 18 March 2014 Sequence, Concurrence, Confluence, Synchronization, Share, Interaction, Continuity, Hierarchy, Priority, Dependency, Repetition, Loop, Parameter, Prerequisite, Configuration, Rule, Customize, Constraint, Resource, Access, Lock, State, History, Roll back, Restore, Refresh, Clone, Temporary, Trace, Batch, Void, Absent, Feedback, Saturate, Sort, Scale, Corrupt, Integrity, Invoke, Timing, Delay, Velocity, Instance, Format, Precision, Duplicate, Transmit, Wait, Source, Trigger, Lost, Reality, Classification How to investigate: comparisons (2)
- 19. The Art of Skillful Investigation Joris Meerts, 18 March 2014 experiment
- 20. The Art of Skillful Investigation Joris Meerts, 18 March 2014 How to investigate: conducting an experiment • Set up a theory • Conduct the experiment • Observe the result
- 21. The Art of Skillful Investigation Joris Meerts, 18 March 2014 How to investigate: observation from a frame of reference • William Kahan • Father of the floating point
- 22. The Art of Skillful Investigation Joris Meerts, 18 March 2014 How to investigate: reasoning • Testers must provide sound evidence • On the basis of theory • On the basis of observations
- 23. The Art of Skillful Investigation Joris Meerts, 18 March 2014 • Jonathan Baron – Thinking and Deciding • Norwood Russell Hanson – Patterns of Discovery • Joy Paul Guilford – The Nature of Human Intelligence • Karl E. Wieck - Theory Construction as Disciplined Imagination • Ron Ritchhart - Intellectual Character : What It Is, Why It Matters, and How to Get It • Malcolm Gladwell – The Ketchup Conundrum • Philip N. Johnson-Laird - The Psychology of Syllogisms • Carl Sagan - The Demon-Haunted World • Shirley Gregor - The Nature of Theory in Information Systems • Paul Feyerabend – Against Method • Peter Naur - Programming as Theory Building • Jacqueline Fawcett - The Relationship of Theory and Research • Richard P. Feynman - Surely You're Joking, Mr. Feynman! • Philip N. Johnson-Laird - Mental Models • Plato - Selected Dialogues of Plato: The Benjamin Jowett Translation • Herbert A. Simon - The Sciences of the Artificial • Russell L. Ackoff - Towards a System of Systems Concepts • Jerome S. Bruner - Beyond the Information Given: Studies in the Psychology of Knowing • National Center for History in the Schools - Historical Thinking Standards • William Kahan - Needed Remedies for the Undebuggability of Large-Scale Floating-Point Computations in Science and Engineering • Rikard Edgren, Martin Jansson, Henrik Emilsson - 37 Sources for Test Ideas References
- 24. The Art of Skillful Investigation Joris Meerts, 18 March 2014 joris.meerts@capgemini.com @testingref www.testingreferences.com Thank you for attending my presentation!
Editor's Notes
- Waarom deze presentatie: onderzoekend testen is de enige benadering van testen die daadwerkelijk over testen gaat.Doelen van de presentatie:Uitleggen wat onderzoekend testen is.Laten zien dat context-driven testen de enige stroming is die concepten uit onderzoekend testen uitwerkt.
- Context-driven testen is de meest realistische basis voor software testen.
- Philip Johnson Laird – Mental Models. Theories have to account for the phenomenathat are observed.I do notgive a rat’sassaboutwhatitshouldbe.
- A normativetheorytellsushowthingsshouldbe in the idealworld.
- Weggelaten uit deze definitie: het doel van testen (dit verandert in de tijd) de specificaties, zoals hiervoor al toegelicht.We set up experiments!Testen gaat niet over proces, activiteiten en producten, testen gaat over het doen van onderzoek. Testers zijn onderzoekers en zouden eigenlijk in hoge mate moeten beschikken over de vaardigheden van een onderzoeker.
- Volgens de Historical Thinking Standards vormt de student een eigen beeld van de geschiedenis.Ik gebruik vrijwel altijd mindmaps, afbeeldingen. Ik maak schetsen van de samenhang van bepaalde zaken, de interactie tussen applicaties en de omgeving. Ik maak schema’s van hoe de organisatie in elkaar zit.Het geestelijk model mag fout zijn, fouten bevatten. Gebruik de fouten om dingen te ontdekken.Maak je eigen model van het systeem, van de software, van het ontwerp, van de omgeving, van het project, de organisatie, de gebruikers.Ahold: incomplete mental model!
- Vergelijkingen zijn een heel belangrijk onderdeel van het bouwen aan een theorie over hoe de software zou moeten werken.Vergelijkingen leren je dingen vanuit verschillende perspectieven te zien. 37 Sources for Test IdeasRikardEdgren, Martin Jansson, HenrikEmilsson.Je kunt het zo gek maken als je zelf wil en ik raad aan om het vooral gek te maken.Analogie (met industriële concepten)Tomatenketchup
- Experimenteren = testen. Waarom wordt er zo weinig geschreven over experimenteren.De waarneming is het meest onderschatte aspect van software testen vs product risico analyse, test volwassenheidsmodelHet uitlokken van de waarnemingalsonderdeel van het experiment. RA600:”willekeurige bak nullen en enen vs specifieke gegevensverzameling.Testen van voorraden.Frame of reference. Dokter Tinus (SBS6).Collega over afronden.
- Als tester hebben we een zwaar wegende bewijslast. Binnen het project ligt de bewijslast grotendeels bij de testers.Plato - Socraticdialogue. Tester moeten leren redeneren.Voorbeeld SVBRon Ritchhart - Intellectual Character : What It Is, Why It Matters, and How to Get It ,Chapter Four: First Days, First Steps—Initiating a Culture of ThinkingInductief, deductief.Sillogisme.Logicalfallacy.