• Simultaneous test design, test execution, and learning.


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

• Simultaneous test design, test execution, and learning.

  1. 1. What IS Exploratory Testing? • Simultaneous test design, test Exploratory Testing execution, and learning. and Leadership • James Bach, 1995 Michael Bolton DevelopSense http://www.developsense.com But maybe it would be a good idea to underscore November 2009 why that’s important… What IS Exploratory Testing? What IS Exploratory Testing? • I follow (and to some degree contributed to) Kaner’s definition, •Simultaneous test design, test execution, which was refined over several peer conferences through 2007: and learning, with an emphasis on learning. Exploratory software testing is… • a style of software testing •Cem Kaner, 2005 • • that emphasizes the personal freedom and responsibility of the individual tester • to continually optimize the value of his or her work • by treating test design, test execution, test result interpretation, and test-related learning Whoa. Maybe it • as mutually supportive activities would be a good But maybe it would be a good idea to be • that run in parallel idea to keep it brief really explicit about what goes on… • throughout the project. most of the time… See Kaner, “Exploratory Testing After 23 Years”, www.kaner.com/pdfs/ETat23.pdf Testing Isn’t Just Checking Oh no! What Does “Non-Sapient” Mean? • Checking is a process of confirming and • A non-sapient activity can be performed verifying existing beliefs • Checking can (and I argue, largely should) be done by automation • It is a non-sapient process by a machine by a human who has been that can’t think instructed NOT to think (but it’s quick and precise) (and that’s slow and erratic) See http://www.developsense.com/2009/08/testing-vs-checking.html 1
  2. 2. What is Checking? What Is Sapience? • A check has three attributes • A sapient activity is one that requires a • It requires an observation thinking human to perform • The observation is linked to a decision rule • We test not only for repeatability, but also for • The observation and the rule can be applied adaptability, value, and threats to value • by a machine • by a sufficiently disengaged human Checking IS Important But… • Despite what the Agilists might have you believe, checking is not new • A good tester doesn’t just ask • D. McCracken (1957) refers to “program checkout” • Jerry Weinberg: checking was important in the early days because • computer time was expensive • A good tester asks • programmers were cheap • the machinery was so unreliable • Checking has been rediscovered by the Agilists • centrally important to test-driven development, refactoring, continuous integration & deployment • excellent checking is surrounded by testing work • CHECks are CHange detECtors Testing IS Exploring Humans can… • Testing, as I see, it is all about exploration, recognize new risks investigate speculate discovery, investigation, and learning empathize anticipate predict suggest judge project • Testing can be assisted by machines, but can’t recognize refocus be done by machines alone contextualize elaborate appreciate • It is a sapient process I can’t do that, strategize evaluate but I can help you become resigned act on your ideas. question charter assess teach learn get frustrated reframe work around a problem invent make conscious decisions model resource See http://www.developsense.com/2009/08/testing-vs-checking.html troubleshoot collaborate refine 2
  3. 3. Machines can’t… The Danger of Scripts • Scripts aren’t necessary for recognize new risks investigate speculate EL skilled (human) testers empathize anticipate predict suggest • Script preparation takes away judge FE recognize refocus project from testing time contextualize elaborate • Bugs found and fixed during appreciate NK strategize evaluate script prep tend to stay fixed become resigned • Scripts separate design, question charter assess HI execution, interpretation, and teach learn get frustrated learning…and thus DE-SKILL reframe work around a problem • Scripts drive inattentional invent T model make conscious decisions troubleshoot collaborate refine resource blindness See Kaner, “The Value of Checklists and The Danger of Scripts” http://www.kaner.com Besides… So why don’t we hear more about E.T.? FEAR • You cannot use a script to • investigate a problem you’ve found • decide that there’s a problem with a script • escape the script problem you’ve identified • Maybe managers fear that E.T. depends on skill • but who benefits from ANY unskilled testing? • determine the best way to phrase a report • Maybe managers fear that E.T. is unstructured • unravel a puzzling situation • but it is structured • Maybe managers fear that E.T. is unaccountable • but it can be entirely accountable • Maybe managers fear that E.T. is unmanageable • but you can manage anything if you put your mind to it Skill vs. Alternatives Heuristics are applied, not followed. supervision The greater the skill of This… The skilled tester the tester, the less -Idea remains in control of prescribed procedure, -Idea the process. supervision, or … favorable environment is required. 1. Do this skill 2. Then do this …not this. 3. Then do this s edure ol 4. Then do this d proc tr Scripte sion of con environment procedure 5. And then this… e illu ters. give th nskilled tes over u 17 3
  4. 4. Exploratory Testing IS Structured A Heuristic Test Strategy Model • We’ve studied the structure of ET, we’ve written Project about it, and we know how to teach it Environment • The structure of ET comes from many sources: • Test design heuristics • Chartering Not procedurally • Time boxing structured, but cognitively structured. • Perceived product risks • The nature of specific tests Tests • The structure of the product being tested In other words, • The process of learning the product it’s not “random”, Quality Product • Development activities but systematic. Criteria Elements • Constraints and resources afforded by the project • The skills, talents, and interests of the tester Perceived • The overall mission of testing Quality 20 A Heuristic Test Strategy Model Oracles Project An oracle is Environment a heuristic (usually works, might fail) principle or mechanism by which Tests someone Quality Product Criteria Elements might recognize (but not decide conclusively) a problem. Bug (n): Something that Perceived Quality 21 bugs someone who matters Consistency (“this agrees with that”) an important theme in oracles Test Coverage Isn’t Just Code Coverage ry isto e ucts rod Test coverage is the amount of the H g eP Ima parabl system space that has been tested. Com ims ons There are as many kinds of coverage c tati as there are ways to model the product. Cla r Expe Use urpose t Performance Maintainability P duc • Structure Capability Installability Portability Pro ndards Reliability Compatibility Localizability • Functional Usability Supportability Sta • Data Security Testability • Platform Scalability • Operations Consistency heuristics rely on the quality of your • Time models of the product and its context. 4
  5. 5. Cost as a Simplifying Factor What Does Rapid ET Look Like? Try quick tests as well as careful tests Concise Documentation Minimizes Waste In my travels, I’ve seen extraordinary emphasis Testing Heuristics Risk Catalog on long cycles of planning without feedback. General This makes testing ineffective and slow. A quick test is a cheap test that has some value, gives fast feedback, but requires little preparation, Coverage Model Risk Model Test Strategy knowledge, or time to perform. Reference Project- Bursts of quick tests represent a great way Specific to discover risks upon which Schedule Issues Bugs Status careful testing can be better focused. Dashboard 26 Accountability for Exploratory Testing: How To Measure Test Coverage Session-Based Test Management (it’s not merely code coverage) • Charter • Identify quality criteria • A clear, concise mission for a test session • Identify session time focused on each criterion • Time Box • Consider product elements (structure, function, • 90-minutes (+/- 45) data, platform, operations, and time) vs. • Reviewable Results • Break them down into coverage areas • a session sheet—a test report whose raw data can be scanned, parsed and • Assess test coverage in terms of compiled by a tool • Level 1: Smoke and sanity • Debriefing • Level 2: Common, core, critical aspects • a conversation between tester and • Level 3: Complex, challenging, harsh, extreme, manager or test lead exceptional For more info, see http://www.satisfice.com/sbtm 27 How To Measure ET Efficiency How To Manage Exploratory Testing Track rough percentage of time Achieve excellent test design by Produces exploring different test designs coverage spent on while actually testing and Checks • Test design and execution interacting with the system Interrupts • Bug investigation and reporting coverage • Setup Test Product Ask why time was spent on each: Ideas Tests • Lots on T might indicate great code, but might indicate poor bug- finding skill Guide testers with personal supervision and • Lots on B might mean code quality problems, but might suggest inefficiency in reporting Product concise documentation of test ideas. Meanwhile, • Lots on S might mean testability or configuration problems for train them so that they can guide themselves and or spec be accountable for increasingly challenging work. customers, or it might mean early days of testing 5
  6. 6. What Is Leadership? What Does A Leader Do? • "Leadership is the process of creating an • Performs complex cognitive tasks environment in which everyone is • Has access to a large number of models empowered." • Applies the models to absorb, process, and • Gerald M. Weinberg, respond to whatever information is available Becoming a Technical Leader • Responds, flexibly and adaptably, to whatever • Leaders require freedom and responsibility complications the situation presents to optimize the quality of their work, while • Empowers (teams of skilled technical) people granting freedom and responsibility to others • Learns rapidly and observes keenly to do the same. • Is introspective and self-critical • Motivates, organizes, and innovates Key Ideas Motivation: How To Kill It • All managers should be leaders, but • Make people feel that change will not be managers are not the only leaders appreciated • Managers who relinquish control foster • Do everything for them so they won’t feel the environments in which leadership can need to do things themselves blossom • Discourage anything that people might enjoy • Managers who seize control and won’t let go doing for its own sake destroy leadership Gerald M. Weinberg, Becoming a Technical Leader Organization: How To Foster Chaos Ideas: How To Suppress the Flow • Encourage such high competition that • Don’t listen when you can criticize co-operation will be unthinkable • Give your own ideas first, and loudest • Keep resources slightly below the necessary • Punish those who offer suggestions minimum • Keep people from working together • Suppress information of general value, or • Above all, tolerate no laughter bury it in an avalanche of meaningless words and paper Gerald M. Weinberg, Becoming a Technical Leader Gerald M. Weinberg, Becoming a Technical Leader 6
  7. 7. Leadership Is Exploratory! People I DO See As Leaders • A leader Not procedurally • People who question what they see and hear • both grants and receives structured, but cognitively structured. • like participants in Edista’s Test Republic freedom with responsibility • doesn’t follow a script • People who exchange their ideas • fosters fault-tolerant environments • like participants in The Bangalore Workshops on In other words, • fosters and practices learning it’s not “random”, Software Testing • practices critical thinking but systematic. • People who practice their craft • like the Weekend Testers • (see the presentation this afternoon!) People I DO NOT See As Leaders How Do Programmers Program? • People who are afraid to speak truth to power • Do we use programming cases? • Those who do not actively question the outdated • Do we follow programming scripts? testing mythodologies • Is there a step-by-step procedure for the development of every program? • Those who disempower other people • Does each programming task have an expected, • those (including, alas, Indian managers) who see testers predicted result? (and especially Indian testers) as hopelessly unskilled • Do we evaluate programmers by counting the lines • anyone involved with scripted testing (unless the script is of code they write? for a machine) • Do we evaluate programmer performance by • certificationists; people who participate in or promote the empty certifications that we currently have “coding error escape rates”? • Western organizations that help promote this stuff • Do we aspire to reduce the cost of programming by bringing in development automation? How Do Managers Manage? So… What Do We Want? • Do we use management cases? • If we want to miss important problems slowly • Do we follow management scripts? • emphasize confirmation • Is there a step-by-step procedure for every management • emphasize repetition action? • then complain about how little time we have • Does each management action have an expected, predicted result? • If we want to find important problems quickly • Do we evaluate managers by counting their • reduce wasted time and wasted effort decisions? • prevent regression problems • Do we evaluate management performance by • emphasize exploration, discovery, investigation “bad decision escape rates”? • train and empower testers • Do we aspire to reduce the cost of management by • grant them freedom and responsibility for the quality of their work bringing in management automation? 7
  8. 8. What if we train our people and they leave? Acknowledgements • James Bach (http://satisfice.com) • Cem Kaner (http://www.kaner.com) • Jerry Weinberg (http://www.geraldmweinberg.com) Questions? More information? Michael Bolton http://www.developsense.com michael@developsense.com Author unknown, but I’m envious of him/her. Readings • Perfect Software and Other Illusions About Testing • Becoming a Technical Leader • Quality Software Management, Vol. 1: Systems Thinking • Quality Software Management, Vol. 2: First Order Measurement • Gerald M. Weinberg • Lessons Learned in Software Testing • Kaner, Bach, and Pettichord • DevelopSense Web Site (and blog), http://www.developsense.com • Michael Bolton • Satisfice Web Site (and blog), http://www.satisfice.com • James Bach • Collaborative Software Testing, http://www.kohl.ca • Jonathan Kohl • Quality Tree Software, http://www.qualitytree.com • Elisabeth Hendrickson 8