Tips & Tricks
Your SlideShare is downloading.
Rapid Software Testing
Like this document? Why not share!
A Rapid Introduction to Rapid Softw...
Qt App Development for Symbian & Me...
by Andreas Jakl
Rapid Software Testing: Strategy
A Rapid Introduction to Rapid Softw...
Git workshop 33degree 2011 krakow
by Luca Milanesio
On Rapid Releases and Software Testing
by Foutse Khomh
Email sent successfully!
Show related SlideShares at end
Rapid Software Testing
May 22, 2011
Comment goes here.
12 hours ago
Are you sure you want to
Your message goes here
Be the first to comment
2 years ago
Number of Embeds
Flagged as inappropriate
Flag as inappropriate
No notes for slide
Transcript of "Rapid Software Testing"
1. Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. V2.1.2 James Bach, Satisfice, Inc. Michael Bolton, DevelopSense firstname.lastname@example.org email@example.com www.satisfice.com www.developsense.com +1 (540) 631-0600 +1 (416) 656-5160 Copyright © 1995-2007, Satisfice, Inc. 1
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Acknowledgements Some of this material was developed in collaboration with Dr. Cem Kaner, of the Florida Institute of Technology. See www.kaner.com and www.testingeducation.org. Doug Hoffman (www.softwarequalitymethods.com) also contributes to and teaches from this material. Many of the ideas in this presentation were also inspired by or augmented by other colleagues including Jonathan Bach, Bret Pettichord, Brian Marick, Dave Gelperin, Elisabeth Hendrickson, Jerry Weinberg, Noel Nyman, and Mary Alton. Some of our exercises were introduced to us by Payson Hall, Jerry Weinberg, Ross Collard, James Lyndsay, Dave Smith, Earl Everett, Brian Marick, Cem Kaner and Joe McMahon. Many ideas were improved by students who took earlier versions of the class going back to 1996. 2 Although the value of the class is greater because of the people who helped us, the sole responsibility for the content of this class belongs to the authors, James Bach and Michael Bolton. This class is essentially a long editorial opinion about testing. We have no authority to say what is right. There are no authorities in our field. We don’t peddle best practices. We don’t believe in them. What we will do is share our experiences and challenge your mind. We hope you will challenge us back. You’ll find the class more rewarding, we think, if you assume that our purpose is to make you stronger, smarter and more confident as a tester. Our purpose is NOT to have you listen to what the instructor says and believe it, but to listen and then think for yourself. All good testers think for themselves. In a way, that’s what testing is about. We look at things differently than everyone else so that we can find problems no one else will find.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Assumptions You test software. You have at least some control over the design of your tests and some time to create new tests. You are worried that your test process is spending too much time and resources on things that aren’t important. Good testing requires thinking. You test under uncertainty and time pressure. Your major goal is to find important problems quickly. You want to get very good at software testing. 3 If any of these assumptions don’t fit, this class might not be for you. We don’t make any assumptions about how experienced you are. The class can work well for novices or experts.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Triangle Rapid Testing Rapid testing is a mind-set and a skill-set of testing focused on how to do testing more quickly, less expensively, with excellent results. This is a general testing methodology. It adapts to any kind of project or product. 4 Rapid testing involves considerations of skill, personal integrity, improved focus on the underlying need for testing tasks, improved appreciation for the stakeholders of testing tasks, and knowledge of the possible techniques and tools that could be brought to bear to improve efficiency.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Our Method of Instruction The Class Presents Our Editorial Opinions: We do not make appeals to authority; we speak only from our experiences, and we appeal to your experience and intelligence. Not All Slides Will be Discussed: There is much more material here than we can cover in detail, so we may skip some of it. (If you want me to go back to something that I skipped, just ask.) We Need to Hear from You: You control what you think and do, so we encourage your questions and challenges the lecture. (Talk to me during the break, too.) If You Want Specifics, Bring Specifics: We invite you to bring real examples of testing problems and test documents to class. (I am happy to show you how a rapid tester would work through them.) The Exercises are the Most Important Part: We use immersive socratic exercises that are designed to fool you if you don’t ask questions. We usually do not provide all the information you need. Asking questions is a fundamental testing skill! 5
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Our Method of Instruction If I call on you, and you don’t want to be put on the spot, say “Pass!” 6
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Primary Goal of this Class To teach you how to test a product when you have to test it right now, under conditions of uncertainty, in a way that stands up to scrutiny. 7 Without scrutiny, this would be easy to do. You could test badly, but no one would ever know or care. What makes testing hard is that someone, perhaps your employer, perhaps the users, or perhaps you yourself, will be judging your work. Maybe this scrutiny will be indirect, such as someone unhappy with the product and cursing your company for building it so poorly. Or perhaps it will be the most intimate form of scrutiny– which is your feeling of pride or disappointment in your own work. If you want your testing to pass muster, you need to work at it.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Secondary Goal of this Class To help you practice thinking like an expert tester. Do that well, and you’ll be able to handle any testing situation. 8
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. KEY IDEA 9
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Universal Testing Method, v1.0 “Try it and see if it works.” Get it Choose Read the spec Set it up where to look See if the product Run it See what’s matches the spec there Run it again, Find problems… maybe See what’s …especially the not there bad ones Procedures Coverage Oracles 10
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Universal Testing Method, v1.5 “Try it and see if it works.” “Try it to learn, sufficiently, everything that matters about whether it can work and how it might not work.” 11
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Universal Testing Method, v1.5 If you don’t have an understanding and an agreement on what is the mission of your testing, then doing it “rapidly” would be pointless. “everything that matters” 12
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. A Heuristic Test Strategy Model Project Environment Tests Quality Product Criteria Elements Perceived Quality 13
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. A Heuristic Test Strategy Model Project Environment Tests Quality Product Criteria Elements Perceived Quality 14
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Calendar The Universal Testing Method, v2.0 “Try it to learn…how it might not work.” 1. Model the test space. 2. Determine coverage. 3. Determine oracles. There are numerous ways 4. Determine test procedures. of doing these 5. Configure the test system. things. Patterns that describe how to do 6. Operate the test system. this are called 7. Observe the test system. “test techniques.” 8. Evaluate the test results. 9. Report test results. 15 By “modeling”, we mean that we must build a representation in our minds of what the product must do. If our model is incorrect or limited, then we automatically will not test what should be tested. The first four items on the list are usually called “test design”. The latter five items are usually called “test execution”.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. KEY IDEA 16
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Testers are Not Savage Beasts Though sometimes we seem a bit “negative” to developers or wishful marketers. 17
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Testers light the way. This is our role. We see things for what they are. We make informed decisions about quality possible, because we think critically about software. 18
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Testers light the way. Quality is value to some person (who matters). A bug is anything about the product that threatens its value. These definitions are designed to be inclusive. Inclusive definitions minimize the chance that you will inadvertently overlook an important problem. For a video lecture, see: http://www.satisfice.com/bbst/videos/BugAdvocacy.wmv 19 The first question about quality is always “whose opinions matter?” If someone who doesn’t matter thinks your product is terrible, you don’t necessarily change anything. So, an important bug is something about a product that really bugs someone important. One implication of this is that, if we see something that we think is a bug and we’re overruled, we don’t matter. That’s a fact: testers don’t matter. That is to say, our standards are not the standards to which the product must adhere; we don’t get to make decisions about quality. This can be hard for some testers to accept, but it’s the truth. We provide information to managers; they get to make the hard decisions. They get the big bucks, and we get to sleep at night.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Testers light the way. Testing is questioning a product in order to evaluate it. The “questions” consist of ordinary questions about the idea or design of the product, or else questions implicit in the various ways of configuring and operating the product. (Asking questions about the product without any intent to operate it is usually called review rather than testing.) The product “answers” by exhibiting behavior, which the tester observes and evaluates. To evaluate a product is to infer from its observed behavior how it will behave in the field, and to identify important problems in the product. 20 Cem Kaner prefers this definition: “Software testing is a process of empirical, technical investigation of the product under test conducted to provide stakeholders with quality-related information.” Kaner’s definition means the same thing as our definition in every material respect. However, we sometimes prefer more concise wording. One surprising implication of both our definitions is that you can test a product that doesn’t yet exist in operable form. The questioning process that precedes what we call test execution is still part of testing if it serves the process of test design and test execution.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Themes of Rapid Testing Put the testers mind at the center of testing. Learn to deal with complexity and ambiguity. Learn to tell a compelling testing story. Develop testing skills through practice, not just talk. Use heuristics to guide and structure your process. Be a service to the project community, not an obstacle. Consider cost vs. value in all your testing activity. Diversify your team and your tactics. Dynamically manage the focus of your work. Your context should drive your choices, both of which evolve over time. 21
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Themes of Rapid Testing A Diversified Strategy Minimizes Tunnel Vision le isib ms y ou inv ble lems ith your pro Prob nd w fi can s… e bias le isib ms inv ble pro 22
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Themes of Rapid Testing Assess Testing Cost vs. Value. No! More Work & Time ybe ma (Cost) Yes! Better Thinking & Better Testing (Value) For a video lecture, see: http://www.satisfice.com/bbst/videos/QualityCost.mp4 23 VALUE Coverage. The value of the elements of the product that are exercised by the test. Power. The probability that if a problem exists in the area covered, the test will reveal it. Integrity. The degree to which the test can be trusted, because its results are valid and consistent across multiple executions of that test. Technical Necessity. The degree to which the information we want to discover exists; and the degree to which that information matters. Business Necessity. The degree to which a test is expected to be performed by your clients. COSTS (remember that time = money) Opportunity Cost. Performing it may prevent you from doing other tests. Development. The cost of getting ready to perform the test. Execution. The cost of performing it as needed. Transfer. The cost of getting someone else ready to run that test. Maintenance. The cost of keeping it running. Accounting. The cost of explaining the test, justifying it, show that it was performed.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. How does Rapid Testing compare with other kinds of testing? When testing is turned into an Management likes to talk elaborate set of rote tasks, about exhaustive testing, but it becomes ponderous without they don’t want to fund it and really being thorough. they don’t know how to do it. More Work & Time Exhaustive Rapid testing may Ponderous Slow, very expensive, not be exhaustive, Slow, expensive, and difficult but it is thorough (Cost) and easier enough and quick enough. It’s less work Slapdash Rapid than ponderous Faster, less expensive, testing. It might be Much faster, cheaper, less work than still challenging and easier slapdash testing. You can always test It fulfills the mission quickly... Better Thinking & Better Testing of testing. But it might be poor (Value) testing. 24 There are big differences between rapid testing and testing that is merely fast. Testing can be fast, but might be shallow or sloppy or ill-considered or unprofessional. If so, we call it slapdash testing. Many organizations start with slapdash testing. It might be wonderful to test products exhaustively or completely or comprehensively. However, we can never test a product completely. To do all the testing that we could possibly imagine would take an enormous amount of time—and even if we had that time available, management wouldn’t want to pay for all of the resources that it would take. Even though the value of exhaustive testing might be high, the prohibitive cost wouldn’t justify the value. Still, there is a perception that exhaustive testing is a good thing, so people sometimes try to achieve it by adding work and time to the testing process. When exhaustive testing is carelessly scaled down, or slapdash testing is made to appear rigorous by piling on mounds of badly designed documentation, it becomes ponderous. It’s expensive, and it’s slow, but it isn’t very good testing. Rapid testing is as exhaustive as it needs to be, and no more.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. KEY IDEA 25
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Test Project Context Where are you and what are you doing there? Mission Find Important Problems Advise about QA Assess Quality Advise about Testing Certify to Standard Advise about Quality Fulfill Process Mandates Maximize Efficiency Satisfy Stakeholders Minimize Cost Assure Accountability Minimize Time Development Requirements Product Product Mission Project Lifecycle Stakeholders Project Management Configuration Management Test Quality Criteria Reference Material Defect Prevention Process Development Team Strategy Logistics Work-products Test Team Test Lab Expertise Test Platforms Loading Test Tools Cohesion Test Library Motivation Problem Tracking System Leadership Office Facilities Project Integration 26
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Modern Project Cycle Project Start “Code Freeze” Alpha Beta Beta Beta Release to Ongoing Requirements Definition Manufacturing Incremental Design Design... Incremental Coding Testing and Retesting 27
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Rapid Testing Starts Right Away Cycles of Work with Ongoing Status Reports •Getting ready to test •Designing & doing tests •Processing test results Do a burst of START testing Make sense of Focus on what Report needs doing your status Compare status STOP against mission 28
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Test Cycle Ship it! Quality Typical Quality Growth Time 29 Near the beginning of the project, rapid testing is often a process of very rapid cycles of determining that there are major problems. Our focus is mostly on learning and developing the infrastructure for better testing, later on. In the middle of the project, we are find a lot of problems. We feel productive. It’s near the end of the project that rapid testing becomes very important, because we have very little time to discover whether the product has been seriously degraded by that most recent change. A year long project can come down to an hour or two of retesting at the end. This is where rapid testing skill becomes very important.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Do You Test Under Time Pressure? You have less time than you’d like. Faster is better. Your plans are often interrupted. You are not fully in control of your time. What is this like? 30 When you’re watching a televised professional team sport, have you noticed that there are no Gantt charts on the sidelines? Soccer teams, baseball teams, and yacht racing teams don’t manage by Gantt charts. That’s because they have to respond in real time to the conditions set up by the other team, and that can’t be predicted or charted.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Excellent Rapid Technical Work Begins with You When the ball comes to you… Do you know you have the ball? Can you receive the pass? Do you know your options? Do you know what your Is your equipment ready? role and mission is? Can you read the Do you know where situation on the field? your teammates are? Are you aware of the Can you let your teammates help you? criticality of the situation? Are you ready to act, right now? 31 A good sports team has to depend on the skills of the players, rather than on rote procedures. Professional sports teams begin with the skills of each player, and then work outwards from that. Skills get practiced offline, but every game is part of the practice too. Strategy matters, plans matter, techniques matter, but the focus is on the player to choose the appropriate techniques and to execute them skillfully.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. …but you don’t have to be great at everything. Rapid test teams are about diverse talents cooperating − We call this the elliptical team, as opposed to the team of perfect circles. − Some important dimensions to vary: − Technical skill − Domain expertise − Temperament (e.g. introvert vs. extrovert) − Testing experience − Project experience − Industry experience − Product knowledge − Educational background − Writing skill − Diversity makes exploration far more powerful − Your team is more powerful because of your unique individual contribution 32
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Skill vs. Alternatives supervision The greater the skill of the tester, the less prescribed procedure, supervision, or favorable environment is required. skill environment procedure 33 How do we deal with people who don’t have the necessary skills? Set an Undemanding Context: We can give them a supportive environment by giving them easy problems to solve, easily tested technology, users with low quality standards, management that doesn’t worry about quality or liability, or developers that don’t put bugs in the software. The quality control school argues that you don’t need to test, or that testing should be super-easy. This is how unskilled tester survive; if they’re not given hard problems to solve, it doesn’t matter if they’re skilled. This is the approach that the quality control school takes. Assign Rote Tasks: We can give unskilled testers comprehensive procedures to follow. We can give them step-by-step procedures to follow, and we can turn testing into more of a clerical task than a bureaucratic one. This is the approach that the factory school takes. Give Personal Support: We can personally guide a tester. In the positive view, we can give a less- experienced tester hands-on mentoring and coaching and relax the supervision as he gains experience and skill. The pathological version of this is to supervise at a distance, or to guide the tester so thoroughly that we end up doing all his thinking for him. But we prefer to train testers. It’s cheaper, easier, and the more skill they have, the less of these other things are needed.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. What might be in a tester’s head? 34
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. KEY IDEA 35
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Sphere Welcome to Epistemology the study of knowledge Epistemology is the study of how we know what we know. The philosophy of science and the theory of testing comes from Epistemology. All good scientists and testers study Epistemology. 36 Why do we talk about Epistemology? Just in case you want to become a true expert in testing. Reading about Epistemology can be a heady, theoretical undertaking. It’s something you do in a café while sipping espresso. Maybe that’s not for you, but if you’re the kind of person who likes to have a deep understanding of your work, get thee to a Starbucks and start reading. Even if you read five pages and stop, it will exercise your mind in important ways. What specifically should you read? We would recommend The Structure of Scientific Revolutions, by Kuhn, or the first 30 pages of Conjectures and Refutations, by Karl Popper. Or try Proofs and Refutations, by Imre Lakatos. These are all books about how scientists come up with theories and know if they are valid.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Test Project Epistemology 101 Spot traps and avoid them. The essential purpose served Mission by the strategy. What we do in this situation Strategy to accomplish the mission. The conditions under which Situation at Hand we must work. 37
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Test Project Epistemology 101 Spot traps and avoid them. Unachievable Strategies Mission Review assumptions. Check for overlooked resources. Strategy Consider radical shortcuts. can’t work Do something now. Share the problem. Make a counteroffer. Situation won’t Promise good faith effort, support strategy not unconditional success. 38
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Scientific Skills posing useful questions observing what’s going on describing what you perceive thinking critically about what you know recognizing and managing bias designing hypotheses and experiments thinking despite already knowing analyzing someone else’s thinking reasoning about cause and effect treating “facts” as merely what we believe we know as of this moment 39
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Testing is about questions… posing them and answering them Product Tests − What is this product? − What would constitute a diversified − What can I control and observe? and practical test strategy? − What should I test? − How can I improve my understanding of how this product works? Problems − If there were an important problem − What quality criteria matter? here, how would I uncover it? − What kinds of problems might I − What document to load? Which button find in this product? to push? What number to enter? − Is what I see a problem? Why? − How powerful is this test? − How important is this problem? − What have I learned from this test that Why should it be fixed? helps me perform powerful new tests? − What just happened? How do I Sometimes stating an examine that more closely? assumption is a useful proxy for asking a question. 40
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. ..also, explanations and predictions. what happened, what might happen, and why 1. Collect data. 2. Find several explanations that account for the data. 3. Find more data that is either consistent or inconsistent with explanations. 4. Choose the best explanation that accounts for the important data, or keep searching. This is called abductive inference. To do it well, you must consider a variety of possible explanations. When you think you know what something is, stop and ask, “What else could it be?” 41
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Confronting Complexity: General Systems Thinking We must be able to simplify things without oversimplifying them. We must know when to say: − “it worked the same way” even when there are many differences. − “that’s not the same bug” or “that’s a different test”, even when there are many similarities. − “that quality is worse, now” or “the product improved, significantly”, even when we have made only a handful of observations. GST is about the relationship between simplicity and complexity, especially in dynamic and open systems. Testers practice GST when we confront complex systems and try to test them with relatively simple tests. 42 General Systems Thinking has been called the science of simplification (Weinberg, 1974). It is concerned with the study of complex and open systems in general, rather than with any specific system. The aim of GST is to help us understand the general relationships between systems and our simplified models of them, so that we can better observe, learn about, interact with and build specific systems. I think the basic textbook of testing is Introduction to General Systems Thinking, by Gerald M. Weinberg. This book doesn’t specifically talk about software testing, but everything in it pertains to what testers do and how they think.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Magic Magic Tricks are Like Programs Our thinking is limited − We misunderstand probabilities Magic tricks work for the same reasons − We use the wrong heuristics that bugs exist − We lack specialized knowledge − We forget details − We don’t pay attention to the right things Studying magic can help you develop The world is hidden the imagination − states to find better bugs. − sequences − processes Testing magic is − attributes indistinguishable from testing sufficiently − variables advanced technology − identities 43
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Observation vs. Inference Observation and inference are easily confused. Observation is direct sensory data, but on a very low level it is guided and manipulated by inferences and heuristics. You sense very little of what there is to sense. You remember little of what you actually sense. Some things you think you see in one instance may be confused with memories of other things you saw at other times. It’s easy to miss bugs that occur right in front of your eyes. It’s easy to think you “saw” a thing when in fact you merely inferred that you must have seen it. 44
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Observation vs. Inference Accept that we’re all fallible, but that we can learn to be better observers by learning from mistakes. Pay special attention to incidents where someone notices something you could have noticed, but did not. Don’t strongly commit to a belief about any important evidence you’ve seen only once. Whenever you describe what you experienced, notice where you’re saying what you saw and heard, and where you are instead jumping to a conclusion about “what was really going on.” Where feasible, look at things in more than one way, and collect more than one kind of information about what happened (such as repeated testing, paired testing, loggers and log files, or video cameras). 45 Our colleague, Jeremy Kominar, is an expert amateur magician. We asked him to provide us with some remarks on the links between software testing and magic. Here’s an edited version of what he had to say--we can’t quote him exactly, because he asked us not to give away his secrets! •Learning to perform magic causes the magician to be aware of not only his own perspective, but also (and more importantly) others’ perception of the trick. This becomes invaluable for testers because it helps us to recognize that the interests and perspectives of many stakeholders must be considered when testing software. The trick may “look good” or “work” from your angle, but there are always other angles to cover. •When learning to perform magic tricks, you generally want to simplify things--minimize the number of moves and figure out how you can reach the same end result without as many steps to get there. In a testing context this idea can be used for creating different usage scenarios—taking multiple paths and variables to attain the same goal. (We might want to minimize the number of steps, but we might also want to vary the paths to shake out more bugs.) •Learning to perform magic and learning to figure out magic tricks require us to develop observational skills, and to recognize things that deceive us. Most non-magicians fail to consider possibilities that come easily to magicians; devices, coins, or cards can be gimmicked. Non-magicians only think about normal cards. You need to think outside of the box to be a magician or to figure out how the magician does his work. This kind of reasoning and deduction are key assets to being a tester. There isn’t really such a thing as magic but there clearly is such a thing as deception. As the magician becomes more experienced and more wise to the practice, he should be able to reverse-engineer tricks by using patterns that he already knows about magic. Once you’ve seen one kind of trick, similar tricks aren’t so mysterious because you have heuristics that you can use to recognize how it’s done.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Screen Wine Glass Wason Models Link Observation and Inference A model is an idea, activity, or object… such as an idea in your mind, a diagram, a list of words, a spreadsheet, a person, a toy, an equation, a demonstration, or a program …that represents another idea, activity, or object… such as something complex that you need to work with or study. …whereby understanding the model may help you understand or manipulate what it represents. - A map helps navigate across a terrain. - 2+2=4 is a model for adding two apples to a basket that already has two apples. - Atmospheric models help predict where hurricanes will go. - A fashion model helps understand how clothing would look on actual humans. - Your beliefs about what you test are a model of what you test. 46
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Thinking Like A Tester: Seeing the Words that Aren’t There Among other things, testers question premises. A suppressed premise is an unstated premise that an argument needs in order to be logical. A suppressed premise is something that should be there, but isn’t… (…or is there, but it’s invisible or implicit.) Among other things, testers bring suppressed premises to light and then question them. A diverse set of models can help us to see the things that “aren’t there.” 47
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Thinking Like A Tester: Spot the missing words! “I performed the tests. All my tests passed. Therefore, the product works.” “The programmer said he fixed the bug. I can’t reproduce it anymore. Therefore it must be fixed.” “Microsoft Word frequently crashes while I am using it. Therefore it’s a bad product.” “Nobody complained about that bug after we shipped it. Therefore it wasn’t important.” 48
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Heuristics bring useful structure to problem-solving skill. adjective: “serving to discover.” noun: “a fallible method for solving a problem or making a decision.” “Heuristic reasoning is not regarded as final and strict but as provisional and plausible only, whose purpose is to discover the solution to the present problem.” - George Polya, How to Solve It 49
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Types of Heuristics Guideword Heuristics: Words or labels that help you access the full spectrum of your knowledge and experience as you analyze something. Trigger Heuristics: Ideas associated with an event or condition that help you recognize when it may be time to take an action or think a particular way. Like an alarm clock for your mind. Subtitle Heuristics: Help you reframe an idea so you can see alternatives and bring out assumptions during a conversation. Heuristic Model: A representation of an idea, object, or system that helps you explore, understand, or control it. Heuristic Procedure or Rule: A plan of action that may help solve a class of problems. 50
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Some Everyday Heuristics It’s dangerous to drink and drive. A bird in hand is worth two in the bush. Nothing ventured, nothing gained. Sometimes people stash their passwords near their computers. Try looking there. Stores are open later during the Holidays. If your computer is behaving strangely, try rebooting. If it’s very strange, reinstall Windows. If it’s a genuinely important task, your boss will follow-up, otherwise, you can ignore it. 51
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. How Heuristics Differ from Other Procedures or Methods A heuristic is not an edict. Heuristics require guidance and control of skilled practitioner. Heuristics are context-dependent. Heuristics may be useful even when they contradict each other– especially when they do! Heuristics can substitute for complete and rigorous analysis. 52
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. KEY IDEA For a video lecture, see: http://www.satisfice.com/bbst/videos/BBSTORACLES.mp4 53
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Oracles An oracle is the principle or mechanism by which you recognize a problem. “..it works” “...it appeared at least once to meet some requirement to some degree.” 54 Jerry Weinberg has suggested that “it works” may mean “We havent tried very hard to make it fail, and we havent been running it very long or under very diverse conditions, but so far we havent seen any failures, though we havent been looking too closely, either.” In this pessimistic view, you have to be on guard for people who say it works without checking even once to see if it could work.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Learn About Oracles Without an oracle you cannot recognize a problem If you think you see a problem, you must be using an oracle. 55
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Consistency (“this agrees with that”) an important theme in oracles History: The present version of the system is consistent with past versions of it. Image: The system is consistent with an image that the organization wants to project. Comparable Products: The system is consistent with comparable systems. Claims: The system is consistent with what important people say it’s supposed to be. Users’ Expectations: The system is consistent with what users want. Product: Each element of the system is consistent with comparable elements in the same system. Purpose: The system is consistent with its purposes, both explicit and implicit. Statutes: The system is consistent with applicable laws. Familiarity: The system is not consistent with the pattern of any familiar problem. Consistency heuristics rely on the quality of your models of the product and its context. 56 Note that we’re saying “system” here, instead of “product”. A product could behave perfectly as a part of some system, but when something else within the system changes, that change could undermine the product’s usefulness or perceived quality. Seeing the product as a component or participant of some larger system opens our eyes to a greater range of potential problems.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Word Save File Notepad Save File Diskmapper All Oracles Are Heuristic We often do not have oracles that establish a definite correct or incorrect result, in advance. That’s why we use abductive inference. No single oracle can tell us whether a program (or a feature) is working correctly at all times and in all circumstances. That’s why we use a variety of oracles. Any program that looks like it’s working, to you, may in fact be failing in some way that happens to fool all of your oracles. That’s why we proceed with humility and critical thinking. You (the tester) can’t know the deep truth about any result. That’s why we report whatever seems likely to be a bug. 57
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Coping With Difficult Oracle Problems Ignore the Problem − Ask “so what?” Maybe the value of the information doesn’t justify the cost. Simplify the Problem − Ask for testability. It usually doesn’t happen by accident. − Built-in oracle. Internal error detection and handling. − Lower the standards. You may be using an unreasonable standard of correctness. Shift the Problem − Parallel testing. Compare with another instance of a comparable algorithm. − Live oracle. Find an expert who can tell if the output is correct. − Reverse the function. (e.g. 2 x 2 = 4, then 4/2 = 2) Divide and Conquer the Problem − Spot check. Perform a detailed inspection on one instance out of a set of outputs. − Verify by parts. Check specific, risky aspects of the output, instead of everything. − Blink test. Compare or review overwhelming batches of data for patterns that stand out. − Easy input. Use input for which the output is easy to analyze. − Easy output. Some output may be obviously wrong, regardless of input. − Unit test first. Gain confidence in the pieces that make the whole. − Test incrementally. Gain confidence by testing over a period of time. 58
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. “Easy Input” Fixed Markers. Use distinctive fixed input patterns that are easy to spot in the output. Statistical Markers. Use populations of data that have distinguishable statistical properties. Self-Referential Data. Use data that embeds metadata about itself. (e.g. counterstrings) Easy Input Regions. For specific inputs, the correct output may be easy to calculate. Outrageous Values. For some inputs, we expect error handling. Idempotent Input. Try a case where the output will be the same as the input. Match. Do the “same thing” twice and look for a match. Progressive Mismatch. Do progressively differing things over time and account for each difference. (code-breaking technique) 59 Fixed Markers. Some programmers use the hex values 0xDEADBEEF as a flag to indicate uninitialized data. Statistical Markers. If you create input that has a specific statistical property, then you may be able to detect missing or corrupted data by examining the statistical properties of the output. Self-referential data. Counterstrings are strings that identify their lengths. *3*5*7*9*12*15* is an example of a counterstring. (Satisfice’s PERLCLIP tool creates counterstrings of arbitrary lengths.) Another example is using JAMES_FIRSTNAME and BACH_LASTNAME as sample data in the first name and last name fields of a database. Easy Input Regions. A number times one is always itself; a number plus zero is always itself; a multiple of ten always ends in 0, and so on. Outrageous values. Examples include letters in fields that expect only numbers, extremely long strings in fields that expect short ones, huge numbers where modest ones are expected, or zero or null values where some positive value is required. Each example should trigger some kind of error handling. Idempotent input. Take some data and process it in some way, such that the output from the first run can be used as input for subsequent runs but shouldn’t go through additional changes. Examples include spell- checking (after all the corrections have been accepted for the first run, no further changes should be suggested) or code formatters (after the code has been formatted once, the same code should pass through the process unchanged). Match. Given the same input data and the same function on that data, we would typically expect the output to be the same. This approach is sometimes used to check various kinds of encoding or checksums. Progressive mismatch. Check for consistency in a function by varying the input in some limited or controlled way, and observe the effect on the output. This is an exploratory approach, sometimes used as a code-breaking technique.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Quality Criteria Capability Compatibility Reliability Supportability Usability Testability Security Maintainability Scalability Portability Performance Localizability Installability Many test approaches focus on Capability (functionality) and underemphasize the other criteria 60 Try pronouncing CRUSSPIC STMPL; sometimes we think of it as the name of an Eastern European hockey player, but however you think of it, it makes this list easier to remember. The point is to be able to cycle through the list at any time to ask a new question about something that we might value in the product, or something that might represent a vulnerability if it were missing or broken.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. KEY IDEA 61
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Coverage Product coverage is the proportion of the product that has been tested. Structure Function Performance Maintainability Capability Portability Data Installability Reliability Localizability Compatibility Platform Usability Supportability Operations Security Testability Scalability Time 62
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Product Elements: Structural Coverage input output Test what it’s made of. platform Print testing example − Files associated with printing − Code modules that implement printing − Code statements inside the modules − Code branches inside the modules
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Product Elements: Functional Coverage input functions functions output Test what it does. platform Print testing example − Print, page setup and print preview − Print range, print copies, zoom − Print all, current page, or specific range
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Product Elements: Data Coverage functions input & output Test what structure it does it to. platform Print testing example − Types of documents − Items in documents, size and structure of documents − Data about how to print (e.g. zoom factor, no. of copies)
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Product Elements: Platform Coverage functions input & output Test what it structure depends upon. platform Print testing example − Printers, spoolers, network behavior − Computers − Operating systems − Printer drivers
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Product Elements: Operations Coverage input output Test how it’s used. platform Print testing example − Use defaults − Use realistic environments − Use realistic scenarios − Use complex flows
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Profiler Product Elements: Time Coverage Test how input output it’s affected by time. platform Print testing example − Try different network or port speeds − Print one document right after another, or after long intervals − Try time-related constraints--spooling, buffering, or timeouts − Try printing hourly, daily, month-end, and year-end reports − Try printing from two workstations at the same time − Try printing again, later.
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Sometimes your coverage is disputed… “No user would do that.” “No user I can think of, who I like, would do that on purpose.” Who aren’t you thinking of? Who don’t you like who might really use this product? What might good users do by accident? 69
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Sometimes it’s really hard to cover… Ask for testability! Controllability Scriptable Observability Interface! Availability Log files! Simplicity Stability Information 70
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. KEY IDEA 71
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Exploratory Testing Is… an approach to software testing… (applicable to any test technique) that emphasizes the personal freedom and responsibility of each tester to continually optimize the value of his work… (optimize how?) by treating learning, test design and test execution as mutually supportive activities that run in parallel throughout the project. 72
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. IP Address ET is a Structured Process Exploratory testing, as I teach it, is a structured process conducted by a skilled tester, or by lesser skilled testers or users working under supervision. The structure of ET comes from many sources: − Test design heuristics − Chartering − Time boxing In other words, − Perceived product risks it’s not “random”, − The nature of specific tests but systematic. − The structure of the product being tested − The process of learning the product − Development activities − Constraints and resources afforded by the project − The skills, talents, and interests of the tester − The overall mission of testing 73
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. ET is a Structured Process In excellent exploratory testing, one structure tends to dominate all the others: Exploratory testers construct a compelling story of their testing. It is this story that gives ET a backbone. 74
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. To test is to compose, edit, narrate, and justify a story. You must tell a story about the product… …about how it failed, and how it might fail... …in ways that matter to your various clients. But also tell a story about testing… …how you configured, operated and observed it… …about what you haven’t tested, yet… …or won’t test, at all… …and about why what you did was good enough. 75
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. The Process of Test Design Testing Story - Test Plan/Report - Work Products - Status Informs Produces Question Knowledge Analysis Experiment - Product Story - Risk Test - Technical Knowledge - Coverage Procedure - Domain Knowledge - Oracles - Resources/Constraints - Configure - General Knowledge - Value/Cost - Operate - Bugs Investigate - Observe bugs - Evaluate Answer Produces Informs 76
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. How to Start? Pick a Useful, Fun, or Easy Starting Point. Testing Story What do I need to produce for my client? What has already been done? Informs Produces Knowledge Analysis Experiment What do I know? What kind of testing Do I have a product? What do I need to know? am I good at? What can I do with What obstacles threaten the product? my success? Produces Informs 77
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Test Design Testing to Learn vs. Testing to Search Testing (primarily) to Learn − Forming a mental model of the product. − Learning what the product is supposed to do. − Inventorying the product elements you may want to test. − Looking at consistency relationships and trying various oracles. − Generating test data. − Considering testability and trying potentially useful tools. − Experimenting with different ways of testing it. − Reporting bugs that you find. Testing (primarily) to Search − Using your detailed product knowledge, and any relevant tools, to systematically exercise the product in every important way. − Using careful observation and good oracles to detect important bugs. − Modifying and diversifying your tests to make them more powerful. − Reporting bugs that you find. 78
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Test Design Testing to Learn vs. Testing to Search Compare these Situations: L S − Starting a new project. L S − Seeing a new feature for the first time. L S − Testing a product deeply to reveal important bugs. L S − Investigating a particular bug. L S − Re-testing a product after a change. S − Repeated execution of detailed procedural test scripts. 79
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Lyndsay Machine High Learning ET: Explore These Things Composition − Affordances: Interfaces to the product. − Dimensions & Variables: Product space and what changes within it. − Relationships & Interactions: functions that cooperate or interfere. − Navigation: Where things are and how to get to them. Conformance − Benefits: What the product is good for-- when it has no bugs in it. − Consistencies: Fulfillment of logical, factual, and cultural expectations. − Oracles: Specific mechanisms or principles by which you can spot bugs. − Bugs and Risks: Specific problems and potential problems that matter. Context (of the Product) − History: Where the product has been, and how it came to be. − Operations: Its users and the conditions under which it will be used. Conditions (of Testing) − Attitudes: What your clients care about and what they want from you. − Complexities & Challenges: Discover the hardest things to test. − Resources: Discover tools and information that might help you test. 80
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Contrasting Approaches In scripted testing, tests are first designed and recorded. Then they Test may be executed at some later time Scripts or by a different tester. In exploratory testing, tests are Test Ideas designed and executed at the same time, and they are not Product necessarily recorded, but may be. 81
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Contrasting Approaches Scripted testing is about Test controlling test execution. Scripts Exploratory testing is about Test Ideas improving test design. Product 82
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Contrasting Approaches Scripted testing is like making a prepared speech. Test It is guided by pre-conceived ideas. Scripts Exploratory testing is like Test Ideas having a conversation. It is self-guided. Product 83
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Mixing Scripting and Exploration freestyle exploratory pure scripted fragmentary vague scripts test cases charters roles When I say “exploratory testing” and don’t qualify it, I mean anything on the exploratory side of this continuum. 84
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Blending Scripted & Exploratory Generic scripts: specify general test procedures and apply them to different parts of a test coverage outline. Vague scripts: specify a test step-by-step, but leave out any detail that does not absolutely need to be pre-specified. Improvisation: have scripts, but encourage deviation from them, too. Fragmentary cases: specify tests as single sentences or phrases. Test Coverage Outline: use outline of product elements and have tester construct tests from it on the fly. Risk Catalog: specify types of problems to look for, then construct tests on the fly to find each one. Exploratory Charters: specify 90 minutes of testing in two sentences or less. Roles: Give each tester a standing role to test a certain part of the product. Leave the rest up to them. Heuristics: Train exploratory testers to use standardized test design heuristics. SBTM: Consider using Session-Based Test Management, a formalized method of exploratory test management. (http://www.satisfice.com/sbtm). 85
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Test Design and Execution Achieve excellent test design by exploring different test designs while actually testing Product Test Ideas Guide testers with personal supervision and concise documentation of test ideas. Meanwhile, Product train them so that they can guide themselves and or spec be accountable for increasingly challenging work. 86
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Allow some disposable time Self-management is good! How often do you account for your progress? If you have any autonomy at all, you can risk investing some time in − learning − thinking − refining approaches − better tests 87
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. Allow some disposable time If it turns out that you’ve made a bad investment…oh well ☺ If it turns out that you’ve made a good investment, you might have − learned something about the product − invented a more powerful test − found a bug − done a better job − surprised and impressed your manager 88
Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc. “Plunge in and Quit” Heuristic Whenever you are called upon to test something very complex or frightening, plunge in! After a little while, if you are very confused or find yourself stuck, quit! This benefits from disposable time– that part of your work not scrutinized in detail. Plunge in and quit means you can start something without committing to finish it successfully, and therefore you don’t need a plan. Several cycles of this is a great way to discover a plan. 89