More Related Content
What's hot
What's hot (20)
Similar to How did i miss that bug rtc
Similar to How did i miss that bug rtc (20)
More from GerieOwen
More from GerieOwen (9)
Recently uploaded
Recently uploaded (20)
How did i miss that bug rtc
- 1. How Did I Miss That Bug? Overcome Cognitive Bias In Testing Gerie Owen & Peter Varhol
- 2. Presenters Gerie Owen gerie@gerieowen.com • Quality Assurance Consultant • Speaker and Blogger on Testing topics • Experienced Tester: • Bug Finder and Bug Misser Peter Varhol peter@petervarhol.com • International speaker on technology topics • Former product manager and university professor
- 3. Meet Our Cats... Max, Stephanie & Monica
- 4. Presentation Outline • Agenda • Why are we talking about missed bugs? • How do we miss bugs? • System 1 and System 2 thinking • Cognitive Biases Affecting Testing • Managing Cognitive Bias for • Testers • Test Leads and Managers • Our Profession
- 5. Why are we talking about missing bugs? • Have you ever missed a bug? • Have you ever been asked how you missed a bug? • Have you ever wondered how you missed a bug?
- 6. Consequences of Missed Bugs • Possible Consequences of Missed Bugs: • Negative Publicity • Lost Sales • Lost Customers • Even Loss of Life MISSED BUGS CAUSE MAYHEM
- 7. My Journey The “HOW” is more important than the “WHY” And now , I invite you to join with me into the journey of How Did I Miss that Bug?
- 8. How Do We Miss Bugs? • Missed test cases • Misunderstanding of requirements • Misjudgment in risk-based testing • Inattention • Fatigue • Burnout • Multi-tasking
- 9. How Do We Test? • What is Software Testing? • Software testing is making judgments about the quality of the software under test • Involves: • Objective comparisons of code to specifications, • AND • Subjective assessments regarding usability, functionality etc
- 10. What IS a Missed Bug? An Error in Judgment! To determine how testers miss bugs, we need to understand how humans make judgments, especially in complex situations.
- 11. How do we make judgments? • Thinking, Fast and Slow – Daniel Kahneman • System 1 thinking – fast, intuitive, and sometimes wrong • System 2 thinking – slower, more deliberate, more accurate
- 12. System 1 vs. System 2 Thinking • System 1 thinking keeps us functioning • Fast decisions, usually right enough • Gullible and biased • System 2 makes deliberate, thoughtful decisions • It is in charge of doubt and unbelieving • But is often lazy • Difficult to engage
- 13. How Do We Apply System 1 and System 2 Thinking? • System 1 thinking: • Is applied in our initial reactions to situations. • May employ Heuristics or rules of thumb • System 2 thinking: • Is applied when we analyze a problem, for example when calculating the answer to a math problem. • System 1 and System 2 can be in conflict: • lead to biases in decision-making.
- 14. Consider This Problem • A bat and ball cost $1.10 • The bat cost one dollar more than the ball • How much does the ball cost?
- 15. How do Biases Impact Testing? • We maintain certain beliefs in testing practice • Which may or may not be factually true • Those biases can affect our testing results • We may be predisposed to believe something that affects our work and our conclusions • How do bias and error work? • We may test the wrong things • Not find errors, or find false errors
- 17. The Representative Bias • Happens when we judge the likelihood of an occurrence in a particular situation by how closely the situation resembles similar situations. • Testers may be influenced by this bias when designing data matrices, perhaps not testing data in all states or not testing enough types of data. • Case Study: Ability to print more than once bug
- 18. The Curse of Knowledge • Happens when we are so knowledgeable about something, that our ability to address it from a less informed, more neutral perspective is diminished. • When testers develop so much domain knowledge that they fail to test from the perspective of a new user. Usability bugs are often missed due to this bias. • Case Study: Date of Death Bug
- 19. The Congruence Bias • The tendency of experimenters to plan and execute tests on just their own hypotheses without considering alternative hypotheses. • This bias is often the root cause of missed negative test cases. Testers write test cases to validate that the functionality works according to the specifications and neglect to validate that the functionality doesn’t work in ways that it should not. • Case Study: Your negative test case or boundary miss
- 20. The Confirmation Bias • The tendency to search for and interpret information in a way that confirms one’s initial perceptions. • Testers’ initial perceptions of the quality of code, the quality of the requirements and the capabilities of developers can impact the ways in which they test. • Case Study: Whose code to you test the most thoroughly?
- 21. The Anchoring Effect • The tendency to become locked on and rely too heavily on one piece of information and therefore exclude other ideas or evidence that contradicts the initial information. • Software testers do this often when they validate code to specifications exclusively without considering ambiguities or errors in the requirements.
- 23. Inattentional Blindness • Chabris and Simon conducted experiments on how focusing on one thing makes us blind to others • Invisible gorilla on the basketball court • Images on a lung x-ray
- 24. Inattentional Blindness • a psychological lack of attention • the tendency to miss obvious inconsistencies when focusing specifically on a particular task. • This happens in software testing when testers miss the blatantly obvious bugs
- 25. Why Do We Develop Biases? • The Blind Spot Bias • We evaluate our own decision-making process differently than we evaluate how others make decisions. • West, Meserve and Stanovich
- 26. How Does This Apply To Testing? •We must manage the way we think throughout the test process. • As individual testers • As test managers • As a professional community
- 27. How Can Testers Manage Their Thought Processes? •Use more System 1 thinking? OR •Use more System 2 thinking?
- 28. Test Methodology and System 2 Thinking • Test methodology is the analytical framework of testing; it invokes our system 2 thinking and places the tester under cognitive load. • The determination of whether the actual results match the expected results becomes an objective assessment.
- 29. How Do We Find Bugs? Focus on System 1 thinking, intuition and emotion
- 30. Focus On System 1 Thinking • Heuristics used with Oracles • Recognize our emotions as indicators of potential bugs • Exploratory Testing
- 31. The Power of Exploratory Testing • Exploratory testing is simultaneous learning, test design, and test execution • Exploratory testers often use tools • record of the exploratory session • generate situations of interest
- 32. The Characteristics of Exploratory Testing • Planned • Learning experience • Discovery process • Different for each application
- 33. How Should We Use Exploratory Testing? • Unstructured • Before beginning test case execution • Minimizes preconceived notions about the application under test • Oracle based • Users’ perspectives • Data flow
- 34. How Should We Use Exploratory Testing? • Structured • Use to create additional test cases • May be done earlier, possible as modules are developed • Session-Based • Time-boxed charters • Multiple testers • Post test review session
- 35. Planning • What are situations of interest? • Usual tasks performed by the user • Things that a user might do • Things covered by charter • Document the plan • It’s not ad hoc testing • Deviate from the plan as necessary
- 36. Learning Experience • What can I learn about the software? • What all the buttons and forms do • How it wants the user to work • Strengths and weaknesses
- 37. Discovery • What are situations of interest? • Usual tasks performed by the user • Things a user might do • Does it behave as expected? • If not, lets explore • And document
- 38. What Can Test Managers Do? • Foster an environment in which the testers feel comfortable and empowered to use System 1 thinking. • Plan for exploratory testing in the test schedule • Encourage Testers to take risks • Reward for Quality of bugs rather than quantity of test cases executed
- 39. What Can the QA Profession Do? A Paradigm Shift • Shift our focus from requirements coverage based test execution to a more intuitive approach • Exploratory testing and business process flow testing becomes the norm rather than the exception • Develop new testing frameworks where risk-based testing is executed through targeted exploratory testing and is balanced with scripted testing • Our purpose should be providing information versus finding bugs
- 40. Question Test Results • Is there any reason to suspect we are evaluating our test results based on self-interest, overconfidence, or attachment to past experiences? • Have we fallen in love with our test results? • Were there any differences of opinion among the team reviewing the test results?
- 41. How Do We Find The Obvious Bugs? • Focus less • Use intuition • Believe what we can’t believe
- 43. Questions?
Editor's Notes
- And so in most organizations, each time a bug, as tiny and insignificant as it may be, crawls into production, mayhem of magnanimous proportions ensues. And sometimes, the focus on finding out why it happened takes priority over the fix. In the name of continuous improvement, we begin the root cause analysis. Root cause analysis can take many forms. In some organizations, it is used effectively to make process improvements. In other organizations, it amounts to a witch hunt, the sole purpose of which is to assign blame. I worked in an organization where the art of the witch hunt and assignment of blame was developed to the level of a science. All bugs escaping into user acceptance testing or production were immediately analyzed to determine root cause, i.e., code, requirements, missed test case, etc. If the root cause was determined to be a tester miss, this was also noted in the test management tool. Metrics were developed to track missed bugs and testers were effectively pulverized for missing bugs. Test leads dreaded the root cause analysis process and testers worked in fear of missing bugs. As a test lead in this environment, I really wanted to help my test teams and reduce our bug misses. I started to think about how we missed bugs.
- Test leads dreaded the root cause analysis process and testers worked in fear of missing bugs. As a test lead in this environment, I really wanted to help my test teams and reduce our bug misses. I started to think about how we missed bugs. The more I thought about it, I realized that the “how” is probably more important than the “why”. And I began the journey into How Did I Miss That Bug? I invite you to journey with me.
- But What else? What if we take a step back and examine how we test?
- So then, what is a missed bug?
- Has anyone ever thought about that? Social Scientists Daniel Kahneman and Amos Tversky and Behavioral Psychologist Dan Ariely have. So as I, as a Test Lead and tester was struggling to understand how my teams and I were missing bugs, Peter had begun studying the role of cognition and bias in testing. He discussed this at several conferences in his presentation, “Moneyball and the Science of Building Great Test Teams”. Has anyone seen that presentation? When I saw it, I started to think about how this might apply to my quest to find out how we miss bugs. So now I’ll invite Peter to tell us about the research.
- Daniel Kahneman, in his book Thinking, Fast and Slow, defines two types of thinking. He calls them System 1 and System 2 thinking. These, of course, are models, and don’t have any physical representation in the brain or elsewhere.
- Daniel Kahneman has developed a model that divides thinking into two components, which he calls System 1 and System 2. System 1 is immediate, reflexive thinking that is for the most part unconscious in nature. We do this sort of thinking many times a day. It keeps us functioning in an environment where we have numerous external stimuli, some important, and many not. System 2 is more deliberate thought. It is engaged for more complex problems, those that require mental effort to evaluate and solve. System 2 makes more accurate evaluations and decisions, but it can’t respond instantly, as is required for many types of day-to-day decisions. And it takes effort, which means it can tire out team members.
- How many of you have ever had pilot training? Years ago, in a PA-28 Cherokee 140 like this one, my instructor put me “under the hood” in practicing recovery from unusual attitudes. With the hood down, he put the plane into unusual flying positions from which I had to recover as quickly as possible. When he brought the hood up, I could see only the instrument panel. I rapidly developed a heuristic that enabled me to quickly identify and correct an unusual attitude. In short, I focused on the turn indicator and artificial horizon, and worked to center both of them. My instructor figured out what I was doing, and I did the same thing the next time. My turn indicator and artificial horizon were centered, but I was still losing over 1000 feet a minute! I was stumped. My instructor had “crossed” the controls, leaving me in a slip that my heuristic couldn’t account for. I was worse than wrong; I couldn’t follow through at all once my heuristic failed. I never forgot that experience.
- Many, many biases have been identified, but let’s discuss some of the ones that are most relevant to testing.
- First let’s look at an example. Reading this obviously involves judgment? Can you read it?
- Usability bugs are often missed due to this bias. I tested an eOrder entry application for an annuity product where upon the death of one spouse the surviving spouse could elect to continue the contract rather than annuitize. This election had to be made within six months of the date of death. The developer placed the date of death about eight screens into the application. So, a user could type in over half of the required information only to find out that it was to late to proceed based upon the date of death.
- For example, if the specification is that a field should accept only alpha characters, the tester must also validate that the field does not accept numeric.
- For example, tester’s may test a less experienced developer’s code more thoroughly than a more experienced developer because finding more bugs in the less experienced developer’s code will confirm the tester’s expectations. When I travel, I usually take the express bus to the airport. When I’m leaving early in the morning, I buy my ticket ahead of time, but on this particular trip I didn’t. So here I was at about 5 min of 4 am. While I was waiting a the traffic light, I pulled $20 out of my purse and put it in my pocket. When got to the Logan express, the bus was already there, so I parked and ran with my bag flying behind me, tossed my $20 under the window and asked for a ticket, the cashier said something I didn’t quite understand but gave me the ticket. When I got on the bus, I noticed it was a different color and that it was a senior ticket. So when I gave the $20 and didn’t say anything when he attempted to question it, he interpreted, falsely, that I was a senior citizen and sold me the wrong ticket.
- Play Gorilla Video next
- I was testing a Smart Grid application and the associated devices including programmable thermostats, load control devices etc. I had a test lab with 18 different boxes with these devices and electric meters associated with them. In order to get load on the meters to test the calculations, we used heaters to simulate high usage. The test lab would get up to over 100 degrees. When the electric company would call a “critical event” the devices were show a red signal. When I tested a new release, I totally missed that the critical event didn’t activate the red signal and change the pricing because I was so focused on verifying the bugs fixes turned over in the release that I totally missed this new regression bug. I suppose fatigue due to the extreme heat may have factored in my missing the bug as well.
- An answer to why we aren’t as smart as we think comes from the research of West, Meserve and Stanovich which showed that we can easily identify biases in other’s decision-making process and we tend to believe that others are more susceptible to biases and preconceived notions than ourselves. We actually have a bias about biases!
- Now that we understand why we are so susceptible to biases and preconceived notions, how do we effectively manage the way we think throughout the test process? Although the research shows that we cannot prevent biases and preconceived notions through awareness and understanding, we, as individual testers, can attempt to manage our thought processes as we approach our test projects.
- In their working paper, How Can Decision Making Be Improved, Milkman et al. review the work of several researchers which suggests various ways of invoking system 2 thinking more heavily than system 1 thinking. These include evaluating multiple options simultaneously rather than accepting or rejecting each option individually, making choices when under less cognitive load and making decisions in groups rather than individually. But will an increased focus on system 2 thinking improve testers’ effectiveness at finding bugs? I believe not. In software testing, our test methodology already focuses us and requires us to use system 2
- Our use of requirements coverage matrices, data matrices, and test case execution coverage metrics are the tools of our trade and are designed to prevent us from missing bugs. These tools and processes are very useful in validating that the code is built to specifications and that all requirements have been coded and tested. But will an increased focus on system 2 thinking improve testers’ effectiveness at finding bugs? I believe not.
- However, is validating that the application functions according to specifications mean that the application is bug free? Does 100 percent test coverage with all test cases passed mean that the application works as the business customer intended? We all know that the answer to both of these questions is not necessarily. Now the question becomes, how do we find the bugs that prevent the application from working as the customer intended. We believe this requires us to refocus on system 1 thinking and intuition.
- As much as heuristics are biases in themselves, when used with oracles, the principles for using a particular heuristic, they can be quite valuable in invoking System 2 thinking. For example, if a typical user, or various users if the application will be used by different users for different functions, is our oracle, we might test workflows and find bugs that we might not find by executing our test cases. How many times have you reported a bug to which the developer responds: “The user would never do that!”? Users make decisions about using applications based on look and feel as well as ease of use. Since users and potential customers use their System 1 thinking to make these decisions, it follows that to test effectively, we, as testers, must also use System 1 thinking and anticipate the emotions of the user. There is no better way to perform this testing than to consider our emotions and what they might be telling us about the application under test. For example if we are feeling frustrated and anxious, perhaps there is a performance or usability issue.
- Instead of asking what test they are instructed to run, exploratory testers ask what’s the best test I can perform, right now?
- Exploratory testing is also about learning. The tester learns the application, maybe something about the subject domain, and perhaps a lot about what’s good and bad about the application.
- If you kill a cockroach in your kitchen, do you assume you’ve killed the last bug? Or do you call the exterminator?
- Daniel Kahneman recommends that you ask three questions to minimize the impact of cognitive biases in your decision making. Here’s how they apply to finding bugs.