The Craft of Bug Investigation


Published on

Although many training classes and conference presentations describe processes and techniques meant to help you find bugs, few explain what to do when you find a good one. How do you know what the underlying problem is? What do you do when you find a bug, and the developer wants you to provide more information? How do you reproduce those pesky, intermittent bugs that come in from customer land? In this hands-on class, Jon Bach helps you practice your investigation and analysis skills—questioning, conjecturing, branching, and backtracking. For those of you who have ever had to tell the story about the big bug that got away, Jon offers up new techniques that may trap it next time so you can earn more credibility, respect, and accolades from stakeholders. Because collaboration and participation are encouraged in this class, bring your mental tester toolkit, tester’s notebook, and an open mind.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The Craft of Bug Investigation

  1. 1. TD AM Tutorial 10/1/2013 8:30:00 AM "The Craft of Bug Investigation" Presented by: Jon Bach eBay Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
  2. 2. Jon Bach eBay, Inc. With more than eighteen years of experience in software testing, Jon Bach has held technical and managerial positions in companies including Hewlett-Packard and Microsoft. In his current role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests (activity flows) in eBay’s core sites to discover important bugs that threaten its core business. He is most notable for creating, with his brother James, Session-Based Test Management, a method to manage and report exploratory testing.
  3. 3. The Craft of (black box) Bug Investigation Jon Bach QE Director, eBay STAR West 2013 Preamble These next few hours are designed with exercises that lead you through ways of thinking that help you narrow down a bug. We’ll discuss techniques for bug isolation, reproducing intermittent bugs, and skills to help you know when you’ve gone as far as is reasonable before handing it off. eBay Inc. confidential 1
  4. 4. A morning of investigation Psychic Reader Art Show Mysterious Spheres Gettysbug Address Checkout Checking Out? Out-of-Bounds eBay Inc. confidential } } } } } Ways we find bugs CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA Project Environment Product Elements Quality Criteria Development Criteria General Test Techniques Customers Information Developer relations Team Equipment & tools Schedule Test Items Deliverables Structure Function Data Platform Operations Time Capability Reliability Usability Security Scalability Performance Installability Compatibility Supportability Testability Maintainability Portability Localizability Function testing Domain testing Stress testing Flow testing Scenario testing Claims testing User testing Risk testing Automatic testing eBay Inc. confidential 2
  5. 5. How *well* do we find bugs? Some Investigation Skills and Tactics “MR.Q COMC GOARABC R&R?” Modeling Chartering Generating/Elaborating Recording Resourcing Observing Overproduction/Abandonment Reporting Questioning Manipulating Abandonment/Recovery Collaboration Refocusing Alternating Branching/Backtracking Conjecturing Investigation is a mindset using this skillset. eBay Inc. confidential How well do we investigate them? Too Much Quality Further time and effort is a waste of resources. floating line Good enough quality bar Further time and effort is crucial. Unacceptable Quality eBay Inc. confidential 3
  6. 6. Abductive Inference Abductive inference means finding the best explanation for a set of data. 1. Collect data. 2. Find several explanations that account for the data. 3. Find more data that is either consistent or inconsistent with explanations. 4. Choose the best explanation that accounts for the important data, or keep searching. Jump to conjectures, not conclusions eBay Inc. confidential Is Checkout Checking Out? This is a dip in bids we saw in Production. What questions do you have to troubleshoot this? eBay Inc. confidential 8 4
  7. 7. Topics for investigation (FORCOSTTTIP) • Feature – Do we know what happens when people bid? • Origin -- When did this start happening? • Recent – Did we just push code to Production? • Consistency / Correlation -- What happened yesterday, tomorrow? Are there other graphs and metrics? • Overlap – Is there any other process happening? • Security – Is it malicious? Intentional? • Terminology -- What kinds of “bids”? US, UK, International? • Third-Party / Silent Partner – What’s not in the room? What sort of information have we NOT heard from? • Truth is out there – does someone already know about this? • Instrumentation – is the graph wrong? Can I get more data? • Prediction – can we test a conjecture with a prediction eBay Inc. confidential 9 Exercise: Psychic Reader eBay Inc. confidential 5
  8. 8. Lessons You can be psychic, too You may be clue-less, for now Try “defocusing” Are you just following orders? eBay Inc. confidential Exercise: Mysterious Spheres eBay Inc. confidential 6
  9. 9. Lessons If it fails, whose fault is it? Assumptions need nourishment Oracles are in “control” “Elsewhere”, is a useful thought eBay Inc. confidential Exercise: Out-of-Bounds eBay Inc. confidential 7
  10. 10. Lessons Be a “model” tester Stagehands work behind curtains Outside data is not inside data Computers are our slaves eBay Inc. confidential Exercise: Art Show What is my Operating Rule? What do you notice? What (crazy) test haven’t you tried? eBay Inc. confidential 8
  11. 11. Lessons “You’re so one dimensional” Jump to *conjectures* Yes, I *do* look FAT in this. (M vs. O) Breaking “rules”! eBay Inc. confidential Exercise: Gettysbug Address Does it work? What is the hidden feature? What story does the data tell? eBay Inc. confidential 9
  12. 12. Lessons The Rumble Strip heuristic Missing a mission? Be “thoughtless” (random is good) Defocus: do *anything*, *differently* eBay Inc. confidential Exercise: What’s the Pattern? Investigation of behavior is like trying to repro a bug… what is the behavior of this app? eBay Inc. confidential 10
  13. 13. Lessons Checking is different than testing Collaboration is a kick Not all characters are created equal Description takes practice eBay Inc. confidential How well do we investigate them? Too Much Quality Further time and effort is a waste of resources. floating line Good enough quality bar Further time and effort is crucial. Unacceptable Quality eBay Inc. confidential 11
  14. 14. How much investigation? 1) Sufficient benefits 2) No critical problems 3) The benefits outweigh the problems 4) In the present situation, and all things considered, improvement would be more harmful than helpful The answer must be “Yes” to all four criteria, but ask “for whom, what, and when?” eBay Inc. confidential Intermittent Bugs: Observation Bad observation Irrelevant observation Bad memory Misattribution Misrepresentation Unreliable oracle Unreliable communication Link eBay Inc. confidential 12
  15. 15. Intermittent Bugs: System Purposeful change, and then back to original Accidental change Platform change Flaky hardware Trespassing system Executable corruption Component competition eBay Inc. confidential Intermittent Bugs: Machine State Frozen conditional Improper Initialization Resource denial Progressive data corruption Progressive destabilization Overflow Occasional functions Different mode or option setting eBay Inc. confidential 13
  16. 16. Intermittent Bugs: Input Accidental Secret boundaries or conditions Different profile Ghost input (alternative source) Simultaneous action as others Compromised input Timing issues Crazy Combinations eBay Inc. confidential Intermittent Bugs: You? You may not be aware of... …variables of influence …sources of distortion in your observations …available tools that might help …boundaries and their characteristics …the system's missing / extra functions …complex / competing algorithms eBay Inc. confidential 14
  17. 17. A few (free) investigation tools Httpwatch -- displays log of requests and responses Firebug – inspect HTML -- JavaScript debugger WAVE – accessibility errors eBay Inc. confidential More info • ET Dynamics: • Test Heuristics and Planning • Context-Driven Software Testing • Center for Software Testing Education and Research • Books related to Exploratory Testing skills and tactics • Scenario testing examples – eBay Inc. confidential 15