Downloaded 14 times
Uploaded byNeil Studd
Supercharging your bug reports
The document is a comprehensive workshop outline on improving bug reporting, emphasizing the importance of clear and persuasive communication when reporting software bugs. It explores various aspects including the definition of bugs, the elements of a good bug report, and the significance of credibility and professionalism in the bug advocacy process. Additionally, it discusses strategies for overcoming obstacles in getting bugs fixed and highlights the value of shared experiences among testers.
Supercharging your bug reports
- 1. Supercharging Your Bug Reports NeilStudd (@neilstudd) TestBash Workshop Day Thursday 26th March 2015
- 2. About me • Tenyears of testing • Working for acquired startups • Senior/lead/mentorship positions • Currently consulting in London • Beta/freelance testing in spare time • Facilitator, Weekend Testing Europe
- 3. This will bea bit like Weekend Testing… • Two hours • Too much content! • Driven by you – we’ll “follow the energy” • Link to resources at end of session • It’s not a lecture – pitch in whenever you want! • Share and compare experiences
- 4.
- 5. Talking about… bugs(or problems, or defects) • Bug: “Any problem with the product that might threaten its value” (Rapid Software Testing) • …or “something that ‘bugs’ somebody who matters” (James Bach) • Problem: “A difference between someone’s desires and the way things seem to them” (Jerry Weinberg) • Or defect, or issue, or fault, or backlog item… • Whatever you choose to call them, we’re going to focus on what happens between discovering one of them and it being fixed.
- 6. So… what’s abug? (BBST Bug Advocacy) Doesn’t match specs or requirements Could embarrass the company Generates incorrect or confusing results Doesn’t match documentation Reduces the chance of a sale Wastes the time of a user Doesn’t do what the programmer intended (coding error) Interferes with development of the product Anything that would lead to a code change if reported Doesn’t meet company standards Interferes with the testability of the product Failure to meet reasonable expectations of a user Doesn’t meet industry standards Interacts badly with other programs or components Failure to meet reasonable expectations of a stakeholder
- 7. Does this stuffreally matter? • Most testers discover/report about bugs • Very few testers have any formal training on bug reporting • Testers have a wealth of shared experiences in this field, but we each log bugs in isolation • Bug reports are like a sales tool for bugs. Need persuasive powers! • Bug reports are often a tester’s primary delivered work product • Reporting skills are vital for credibility
- 8. Credibility We GAIN credibilityby: • Using clarity and accuracy in writing • Pinpointing the exact steps needed for a subtle bug to occur • Giving decision-makers enough information to cast a judgement • Displaying professionalism - not creating a blame culture • Owning up to our mistakes We LOSE credibility by: • Wasting a developer’s time • Making accusatory statements • Making false assumptions or inflating the status of a bug without reason • Being the boy who cried “bug” When a contentious bug arises – your credibility may directly affect the outcome!
- 9. Bug Advocacy =‘Selling’ a problem • Advocacy: “Public support for a particular cause or policy” • Make a case for the fix; pick your battles • Take ownership of the problem as a champion for quality • Testers don’t “break the product” but are well-placed to push for fix • Effective communication is key to good decision-making • Motivate the buyer: Make them WANT to fix the bug • Overcome/pre-empt objections, removing reasons for NOT fixing the bugs
- 10. What makes agood bug report? • What sort of information do you include? • What information do you omit? • Think of adjectives which describe a good bug report. • If it helps, think about your company’s bug-tracker: • What fields are mandatory? • What fields are mandatory, but you wish they weren’t? • What fields are missing that you’d like to see?
- 11. PEOPLE WORKing: Heuristicfor reports • P • E • O • P • L • E • Work Presented by Michael Bolton at EuroSTAR 2014 Co-authors include James Bach & Alexei Barantsev Problem Example Oracle Polite Literate Extrapolation Workaround
- 12. PEOPLE WORKing: Problem(1/2) • Title/Summary • A summary of what you’ve observed • What happens, under what conditions • Describe the relationship between perceived and desired • Balancing length restrictions versus ease of comprehension • Grabs attention, starts a conversation, serves as a call-to-action • Searchability: “Where’s that bug for the homepage dropdown…?”:
- 13.
- 14. PEOPLE WORKing: Problem(2/2) Alan Page, “How We Test Software At Microsoft”: • Title is used for review by feature/product triage team • In the space of ~80chars, needs to accurately summarise the report • “At Microsoft, many testers quickly fill out all fields of the bug report but take extra time to carefully word a title for their bug.” • Program crash – too short • When running many instances of the program at the same time, a crash occurs in one of the dialogs – both wordy and vague • Program crash in Settings dialog box under low-memory conditions – specific, accurate, and tells enough of the story to understand the bug
- 15. PEOPLE WORKing: Example •An illustration of the problem, as clearly and briefly as possible • Narrow scope: Find critical variables, drop steps, subtle variations • “What is the fastest, least effortful way to report this problem that still completely addresses what people need to know in order to get to it” – Michael Bolton • If looking for the “fastest” report, consider how your reporting tool might be working against you (mandatory fields; fields where you add the same value every time) • Commonly “Steps to reproduce” and/or screenshots • Record information (steps, logs, screenshots) as soon as you see the issue. It might not be as reproducible as you think!
- 16. Mongoose or antelope?(Simon Tatham) • Mongoose: Attacks frantically, because it’s better than nothing • Antelope: Freezes, stops, thinks about the best course of action • When a bug occurs… be the antelope!
- 17. PEOPLE WORKing: Oracle •What can you apply to the situation to tell whether it’s a problem? • Oracle: a fallible way to recognise a problem when it occurs in testing • Consistency heuristic: FEW HICCUPPS (e.g. Comparable products) • Our feelings can be heuristic triggers; emotions help us to reason • Cite good oracles, which indicate a potential threat to product value
- 18. PEOPLE WORKing: Polite •A bug report is usually bad news. Don’t kick them while they’re down. • Avoid antagonising or apportioning blame • Own the problem for the team (“we have a problem”) • Polite also means not wasting time: • Don’t write in a way that’s hard to understand • Don’t omit information which was obviously necessary • Don’t waffle, get to the point, make the key facts easy to find
- 19. PEOPLE WORKing: Literate •If you just have a title and an example, the natural reaction in many cases is “So what?” – why should anyone care about this bug? • Tell the “story” of the problem: • Characters: The product, at least one person • Build empathy for the person (they must matter, we must care about them) • Reveal the problem (the problem must matter too) • Develop a plot (the threat to value; use extrapolation)
- 20. PEOPLE WORKing: Extrapolation •Is bug more serious or more general than first seems? (Fix chance ++) • Once the bug has occurred, is system vulnerable to other problems? • Cem Kaner – RIMGEA • Replicate • Isolate • Maximize (how could this be the biggest, baddest problem possible?) • Generalize (where else might this bug manifest in the code?) • Externalize (where else might this bug manifest in the wild?) • And say it clearly/dispassionately • BUT be realistic – don’t exaggerate, provide salient facts • Use safety language (e.g. can/could) but do it sparingly
- 21. PEOPLE WORKing: Workaround •Is there an alternative solution which could lessen severity? • Just provide information about it, it’s management’s decision as to whether it’s good enough • Without a workaround, it could be a showstopper
- 22. PEOPLE WORKing: Heuristicfor reports • Problem • Example • Oracle • Polite • Literate • Extrapolation • Workaround • Is this a good list? • What might you add? • Review some of your own bugs against these criteria
- 23. It’s logged –now how to get it fixed? • Congratulations on logging a brilliant, eloquent, convincing report. • Bug reporting is not just about bug reports! • Triage stage: formal/informal, intra-team or “Bug Review Board” • Brainstorm: What are the hurdles to actually getting a bug fixed? OBSTACLE POSSIBLE SOLUTION Can’t reproduce Better/clearer example, walkthrough, … Don’t understand problem …? Not enough time …? …come up with more as a group
- 24. Invalid / “Won’tFix” (1/2) • “It’s not a bug, it’s a feature!” – use oracles • “It’s unrealistic / nobody uses it in that way” – get data • “Nobody’s reported that” – are you sure? Check with Support • “It’s too hard / too risky to fix that” – goalposts shifts as release nears • Have they understood the report? Were you clear/brief enough?
- 25. Invalid / “Won’tFix” (2/2) • Learn when to let go. • If the people in charge are saying it’s not important, and your best information can’t convince them, it’s not your problem.
- 26. Precision • Say exactlywhat you mean; clarify what you mean when you say it! • Particularly for product-specific / domain language. Get that wrong and people might lose trust in whether you know what you’re talking about. • Be aware of what’s being left open to interpretation. • “Mary HAD a little lamb” • James Bach – “Two scoops of raisins” • Write in a way that the problem is fully understood, so that the fix doesn’t just fix one small variation. • If you’re speculating – say so, separate this from facts.
- 27. Precision, but alsobe brief… The eventual fix was here all along… everyone tuned out!
- 28. Cannot Reproduce (1/3) •Bugs are rarely truly “random”, even something like this… • Not random, but the circumstances under which it occurs is obtuse. • (Debug, monitor value of x, you’d notice the pattern quickly.) • The trick is to find ways to break down the pattern…
- 29. Cannot Reproduce: Findingthe patterns… (2/3) • Some emails fail to send • Printer sometimes jams when printing • Xerox machine doesn’t scan some documents correctly • OpenOffice keeps refusing to print files • My savegame file sometimes gets deleted Cannot send email over 500 miles Print this file; your printer will jam! Xerox machine changes numbers on scanned documents OpenOffice can’t print on Tuesdays Savegame file deleted when PlayStation controller battery dies
- 30. Cannot Reproduce: Tipsfor Progressing (3/3) • “Works on my machine!” • Can YOU still reproduce it? (Did you give yourself enough info?) • What are you doing which isn’t in the bug report? (Be obtuse!) • What have THEY tried to reproduce it? (Pairing) • Desktop: How do your environments differ? (Too clean? Too dirty?) • Web: Browser? Version? Extensions? Dev Tools?
- 31. Duplicate (1/2) • Whydo duplicates happen? • Didn’t search • No time to search • Can’t search (repo doesn’t support it) • Searched but didn’t find anything – know your system’s search grammar • Search is key… so apply SEO techniques when creating content! • Keywords – consistent terminology • Cross-referencing other issues (in text, or by linking bugs if system allows it) • Errors in screenshots aren’t searchable; replicate it in text
- 32. Duplicate: Get errormessages in text! (2/2) …or type it! (If you have to. First few lines of stack trace + screenshot)
- 33. Formality • Vary asappropriate • “You are not a bureaucrat” (Michael Bolton) • Expected: Total field should update and display correct result • Actual: Total field updates and displays incorrect result • If you’re forced to complete meaningless mandatory fields, ask why • Ideally as lightweight as possible • Different team dynamics will operate at different levels
- 34. Formality – IanMacKinnon “I think the best barometer for the culture of a company is in the formality of the bug reports.”
- 35. Formality - “ViewSource”
- 37. Going round thefixed / not-fixed loop • Fixed > Rejected > Fixed > Rejected > Fixed > Done • Michael Bolton, Christian Legget: “Boomerang” bugs • My experience: “Bouncing” bugs (like a ball, the bounce gets smaller) • How to avoid? • Dev-box testing – get at least a cursory hands-on before commit • Understand why the developer believed it was fixed (environmental?) • If it’s a regression then an automated check could help • If it’s too hard to fix as desired, is there an alternative solution which could at least lessen the severity?
- 38. “We never getrid of problems… The best we can hope for is that the problems we substitute are less troublesome than the ones we ‘solve’.” - Jerry Weinberg, Are Your Lights On?
- 39. “Trivial” bugs • Shouldyou log them? • Open discussion
- 40. Do you evenneed a report? • Overhead of creating understandable report, waiting for time to pass, reviewing it, getting back into headspace… • Consider: Post-its, mention-in-passing, tea! • Investigate whether it really is small, or whether you think it is • Take responsibility for the full stack of related activities; if you tell a dev it’s going to be “quick”, make sure it is (for them). • A quick fix does not necessarily mean a quick test • Respect your colleagues! “Quick fix” still has context-switch cost
- 41. Honing your bug-reportingskills • Paid projects: uTest, TryMyUI • Live bug bashes: The Defectives, hack days (be a “tester-for-hire”) • Beta testing: Public betas for applications, websites, games • Peer review: Within your team (gamify it?), Weekend Testing
- 42. Bug counting andmetrics (1/2) • The Daily WTF: “The Defect Black Market” • “Don’t log that, we can’t fix it and it will make our stats look bad” • If you didn’t log any bugs… what does that mean?
- 43. Bug counting andmetrics (2/2) • If you’re close to the team, you have better metrics available to you • If you’re not close to the team, bug counts alone aren’t enough Ask: • “What are you actually trying to measure?” • “How do you plan to add value with that information? To whom?” • “What information might serve you better?” • …What else?
- 44. “The best testerisn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed” - Cem Kaner
- 45. Bug counting andmetrics
- 46. If all elsefails… • If there’s a culture of bugs building up, how can you fix it? • “Cheese Day” (Blizzard) • Beer/bacon bounties! • To conclude: Your own experiences?
- 47. Whatever Works • Ameans to an end – focus on whatever eases your path to the end • How do you describe your techniques to newcomers? • How do you know it’s working? • How do you recognise what isn’t? • Don’t change for the sake of it
- 48. Thanks for takingpart! Resources from today’s session: http://blog.neiltest.com/testbash2015/ Twitter: @neilstudd Email: neil@neilstudd.com
Editor's Notes
- #3 Small companies which have been acquired – so I’ve seen good/bad examples in both lightweight bugtrackers and heavyweight in-house behemoths Freelance including uTest – many projects are pay-per-bug (for better or worse) so incentivised to produce good reports, but I still see a lot of poor stuff Along the way I’ve seen recurring patterns and anti-patterns in bug reporting
- #4 Not a lecture – I will say things to provoke discussion, encourage debate. Don’t be afraid! These slides are my cue cards, my experiences. Yours may differ. That’s kind of the point.
- #5 To 00:10-00:15 Introduce everyone in the room Help me to get a feel for who’s clustered together from the same companies
- #6 Different names, often the differences are purely semantic – it’s what a particular organisation is used to. If I join a large org, I’m not going to force a name-shift for the sake of it. Weinberg – Quality is value to some person
- #7 Bugs happen; it’s an inevitability. We strive to not do any of these things, but we’re imperfect and subjected to pressures (esp.time/resources)
- #8 We discover problems, rank them according to other problems that we know of, and our reports will assist stakeholders in prioritising and judging product quality. Developers pair on coding activities; pair-testing (and pair bug-reporting, or at least peer review) should be done more Some people are naturally adept at it. I’ve got an English/arts background but even I’ve had to refine my working practices, and they evolve as I work with new teams. Credibility: Exaggerating in bug reports, incorrectly jumping to conclusions about cause/fix, giving too much (or not enough) info such that it leads a developer astray. Sales techniques: Bug Advocacy. Taking ownership of the issue; testers don’t “break the product” but they are often in a position to champion quality. Primary work product – Like it or not, this is how most developers will form an impression of you
- #9 Credibility is a lot like currency – earn/spend it, and it can be harder to earn than spend! Wasting dev time – could’ve just talked to them for clarification Contentious bug: Credibility gives people the faith to read on, reconsider, etc. Consider two parallel universes, where a tester’s credibility varies in each. My example of being on a team where none of the other testers held much credibility with dev team
- #10 You found it, you’re best placed to fight for it. Make them want to fix the bug – even better if they think it was their idea to fix it! (All guns blazing can be counterproductive… try softly softly)
- #11 Draw/discuss until 00:45 latest 10-15 minutes on sheets of paper, then we’ll discuss what each group has written Unique ID/reference, preferably memorable. Large orgs can have 8-9 digit IDs… Look to systems which allow project/release-specific prefixes, e.g. JIRA has MOT-123, keeps them memorable. (Barking 3-digit numbers on current project) Title/Summary/Headline Date reported Reported by (who you are) Area/Component – particularly for large products. Sometimes just replaced by a prefix on the title Steps – including Expected/Actual? – how do you know what is expected? Michael Bolton – “Don’t be a bureaucrat” Release/build identifiers – which build was it seen in? Particularly important if you’re sure it’s a new bug, or where there are different daily builds circulating Environment Severity (as you perceive it) / Priority (as the PM perceives it) – many orgs use these interchangeably but they should be better understood sometimes deeply political. Once worked with a customer who used P1-P5 (with 1 being most severe, a P1 would result in people being woken up around the world to resolve it and someone being fired). So they would often ask for bugs to be classified as “a high-priority P2”… “Hints To Tester” – a mandatory field completed by the dev who fixes the issue. They state any assumptions they’ve made, any special-cases they had to code for, a list of places that the code is referenced etc. (Not just talking about written bug reports.)
- #12 Heuristic for bug reporting – a mnemonic We’ll look at each one in detail Michael did a half-day class at EuroSTAR, but we’ll push through quickly as there’s other stuff to look at afterwards
- #13 Perceived/desired: attributed to Jerry Weinberg’s work earlier, though the original quote comes from John Dewey’s “How We Think”, from 1933! Eric Jacobson blog post – “Do bug titles matter?” – There’s no such thing as the “best” bug title, just needs to be unique, describe the problem “to some extent” and include key words. So we agree on the last point, for sure.
- #14 Michael Bolton’s blog post “Not-so-great Expectations”
- #15 Title used for review = In many orgs, particularly large ones with many products and a formal review process, don’t expect any guarantees that the detail of your report will get read. Lots to get through.
- #16 e.g. is it browser-specific? Only for users with particular credentials? Particular devices/screen sizes?) Filler information decreases chance of reading You can’t search on screenshots! But they can create an “oh, THAT bug” effect to aid speed to comprehension Clarify what is happening at each step. “Do X, you should see Y appear” Other ways it can work against you: extra fields for tracking sprint information, time-tracking information which can be used for burndowns but also nefarious means. Who’s actually using the info? Anybody?
- #17 Simon’s example = people who contact techincal support. Someone who managed to delete all of their Word docs, so reinstalled Word and did a defrag / disk cleanup
- #18 All oracles are heuristic – a rule of thumb. Cannot show that there is no problem – but they can help us to recognise when there might be a problem FEW HICCUPPS - Familiar problems, Explainability, World, History, Image, Comparable products, Claims, User’s Desires, Product, Purpose, Statutes Feelings as triggers / emotions as signals: Impatience > a threat to performance? Frustration > a threat to capability? Confusion > a threat to usability/tesability? Fear > a threat to security? Surprise > a threat to reliability? Annoyance > a threat to charisma? Boredom > an insignificant test? Tiredness > time for a break? Anxiety > a need for a particular skill? Curiosity > a pointer to useful investigation? ==================
- #20 This can mean things like comprehensibility and clarity, though in this mnemonic it specifically refers to telling the story of the problem (and relationships) People: when stakeholders say “Nobody would do that”, have knowledge of users/personas who may willingly/unknowingly suffer at the hands of this problem
- #21 Too much safety language becomes weasel words Maximize – my RST example, accessibility issue in writing software. On surface doesn’t fit with product’s primary usage, BUT if they ever wanted to sell software to a US government dept, section 508 compliance would prevent it
- #23 01:10
- #24 01:20 So now it’ll be fixed, and in record time, won’t it…..? (Or not.) Brainstorm on whiteboard/flipchart. Focus on reds, get a quick bit of green though the slides that follow will talk about them Not important enough not worth the effort not a bug Duplicate not ready to be tested yet
- #25 Nobody’s reported it: One person did, so Support wrote a KB article. Now every time it’s reported, they get pointed to the KB article, a lot of our customers were living with the problem! Unrealistic / edge-case -- get data to show that it isn’t. Analytics/metrics about your users? (e.g. web stuff – browser stats. “Nobody uses Safari any more” > “Actually I’ve got site installation reports which show we work with 3 huge design agencies who are all Mac/Safari”)
- #26 Defect prioritisation example Other people’s examples?
- #27 Speculation: “Doctor, I need a prescription for Xyz” – you tell them the symptoms and they decide!
- #28 Too much info can also be a killer… if someone sees a wall of text, even if it’s important they may never know because they fall asleep or can’t comprehend. Concise! Prominent issue on project Raised early, with examples, and a suggestion of a possible workaround Grammatically, syntactically, it’s great (if I do say so myself) It lingered for weeks/months. Everyone encountered it. “We’ve logged it, trying to work out what to do with it” After deliberating on it, the team settled on implementing a quick workaround – without even realising it was the workaround I’d suggested in the 4th paragraph!
- #30 If you’re seeing these sort of headlines, without more detail to back them up, usually a sign of a tester who gave up too soon. If you – the person tasked with revealing important issues within your product – won’t go the distance, what’s to say a dev will? Once you find the pattern, they might still sound bonkers but reproducibility enables reliable debug > can be OK in headline if detail tells more, but leaving yourself open to cannot reproduce / works for me
- #31 If you can’t reproduce it yourself – still log it, with the information at-hand. State clearly in the bug that you can’t reproduce it. Also state what combinations of factors you’ve tried in order to reproduce, can help a dev to narrow it down. Especially report any error messages – aids searchability for the next person. If you’re lucky, someone who knows the code might be able to reverse-engineer a situation which makes it happen. Replicate in the presence of a dev, which might help them see what you’re doing differently. (James example from TestBash 2013 – crashing screen recorder tool – doing “the same thing” as a dev but when dev watched him, he’d actually been moving mouse frantically/quickly which generated an overload in screen events)
- #32 Bug repo capabilities/limitations, especially around phrase searching, AND/OR, fuzzy searching / wildcards, deep-searching within comments, searching for stack traces etc. If some refer to Dashboard and some call Homepage, try to standardise on one. (Or include both within the text, so that a search for either will find it)
- #33 When you encounter an error message, don’t just attach a screenshot because you won’t be able to find that message when searching for similar errors in future. Surprisingly little-known tip: you can do ctrl-c on Windows dialogs to extract all the text. If it’s buried deeper (e.g. text on main window, not a popup dialog) you may be able to use an object inspector if framework supports it. Know your framework and support tools available. (WPF > Snoop)
- #34 As appropriate…
- #35 Good enough? Maybe. Doesn’t describe what “don’t work” means, or in what contexts, but maybe it was a well-known bug
- #37 Christmas Eve / “scrooge” Developers responded well to it – knew they would Trivial but it hadn’t got done, had to get it in queue without it looking like I’m logging tiny little things
- #40 Pros: If you’re cornered by a stakeholder, gives you protection of saying it was raised / triaged / de-prioritised There’s often a “totting-up” effect: a product with many trivial issues can have a large loss of confidence Protecting a tester’s reputation / bottom-line? Cons: Longer to log than fix (might be true, we’ll talk about alternatives in a moment) If they’re not important enough to fix now, what makes you think they’ll be important later? Are we biased because we found them? Do we think that a released bug – however minor – reflects badly on us? If they’re trivial, could time be better spent elsewhere (e.g. new features)?
- #43 On many long-running projects I’ve been on (e.g. annual releases), peaks in bug reporting are just indicative of when people are looking at the software, not when the software stability is varying. If you didn’t find a bug: Eric Jacobson – maybe there aren’t any maybe that’s not your mission (e.g. performance/survey) maybe you are getting issues fixed before they become issues, or by talking to devs Maybe you’re not the best person to be testing it, do you need more domain knowledge (pair with someone who has it) Defect Black Market – for each bug that was found and fixed, the tester/dev would each get $10. Within days, devs/testers were collaborating – dev: “if I make a little mistake here, it’ll cause 8 test cases to fail” – program didn’t last long! If you’re close to the team, you have more useful information than metrics at your fingertips If you’re a high-level executive, you don’t know the team well enough to do anything useful with the metrics
- #46 At first he looks like a jerk But the more you think about it, it’s his boss that’s the jerk… under what circumstances would that salary calculation NOT lead to that??