Bug best practice


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Bug best practice

  2. 2. AGENDABug filing best practiceYou should learn….Value added for bug managementDefect managementBug follow up best practice
  3. 3. BUG IS IMPORTANTYour performance is partially judged by the quality andamount of the bug you filed.As an engineer, reputation is the most important thing. Atester’s reputation is as good as the quality of your bug canget.Most of the time, it is the only way you communicate with thedeveloper.Who will read your bug report? – developer, your managers,developer’s manager, product manager, marketing folks,customers……
  4. 4. WHERE ARE BUGSCOME FROM?Laziness (does not perform error handling)Inexperience (does not know the impact of code changes)Carelessness (blind cut-n-paste)Bad design/infrastructure (bad module design  un-maintainable code)Incomplete documentation (misunderstood customerrequirements, missing information in func spec)Complexity of software (hard to maintain/enhance/test)Bad tools (compiler, CM tool, merge tool, etc)Lack unit-level testing (misses lots of basic bugs)Time pressure (fatigue)  Laziness and carelessness Code scales linearly, the complexity scales exponentially
  5. 5. THE QUALITY OF ATESTERScript QualityTest Case QualityBug QualityCase ExecutionQuality
  6. 6. HOW TO FIND A BUG?You think it is a bug because?• Understand the product/function spec correctly• Understand the use case scenarios correctly• Put yourself in a user’s perspective.• Your experience counts a lot, 上 兵纸 谈 is harmfulBad bugs cause you trouble• Junk bug is counter productivity, and ruin yourreputation• Bad documented bugs cause you a lot of time andenergy, like…. living in a nightmare.
  7. 7. HOW TO FIND A BUG?Debugging is like watching Magic Tricks(Michael Amma17:40 - 19:00)James Brown 5:10
  8. 8. WHY I CAN FIND HIGH QUALITY BUGS BUTYOU CAN’T?High quality bug• Affect the major functionalityQoS still work after delete the configuration• Not CosmeticBug 3744 - WebUI 的导航栏没有滚动条 , 不便于用户浏览操作Because I understand the functionality deeper than youBecause I use more testing types than you.Because I design more use case than youBecause I carefully execute the cases than youBecause I pay more attention to the details than you.Because once I found a crack, I’ll keep on going, but you didn’t
  9. 9. NONREPRODUCIBLE BUGS –HEADACHE?Random is not equal to irreproducible
  10. 10. NONREPRODUCIBLE BUGS –HEADACHE?Always report non-reproducible bugs, they might be timebombs.Non-reproducible bugs are always reproducible.• Timing and delay?• Only happen during installation• Depend on some data like corrupted database?• Sequence of the execution?• Interaction with other programs in the background?
  11. 11. NONREPRODUCIBLE BUGS –ELABORATEa. I saw this one, but was never able to make it happen again.b. I see this when I run with this setup, but sometimes I dont.c. This is the same anomaly I see under several setups, but notall the time.d. Under X setup, this happens sometimes and not other times.There may be something different from one X to the other, butI not seeing it.What is the impact if this happens all the time?....
  13. 13. AVOID CRITICAL BUG IN THE LATEDEVELOPMENTExecute test cases based on prioritiesDiscover critical bug in the early development stage• It gets fixed early, less resource waste• It gives developer more lead time – critical bugs are hard to fix
  14. 14. MOST IMPORTANTTHINGS ON A BUGSubject, Subject, Subject• One line• Exactly pinpoint the problem.• Accurately describe the problemBug 3747 - WebUI 界面上的字体、图标等在 Linux 系统下的显示不正常Bug 3774 - 客户端能够通过 Admin 端口与 DUT 建立 PPTP 连接 , 不符合设计规范Bug 3768 - admin route: 不能为同一个服务创建多条 admin 路由Bug 3788 - 在配置 DUT 的时候, DUT 死机Which one is better?
  15. 15. WHAT IS THE BUGQUALITYReproducible• Testers filed bug need to be reproducible in developer’senvironment.• Customer found bugs sometime is hard to reproduce, but thatnot tester’s concern.• Test case is not the step to reproduce.• Need to find the main cause by ruling our the unrelatedfactors. Need to pinpoint the root cause (Strong Analyticalskills).• 蛛 迹只是 象,需要 一 追踪,找到 的本丝马 现 进 步 问题 质。ISIC crash the box, can you find out which packet, and why?
  16. 16. HOW TO WRITE AREPRODUCIBLE BUGREPORTA short paragraph for detail description.A topology file with detail annotation. (IP address, DUT typeetc)Step to reproduce• Step 1. execution step with detail output.• Step 2… execution step with detail output• Seep 3: where the problem is. What is the expected output,what is the actual output.Detail DUT configuration dump,Anything else (document) you think can claim thevalidation of your bug?Review each others bug reports
  17. 17. AGENDABug filing best practiceBug follow up best practiceDefect managementIntegration of bug tracking system with other engineeringtools
  18. 18. UNDERSTAND BUGSTATE FIRST!N:A:I: Information needed: Take action ASAPR: Need Tester’s verify. Take action ASAPV:Monitor your bug daily: usually you will get an email reminderwhenever a bug state is changed. Pay special attention to “I”and “R”Best practice: I and R state bug need to be updated within 24hours.Ask your team lead for help, monitoring bug every day is hard.
  19. 19. INTERACT WITHDEVELOPERS -1Understand the developer’s response.It is like writing an email, need to be polite. Need to have startparagraph and end paragraph.It needs to be thorough and accurate (provide as manyuseful info as possible).A quick and good response help you gain the respect fromdeveloper.Ask your team lead for help is a good idea.
  20. 20. INTERACT WITHDEVELOPERS -2Meet the programmers who will read your reportThe best approach may be to demonstrate your bugs to theprogrammers.When the programmer say it is fixed, make sure it isn’t stillbrokenVerify bug fixed promptlyWhen fixes fail, talk with the programmer.Bug report should be closed by testers..
  21. 21. AN EXAMPLE OFQUERYState-Changed-From-To: open->infoState-Changed-By: murphywState-Changed-When: Sun, 24 Jun 2007 23:14:08 -0700State-Changed-Why:1. ‘set policy id 1 attack "CS:ftp" action close’, does this CLI mean thatwe should drop FTP traffic?2. Please help to provide debug information on successful case.3. Please turn on snoop and snoop detail during debugging.Thanks!How Would you response?
  22. 22. AGENDABug filing best practiceBug follow up best practiceDefect managementIntegration of bug tracking system with other engineeringtools
  23. 23. DEFECTMANAGEMENTHow to know how many bugs filed per person, per day – bug searchis your best friend.How to know how many bugs filed for certain type of testing• Search for testing type – regression; performance;functionality;…• Add key word in front of the bug subject field then search thekeywordREG: extended access list can not block traffic.Pre-Test:-Pascal-when the packets cant pass through VPNtunnel with deep inspect• Be careful when add the interest field: those are the personwill be notified every time this bug state is changed.• If there is a field called attribute, use it. It helps to categorizeand manage bugs we filed.
  24. 24. FILING BUG REPORTSFile every bug found (except duplicate bugs), annotate if abug:• Blocker: High priority bug preventing further testing• Regression: Bugs that introduce in new build• Unreproducable: Bugs that cannot be reproduced• Customer found: Bugs found by customers (Beta/FCS)Pick right Priority (usage) & Severity (impact)• P3/S1 : Login 200 times with 200 characters user name willcrash the device with exception core dump• P1/S3 : GUI splash screen misspelled company nameBug report is a statistics tool, use to predict readiness ofsoftware and future releases
  26. 26. BUG TRENDCONVERGENCEBug FiledRelease Date
  27. 27. BUG TRENDCONVERGENCEBug FixedRelease DateBug Filed
  28. 28. BUG TRENDCONVERGENCE发现 Bug正常测试 分散测试发布日期Bug 修复Bug 严重程度日期Bug 数目
  29. 29. BUG TRENDCONVERGENCE发现 Bug正常测试 分散测试发布日期Bug 修复Bug 严重程度 日期Bug 数目
  30. 30. BUG TRENDCONVERGENCE发现 Bug正常测试 分散测试发布日期Bug 修复 Bug 严重程度 日期Bug 数目
  33. 33. BUG FIELDS FORMETRICSStatusDate ReportedWho FoundAssigned ToProgram AreaSeverityPriorityResolution
  34. 34. INTERESTING BUGMETRICSPriority 1 Find Rate− This is the find rate of bugs that at any point in timebecome priority 1. It is a retrospective analysis.Open Bug Aging Report − A histogram of how long bugshave been open. This is most useful early in long projects.Deferred Bug Aging Report − A histogram of how longdeferred bugs have been waiting on the list. It helps spotchronically deferred bugs.Promotions/Demotions/Deferrals Chart − Three lines thathelp us see the triage process at work.
  35. 35. AVOID CONTROL METRICS;EMBRACE INQUIRY METRICSA control metric is any metric that drives decisions. − Anymetric you use to control a self-aware system will be used bythat system to control YOU.An inquiry metric is any metric that helps you ask the rightquestions at the right time. − An inquiry metric might looklike a control metric. The difference is how you use it andwhat you infer from it.
  36. 36. EXAMPLE CONTROLMETRICS“The developer who creates the fewest bugs will receive abonus.”“Testers must average at least three bugs found per day.”“The product may not be released unless we’ve gone at leastone week without finding a bug.”“The product may not be released with more than 10 highseverity bugs.”
  37. 37. EXAMPLE INQUIRYMETRICSWhy do some developers have more bugs reported on theircode than others? How might it be good to have more bugsreported? How might it be bad?”“How do find rates differ among testers? What makes themdiffer?”“What does the bug find rate tell us about the readiness ofthe product? Could the find rate be falling because testing isslack, and not because the product is good?”“What are the high severity bugs? Should we fix them?”
  38. 38. INQUIRY METRICSINVITE LEARNINGObserve: You try to see whats happening.Inquire: You conjecture about the meaning and significanceof the observations. You collect additional observations tocorroborate or refute our conjectures.Model: You form and improve theories about why the systembehaves as it does, how you know that it behaves that way,and what you can do about it.
  39. 39. QUESTIONS BUG METRICS MIGHT HELPWITH TEST PROCESSProductivity• − Are the testers working at full speed?• − Are they speeding up or slowing down?• − Is anything blocking their work?• − Are the testers burning out or staying sharp?• − Are testers and developers getting along together?Status• − How close is the test effort to reaching a point of diminishingreturns?• − What kinds of problems are being found?• − Is the bug tracking system being used properly?Focus• − Are the testers focused on the areas of greatest risk or need?• − Could the testers use some outside help?
  40. 40. QUESTIONS BUG METRICS MIGHT HELPWITHQUALITY IMPROVEMENTProductivity• − Are the developers working at full speed?• − Are they speeding up or slowing down?• − Is anything blocking their work?Status• − Is the product improving?• − How close is the product to being good enough?• − What must happen in order to meet the schedule?Focus• − Is the work distributed effectively among the developers?• − Do any areas of the product need more attention to improvemore quickly?• − Is the triage process working well?
  41. 41. DAILY METRICS!Consider producing a bug metric summary:• − Daily• − Automatic• − Archived• − Web BasedThis becomes a resource for retrospective analysis.Additionally, daily graphs can help you notice fast breakingdynamics.Include weekends in the graphs.
  49. 49. AGENDABug filing best practiceBug follow up best practiceDefect managementIntegration of bug tracking system with other engineeringtools
  50. 50. VALUE ADDEDDEFECT/CODEINTEGRATIONBug system can be integrated with CVS, Subversion etc codecontrol toolWhen check in code, developer can specify the bug id, thebug will be automatically updated with code diff.When check in code, developer can specify the bug id, andbug’s state will be automatically changed to “R”, and triggertester to verify it.Other scenarios you can think of?This is the area that we can provide the value added for ourcustomer
  51. 51. THINGS YOU SHOULDREMEMBER AFTER THISBug quality means it can be reproduced by developers intheir environment.Most important thing on a bug is its subjectJunk bug is counter productive, and ruin your reputation.I, R state bug need to be updated ASAP.Use Sigma Bug Template to file bugs.
  52. 52. BUG REVIEW CHECKLIST - 1Is the summary short (about 50-70 characters) and descriptive?Can you understand the report?• As you read the description, do you understand what the reporterdid?• Can you envision what the program did in response?• Do you understand what the failure was?Is it obvious where to start (what state to bring the program to)to replicate the bug?Is it obvious what files to use (if any)? Is it obvious what youwould type?Is the replication sequence provided as a numbered set of steps,which tell you exactly what to do and, when useful, what youwill see?
  53. 53. BUG REVIEW CHECKLIST - 2Does the report include unnecessary information, personalopinions or anecdotes that seem out of place?Is the tone of the report insulting? Are any words in thereport potentially insulting?Does the report seem too long? Too short? Does it seem tohave a lot of unnecessary steps? (This is your firstimpression—you might be mistaken. After all, you haven’treplicated it yet. But does it LOOK like there’s a lot of excessin the report?)Does the report seem overly general (“Insert a file and youwill see” – what file? What kind of file? Is there an example,like “Insert a file like blah.foo or blah2.fee”?)
  54. 54. BUG REVIEW CHECKLIST – 3Can you replicate the bug?Did you need additional information or steps?Did you get lost or wonder whether you had done a stepcorrectly? Would additional feedback (like, “the program willrespond like this...”) have helped?Did you have to guess about what to do next?Did you have to change your configuration or environment inany way that wasn’t specified in the report?Did some steps appear unnecessary? Were they unnecessary?Did the description accurately describe the failure?Did the summary accurate describe the failure?Does the description include non-factual information (such asthe tester’s guesses about the underlying fault) and if so, doesthis information seem credible and useful or not?
  55. 55. BUG REVIEW CHECKLIST – 4Does the description include non-factual information (such asthe tester’s guesses about the underlying fault) and if so, doesthis information seem credible and useful or not?Does the description include statements about why this bugwould be important to the customer or to someone else?
  56. 56. VERSION CONTROL1st draft, Dec 7th, 2007, Liang Gao2nd draft, Dec 27th, 2007, Liang Gao