Art_of_DefectReporting080214

OAK~TQ Seminars 2008
From Tester with Love
The Art of Defect
Reporting
K. L Mohan Kumar
14 February 2008

Software Bug - DefinitionSoftware Bug - Definition
 A software “bug” is an error, flaw, mistake,A software “bug” is an error, flaw, mistake,
failure, or fault in a computer program thatfailure, or fault in a computer program that
prevents it from behaving as intendedprevents it from behaving as intended
 Very often producing an incorrect resultVery often producing an incorrect result

Earliest Bug Report ?

Defect Life CycleDefect Life Cycle
Enter Defect
Analyze and
classification
Valid Defects?
Fix defect and
deliver to testing
Fix OK?
Close
Re-open
YesYes
ON
NO

Defect ReportingDefect Reporting
Is it Science or Art?Is it Science or Art?

Defect Report – Why?Defect Report – Why?
 To communicate the defect to the intendedTo communicate the defect to the intended
audienceaudience
 DeveloperDeveloper
 For fixingFor fixing
 Test ManagerTest Manager
 To evaluate the reliability and stability of the applicationTo evaluate the reliability and stability of the application
 ManagementManagement
 To evaluate the business risk of the defectTo evaluate the business risk of the defect
 Missed timelines, Implementation delaysMissed timelines, Implementation delays
 Cost of fixing, maintenance costs …..Cost of fixing, maintenance costs …..

Impact of improper Defect ReportsImpact of improper Defect Reports
 Increased turnaround time for fixesIncreased turnaround time for fixes
 Loss of productivity in testingLoss of productivity in testing
 Pile up of defectsPile up of defects
 Improper fixImproper fix
 Possibility of defects escaping into productionPossibility of defects escaping into production
 Friction with development teamFriction with development team
 Desperation to both Developer and TesterDesperation to both Developer and Tester
 Improper classificationImproper classification
 Decreases test effectivenessDecreases test effectiveness

Objective of Defect ReportObjective of Defect Report
 Should meet the purposeShould meet the purpose
 Should be effective and efficientShould be effective and efficient
 Should afford enough information to do a goodShould afford enough information to do a good
analysis and proper fixanalysis and proper fix
 Should unearth the real defectShould unearth the real defect
 Should help the developer fix the defectShould help the developer fix the defect

Common Traps in Defect ReportingCommon Traps in Defect Reporting
1.1. Too brief descriptionToo brief description
Brevity is the soul of WitBrevity is the soul of Wit
But ….But ….
Not when reporting defectsNot when reporting defects

Sample …Sample …
 In the "TO DO" list, the User can't delete theIn the "TO DO" list, the User can't delete the
entriesentries
 Developer Comment:Developer Comment:
 Tell me what were you trying to do?Tell me what were you trying to do?
 Action Taken:Action Taken:
 Closed without fixClosed without fix

2. Verbose Reports2. Verbose Reports
Verbose and flowery language is best for proseVerbose and flowery language is best for prose
but ….but ….
does not cut ice with defect reportingdoes not cut ice with defect reporting

 I logged in as a Retail User. From the different optionsI logged in as a Retail User. From the different options
presented, I chose the Message log which has compose,presented, I chose the Message log which has compose,
display amongst other buttons. I intended to compose andisplay amongst other buttons. I intended to compose an
email to the Relationship Manager and so went aheademail to the Relationship Manager and so went ahead
composing the message and sending it. On clicking sendcomposing the message and sending it. On clicking send
button, the window vanished without a trace. The applicationbutton, the window vanished without a trace. The application
promptly gave me the feedback that the mail was successfullypromptly gave me the feedback that the mail was successfully
sent and brought me back to the main screen. I had a vaguesent and brought me back to the main screen. I had a vague
premonition that the message send was not successful, so Ipremonition that the message send was not successful, so I
thought I would investigate. I clicked cancel button to go backthought I would investigate. I clicked cancel button to go back
to the message log. And viola! I find the mail "Job!" in theto the message log. And viola! I find the mail "Job!" in the
message log.message log.
 Developer Action: So? What is the problem?Developer Action: So? What is the problem?
 Action taken: Updated the status as invalid observation!Action taken: Updated the status as invalid observation!

3. Gibberish Communication3. Gibberish Communication
Gobbledygook is fine for puzzlesGobbledygook is fine for puzzles
But ….But ….
Means nothing to the audienceMeans nothing to the audience

 Error Details:Error Details:
 10012: File=pp0001.c Ver=1.00 Rtn=fnBayGetRtnPtr Line=392310012: File=pp0001.c Ver=1.00 Rtn=fnBayGetRtnPtr Line=3923
File=pp0001.c Ver=1.00 Rtn=fnBayGetLRDetails Line=3477File=pp0001.c Ver=1.00 Rtn=fnBayGetLRDetails Line=3477
File=pp0001.c Ver=1.00 Rtn=fnEvalExpr Line=4041 File=pp0001.cFile=pp0001.c Ver=1.00 Rtn=fnEvalExpr Line=4041 File=pp0001.c
Ver=1.00 Rtn=fnBayIfAppDir Line=1839 File=parse.c Ver=1.08Ver=1.00 Rtn=fnBayIfAppDir Line=1839 File=parse.c Ver=1.08
Rtn=IWAppDirectiveHandler Line=2786 File=parse.c Ver=1.08Rtn=IWAppDirectiveHandler Line=2786 File=parse.c Ver=1.08
Rtn=IWProcessDirectives Line=442 File=parse.c Ver=1.08Rtn=IWProcessDirectives Line=442 File=parse.c Ver=1.08
Rtn=IWParseHTTBuffer Line=3445 File=seyacc.c Ver=1.00Rtn=IWParseHTTBuffer Line=3445 File=seyacc.c Ver=1.00
Rtn=SECallDSParser Line=4364 File=seyacc.c Ver=1.00 Rtn=ExitStmntRtn=SECallDSParser Line=4364 File=seyacc.c Ver=1.00 Rtn=ExitStmnt
Line=4039 File=seyacc.c Ver=1.00 Rtn=SEStartScriptWExeInternalLine=4039 File=seyacc.c Ver=1.00 Rtn=SEStartScriptWExeInternal
Line=377 File=thrdmain.c Ver=1.42 Rtn=IWProcessHTTFile Line=1051Line=377 File=thrdmain.c Ver=1.42 Rtn=IWProcessHTTFile Line=1051
HTT File Name = {../htt/style1/cmstcoch.htt} and Current TO Name =HTT File Name = {../htt/style1/cmstcoch.htt} and Current TO Name =
{bay_load_htt} File=thrdmain.c Ver=1.42 Rtn=IWThreadMain Line=346{bay_load_htt} File=thrdmain.c Ver=1.42 Rtn=IWThreadMain Line=346
 Developer Comments: Can you give me a reasonable description of theDeveloper Comments: Can you give me a reasonable description of the
problem?problem?
 Action taken: Defect left in Entered statusAction taken: Defect left in Entered status

4. Clubbing more than one defect in a single4. Clubbing more than one defect in a single
reportreport
Developer appreciates a bouquet of flowersDeveloper appreciates a bouquet of flowers
NotNot
Bouquet of defectsBouquet of defects

Create a Product Scheme through menu HGSPMCreate a Product Scheme through menu HGSPM
When we try to verify scheme, last page gettingWhen we try to verify scheme, last page getting
as run time error (sic)as run time error (sic)
Similar issue in accountSimilar issue in account verification whereverification where
verification not happening due to run timeverification not happening due to run time
errorerror
Action taken: Only the first defect was fixedAction taken: Only the first defect was fixed

5. Passing Judgmental comments or Wise Cracks5. Passing Judgmental comments or Wise Cracks
in Defect Reportsin Defect Reports
Helps in isolating the testerHelps in isolating the tester
NotNot
The defectThe defect

 The page design is not done aesthetically. TheThe page design is not done aesthetically. The
navigation is not goodnavigation is not good
 Developer Comment: Talk to the UI designerDeveloper Comment: Talk to the UI designer
 Action taken: Defect closed without fixAction taken: Defect closed without fix
 The browser crashed when the applicationThe browser crashed when the application
window was resized. Did I expect anythingwindow was resized. Did I expect anything
better??better??
 Developer Comment: UnprintableDeveloper Comment: Unprintable
 Action taken: Escalation to Test ManagerAction taken: Escalation to Test Manager

6. Reporting based on perception rather than6. Reporting based on perception rather than
factsfacts
Facts make the defect more powerful thanFacts make the defect more powerful than
perceptionsperceptions

 Application should support any browser. ItApplication should support any browser. It
works only on IE.works only on IE.
 Pls. see the Requirement documentPls. see the Requirement document
 Action takenAction taken
 Defect closed without fixDefect closed without fix

7. Using generalizations in Defect Reports7. Using generalizations in Defect Reports
Using terms like “Occasionally”,Using terms like “Occasionally”,
“sometimes”, infrequently etc.“sometimes”, infrequently etc.
 Where did it happenWhere did it happen
 Which other programs were running at that time?Which other programs were running at that time?
 It always works fine on my computerIt always works fine on my computer
 Action taken:Action taken:
 Closed with remark “Not simulated”Closed with remark “Not simulated”

The art of defect reporting -The art of defect reporting -
Essentials of good defect reportEssentials of good defect report
 Preliminaries (Before raising the defects)Preliminaries (Before raising the defects)
 Reproduce the defect consistentlyReproduce the defect consistently
 Establish the correct sequence of reproducing the defectEstablish the correct sequence of reproducing the defect
 Preclude all setup issues and other extraneous conditionsPreclude all setup issues and other extraneous conditions
 Check if the same defect is already reportedCheck if the same defect is already reported
 Collect necessary supplementary information and evidenceCollect necessary supplementary information and evidence
to support the defect reportto support the defect report
 Screens shots, Application Logs, trace files, Spool files, memoryScreens shots, Application Logs, trace files, Spool files, memory
dumps, etc.dumps, etc.
 Report the defectReport the defect

Structure of a good defect reportStructure of a good defect report
 Summary of the defectSummary of the defect
 Provide a decent summary of the problem with as minimum words as possible.Provide a decent summary of the problem with as minimum words as possible.
The defect should be easily and uniquely identifiable.The defect should be easily and uniquely identifiable.
 Description of the defectDescription of the defect
 The test case id against which the defect was raised, the resource (URL) toThe test case id against which the defect was raised, the resource (URL) to
indicate the test environment in which it occurred …indicate the test environment in which it occurred …
 Describe in detail the problem being reported. Use correct and appropriateDescribe in detail the problem being reported. Use correct and appropriate
jargon while reporting.jargon while reporting.
 Steps to reproduce the defectSteps to reproduce the defect
 Minimum number of easy to follow steps that will reproduce the bug. IncludeMinimum number of easy to follow steps that will reproduce the bug. Include
details of special setups if any.details of special setups if any.
 Expected resultsExpected results
 What the application should have done if the bug was not presentWhat the application should have done if the bug was not present
 Actual resultsActual results
 What the application did after executing the test procedureWhat the application did after executing the test procedure
 Attach the additional information or provide pointers to that to makes itAttach the additional information or provide pointers to that to makes it
easy to analyze and fix the problem.easy to analyze and fix the problem.

Sample (from Wikipedia)Sample (from Wikipedia)
 Summary:Summary:
"Cancelling a File Copy dialog crashes File Manager""Cancelling a File Copy dialog crashes File Manager"
 Description:Description:
 Test Case Id: < >Test Case Id: < >
 Resource : Url informationResource : Url information
Drag-selecting any page crashes Mac builds in the NSGetFactory function.Drag-selecting any page crashes Mac builds in the NSGetFactory function.
 Steps to Reproduce:Steps to Reproduce:
1) View any web page. (I used the default sample page,1) View any web page. (I used the default sample page,
resource:/res/samples/test0.html)resource:/res/samples/test0.html)
2) Drag-select the page. (Specifically, while holding down2) Drag-select the page. (Specifically, while holding down
the mouse button, drag the mouse pointer downwards from anythe mouse button, drag the mouse pointer downwards from any
point in the browser's content region to the bottom of thepoint in the browser's content region to the bottom of the
browser's content region.)browser's content region.)

Sample (from Wikipedia)Sample (from Wikipedia)
 Actual Results:Actual Results:
 The application crashed.The application crashed.
 Expected Results:Expected Results:
 The window should scroll downwards. Scrolled content should beThe window should scroll downwards. Scrolled content should be
selected.selected.
(Or, at least, the application should not crash.)(Or, at least, the application should not crash.)
 Build Date & Platform: Build 2006-08-10 on Mac OS 10.4.3Build Date & Platform: Build 2006-08-10 on Mac OS 10.4.3
 Additional Builds and Platforms:Additional Builds and Platforms:
 Doesn't Occur On Build 2006-08-10 on Windows XP Home (ServiceDoesn't Occur On Build 2006-08-10 on Windows XP Home (Service
Pack 2)Pack 2)
 Additional Information:Additional Information:
 Windows: access violation in apprunner.exeWindows: access violation in apprunner.exe
 Mac OS X: "Crash Reporter" log that appears upon crash is attached.Mac OS X: "Crash Reporter" log that appears upon crash is attached.

Role of Defect Tracking ToolRole of Defect Tracking Tool
 Facilitates tracking from origin to closureFacilitates tracking from origin to closure
 Facilitates communication about defectsFacilitates communication about defects
 Facilitates analysis and reporting optionsFacilitates analysis and reporting options
 Keeps historyKeeps history
 Allows developer to fixAllows developer to fix
 Facilitates documentation on origin of defectFacilitates documentation on origin of defect
and action takenand action taken

Characteristics of a good defectCharacteristics of a good defect
management toolmanagement tool
 Generating defect ids for easy trackingGenerating defect ids for easy tracking
 Configurable workflowConfigurable workflow
 User authentication to prevent unauthorizedUser authentication to prevent unauthorized
accessaccess
 e-mail interface for facilitate follow upe-mail interface for facilitate follow up
 Integration with test case management toolIntegration with test case management tool

Case Study: Finacle ProductCase Study: Finacle Product
DevelopmentDevelopment

Finacle Universal Banking SolutionFinacle Universal Banking Solution
Trade Finance
Consumer
Banking
Branch Solution
E-banking
Solution
Mobile Banking
Solution
Integrated
Alerts
Branch ATM/POS Kiosk Fax IVR Call Centre e-mail Internet Portal Mobile
Customer Touch Points
Regulatory
Reporting
Document
Imaging
Business
Intelligence
Allied Solutions
SOA Integration Framework (Web Services, MoM)
Payment
Networks
Credit
Bureaus
Market Feeds
Card Issuers
Rating
Agencies
Regulators
Backend
systems
External World
Enterprise Customer Information and CRM
Common Services
Master Data/ User Definition/ Multi-entity/Multi-Calendar
Accounting Backbone
Multi-currency / General Ledger
Call Centre
Solution
Treasury
Corporate
Banking
Enterprise Workflow (BPEL/BPML)
Money Markets
Infrastructure Services
Workflow (BPEL),/Reporting and MIS/Security/
Finacle Studio/Communication
Functional
Modules
Infrastructure&
Common
Facilities
Channels
and
Integration
Derivatives
Securities
Foreign Exchange
Wealth
Management
Insurance
Mutual Funds
Structured
Products
Islamic
Products
Mudarabah
Murabaha Analyzed Pricing
Risk Management
AML/ KYC
Musharaka
Istisna
Current/
Overdrafts
Trade Finance
Savings/
Checking
Commercial
Lending
Retail Assets
Mortgages
Term Deposits
Document
Imaging

Defect Tracking ToolDefect Tracking Tool
 BBU ToolsBBU Tools
 Supports:Supports:
 Life cycle maintenance (Workflow maintenance)Life cycle maintenance (Workflow maintenance)
 Ticket maintenanceTicket maintenance
 Tracker maintenanceTracker maintenance
 Configurable ReportsConfigurable Reports
 Recon trackerRecon tracker
 e-mail interfacee-mail interface
 Test case maintenanceTest case maintenance

Defect Management CycleDefect Management Cycle
Tester Enters
Defect
Test Lead
Analyses Defect
Valid Defect? Mark for closure
Needs S/W fix?
Development Lead
converts to tracker
and assigns to
developer
**
Developer fixes
Defect and change
status of defect
**
Test Lead Assigns
to tester for
retesting
Fix OK?
1
1
Tester Update
closes defect
**
Tester reopens
Defect
**
Yes
Yes
NO
NO
Yes
NO
** means e-Mail generation

Sample Screens: Ticket MaintenanceSample Screens: Ticket Maintenance

Sample Screen:Tracker MaintenanceSample Screen:Tracker Maintenance

Sample Screen: Tester’s closingSample Screen: Tester’s closing
remarksremarks

Best PracticesBest Practices
 Defect Tracking completely processizedDefect Tracking completely processized
 Tracking Defect fixes across versionsTracking Defect fixes across versions
 Availability of historical dataAvailability of historical data
 Reduction in invalid observationsReduction in invalid observations
 Less than 1% of defects get abandoned due toLess than 1% of defects get abandoned due to
faulty defect reportsfaulty defect reports

Q & A SessionQ & A Session

Art_of_DefectReporting080214

Recommended

Recommended

More Related Content

What's hot

What's hot (14)

Similar to Art_of_DefectReporting080214

Similar to Art_of_DefectReporting080214 (20)

More from OAK Systems Pvt Ltd

More from OAK Systems Pvt Ltd (6)

Recently uploaded

Recently uploaded (20)

Art_of_DefectReporting080214