What makes a
good bug report?
Nicolas Bettenburg Sascha Just Adrian Schröter
Saarland University Saarland University Saarland University
Cathrin Weiss Rahul Premraj Thomas Zimmermann
Saarland University Saarland University Saarland University
1

What makes a
good bug report?
Nicolas Bettenburg Sascha Just Adrian Schröter
Queen’s University Saarland University University of Victoria
Cathrin Weiss Rahul Premraj Thomas Zimmermann
University of Zurich Free Uni. Amsterdam University of Calgary

What makes a
good bug report?
Nicolas Bettenburg Sascha Just Adrian Schröter
Queen’s University Saarland University University of Victoria
Cathrin Weiss Rahul Premraj Thomas Zimmermann
University of Zurich Free Uni. Amsterdam Microsoft Research

Bug 31021
Summary: Tree - Selection listener stops default expansion
Product: [Eclipse] Platform Reporter: Veronika Irvine <veronika_irvine@ca.ibm.com>
Component: SWT Assignee: Steve Northover <steve_northover@ca.ibm.com>
Status: RESOLVED WONTFIX QA Contact:
Severity: normal
Priority: P3
Version: 2.1
Target Milestone: ---
Hardware: PC
OS: Windows XP
Whiteboard:
Opened: 2003-02-05 14:52 -
Description:
0400
I20030205
Run the following example. Double click on a tree item and notice that it does
not expand.
Comment out the Selection listener and now double click on any tree item and
notice that it expands.
public static void main(String[] args) {
Display display = new Display();
Shell shell = new Shell(display);
shell.setLayout(new FillLayout());
Tree tree = new Tree(shell, SWT.BORDER);
for (int i = 0; i < 4; i++) {
TreeItem item = new TreeItem(tree, SWT.NONE);
item.setText(\"item \" + i);
for (int j = 0; j < 4; j++) {
TreeItem subItem = new TreeItem(item, SWT.NONE);
subItem.setText(\"item \" + j);
for (int k = 0; k < 4; k++) {
TreeItem subsubItem = new TreeItem(subItem,
SWT.NONE);
subsubItem.setText(\"item \" + k);
}
}
}
tree.addSelectionListener(new SelectionAdapter() {});
shell.setSize(200, 200);
shell.open();
while (!shell.isDisposed()) {
if (!display.readAndDispatch())
display.sleep();
}
display.dispose();
}
------- Comment #1 From Steve Northover 2004-10-07 19:39:16 -0400 -------
This behavior can't be changed now because application code now relies on it.
Sorry.
4

Home | New | Search | Find | Reports | Requests | New Account | Log In | Terms of Use
Bug 31021
Summary: Tree - Selection listener stops default expansion
Product: [Eclipse] Platform Reporter: Veronika Irvine <veronika_irvine@ca.ibm.com>
Component: SWT Assignee: Steve Northover <steve_northover@ca.ibm.com>
Status: RESOLVED WONTFIX QA Contact:
Severity: normal
Priority: P3
Version: 2.1
Target Milestone: ---
Hardware: PC
OS: Windows XP
Whiteboard:
Opened: 2003-02-05 14:52 -
Description:
0400
I20030205
Run the following example. Double click on a tree item and notice that it does
not expand.
Comment out the Selection listener and now double click on any tree item and
notice that it expands.
public static void main(String[] args) {
Display display = new Display();
Shell shell = new Shell(display);
shell.setLayout(new FillLayout());
Tree tree = new Tree(shell, SWT.BORDER);
for (int i = 0; i < 4; i++) {
TreeItem item = new TreeItem(tree, SWT.NONE);
item.setText(\"item \" + i);
for (int j = 0; j < 4; j++) {
TreeItem subItem = new TreeItem(item, SWT.NONE);
subItem.setText(\"item \" + j);
for (int k = 0; k < 4; k++) {
TreeItem subsubItem = new TreeItem(subItem,
SWT.NONE);
subsubItem.setText(\"item \" + k);
}
}
}
tree.addSelectionListener(new SelectionAdapter() {});
shell.setSize(200, 200);
shell.open();
while (!shell.isDisposed()) {
if (!display.readAndDispatch())
display.sleep();
}
display.dispose();
}
------- Comment #1 From Steve Northover 2004-10-07 19:39:16 -0400 -------
This behavior can't be changed now because application code now relies on it.
Sorry.
4

Whiteboard:
Bug 31021
Summary: Tree - Selection listener stops default expansion Opened: 2003-02-05 14:52 -
Description:
Product: [Eclipse] Platform Reporter: Veronika Irvine <veronika_irvine@ca.ibm.com> 0400
Component: SWT Assignee: Steve Northover <steve_northover@ca.ibm.com>
Status: RESOLVED WONTFIX
I20030205 QA Contact:
Severity: normal
Priority: P3
Run the following example. Double click on a tree item and notice that it does
Version: 2.1
not expand.
Target Milestone: ---
Hardware: PC
Comment out the Selection listener and now double click on any tree item and
OS: Windows XP
notice that it expands.
Whiteboard:
public static void main(String[] 2003-02-05 14:52 - Opened:
args) {
Description:
Display display = new Display(); 0400
I20030205
Shell shell = new Shell(display);
shell.setLayout(new FillLayout());
Run the following example. Double click on a tree item and notice that it does
not expand. Tree tree = new Tree(shell, SWT.BORDER);
for (int i = 0; i < 4; i++) {
Comment out the Selection listener and now double click on any tree item and
notice that it expands. TreeItem item = new TreeItem(tree, SWT.NONE);
public static void main(String[] args) {
item.setText(\"item \" + i);
Display display = new Display(); for (int j = 0; j < 4; j++) {
Shell shell = new Shell(display);
shell.setLayout(new FillLayout()); TreeItem subItem = new TreeItem(item, SWT.NONE);
Tree tree = new Tree(shell, SWT.BORDER); subItem.setText(\"item \" + j);
for (int i = 0; i < 4; i++) {
TreeItem item = new TreeItem(tree, SWT.NONE); for (int k = 0; k < 4; k++) {
item.setText(\"item \" + i);
for (int j = 0; j < 4; j++) {
TreeItem subsubItem = new TreeItem(subItem,
SWT.NONE); subItem = new TreeItem(item, SWT.NONE);
TreeItem
subItem.setText(\"item \" + j);
for (int k = 0; k < 4; k++) {
subsubItem.setText(\"item \" + k);
TreeItem subsubItem = } new TreeItem(subItem,
SWT.NONE);
}
subsubItem.setText(\"item \" + k);
}
} }
} tree.addSelectionListener(new SelectionAdapter() {});
tree.addSelectionListener(new SelectionAdapter() {});
shell.setSize(200, 200);
shell.setSize(200, 200);
shell.open(); shell.open();
while (!shell.isDisposed()) {
while (!shell.isDisposed()) {
if (!display.readAndDispatch())
display.sleep(); if (!display.readAndDispatch())
}
display.dispose(); display.sleep();
} }
display.dispose();
------- Comment #1 From Steve Northover 2004-10-07 19:39:16 -0400 -------
}
This behavior can't be changed now because application code now relies on it.
Sorry.
------- Comment #1 From Steve Northover 2004-10-07 19:39:16 -0400 -------
4

INVALID
Severity: normal
Priority: P3 CC: norbert.ploett@siemens.com
Bug 175222
Version: 3.1.1
Summary:
Target Milestone: using CDT.
Product:
Create a new plugin
[Tools] CDT
---
Reporter: Sabeesh C.S. <sabeesh.cs@ditro.com>
Component: cdt-build Assignee: cdt-build-inbox@eclipse.org <cdt-build-
Hardware: PC inbox@eclipse.org>
Status: RESOLVED QA Contact:
OS: INVALID Linux
Severity: normal
Whiteboard:
Priority: P3 CC: norbert.ploett@siemens.com
Version: 3.1.1
Target Milestone: ---
Hardware: PC
OS: Linux Opened: 2007-02-22 23:35 -
Description:
Whiteboard:
0400
Opened: 2007-02-22 23:35 -
Description:
0400
I wand to create a new plugin in Eclipse using CDT. Shall it possible. I had
I wand to create a new plugin in Eclipse using CDT. Shall it possible. I had
made a R&D in eclipse documentation. I had get an idea about create a plugin
made a R&D in eclipse documentation. I had get an idea about create a plugin
using Java. But i wand to create a new plugin ( user defined plugin ) using
using Java. But i wand to create a new plugin ( user defined plugin ) using
CDT. After that I wand to impliment it in my programe. If it possible?. Any
one can help me please...
CDT. After that I wand to impliment it in my programe. If it possible?. Any
------- Comment #1 From Norbert Plött 2007-02-23 01:55:42 -0400 -------
one can help me please...
Saabeesh,
the way I understand it you want to create a plug-in and do the code
implementation in C/C++ using the CDT. Eclipse supports only Java
implementations. In theory you could probably call into your C/C++ library via
------- Comment #1 From Norbert Plött 2007-02-23 01:55:42 -0400 -------
the Java native interface but this is probably not what you want to do.
In any case you would need to create at least a Java wrapper around your
Saabeesh,
implementation.
Do I read you right?
the way I understand it you want to create a plug-in and do the code
------- Comment #2 From Chris Recoskie 2007-02-23 08:14:04 -0400 -------
I implementation described here... this question the on the
don't see a bug or enhancement in C/C++ using belongs CDT. Eclipse supports only Java
newsgroup.
implementations. In theory you could probably call into your C/C++ library via
the Java native interface but this is probably not what you want to do.
In any case you would need to create at least a Java wrapper around your
implementation.
5

1 2
Survey Cuezilla
6

1 2
Survey Cuezilla
6

Survey the users
872 developers 1,354 reporters
with at least 50 bugs who have reported
assigned to them. at least 30 bugs.
7

Survey the users
872 developers 1,354 reporters
156 responses (19%) 310 responses (26%)
8

Survey’s aim
9

Survey’s aim
Learn developers’ problems
and information needs—and
what reporters provide.
9

Problems faced
You were given wrong There were errors in The reporter used Others
product name code examples bad grammar duplicates
component name steps to reproduce unstructured text spam
operating system test cases prose text incomplete info.
observed behaviour stack traces too long text viruses/worms
expected behaviour no spell check
10

Problems faced
Question 1
Which of the following problems have your encountered when fixing bugs?
(select as many as you wish)
You were given wrong There were errors in The reporter used Others
product name code examples bad grammar duplicates
component name steps to reproduce unstructured text spam
operating system test cases prose text incomplete info.
observed behaviour stack traces too long text viruses/worms
expected behaviour no spell check
10

Problems faced
Question 1
Which of the following problems have your encountered when fixing bugs?
(select as many as you wish)
Question 2
Which three caused you the most delay in fixing bugs?
(select at most three items)
You were given wrong There were errors in The reporter used Others
product name code examples bad grammar duplicates
component name steps to reproduce unstructured text spam
operating system test cases prose text incomplete info.
observed behaviour stack traces too long text viruses/worms
expected behaviour no spell check
10

Responses
Most delay caused
Encountered when fixing a bug
11

Responses
Most delay caused
wrong system information
viruses/spam
Encountered when fixing a bug
11

Responses
Most delay caused
bad grammar
errors in test cases
duplicates
wrong system information
viruses/spam
Encountered when fixing a bug
11

Responses
Most delay caused
wrong observed behaviour
wrong expected behaviour
wrong product information
bad grammar
errors in test cases
duplicates
wrong system information
viruses/spam
Encountered when fixing a bug
11

Responses incomplete information
Most delay caused
erroneous steps to reproduce
wrong observed behaviour
wrong expected behaviour
wrong product information
bad grammar
errors in test cases
duplicates
wrong system information
viruses/spam
Encountered when fixing a bug
11

Information needs
product hardware observed behaviour screen shots
component operating system expected behaviour code examples
version summary steps to reproduce error reports
severity build information stack traces test cases
12

Information needs
Question 1
3
Which of the following items have you previously used when fixing bugs?
(select as many as you wish)
product hardware observed behaviour screen shots
component operating system expected behaviour code examples
version summary steps to reproduce error reports
severity build information stack traces test cases
12

Information needs
Question 1
3
Which of the following items have you previously used when fixing bugs?
(select as many as you wish)
Question 4
Which three items helped you the most?
(select at most three items)
product hardware observed behaviour screen shots
component operating system expected behaviour code examples
version summary steps to reproduce error reports
severity build information stack traces test cases
12

Responses
Helped most
13 Used when fixing a bug

Responses
Helped most
version
hardware severity
13 Used when fixing a bug

Responses
Helped most
screenshots
observed behavior
test cases/examples
expected behavior
version
hardware severity
13 Used when fixing a bug

Responses
Helped most
steps to reproduce
stack traces
screenshots
observed behavior
test cases/examples
expected behavior
version
hardware severity
13 Used when fixing a bug

Questions to reporters
Same options as for developers
product hardware observed behaviour screen shots
component operating system expected behaviour code examples
version summary steps to reproduce error reports
severity build information stack traces test cases
14

Questions to reporters
Question 1
What information have you previously provided in bug reports?
(select as many as you wish)
Same options as for developers
product hardware observed behaviour screen shots
component operating system expected behaviour code examples
version summary steps to reproduce error reports
severity build information stack traces test cases
14

Questions to reporters
Question 1
What information have you previously provided in bug reports?
(select as many as you wish)
Question 2
What of the above information was most difficult to provide?
(select at most three items)
Question 3
In your opinion, what information is most relevant to developers?
(select at most three items)
15

Rated most helpful
by developers
steps to reproduce steps to reproduce steps to reproduce
observed behaviour stack traces test cases
expected behaviour test cases observed behaviour
product observed behaviour stack traces
version screenshots expected behaviour
operating system expected behaviour version
summary code examples code examples
component summary error reports
severity version build information
build information error 16
reports summary

Provided by Rated most helpful
most reporters by developers
steps to reproduce steps to reproduce steps to reproduce
observed behaviour stack traces test cases
expected behaviour test cases observed behaviour
product observed behaviour stack traces
version screenshots expected behaviour
operating system expected behaviour version
summary code examples code examples
component summary error reports
severity version build information
build information error 16
reports summary

Provided by Rated most helpful Rated most helpful
most reporters by developers by reporters
steps to reproduce steps to reproduce steps to reproduce
observed behaviour stack traces test cases
expected behaviour test cases observed behaviour
product observed behaviour stack traces
version screenshots expected behaviour
operating system expected behaviour version
summary code examples code examples
component summary error reports
severity version build information
build information error 16
reports summary

Provided by Rated most helpful Rated helpful
most reporters by developers by most reporters
steps to reproduce steps to reproduce steps to reproduce
observed behaviour stack traces test cases
expected behaviour test cases observed behaviour
product observed behaviour stack traces
version screenshots expected behaviour
operating system expected behaviour version
summary code examples code examples
component summary error reports
severity version build information
build information error 17
reports summary

Provided by Rated most helpful Rated helpful
most reporters by developers by most reporters
steps to reproduce steps to reproduce steps to reproduce
observed behaviour stack traces test cases
expected behaviour test cases observed behaviour
productReporters need betterstack traces
observed behaviour
version tool assistance. expected behaviour
screenshots
operating system expected behaviour version
summary code examples code examples
component summary error reports
severity version build information
build information error 17
reports summary

Provided by Rated most helpful Rated helpful
most reporters by developers by most reporters
steps to reproduce steps to reproduce steps to reproduce
observed behaviour stack traces test cases
expected behaviour test cases observed behaviour
product observed behaviour stack traces
version screenshots expected behaviour
operating system expected behaviour version
summary code examples code examples
component summary error reports
severity version build information
build information error 18
reports summary

1 2
Survey Cuezilla
19

Assist reporters
20

Assist reporters
20

Assist reporters
20

Rating bug reports
Create a new plugin using CDT.
I wand to create a new plugin in Eclipse using CDT. Shall it
possible. I had made a R&D in eclipse documentation. I had get an
idea about create a plugin using Java. But i wand to create a new
plugin ( user defined plugin ) using CDT. After that I wand to
impliment it in my programe. If it possible?. Any one can help me
please...
21

Rating bug reports
Create a new plugin using CDT.
I wand to create a new plugin in Eclipse using CDT. Shall it
possible. I had made a R&D in eclipse documentation. I had get an
idea about create a plugin using Java. But i wand to create a new
plugin ( user defined plugin ) using CDT. After that I wand to
impliment it in my programe. If it possible?. Any one can help me
please...
Poor Average Good
21

Rating bug reports
Total reports voted: 289 Total votes received: 1,186
Create a new plugin using CDT.
I wand to create a new plugin in Eclipse using CDT. Shall it
possible. I had made a R&D in eclipse documentation. I had get an
idea about create a plugin using Java. But i wand to create a new
plugin ( user defined plugin ) using CDT. After that I wand to
impliment it in my programe. If it possible?. Any one can help me
please...
Poor Average Good
21

CUE
Bug 31021
Summary: Tree - Selection listener stops default expansion
Product: [Eclipse] Platform Reporter: Veronika Irvine <veronika_irvine@ca.ibm.com>
Component: SWT Assignee: Steve Northover <steve_northover@ca.ibm.com>
Status: RESOLVED WONTFIX QA Contact:
ZILLA
Severity: normal
Priority: P3
Version: 2.1
Target Milestone: ---
Hardware: PC
OS: Windows XP
Whiteboard:
Opened: 2003-02-05 14:52 -
Description:
0400
I20030205
Run the following example. Double click on a tree item and notice that it does
not expand.
Comment out the Selection listener and now double click on any tree item and
notice that it expands.
public static void main(String[] args) {
Display display = new Display();
Shell shell = new Shell(display);
shell.setLayout(new FillLayout());
Tree tree = new Tree(shell, SWT.BORDER);
for (int i = 0; i < 4; i++) {
TreeItem item = new TreeItem(tree, SWT.NONE);
item.setText(\"item \" + i);
for (int j = 0; j < 4; j++) {
TreeItem subItem = new TreeItem(item, SWT.NONE);
subItem.setText(\"item \" + j);
for (int k = 0; k < 4; k++) {
TreeItem subsubItem = new TreeItem(subItem,
SWT.NONE);
subsubItem.setText(\"item \" + k);
}
}
}
tree.addSelectionListener(new SelectionAdapter() {});
shell.setSize(200, 200);
shell.open();
while (!shell.isDisposed()) {
if (!display.readAndDispatch())
display.sleep();
}
display.dispose();
}
------- Comment #1 From Steve Northover 2004-10-07 19:39:16 -0400 -------
This behavior can't be changed now because application code now relies on it.
Sorry.
22

CUE
Bug 31021
Summary: Tree - Selection listener stops default expansion
Product: [Eclipse] Platform Reporter: Veronika Irvine <veronika_irvine@ca.ibm.com>
Component: SWT Assignee: Steve Northover <steve_northover@ca.ibm.com>
Status: RESOLVED WONTFIX QA Contact:
ZILLA
Severity: normal
Priority: P3
Version: 2.1
Target Milestone: ---
Hardware: PC
OS: Windows XP
Whiteboard:
Opened: 2003-02-05 14:52 -
Description:
0400
I20030205
Run the following example. Double click on a tree item and notice that it does
not expand.
Comment out the Selection listener and now double click on any tree item and
notice that it expands.
public static void main(String[] args) {
Display display = new Display();
Shell shell = new Shell(display);
shell.setLayout(new FillLayout());
Tree tree = new Tree(shell, SWT.BORDER);
for (int i = 0; i < 4; i++) {
TreeItem item = new TreeItem(tree, SWT.NONE);
item.setText(\"item \" + i);
for (int j = 0; j < 4; j++) {
TreeItem subItem = new TreeItem(item, SWT.NONE);
subItem.setText(\"item \" + j);
for (int k = 0; k < 4; k++) {
TreeItem subsubItem = new TreeItem(subItem,
SWT.NONE);
subsubItem.setText(\"item \" + k);
}
}
}
tree.addSelectionListener(new SelectionAdapter() {});
shell.setSize(200, 200);
shell.open();
while (!shell.isDisposed()) {
if (!display.readAndDispatch())
display.sleep();
}
display.dispose();
}
------- Comment #1 From Steve Northover 2004-10-07 19:39:16 -0400 -------
This behavior can't be changed now because application code now relies on it.
Sorry.
22

detects Itemisations
Keywords
CUE
Code samples
Stack traces
ZILLA
Patches
Screenshots
computes Readability
23

Build and evaluate Cuezilla
build
CUE
ZILLA
24

Build and evaluate Cuezilla
build
CUE
ZILLA
evaluate
CUE
ZILLA
24

Evaluation Setup
Within projects
25

Evaluation Setup
Within projects
48% agreement
91% off by one
25

Make recommendations
CUE
ZILLA
+ Stack trace
26

Make recommendations
CUE
ZILLA
+ Stack trace
26

Incentives to reporters
27

Incentives to reporters
• Bug reports containing stack traces get
fixed sooner.
• Bug reports that are easier to read have
lower lifetimes.
• Code samples in your bug report increase
chances of it getting fixed.
28

Duplicates
“ Duplicates often add useful information. That
this information is filed under a new report is
not ideal.
Apache Developer ”
29

Future of bug tracking
lies in thinking
outside the box!
30

Summary
Provided by Rated most helpful Rated helpful
most reporters by developers by most reporters
Responses
Helped most
steps to reproduce
steps to reproduce steps to reproduce steps to reproduce
stack traces observed behaviour stack traces test cases
expected behaviour test cases observed behaviour
screenshots
product observed behaviour stack traces
observed behavior
version screenshots expected behaviour
test cases/examples
operating system expected behaviour version
expected behavior
summary code examples code examples
component summary error reports
version
severity version build information
hardware severity
build information error reports summary
14 Used when fixing a bug
Assist reporters Future of bug tracking
lies in thinking
outside the box!
29
31