Predicting
Recurring
Crash Stacks
Hyunmin Seo and Sunghun Kim
The Hong Kong University Of Science And Technology
September 7th, 2012
Automated Software Engineering 2012
Essen, Germany

Recurring Crashes
Bug
Report
52831
1
3.6b3 Patch 3.6b4
Crash Point Crash Point
nsXULTreeAccessible:: nsXULTreeAccessible::
GetTreeItemAccessible GetTreeItemAccessible
2

Bad Fixes
• Bad fixes comprise
as much as 9% of all bugs
(Gu et al. ICSE 2010)
• 14.8%∼24.4% of fixes for
post-release bugs are incorrect
(Yin et al. FSE 2011)
3

Motivation
• How often do bad fixes occur?
• How can we help to prevent it?
4

Crash Reporting System
(CRS)
5

Mozilla CRS
CR
RELEASE
CRS
CR SERVER
6

Mozilla CRS
CRS
SERVER
7

Mozilla CRS
Bug
Report
#50001
Patch File
NEXT RELEASE 8

How often are bad fixes?
Crash Bug
Reporting Reporting
System System
19 sub-versions of Firefox 3.6
70 Bug Reports
79 Crash Points
9

Have all crashes
disappeared after fixes?
?
Crash
Report
Bug
Report
#5000
1
Before fix Patch After fix
10

Recurring Crash Examples
BUGID CRASH POINT ver1 ver2 ver3
nsHtml5ElementName:: 3.6.8 3.6.9 3.6.10
538722 initializeStatics 677 0 0
3.6.6 3.6.7 3.6.8
554544 nsTextFrame::Reflow
773 186 497
nsXULTreeAccessible:: 3.6b3 3.6b4 3.6b5
528311 GetTreeItemAccessible 70 168 0
48.1 % (38/79)
11

Crash Paths
• The same crash point but different
crash paths
• The fix may miss some paths
12

BUGID CRASH POINT ver1 ver2 ver3
nsXULTreeAccessible:: 3.6b3 3.6b4 3.6b5
528311 GetTreeItemAccessible 70 168 0
35 175
30 150
# of Crash Reports
# of Crash Reports
25 125
20 100
15 75
10 50
5 25
0 0
#1 #2 #3 #4 #5 #1 #2 #3 #4 #5
Sub-group Sub-group
3.6b3 3.6b4 13

Comment in Bug Report
“I don’t know how this bit (crash
trace) got lost from the patch I
ended up checking in, but it’s pretty
essential...”
A comment in bug report #523528
14

Incomplete Fixes
• We call this as incomplete fixes
• “incomplete” in terms of fix
locations
• How can we help to prevent this?
15

Approach Overview
Covered
Bug
Report
#5000
1
Patch File
Missing
16

Idea behind Classification
A fix has nothing to do
if it is not executed
✓
Fix Location
17

Stack Expansion
L-1 L-2 L-3 Entry
A
if
G H I Path 1 Path 2
Block1 Block2
B G ( )
J Y ( ) B ( )
X ( )
C
Exit
D ✓
CFG of A
F
E
Covered ( if S F )
Crash Stack Missing ( otherwise )
18

Experimental Design
RQ1 - How good is the classification?
RQ2 - How can this help developers?
19

Subjects
Name Description
Subject 19 releases of Firefox 3.6
Release Date Oct 2009 ~ Mar 2011
Programming Language C / C++
LOC 3.2M ~ 4.4M
Name Value
# of crash buckets 33
# of total sub-groups 1159
# of recurring sub-groups 354
20

Experimental Design
RQ1 - How good is the classification?
RQ2 - How can this help developers?
21

RQ1 - Prediction Result
L4 Prediction Actual
292 167
Precision 0.57
Recall 0.49
F-measure 0.53
22

RQ1 - Expansion Levels
0.9
0.8
0.7
0.6
0.5
Value
0.4
0.3 PRECISION
0.2 RECALL
0.1 F-MEASURE
0
L-0 L-1 L-2 L-3 L-4 L-5 L-7 L-10 L-∞
Expansion Level
23

Experimental Design
RQ1 - How good is the classification?
RQ2 - How can this help developers?
24

IEnumConnectionPoints
trace a
RQ2 - Case Study
_RemoteNext_Thunk
trace b
IEnumOleUndoUnits
_Next_Stub
nsAccessibleWrap nsRootAccessible
::Next ::HandleEvent
nsXULTreeAccessible
::GetChildAt ✓ nsRootAccessible::
HandleEventWithTarget
First Fix (#528311)
in 3.6b4 Second Fix (#528311)
nsXULTreeAccessible:: ✓ in 3.6b5
GetTreeItemAccessible
crash point
286 NS_ENSURE_ARG_POINTER(aChild); 545 *aAccessible = nsnull;
287 *aChild = nsnull; 546
288 547- if (aRow < 0)
289+ if (IsDefunct()) 547+ if (aRow < 0 || IsDefunct ())
290+ return NS_ERROR_FAILURE; 548 return;
291 549
292 PRInt32 childCount = 0; 550 PRInt32 rowCount = 0;
25

RQ2 - Developer
Feedback
Firefox developer emails and mailing lists
21 responses - 3 very useful, 7 useful
10“It should be an interesting feature 1 not useful
requested more information, and useful
like any automation tool. It should make the
engineering work easier and keep users less
annoyed.”
“The first patch fixed the known steps but
missed the fact that other routes led to the
same state inconsistency. ... If you have a
system that automates that process it would
indeed be helpful.” 26

Threats to validity
• The subject is open source software
• Collected crash data might be biased
• Oracle data set is incomplete
27

Discussion – Future Work
nsJARInputThunk::EnsureJarStream
nsZipReaderCache::GetZip
nsJAR::Open
nsZipArchive::OpenArchive
crash point ✓ nsZipArchive::BuildFileList
539 //-- Read the central directory headers
540 buf = startp + centralOffset;
541+ if (endp - buf < sizeof(PRUint32))
542+ return NS_ERROR_FILE_CORRUPTED;
543 PRUint32 sig = xtolong(buf); // crash point
544 while (sig == CENTRALSIG) {
28

Related Work
• Crash bucketing
(Dang et al., ICSE 2012)
• Post-mortem crash analysis
(Manevich et al., FSE 2004)
• Bug fix verification
(Gu et al., ICSE 2010)
29

Conclusions
• 48% of fixed crashes in Firefox recurred.
• We present an approach to predict recurring
crashes
• RQ1 - How good is the classification?
• Our approach yields reasonable accuracy - 0.57
precision and 0.49 recall
• RQ2 - How can this help developers?
• Our case studies and developers’ feedback show the
idea is useful 30