Icpc2010 bettenburg

Studying the Impact of Social
Structures on Software Quality
1
Nicolas Bettenburg and Ahmed E. Hassan
Queen’s University, Kingston, Canada

James Larus
Microsoft Research
“If software developers
were angels, debugging
would be unnecessary!”
(in ‘Why programs fail’, Elsevier 2009)
2

Software errors cost
companies millions of dollars
each year!
3

Measure the
source code...
complexity
cohesion
coupling
churn
...
bugs
5

6
...and create a Prediction Model

Measurements
Risk of Bugs
New Source Code
7

We can look inside the
Box and study the model!
8
vs
8
We can also use a
prediction model to
understand factors!

In this work:
SOCIAL
STRUCTURES
9

300,000 Bug Reports (BugZilla)
224,643 Transactions (CVS)
Linked through
Change-Log Analysis
11

Eclipse 3.0
time
6 months
Measure
Discussion
Metrics
6 months
Measure
Post-Release
Bugs
12

Discussion
CONTENT
Social
STRUCTURES
Communication
DYNAMICS
Measures of
WORKFLOW
4 Dimensions
of Measures
13

Bug 137808
Summary: Exceptions from createFromString lock-up the editor
Product: [Modeling] EMF Reporter: Patrick Sodre <psodre@gmail.com>
Component: Core Assignee: Marcelo Paternostro <marcelop@ca.ibm.com>
Status: VERIFIED FIXED QA Contact:
Severity: normal
Priority: P3 CC: merks@ca.ibm.com
Version: 2.2
Target Milestone: ---
Hardware: PC
OS: Windows XP
Whiteboard:
Description:
Opened: 2006-04-20 14:25 -
0400
As discussed on the newsgroup under the Thread with the same name I am opening
this bug entry. Here is a history of the thread.
-- From Ed Merks
Patrick,
The value is checked before it's applied and can't be applied until it's valid.
But this BigDecimal cases behaves oddly because the exception thrown by
new BigDecimal("badvalue")
has a null message and the property editor relies on returning a non-null
message string to indicate there is an error.
Please open a bugzilla which I'll fix like this:
### Eclipse Workspace Patch 1.0
#P org.eclipse.emf.edit.ui
Index: src/org/eclipse/emf/edit/ui/provider/PropertyDescriptor.java
===================================================================
RCS file:
/cvsroot/tools/org.eclipse.emf/plugins/org.eclipse.emf.edit.ui/src/org/eclipse/emf/edit/ui/provider/PropertyDescriptor.java,v
retrieving revision 1.10
diff -u -r1.10 PropertyDescriptor.java
--- src/org/eclipse/emf/edit/ui/provider/PropertyDescriptor.java 21 Mar 2006
16:42:30 -0000 1.10
+++ src/org/eclipse/emf/edit/ui/provider/PropertyDescriptor.java 20 Apr 2006
Source Code
Metadata

11:59:10 -0000
@@ -162,7 +162,8 @@
}
catch (Exception exception)
{
- return exception.getMessage();
+ String message = exception.getMessage();
+ return message == null ? exception.toString() : message;
}
}
Diagnostic diagnostic =
Diagnostician.INSTANCE.validate(EDataTypeCellEditor.this.eDataType, value);
Patrick Sodre wrote:
Hi,
It seems that if the user inputs an invalid parameter that gets created from
"createFromString" the Editor locks-up until the user explicitly calls "restore
Default Value".
Is this the expected behavior or could something better be done? For
instance if an exception is thrown restore the value back to what it was before
after displaying a pop-up error message.
I understand that for DataTypes defined by the user he/she should take care
of catching the exceptions but for the default ones like BigInteger/BigDecimal
I think the EMF runtime could do some of the grunt work...
If you think this is something worth pursuing I could post an entry in
Bugzilla.
Regards,
Patrick Sodre
Below is the stack trace that I got from the Editor...
java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:368)
at
org.eclipse.emf.ecore.impl.EcoreFactoryImpl.createEBigDecimalFromString(EcoreFactoryImpl.java:559)
at
org.eclipse.emf.ecore.impl.EcoreFactoryImpl.createFromString(EcoreFactoryImpl.java:116)
at
org.eclipse.emf.edit.ui.provider.PropertyDescriptor$EDataTypeCellEditor.doGetValue(PropertyDescriptor.java:183)
at org.eclipse.jface.viewers.CellEditor.getValue(CellEditor.java:449)
at
org.eclipse.ui.views.properties.PropertySheetEntry.applyEditorValue(PropertySheetEntry.java:135)
at
org.eclipse.ui.views.properties.PropertySheetViewer.applyEditorValue(PropertySheetViewer.java:249)
at
------- Comment #1 From Ed Merks 2006-04-20 15:09:23 -0400 -------
The fix has been committed to CVS. Thanks for reporting this problem.
Patch
Stack Trace

Discussion
CONTENT
Social
STRUCTURES
Communication
DYNAMICS
Measures of
WORKFLOW
4 Dimensions
of Measures
16

# PARTICIPANTS IN
DISCUSSION
ROLE OF
PARTICIPANTS
CENTRALITY
Bug 137808
Severity: normal
Version: 2.2
Hardware: PC
OS: Windows XP
Whiteboard:
Description:
Opened: 2006-04-20 14:25 -
0400
-- From Ed Merks
Patrick,
===================================================================
RCS file:
16:42:30 -0000 1.10
11:59:10 -0000
@@ -162,7 +162,8 @@
}
{
}
}
Hi,
Default Value".
Bugzilla.
Regards,
Patrick Sodre
at
at
at
at
at
at
REPUTATION
17

Discussion
CONTENT
Social
STRUCTURES
Communication
DYNAMICS
Measures of
WORKFLOW
4 Dimensions
of Measures
18

Bug 137808
Severity: normal
Version: 2.2
Hardware: PC
OS: Windows XP
Whiteboard:
Description:
Opened: 2006-04-20 14:25 -
0400
-- From Ed Merks
Patrick,
===================================================================
RCS file:
16:42:30 -0000 1.10
11:59:10 -0000
@@ -162,7 +162,8 @@
}
{
}
}
Hi,
Default Value".
Bugzilla.
Regards,
Patrick Sodre
at
at
at
at
at
at
# MESSAGES
MESSAGE
LENGTH
INTERESTINGNESS
REPLY TIME
19

Bug 137808
Severity: normal
Version: 2.2
Hardware: PC
OS: Windows XP
Whiteboard:
Description:
Opened: 2006-04-20 14:25 -
0400
-- From Ed Merks
Patrick,
===================================================================
RCS file:
16:42:30 -0000 1.10
11:59:10 -0000
@@ -162,7 +162,8 @@
}
{
}
}
Hi,
Default Value".
Bugzilla.
Regards,
Patrick Sodre
at
at
at
at
at
at
The fix has been committed to CVS. Thanks for reporting this problem. 20
# MESSAGES
MESSAGE
LENGTH
INTERESTINGNESS
REPLY TIME

Bug 137808
Severity: normal
Version: 2.2
Hardware: PC
OS: Windows XP
Whiteboard:
Description:
Opened: 2006-04-20 14:25 -
0400
-- From Ed Merks
Patrick,
===================================================================
RCS file:
16:42:30 -0000 1.10
11:59:10 -0000
@@ -162,7 +162,8 @@
}
{
}
}
Hi,
Default Value".
Bugzilla.
Regards,
Patrick Sodre
at
at
at
at
at
at
# MESSAGES
MESSAGE
LENGTH
INTERESTINGNESS
REPLY TIME

Discussion
CONTENT
Social
STRUCTURES
Communication
DYNAMICS
Measures of
WORKFLOW
4 Dimensions
of Measures
23

24
2.3 Life Cycle of Bug Reports
UNCONFIRMED
ASSIGNED
FIXED
WORKSFORME
WONTFIX
INVALID
DUPLICATE
NEW / REOPEN VERIFIEDRESOLVED
CLOSED

REGRESSION MODEL
the amount of post-release defects into relation of our pre-
release measures. The complete linear regression model has
the form
Defects =

i
αi · ContentMeasurei
+

j
βj · StructuralMeasurej
+

k
γk · ComDynMeasurek
+

l
δl · WorkflowMeasurel +
Based on this model, we will investigate the statistical
relationships between the social interaction measures, which
are represented by the regression variables in the model, and
post release defects, represented by the dependent variable 25

REGRESSION MODEL
the form
Defects =

i
+

j
+

k
+

l
post release defects, represented by the dependent variable
DependentVariable
(what is predicted)
25

REGRESSION MODEL
the form
Defects =

i
+

j
+

k
+

l
DependentVariable
(what is predicted)
Regression
Coefﬁcients
25

REGRESSION MODEL
the form
Defects =

i
+

j
+

k
+

l
DependentVariable
(what is predicted)
Regression
Coefﬁcients
Independent
Variables
25

APPROACH
Hierarchical Analysis
(2) Build linear model step-by-step, each time
adding new independent variables.
(3) At each step, evaluate how well the set of
independent variables explains the data.
Goal: Investigate the effect of each
variable on the model.
Successfully used in the past! (Cataldo, Mockus)
(1) Eliminate variables with high correlation.
26

APPROACH
!
!#
$%'
()*+,-.
()-*!
(/01-2
/01-2)
(1,0-.
1,0-.)
(34(5
(/0,1
(6.7)
(+).,)
)(0-.(1
(!)8
,./39
,./39.
63.(
63.(.
4(1
4(1.
:0
:0.
;;;;;;;;;;;
;;;;;;;;
;;;;;;;;;
;;;;;;
;;;;;;;;; ;;; ;;;;;;;;; ;;;;;;
;;;;;;;;;;;; ;;;; ;;;;;;;;;;; ;;;;;;;
;;;;;;;;;;;; ;;;;;;;;;
;;;;;;;; ;;;;;; ;;;;;; ;;;;
;;;;; ;;;;;; ;;;;;; ;;;;;;
;;;;;; ;;;;;; ; ;;;;;;;;; ;;;;;;;;;
;;;;;; ;;; ;;;; ;;;;; ;;;;;;;;;
;;;;;;;;; ;;; ;;;;;; ;;;;;;;;;;;; ;;; ;;;;
;;;;;;;;; ;;;;;;; ;;;;;;;
;;;;; ;;;; ;;;; ;;;;;;;;
;;;;;; ; ;;; ;;;;;;;;;;;;;
;;;;;;;;;;;; ;;;;;;;;
;;;;;;;;;;;;
;;;;;;;;; ;;;;;;;;;;;;;
;;;;;;;;; ;;; ;;;;;;;;;;;;;;; ;;;
;;;
:0. :0 4(1. 4(1 63.(. 63.( ,./39. ,./39 (!)8 )(0-.(1 (+).,) (6.7) (/0,1 (34(5 1,0-.) (1,0-. /01-2) (/01-2 ()-*! ()*+,-. $%'
!##
=
=
==
27

APPROACH
!
!#
$%'
()*+,-.
()-*!
(/01-2
/01-2)
(1,0-.
1,0-.)
(34(5
(/0,1
(6.7)
(+).,)
)(0-.(1
(!)8
,./39
,./39.
63.(
63.(.
4(1
4(1.
:0
:0.
;;;;;;;;;;;
;;;;;;;;
;;;;;;;;;
;;;;;;
;;;;;;;;; ;;; ;;;;;;;;; ;;;;;;
;;;;;;;;;;;; ;;;; ;;;;;;;;;;; ;;;;;;;
;;;;;;;;;;;; ;;;;;;;;;
;;;;;;;; ;;;;;; ;;;;;; ;;;;
;;;;; ;;;;;; ;;;;;; ;;;;;;
;;;;;; ;;;;;; ; ;;;;;;;;; ;;;;;;;;;
;;;;;; ;;; ;;;; ;;;;; ;;;;;;;;;
;;;;;;;;; ;;; ;;;;;; ;;;;;;;;;;;; ;;; ;;;;
;;;;;;;;; ;;;;;;; ;;;;;;;
;;;;; ;;;; ;;;; ;;;;;;;;
;;;;;; ; ;;; ;;;;;;;;;;;;;
;;;;;;;;;;;; ;;;;;;;;
;;;;;;;;;;;;
;;;;;;;;; ;;;;;;;;;;;;;
;;;;;;;;; ;;; ;;;;;;;;;;;;;;; ;;;
;;;
:0. :0 4(1. 4(1 63.(. 63.( ,./39. ,./39 (!)8 )(0-.(1 (+).,) (6.7) (/0,1 (34(5 1,0-.) (1,0-. /01-2) (/01-2 ()-*! ()*+,-. $%'
!##
=
=
==
Correlogram shows
correlation “hotspots”
27

APPROACH
(1) Eliminate variables with high correlation.!
$%'
()*+,-.
()-*!
(/01-2
/01-2)
;;;;; ;;;;;; ;;;;;; ;;;;;;
;;;;;; ;;;;;; ; ;;;;;;;;; ;;;;;;;;;
;;;;;; ;;; ;;;; ;;;;; ;;;;;;;;;
;;;;; ;;;; ;;;; ;;;;;;;;
;;;;;; ; ;;; ;;;;;;;;;;;;;
:0. :0 4(1. 4(1 63.(. 63.( ,./39. ,./39 (!)8 )(0-.(1 (+).,) (6.7) (/0,1 (34(5 1,0-.) (1,0-. /01-2) (/01-2 ()-*! ()*+,-
igure 1. Pairwise correlations of social interaction measures with levels * p0.05, ** p0.01, *** p0.001. Strength of correlatio
olour intensities; negative correlations are marked with a dashed outline.
Variance Inflation Factor
log(Yi) Model 1 Model 2 Model 3
NSOURCE 3.38 3.38 3.40
NSCOM 3.34 3.34 3.36
NPATCH 3.94 3.88 3.90
PATCHS 3.84 3.82 3.84
NTRACE 4.62 4.60 4.57
TRACES 4.78 4.75 4.70
NLINK 2.24 2.22 1.90
NDEVS 9.32 9.27 1.91
NUSERS 4.55 4.54 2.30
SNACENT 10.66 10.65 —
NMSG 11.63 — —
REPLY 1.17 1.17 1.17
REPLYE 2.04 1.91 1.90
DLEN 4.21 1.91 1.87
DLENE 4.65 1.98 1.96
INT 2.82 2.82 2.60
INTE 1.71 1.71 1.71
WA 2.26 1.99 1.96
WAE 2.08 2.06 2.02
Table II
STEP-WISE ANALYSIS OF MULTICOLLINEARITY.
model that have a variance inflation factor g
We start our analysis with a regression mode
all our variables. The variance inflation fa
model are presented in Table II, Model 1. W
variables that have a variance inflation facto
10. We remove the highest one (NMSG) from
model and recompute the variance inflation f
reduced set of variables. The resulting mode
Table II) contains only one more variable w
factor larger than 10. We remove the regre
(SNACENT) from the model and recomput
factors. In the resulting model (Model 3 in
variables have an inflation factor larger than 5
our analysis of multicollinearity.
C. Hierarchical Analysis Design
After having determined the reduced set
variables with low multicollinearity, we pro
(1) Build model with all
independent variables
(2) Measure VIF of each
independent variable
(3) Remove variable with
highestVIF
Repeat until allVIF below
threshold (t=10)
28

APPROACH
(2) Build linear models step-by-step
log(Yi) MB M1 M2 M3 M4 M5
CHURN 4.996 *** 4.631 *** 4.658 *** 5.303 *** 3.688 *** 4.470 ***
NSOURCE 1.694 *** 1.698 *** 1.772 *** 1.769 *** 1.667 ***
NTRACE 0.79 0.768 0.864 0.881 1.115
NPATCH 0.209 * 0.210 * 0.284 + 0.231 * 0.291
NSCOM 1.218 1.194 1.246 1.208 1.244
PATCHS 12.607 * 12.626 * 11.200 * 12.736 * 18.207 **
TRACES 1.016 1.012 1.004 0.989 0.975
NLINK 1.764 *** 1.613 ** 1.600 ** 1.666 ** 1.596 +
NPART 2.481 2.888 4.480 4.542
NDEVS 0.475 0.582 0.385 0.274
NUSERS 0.749 0.803 0.692 0.792
REPLY 1.019 0.986 0.982
REPLYE 0.117 *** 0.082 *** 0.044 ***
DLEN 0.936 0.898 * 0.876 +
DLENE 2.499 1.251 2.044
INT 0.829 ** 0.821 ** 0.963
INTE 1.109 1.013 1.306
WA 1.432 *** 1.224 +
WAE 2.718 * 2.169
CON1-3 Fig. 2 ***
χ2 559.01 *** 698.5 *** 700.15 731.5 *** 752.3 *** 1055.19 ***
Dev. Expl. 10.71% 13.38 % 13.41 % 14.02 % 14.41 % 26.07 %
∆χ2 139.48 1.652 31.357 20.28 302.87
*** p0.001, ** p0.01, * p0.05, + p 0.1
Baseline Model:
Code Churn
(Ball, Naggapan)
29

APPROACH
CHURN 4.996 *** 4.631 *** 4.658 *** 5.303 *** 3.688 *** 4.470 ***
NSOURCE 1.694 *** 1.698 *** 1.772 *** 1.769 *** 1.667 ***
NTRACE 0.79 0.768 0.864 0.881 1.115
NPATCH 0.209 * 0.210 * 0.284 + 0.231 * 0.291
NSCOM 1.218 1.194 1.246 1.208 1.244
PATCHS 12.607 * 12.626 * 11.200 * 12.736 * 18.207 **
TRACES 1.016 1.012 1.004 0.989 0.975
NLINK 1.764 *** 1.613 ** 1.600 ** 1.666 ** 1.596 +
NPART 2.481 2.888 4.480 4.542
NDEVS 0.475 0.582 0.385 0.274
NUSERS 0.749 0.803 0.692 0.792
REPLY 1.019 0.986 0.982
REPLYE 0.117 *** 0.082 *** 0.044 ***
DLEN 0.936 0.898 * 0.876 +
DLENE 2.499 1.251 2.044
INT 0.829 ** 0.821 ** 0.963
INTE 1.109 1.013 1.306
WA 1.432 *** 1.224 +
WAE 2.718 * 2.169
CON1-3 Fig. 2 ***
χ2 559.01 *** 698.5 *** 700.15 731.5 *** 752.3 *** 1055.19 ***
Dev. Expl. 10.71% 13.38 % 13.41 % 14.02 % 14.41 % 26.07 %
∆χ2 139.48 1.652 31.357 20.28 302.87
*** p0.001, ** p0.01, * p0.05, + p 0.1
Adding ﬁrst dimension
Discussion Content
Increase in explanatory power
statistically signiﬁcant
30

APPROACH
CHURN 4.996 *** 4.631 *** 4.658 *** 5.303 *** 3.688 *** 4.470 ***
NSOURCE 1.694 *** 1.698 *** 1.772 *** 1.769 *** 1.667 ***
NTRACE 0.79 0.768 0.864 0.881 1.115
NPATCH 0.209 * 0.210 * 0.284 + 0.231 * 0.291
NSCOM 1.218 1.194 1.246 1.208 1.244
PATCHS 12.607 * 12.626 * 11.200 * 12.736 * 18.207 **
TRACES 1.016 1.012 1.004 0.989 0.975
NLINK 1.764 *** 1.613 ** 1.600 ** 1.666 ** 1.596 +
NPART 2.481 2.888 4.480 4.542
NDEVS 0.475 0.582 0.385 0.274
NUSERS 0.749 0.803 0.692 0.792
REPLY 1.019 0.986 0.982
REPLYE 0.117 *** 0.082 *** 0.044 ***
DLEN 0.936 0.898 * 0.876 +
DLENE 2.499 1.251 2.044
INT 0.829 ** 0.821 ** 0.963
INTE 1.109 1.013 1.306
WA 1.432 *** 1.224 +
WAE 2.718 * 2.169
CON1-3 Fig. 2 ***
χ2 559.01 *** 698.5 *** 700.15 731.5 *** 752.3 *** 1055.19 ***
Dev. Expl. 10.71% 13.38 % 13.41 % 14.02 % 14.41 % 26.07 %
∆χ2 139.48 1.652 31.357 20.28 302.87
*** p0.001, ** p0.01, * p0.05, + p 0.1
Adding second
dimension
Discussion Content
Increase in
explanatory power:
not statistically
signiﬁcant!
31

APPROACH
CHURN 4.996 *** 4.631 *** 4.658 *** 5.303 *** 3.688 *** 4.470 ***
NSOURCE 1.694 *** 1.698 *** 1.772 *** 1.769 *** 1.667 ***
NTRACE 0.79 0.768 0.864 0.881 1.115
NPATCH 0.209 * 0.210 * 0.284 + 0.231 * 0.291
NSCOM 1.218 1.194 1.246 1.208 1.244
PATCHS 12.607 * 12.626 * 11.200 * 12.736 * 18.207 **
TRACES 1.016 1.012 1.004 0.989 0.975
NLINK 1.764 *** 1.613 ** 1.600 ** 1.666 ** 1.596 +
NPART 2.481 2.888 4.480 4.542
NDEVS 0.475 0.582 0.385 0.274
NUSERS 0.749 0.803 0.692 0.792
REPLY 1.019 0.986 0.982
REPLYE 0.117 *** 0.082 *** 0.044 ***
DLEN 0.936 0.898 * 0.876 +
DLENE 2.499 1.251 2.044
INT 0.829 ** 0.821 ** 0.963
INTE 1.109 1.013 1.306
WA 1.432 *** 1.224 +
WAE 2.718 * 2.169
CON1-3 Fig. 2 ***
χ2 559.01 *** 698.5 *** 700.15 731.5 *** 752.3 *** 1055.19 ***
Dev. Expl. 10.71% 13.38 % 13.41 % 14.02 % 14.41 % 26.07 %
∆χ2 139.48 1.652 31.357 20.28 302.87
*** p0.001, ** p0.01, * p0.05, + p 0.1
32
Adding
third
dimension
Increase in
explanatory
power
signiﬁcant!

APPROACH
CHURN 4.996 *** 4.631 *** 4.658 *** 5.303 *** 3.688 *** 4.470 ***
NSOURCE 1.694 *** 1.698 *** 1.772 *** 1.769 *** 1.667 ***
NTRACE 0.79 0.768 0.864 0.881 1.115
NPATCH 0.209 * 0.210 * 0.284 + 0.231 * 0.291
NSCOM 1.218 1.194 1.246 1.208 1.244
PATCHS 12.607 * 12.626 * 11.200 * 12.736 * 18.207 **
TRACES 1.016 1.012 1.004 0.989 0.975
NLINK 1.764 *** 1.613 ** 1.600 ** 1.666 ** 1.596 +
NPART 2.481 2.888 4.480 4.542
NDEVS 0.475 0.582 0.385 0.274
NUSERS 0.749 0.803 0.692 0.792
REPLY 1.019 0.986 0.982
REPLYE 0.117 *** 0.082 *** 0.044 ***
DLEN 0.936 0.898 * 0.876 +
DLENE 2.499 1.251 2.044
INT 0.829 ** 0.821 ** 0.963
INTE 1.109 1.013 1.306
WA 1.432 *** 1.224 +
WAE 2.718 * 2.169
CON1-3 Fig. 2 ***
χ2 559.01 *** 698.5 *** 700.15 731.5 *** 752.3 *** 1055.19 ***
Dev. Expl. 10.71% 13.38 % 13.41 % 14.02 % 14.41 % 26.07 %
∆χ2 139.48 1.652 31.357 20.28 302.87
*** p0.001, ** p0.01, * p0.05, + p 0.1
33

Model based on
Social Structures
14.41%Variation
Explained
Model based on
Code Metrics
17.04%Variation
Explained
34
(Zimmermann 2007)

Model based on
Social Structures
14.41%Variation
Explained
Model based on
Code Metrics
17.04%Variation
Explained
What happens when we combine both?
34
(Zimmermann 2007)

Model based on
Social Structures
14.41%Variation
Explained
Traditional Model
(Zimmermann,
PROMISE 2007)
17.04%Variation
Explained
Combined Model
30.77%Variation
Explained
35

Strongest
Variables
(Odds Ratios)
Size of Patches
Workﬂow
Code Churn
# Code Examples
# Links
36
Length of the Discussion
Interestigness of Report
# Patches
Reply Time

Strongest
Variables
(Odds Ratios)
Size of Patches
Workﬂow
Code Churn
# Code Examples
# Links
37
Length of the Discussion
Interestigness of Report
# Patches
Reply Time
Correlation,
not Causation!

Icpc2010 bettenburg

Recommended

Recommended

More Related Content

What's hot

What's hot (7)

Viewers also liked

Viewers also liked (6)

Similar to Icpc2010 bettenburg

Similar to Icpc2010 bettenburg (20)

More from SAIL_QU

More from SAIL_QU (20)

Icpc2010 bettenburg