What do programmers believe? Where do they get their beliefs from? Do these beliefs arise out of experience, peer opinion, or research results? Do these beliefs jive with actual empirical evidence from the developers' software projects and workplace? Watch this talk for answers, they may surprise you.

1. Belief & Evidence
in
Empirical
Software Engineering
Premkumar Devanbu
Christian Bird
Tom Zimmermann
1

2. The Human Condition
Belief…
EVIDENCE!
2

3. Actually, Ulcers
are caused
by Bacterial
Infections!
Ulcers are caused
By Stress,
Spicy foods,
alchohol…
No WAY.
4
Marshall, B. J. & Warren, J .R. (1984) Unidentified curved bacilli in the stomach of
patients with gastritis and peptic ulceration. Lancet 1:1311-1315.

4. Autism is caused
By
Vaccinations!
No WAY.
Wakefield et al, 1994: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and
pervasive developmental disorder in children
6

5. Belief & Evidence in
Medicine
Extra-ordinary claims require extra-ordinary evidence.
(Wacholder et al, Ioannides, et al).
Moral Imperatives for Experimental Power.
(Chaloner et al)
Compendia of Evidence for Practitioners.
(See healthypeople.gov)
9

6. What about Software
Engineers..
• Highly Trained, Opinionated, Professionals
• Increasing evidence on important SE Issues (but no
such thing as goodprogramming.gov )
• Do software engineers pay attention to evidence? To
research?
10

7. Bugs Occur
when we
Don’t do X!
But Not
in your
Project
Repo!
Evidence
11

8. I’m going to
do X to avoid
bugs
But..data
says
Otherwise
!
Evidence
12

9. Survey
Programmers
Evidence
Mine+Analyze
Repositories
14
Our Approach
??

10. Survey
Programmers What are the
Beliefs?
How strong?
Where do they
originate?
15
Our Approach

11. SUMMARY
• SURVEY: What do programmers believe?
• SURVEY: How strongly do they believe it?
• SURVEY: Where do they get these beliefs from?
• DATA ANALYSIS: Do their beliefs correspond with
quantitative project evidence?
16
First Such Attempt!

12. What do programmers
believe?Published Findings in
Empirical Software
Engineering
Findings replicable
in Microsoft
project data
Survey
Questions
Uh…ABOUT
WHAT?
17
Actionable
Result
Developers could
form
informed opinion

13. Sample Question
Geographically distributed teams produce code of as
good quality as non-distributed teams.
1. Strongly Agree
2. Agree
3. Neutral
4. Disagree
5. Strongly Disagree
18

14. Responses
• 2500 surveyed, 564 Responses (22%)
• 497 Male, 53 Female
• 267 Bachelors, 211 Masters, 29 PhD
• 386 US, 66 EU, 48 IN, 39 CN, 25 (Other).
19

15. Least Controversial
1. Code Reviews improve Code Quality
2. Coding Standards improve code quality
3. Static Analysis tools improve code quality
20

16. Most Controversial
1. Code Quality depends on programming language
2. Fixing Defects is riskier than adding new features
3. Geographically distributed teams produce code of as
good quality as non-distributed teams.
21

17. Opinion Source
Code quality (defect occurrence) depends on
which programming language is used.
1. Strongly Agree
2. Agree
3. Neutral
4. Disagree
5. Strongly Disagree
Where do they
originate?
22

18. Opinion Source
What factors played a role in your previous
answer? Please choose the relevant factors, and
rank them:
1. Research Papers
2. Articles in Industry Magazines
3. What I hear from my mentors/managers
4. What I hear from my peers
5. Personal Experience
6. Other
23

19. Opinion Formation
24

20. I
BELIEVE
..but
EVIDENCE
Says….
Evidence
25
Same? Different?

21. A Tale of Two Projects
• Project-A: Operating System; 400,000 files, 150 Million
SLOC, began in the Puget Sound area.
• Project-B: Web Service, 430,000 files, 85 Million SLOC,
always distributed.
• Both practice distributed (many buildings, cities, regions,
and countries) development.
• Project-B is a bit more distributed than Project-A.
26

22. A Tale of Two Projects
• Project-A: Operating System; 400,000 files, 150 Million
SLOC, began in the Puget Sound area.
• Project-B: Web Service, 430,000 files, 85 Million SLOC,
always distributed.
• Both practice distributed (many buildings, cities, regions,
and countries) development.
• Project-B is a bit more distributed than Project-A.
Single
Building
Single
City
Single
Region
Single
Country
Project A 56% 90% 91% 92%
Project B 76% 80% 83% 85%
76% of files were
committed by Devs
in one building
85% of files were
committed by Devs
in one country
27

23. A Tale of Two Projects
• Project-A: Operating System; 400,000 files, 150 Million
SLOC, began in the Puget Sound area.
• Project-B: Web Service, 430,000 files, 85 Million SLOC,
always distributed.
• Both practice distributed (many buildings, cities, regions,
and countries) development.
• Project-B is a bit more distributed than Project-A.
”Geographically distributed teams produce
code whose quality (defect occurrence) is just
as good as teams that are not geographically
distributed”
Project-A members tended to DISAGREE
Project-B members tended to AGREE
p < 0.001
28

24. A Tale of Two Projects
• Project-A: Operating System; 400,000 files, 150 Million
SLOC, began in the Puget Sound area.
• Project-B: Web Service, 430,000 files, 85 Million SLOC,
always distributed.
• Both practice distributed (many buildings, cities, regions,
and countries) development.
• Project-B is a bit more distributed than Project-A.
Evidence?
29

25. Evidence…
Single
Building
Single
City
Single
Region
Single
Country
Project A better worse worse worse
Project B better worse worse worse
When Controlling for size, number of commits, number of
authors, and ownership.
Files developed in
single building have
fewer defects!
Files developed in
single city have
more defects!
All Effects
<< Small!
(Cohen’s D)
30

26. Observations
1. Developers beliefs vs. Evidence disparity?
2. Source of opinions: Experience vs. Evidence.
3. Emphasizes importance of Evidence-based Software Engineering..
(Kitchenham, 2004,
Dybå et al, 2005,
Jørgensen et al 2015 )
31

27. Future Work
1. Is developer belief driven by Salience?
2. Are developer beliefs self-serving?
3. Should we re-design our experiments?
4. Should we be making sites like
goodprogramming.gov ?
32

28. Observations
1. Developers beliefs vs. Evidence disparity.
2. Source of opinions: NOT necessarily Scientific Evidence.
3. Emphasizes importance of Evidence-based Software Engineering..
(Kitchenham, 2004,
Dybå et al, 2005,
Jørgensen et al 2015 )
33