The Scientific Method in
Software Evaluation
Speaker: Benjamin Heitmann
<benjamin.heitmann@deri.org>
1

Motivation
• The Scientific Method guarantees a certain
degree of objectivity in Science
• following it improves quality of work
• our work takes the form of Software
• part of the scientific method is evaluation
• How can Software be evaluated?
2
Benjamin Heitmann: The Scientific Method in Software Evaluation
2

Overview
• Is Computer Science a Science ?
hy
• The Scientific Method W
• Reasons to (not) Experiment
• Evaluating the Design ow
H
• Evaluating the Implementation
3
Benjamin Heitmann: The Scientific Method in Software Evaluation
3

Is Computer Science a
Science?
Reference:
Denning, Is Computer Science Science ?
4

Denning’s Definition
Computer Science:
the science of information processes and
their interactions with the world
• studies artificial and natural information
processes
• blend of Science, Engineering, Mathematics
and Art
5 Chapter 1 of 5
Benjamin Heitmann: The Scientific Method in Software Evaluation
5

Other Definitions
• the study of phenomena related to
computers [Newell, Perlis and Simon, 1967]
• the study of information structures
[Wegner, 1968]
• the study and management of complexity
[Dijkstra, 1969]
6 Chapter 1 of 5
Benjamin Heitmann: The Scientific Method in Software Evaluation
6

Real Science
Francis Bacon:
Scientific Method:
• Process of forming hypotheses and verifying
them through experimentation
• Successful hypotheses become models
explaining and predicting the world
7
Benjamin Heitmann: The Scientific Method in Software Evaluation Chapter 1 of 5
7

Example
• Example: the study of algorithms
• e.g.: sort algorithms
• Hypothesis:
An algorithm executes with certain
performance, time and storage requirements
• Verification: by experimentation with real
world data
8
Benjamin Heitmann: The Scientific Method in Software Evaluation Chapter 1 of 5
8

Non-obvious problems solved
by computing principles
Computation: Communication:
Non-computability of Lossy but high-fidelity
some important audio compression
problems
Secure cryptographic key
Optimal algorithms for exchange in open
some common problems networks
Design: Objects and
information hiding
9 Chapter 1 of 5
Benjamin Heitmann: The Scientific Method in Software Evaluation
9

Interdisciplinary
Research
Constantly new relationships with other
research fields:
• Bioinformatics
• Cognitive science
• Computer linguistics
• Immersive computing
• Quantum computing
Benjamin Heitmann: The Scientific Method in Software Evaluation 10 Chapter 1 of 5
10

True Science needs
Validation
Tichy:
50 % of CS papers proposing models and
hypotheses are not validated (papers
published before 1995)
In other fields, only 15 % not validated.
Benjamin Heitmann: The Scientific Method in Software Evaluation 11 Chapter 1 of 5
11

The Scientific Method
Reference:
Dodig-Crnkovic, Scientific Methods in Computer Science
12

Diagram of the Scientific
Method
1 2 3
Existing
Theories and Hypothesis Predictions
Observations
adjust
Hypothesis
4 Tests and new
6 Select among
competing
Observations
theories
5 Confirm old
theory or Recursive
consistency
propose new
and cyclic
achieved
theory
13 Chapter 2 of 5
13

Advantages
• Reproducible Results
• Impartiality
• Acceptance based on results of logical
reasoning, observations and experiments
Benjamin Heitmann: The Scientific Method in Software Evaluation 14 Chapter 2 of 5
14

Scientific Methods of
Computer Science
• Overlying principle: Modeling
• Theoretical CS
• Experimental CS
• Computer Simulation
Benjamin Heitmann: The Scientific Method in Software Evaluation 15 Chapter 2 of 5
15

Reasons to (not)
experiment
Reference:
Tichy, Should Computer Scientists Experiment More?
16

Reasons for
experimentation
• Falsification of theories
• Revealing of assumptions
• Eliminate alternative explanations
• Enable induction: deriving theories from
observation
Benjamin Heitmann: The Scientific Method in Software Evaluation 17 Chapter 3 of 5
17

But “experiments cost
to much”
• Experimentation requires more resources
then theory, so why do it?
• Rebuttal:
• Meaningful experiments can fit into small
budgets
• Expensive experiments can be worth
more than their cost
Benjamin Heitmann: The Scientific Method in Software Evaluation 18 Chapter 3 of 5
18

But “too many variables”
• Impossible to control all variables
• useless results
• Rebuttal:
• create benchmarks
• control variables and errors like e.g. in
medicine
Benjamin Heitmann: The Scientific Method in Software Evaluation 19 Chapter 3 of 5
19

But “experimentation
slows progress”
• Slower flow of ideas, if experimentation is
required
• Rebuttal: better identification of
questionable ideas
Benjamin Heitmann: The Scientific Method in Software Evaluation 20 Chapter 3 of 5
20

But “technology changes
to fast”
• Experiment duration exceeds lifetime of
software tool / product
• Rebuttal: question is too narrow
Benjamin Heitmann: The Scientific Method in Software Evaluation 21 Chapter 3 of 5
21

Evaluating the Design
Reference:
Leist, Zellner; Evaluation of Current Architecture
Frameworks
22

Methodology
1. establish theoretical foundation
2. create list of features based on foundation
3. analyse if subjects exhibit features
Benjamin Heitmann: The Scientific Method in Software Evaluation 23 Chapter 4 of 5
23

Evaluating Enterprise
Architecture Descriptions
• Theoretical foundation: Method Engineering
• List of features:
• Meta Model
• Procedure Model
• Modeling Technique
• Role document
• Specification Document
Benjamin Heitmann: The Scientific Method in Software Evaluation 24 Chapter 4 of 5
24

Another in
Table 1 Evaluation of current Architectures
very balan
shows for
C4ISR/DoDAF
methodic p
appropriate
Zachman
TOGAF
objectives
TEAF
FEAF
MDA
ARIS
further rese
focus on si
appropriate
\"
! ! ! ! ! !
Specification
document
7. R
# \" # \" #
! !
Meta model
\" # # \" \"
! !
Role [1] Arch
(MDA
\" # \" # \"
! !
Technique
http:/
\" #
! ! ! ! !
Procedure model
(Acc
! Fully accomplished
Legend:
[2] Blevi
# Partly accomplished
and M
\" Not accomplished
ADM
For most of the architecture frameworks the strengths and 2005
weaknesses are very balanced. The adequacy of a architecture [3] Bran
framework to support a development project could be evaluated archi
only in consideration of specific circumstances of a case. [4] Brink
25 Chapter 4 of 5
Meth
The results of the evaluation indicate at the same time potentials 25

Evaluating the
Implementation
Reference:
Bansiya, Evaluating Framework Architecture Structural
Stability
26

Methodology
1. establish hypothesis
2. Identify characteristics related to hypothesis
3. define metrics to assess each characteristic
4. collect data from metrics
5. determine if data supports hypothesis
Benjamin Heitmann: The Scientific Method in Software Evaluation 27 Chapter 5 of 5
27

Evaluating Structural Stability of Windows
Application Frameworks
• Hypothesis: extent of change stabilises with
maturity
Characteristics Metric
Design Size in Classes count
number of single/
count
multiple inheritances
average inheritance divide sum of maximum
path lengths of all classes
depth/width by number of classes
Benjamin Heitmann: The Scientific Method in Software Evaluation 28 Chapter 5 of 5
28

taking the difference of the aggregate-change (Vi) of a version (i,with i > 0) with the aggregate-
change (V1) value of the first version.
Table 4. Normalized architecture assessment metric values of MFC and OWL frameworks
Design Metric MFC MFC MFC MFC MFC OWL OWL OWL OWL
1.0 2.0 3.0 4.0 5.0 4.0 4.5 5.0 5.2
1 1.28 1.43 1.56 1.13 1 1.73 2.51 1.00
Design Size
1 1 1 5 1.20 1 1.33 1.81 0.93
Hierarchies
1 1.27 1.39 1.49 1.11 1 1.65 3.61 0.98
Single Inheritances
1 1.00 1.00 1.00 1.00 1 1.44 2.31 1.00
Multiple Inheritance
1 1.26 1.16 0.95 0.99 1 1.04 1.75 0.99
Depth of Inheritance
1 0.99 0.98 0.96 0.98 1 0.94 1.33 0.98
Width of Inheritance
1 1.26 1.16 0.95 0.99 1 0.97 1.61 0.99
Number of Parents
1 1.70 1.26 1.20 0.96 1 0.87 1.55 0.99
Number of Methods
1 1.26 1.19 1.03 0.96 1 1.21 2.09 1.00
Class Coupling
9.0 11.01 10.57 14.14 9.31 9.0 11.18 18.57 8.86
Aggregate Change (Vi )
Extent-of-Change(Vi -V1) 0.0 2.01 1.57 5.14 0.31 0.0 2.18 9.57 -0.14
The extent-of-change measure can be used as an indicator (index) of framework architecture
stability. The value of the measure indicates the relative stability of the architecture structure, while
29 Chapter 5 of 5
higher values of the measure are reflective of higher instability in the structure, values closer to zero
29

Conclusion
• Computer Science is a true science
• In order to have measurable results you have
to choose hypotheses which can actually be
measured.
• Remember falsifiability tests!
• Measure your own progress, by evaluating
your approach
Benjamin Heitmann: The Scientific Method in Software Evaluation 30
30

References:
Denning, Is Computer Science Science? : http://cs.gmu.edu/cne/pjd/PUBS/
CACMcols/cacmApr05.pdf
Dodig-Crnkovic, Scientific Methods in Computer Science: http://
www.mrtc.mdh.se/publications/0446.pdf
Tichy, Should Computer Scientists Experiment More? : http://
ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=675631
Leist, Zellner, Evaluation of Current Architecture Frameworks : http://
portal.acm.org/citation.cfm?id=1141635
Bansiya, Evaluating Framework Architecture Structural Stability: http://
portal.acm.org/citation.cfm?doid=351936.351954
Brown, Wallnau, A Framework for Evaluating Software Technology: http://
ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=536457
Benjamin Heitmann: The Scientific Method in Software Evaluation
31