Your SlideShare is downloading. ×
0
The Scientific Method in
  Software Evaluation
     Speaker: Benjamin Heitmann
    <benjamin.heitmann@deri.org>




       ...
Motivation
               • The Scientific Method guarantees a certain
                     degree of objectivity in Scienc...
Overview
               • Is Computer Science a Science ?
                                                                ...
Is Computer Science a
      Science?
               Reference:
 Denning, Is Computer Science Science ?



                ...
Denning’s Definition
                     Computer Science:
                     the science of information processes and
 ...
Other Definitions

               •     the study of phenomena related to
                     computers [Newell, Perlis an...
Real Science
                     Francis Bacon:
                          Scientific Method:
               • Process of f...
Example
               • Example: the study of algorithms
               • e.g.: sort algorithms
               • Hypothes...
Non-obvious problems solved
                  by computing principles
                    Computation:                    ...
Interdisciplinary
                                  Research
                     Constantly new relationships with other
...
True Science needs
                           Validation
                     Tichy:
                     50 % of CS paper...
The Scientific Method
                      Reference:
Dodig-Crnkovic, Scientific Methods in Computer Science




          ...
Diagram of the Scientific
                    Method
1                          2                            3
      Existi...
Advantages

               • Reproducible Results
               • Impartiality
               • Acceptance based on resul...
Scientific Methods of
                    Computer Science

               • Overlying principle: Modeling
               •...
Reasons to (not)
           experiment
                   Reference:
Tichy, Should Computer Scientists Experiment More?


...
Reasons for
                             experimentation
               • Falsification of theories
               • Reveal...
But “experiments cost
                      to much”
               • Experimentation requires more resources
            ...
But “too many variables”
               • Impossible to control all variables
               • useless results
           ...
But “experimentation
                    slows progress”

               • Slower flow of ideas, if experimentation is
    ...
But “technology changes
                    to fast”

               • Experiment duration exceeds lifetime of
           ...
Evaluating the Design

                     Reference:
Leist, Zellner; Evaluation of Current Architecture
                ...
Methodology

               1. establish theoretical foundation
               2. create list of features based on foundat...
Evaluating Enterprise
                      Architecture Descriptions
               • Theoretical foundation: Method Engi...
Another in
         Table 1 Evaluation of current Architectures
                                                          ...
Evaluating the
           Implementation
                       Reference:
Bansiya, Evaluating Framework Architecture Stru...
Methodology

               1. establish hypothesis
               2. Identify characteristics related to hypothesis
     ...
Evaluating Structural Stability of Windows
                       Application Frameworks

               • Hypothesis: ext...
taking the difference of the aggregate-change (Vi) of a version (i,with i > 0) with the aggregate-

change (V1) value of t...
Conclusion
               • Computer Science is a true science
               • In order to have measurable results you ha...
References:


                     Denning, Is Computer Science Science? : http://cs.gmu.edu/cne/pjd/PUBS/
               ...
Upcoming SlideShare
Loading in...5
×

Applying the scientific method in Software Evaluation

3,322

Published on

Is Computer Science a real Science? If yes, then how does the scientific method apply to computer science. What are the benefits of doing experiments as a computer scientist?
And how can we apply the scientific method to the evaluation of design and implementation of software?

Published in: Economy & Finance, Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,322
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
189
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "Applying the scientific method in Software Evaluation"

  1. 1. The Scientific Method in Software Evaluation Speaker: Benjamin Heitmann <benjamin.heitmann@deri.org> 1
  2. 2. Motivation • The Scientific Method guarantees a certain degree of objectivity in Science • following it improves quality of work • our work takes the form of Software • part of the scientific method is evaluation • How can Software be evaluated? 2 Benjamin Heitmann: The Scientific Method in Software Evaluation 2
  3. 3. Overview • Is Computer Science a Science ? hy • The Scientific Method W • Reasons to (not) Experiment • Evaluating the Design ow H • Evaluating the Implementation 3 Benjamin Heitmann: The Scientific Method in Software Evaluation 3
  4. 4. Is Computer Science a Science? Reference: Denning, Is Computer Science Science ? 4
  5. 5. Denning’s Definition Computer Science: the science of information processes and their interactions with the world • studies artificial and natural information processes • blend of Science, Engineering, Mathematics and Art 5 Chapter 1 of 5 Benjamin Heitmann: The Scientific Method in Software Evaluation 5
  6. 6. Other Definitions • the study of phenomena related to computers [Newell, Perlis and Simon, 1967] • the study of information structures [Wegner, 1968] • the study and management of complexity [Dijkstra, 1969] 6 Chapter 1 of 5 Benjamin Heitmann: The Scientific Method in Software Evaluation 6
  7. 7. Real Science Francis Bacon: Scientific Method: • Process of forming hypotheses and verifying them through experimentation • Successful hypotheses become models explaining and predicting the world 7 Benjamin Heitmann: The Scientific Method in Software Evaluation Chapter 1 of 5 7
  8. 8. Example • Example: the study of algorithms • e.g.: sort algorithms • Hypothesis: An algorithm executes with certain performance, time and storage requirements • Verification: by experimentation with real world data 8 Benjamin Heitmann: The Scientific Method in Software Evaluation Chapter 1 of 5 8
  9. 9. Non-obvious problems solved by computing principles Computation: Communication: Non-computability of Lossy but high-fidelity some important audio compression problems Secure cryptographic key Optimal algorithms for exchange in open some common problems networks Design: Objects and information hiding 9 Chapter 1 of 5 Benjamin Heitmann: The Scientific Method in Software Evaluation 9
  10. 10. Interdisciplinary Research Constantly new relationships with other research fields: • Bioinformatics • Cognitive science • Computer linguistics • Immersive computing • Quantum computing Benjamin Heitmann: The Scientific Method in Software Evaluation 10 Chapter 1 of 5 10
  11. 11. True Science needs Validation Tichy: 50 % of CS papers proposing models and hypotheses are not validated (papers published before 1995) In other fields, only 15 % not validated. Benjamin Heitmann: The Scientific Method in Software Evaluation 11 Chapter 1 of 5 11
  12. 12. The Scientific Method Reference: Dodig-Crnkovic, Scientific Methods in Computer Science 12
  13. 13. Diagram of the Scientific Method 1 2 3 Existing Theories and Hypothesis Predictions Observations adjust Hypothesis 4 Tests and new 6 Select among competing Observations theories 5 Confirm old theory or Recursive consistency propose new and cyclic achieved theory 13 Chapter 2 of 5 13
  14. 14. Advantages • Reproducible Results • Impartiality • Acceptance based on results of logical reasoning, observations and experiments Benjamin Heitmann: The Scientific Method in Software Evaluation 14 Chapter 2 of 5 14
  15. 15. Scientific Methods of Computer Science • Overlying principle: Modeling • Theoretical CS • Experimental CS • Computer Simulation Benjamin Heitmann: The Scientific Method in Software Evaluation 15 Chapter 2 of 5 15
  16. 16. Reasons to (not) experiment Reference: Tichy, Should Computer Scientists Experiment More? 16
  17. 17. Reasons for experimentation • Falsification of theories • Revealing of assumptions • Eliminate alternative explanations • Enable induction: deriving theories from observation Benjamin Heitmann: The Scientific Method in Software Evaluation 17 Chapter 3 of 5 17
  18. 18. But “experiments cost to much” • Experimentation requires more resources then theory, so why do it? • Rebuttal: • Meaningful experiments can fit into small budgets • Expensive experiments can be worth more than their cost Benjamin Heitmann: The Scientific Method in Software Evaluation 18 Chapter 3 of 5 18
  19. 19. But “too many variables” • Impossible to control all variables • useless results • Rebuttal: • create benchmarks • control variables and errors like e.g. in medicine Benjamin Heitmann: The Scientific Method in Software Evaluation 19 Chapter 3 of 5 19
  20. 20. But “experimentation slows progress” • Slower flow of ideas, if experimentation is required • Rebuttal: better identification of questionable ideas Benjamin Heitmann: The Scientific Method in Software Evaluation 20 Chapter 3 of 5 20
  21. 21. But “technology changes to fast” • Experiment duration exceeds lifetime of software tool / product • Rebuttal: question is too narrow Benjamin Heitmann: The Scientific Method in Software Evaluation 21 Chapter 3 of 5 21
  22. 22. Evaluating the Design Reference: Leist, Zellner; Evaluation of Current Architecture Frameworks 22
  23. 23. Methodology 1. establish theoretical foundation 2. create list of features based on foundation 3. analyse if subjects exhibit features Benjamin Heitmann: The Scientific Method in Software Evaluation 23 Chapter 4 of 5 23
  24. 24. Evaluating Enterprise Architecture Descriptions • Theoretical foundation: Method Engineering • List of features: • Meta Model • Procedure Model • Modeling Technique • Role document • Specification Document Benjamin Heitmann: The Scientific Method in Software Evaluation 24 Chapter 4 of 5 24
  25. 25. Another in Table 1 Evaluation of current Architectures very balan shows for C4ISR/DoDAF methodic p appropriate Zachman TOGAF objectives TEAF FEAF MDA ARIS further rese focus on si appropriate quot; ! ! ! ! ! ! Specification document 7. R # quot; # quot; # ! ! Meta model quot; # # quot; quot; ! ! Role [1] Arch (MDA quot; # quot; # quot; ! ! Technique http:/ quot; # ! ! ! ! ! Procedure model (Acc ! Fully accomplished Legend: [2] Blevi # Partly accomplished and M quot; Not accomplished ADM For most of the architecture frameworks the strengths and 2005 weaknesses are very balanced. The adequacy of a architecture [3] Bran framework to support a development project could be evaluated archi only in consideration of specific circumstances of a case. [4] Brink 25 Chapter 4 of 5 Meth The results of the evaluation indicate at the same time potentials 25
  26. 26. Evaluating the Implementation Reference: Bansiya, Evaluating Framework Architecture Structural Stability 26
  27. 27. Methodology 1. establish hypothesis 2. Identify characteristics related to hypothesis 3. define metrics to assess each characteristic 4. collect data from metrics 5. determine if data supports hypothesis Benjamin Heitmann: The Scientific Method in Software Evaluation 27 Chapter 5 of 5 27
  28. 28. Evaluating Structural Stability of Windows Application Frameworks • Hypothesis: extent of change stabilises with maturity Characteristics Metric Design Size in Classes count number of single/ count multiple inheritances average inheritance divide sum of maximum path lengths of all classes depth/width by number of classes Benjamin Heitmann: The Scientific Method in Software Evaluation 28 Chapter 5 of 5 28
  29. 29. taking the difference of the aggregate-change (Vi) of a version (i,with i > 0) with the aggregate- change (V1) value of the first version. Table 4. Normalized architecture assessment metric values of MFC and OWL frameworks Design Metric MFC MFC MFC MFC MFC OWL OWL OWL OWL 1.0 2.0 3.0 4.0 5.0 4.0 4.5 5.0 5.2 1 1.28 1.43 1.56 1.13 1 1.73 2.51 1.00 Design Size 1 1 1 5 1.20 1 1.33 1.81 0.93 Hierarchies 1 1.27 1.39 1.49 1.11 1 1.65 3.61 0.98 Single Inheritances 1 1.00 1.00 1.00 1.00 1 1.44 2.31 1.00 Multiple Inheritance 1 1.26 1.16 0.95 0.99 1 1.04 1.75 0.99 Depth of Inheritance 1 0.99 0.98 0.96 0.98 1 0.94 1.33 0.98 Width of Inheritance 1 1.26 1.16 0.95 0.99 1 0.97 1.61 0.99 Number of Parents 1 1.70 1.26 1.20 0.96 1 0.87 1.55 0.99 Number of Methods 1 1.26 1.19 1.03 0.96 1 1.21 2.09 1.00 Class Coupling 9.0 11.01 10.57 14.14 9.31 9.0 11.18 18.57 8.86 Aggregate Change (Vi ) Extent-of-Change(Vi -V1) 0.0 2.01 1.57 5.14 0.31 0.0 2.18 9.57 -0.14 The extent-of-change measure can be used as an indicator (index) of framework architecture stability. The value of the measure indicates the relative stability of the architecture structure, while 29 Chapter 5 of 5 higher values of the measure are reflective of higher instability in the structure, values closer to zero 29
  30. 30. Conclusion • Computer Science is a true science • In order to have measurable results you have to choose hypotheses which can actually be measured. • Remember falsifiability tests! • Measure your own progress, by evaluating your approach Benjamin Heitmann: The Scientific Method in Software Evaluation 30 30
  31. 31. References: Denning, Is Computer Science Science? : http://cs.gmu.edu/cne/pjd/PUBS/ CACMcols/cacmApr05.pdf Dodig-Crnkovic, Scientific Methods in Computer Science: http:// www.mrtc.mdh.se/publications/0446.pdf Tichy, Should Computer Scientists Experiment More? : http:// ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=675631 Leist, Zellner, Evaluation of Current Architecture Frameworks : http:// portal.acm.org/citation.cfm?id=1141635 Bansiya, Evaluating Framework Architecture Structural Stability: http:// portal.acm.org/citation.cfm?doid=351936.351954 Brown, Wallnau, A Framework for Evaluating Software Technology: http:// ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=536457 Benjamin Heitmann: The Scientific Method in Software Evaluation 31
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×