• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Applying the scientific method in Software Evaluation
 

Applying the scientific method in Software Evaluation

on

  • 6,356 views

Is Computer Science a real Science? If yes, then how does the scientific method apply to computer science. What are the benefits of doing experiments as a computer scientist? ...

Is Computer Science a real Science? If yes, then how does the scientific method apply to computer science. What are the benefits of doing experiments as a computer scientist?
And how can we apply the scientific method to the evaluation of design and implementation of software?

Statistics

Views

Total Views
6,356
Views on SlideShare
6,342
Embed Views
14

Actions

Likes
4
Downloads
181
Comments
0

2 Embeds 14

http://www.slideshare.net 13
http://www.linkedin.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Applying the scientific method in Software Evaluation Applying the scientific method in Software Evaluation Presentation Transcript

    • The Scientific Method in Software Evaluation Speaker: Benjamin Heitmann <benjamin.heitmann@deri.org> 1
    • Motivation • The Scientific Method guarantees a certain degree of objectivity in Science • following it improves quality of work • our work takes the form of Software • part of the scientific method is evaluation • How can Software be evaluated? 2 Benjamin Heitmann: The Scientific Method in Software Evaluation 2
    • Overview • Is Computer Science a Science ? hy • The Scientific Method W • Reasons to (not) Experiment • Evaluating the Design ow H • Evaluating the Implementation 3 Benjamin Heitmann: The Scientific Method in Software Evaluation 3
    • Is Computer Science a Science? Reference: Denning, Is Computer Science Science ? 4
    • Denning’s Definition Computer Science: the science of information processes and their interactions with the world • studies artificial and natural information processes • blend of Science, Engineering, Mathematics and Art 5 Chapter 1 of 5 Benjamin Heitmann: The Scientific Method in Software Evaluation 5
    • Other Definitions • the study of phenomena related to computers [Newell, Perlis and Simon, 1967] • the study of information structures [Wegner, 1968] • the study and management of complexity [Dijkstra, 1969] 6 Chapter 1 of 5 Benjamin Heitmann: The Scientific Method in Software Evaluation 6
    • Real Science Francis Bacon: Scientific Method: • Process of forming hypotheses and verifying them through experimentation • Successful hypotheses become models explaining and predicting the world 7 Benjamin Heitmann: The Scientific Method in Software Evaluation Chapter 1 of 5 7
    • Example • Example: the study of algorithms • e.g.: sort algorithms • Hypothesis: An algorithm executes with certain performance, time and storage requirements • Verification: by experimentation with real world data 8 Benjamin Heitmann: The Scientific Method in Software Evaluation Chapter 1 of 5 8
    • Non-obvious problems solved by computing principles Computation: Communication: Non-computability of Lossy but high-fidelity some important audio compression problems Secure cryptographic key Optimal algorithms for exchange in open some common problems networks Design: Objects and information hiding 9 Chapter 1 of 5 Benjamin Heitmann: The Scientific Method in Software Evaluation 9
    • Interdisciplinary Research Constantly new relationships with other research fields: • Bioinformatics • Cognitive science • Computer linguistics • Immersive computing • Quantum computing Benjamin Heitmann: The Scientific Method in Software Evaluation 10 Chapter 1 of 5 10
    • True Science needs Validation Tichy: 50 % of CS papers proposing models and hypotheses are not validated (papers published before 1995) In other fields, only 15 % not validated. Benjamin Heitmann: The Scientific Method in Software Evaluation 11 Chapter 1 of 5 11
    • The Scientific Method Reference: Dodig-Crnkovic, Scientific Methods in Computer Science 12
    • Diagram of the Scientific Method 1 2 3 Existing Theories and Hypothesis Predictions Observations adjust Hypothesis 4 Tests and new 6 Select among competing Observations theories 5 Confirm old theory or Recursive consistency propose new and cyclic achieved theory 13 Chapter 2 of 5 13
    • Advantages • Reproducible Results • Impartiality • Acceptance based on results of logical reasoning, observations and experiments Benjamin Heitmann: The Scientific Method in Software Evaluation 14 Chapter 2 of 5 14
    • Scientific Methods of Computer Science • Overlying principle: Modeling • Theoretical CS • Experimental CS • Computer Simulation Benjamin Heitmann: The Scientific Method in Software Evaluation 15 Chapter 2 of 5 15
    • Reasons to (not) experiment Reference: Tichy, Should Computer Scientists Experiment More? 16
    • Reasons for experimentation • Falsification of theories • Revealing of assumptions • Eliminate alternative explanations • Enable induction: deriving theories from observation Benjamin Heitmann: The Scientific Method in Software Evaluation 17 Chapter 3 of 5 17
    • But “experiments cost to much” • Experimentation requires more resources then theory, so why do it? • Rebuttal: • Meaningful experiments can fit into small budgets • Expensive experiments can be worth more than their cost Benjamin Heitmann: The Scientific Method in Software Evaluation 18 Chapter 3 of 5 18
    • But “too many variables” • Impossible to control all variables • useless results • Rebuttal: • create benchmarks • control variables and errors like e.g. in medicine Benjamin Heitmann: The Scientific Method in Software Evaluation 19 Chapter 3 of 5 19
    • But “experimentation slows progress” • Slower flow of ideas, if experimentation is required • Rebuttal: better identification of questionable ideas Benjamin Heitmann: The Scientific Method in Software Evaluation 20 Chapter 3 of 5 20
    • But “technology changes to fast” • Experiment duration exceeds lifetime of software tool / product • Rebuttal: question is too narrow Benjamin Heitmann: The Scientific Method in Software Evaluation 21 Chapter 3 of 5 21
    • Evaluating the Design Reference: Leist, Zellner; Evaluation of Current Architecture Frameworks 22
    • Methodology 1. establish theoretical foundation 2. create list of features based on foundation 3. analyse if subjects exhibit features Benjamin Heitmann: The Scientific Method in Software Evaluation 23 Chapter 4 of 5 23
    • Evaluating Enterprise Architecture Descriptions • Theoretical foundation: Method Engineering • List of features: • Meta Model • Procedure Model • Modeling Technique • Role document • Specification Document Benjamin Heitmann: The Scientific Method in Software Evaluation 24 Chapter 4 of 5 24
    • Another in Table 1 Evaluation of current Architectures very balan shows for C4ISR/DoDAF methodic p appropriate Zachman TOGAF objectives TEAF FEAF MDA ARIS further rese focus on si appropriate quot; ! ! ! ! ! ! Specification document 7. R # quot; # quot; # ! ! Meta model quot; # # quot; quot; ! ! Role [1] Arch (MDA quot; # quot; # quot; ! ! Technique http:/ quot; # ! ! ! ! ! Procedure model (Acc ! Fully accomplished Legend: [2] Blevi # Partly accomplished and M quot; Not accomplished ADM For most of the architecture frameworks the strengths and 2005 weaknesses are very balanced. The adequacy of a architecture [3] Bran framework to support a development project could be evaluated archi only in consideration of specific circumstances of a case. [4] Brink 25 Chapter 4 of 5 Meth The results of the evaluation indicate at the same time potentials 25
    • Evaluating the Implementation Reference: Bansiya, Evaluating Framework Architecture Structural Stability 26
    • Methodology 1. establish hypothesis 2. Identify characteristics related to hypothesis 3. define metrics to assess each characteristic 4. collect data from metrics 5. determine if data supports hypothesis Benjamin Heitmann: The Scientific Method in Software Evaluation 27 Chapter 5 of 5 27
    • Evaluating Structural Stability of Windows Application Frameworks • Hypothesis: extent of change stabilises with maturity Characteristics Metric Design Size in Classes count number of single/ count multiple inheritances average inheritance divide sum of maximum path lengths of all classes depth/width by number of classes Benjamin Heitmann: The Scientific Method in Software Evaluation 28 Chapter 5 of 5 28
    • taking the difference of the aggregate-change (Vi) of a version (i,with i > 0) with the aggregate- change (V1) value of the first version. Table 4. Normalized architecture assessment metric values of MFC and OWL frameworks Design Metric MFC MFC MFC MFC MFC OWL OWL OWL OWL 1.0 2.0 3.0 4.0 5.0 4.0 4.5 5.0 5.2 1 1.28 1.43 1.56 1.13 1 1.73 2.51 1.00 Design Size 1 1 1 5 1.20 1 1.33 1.81 0.93 Hierarchies 1 1.27 1.39 1.49 1.11 1 1.65 3.61 0.98 Single Inheritances 1 1.00 1.00 1.00 1.00 1 1.44 2.31 1.00 Multiple Inheritance 1 1.26 1.16 0.95 0.99 1 1.04 1.75 0.99 Depth of Inheritance 1 0.99 0.98 0.96 0.98 1 0.94 1.33 0.98 Width of Inheritance 1 1.26 1.16 0.95 0.99 1 0.97 1.61 0.99 Number of Parents 1 1.70 1.26 1.20 0.96 1 0.87 1.55 0.99 Number of Methods 1 1.26 1.19 1.03 0.96 1 1.21 2.09 1.00 Class Coupling 9.0 11.01 10.57 14.14 9.31 9.0 11.18 18.57 8.86 Aggregate Change (Vi ) Extent-of-Change(Vi -V1) 0.0 2.01 1.57 5.14 0.31 0.0 2.18 9.57 -0.14 The extent-of-change measure can be used as an indicator (index) of framework architecture stability. The value of the measure indicates the relative stability of the architecture structure, while 29 Chapter 5 of 5 higher values of the measure are reflective of higher instability in the structure, values closer to zero 29
    • Conclusion • Computer Science is a true science • In order to have measurable results you have to choose hypotheses which can actually be measured. • Remember falsifiability tests! • Measure your own progress, by evaluating your approach Benjamin Heitmann: The Scientific Method in Software Evaluation 30 30
    • References: Denning, Is Computer Science Science? : http://cs.gmu.edu/cne/pjd/PUBS/ CACMcols/cacmApr05.pdf Dodig-Crnkovic, Scientific Methods in Computer Science: http:// www.mrtc.mdh.se/publications/0446.pdf Tichy, Should Computer Scientists Experiment More? : http:// ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=675631 Leist, Zellner, Evaluation of Current Architecture Frameworks : http:// portal.acm.org/citation.cfm?id=1141635 Bansiya, Evaluating Framework Architecture Structural Stability: http:// portal.acm.org/citation.cfm?doid=351936.351954 Brown, Wallnau, A Framework for Evaluating Software Technology: http:// ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=536457 Benjamin Heitmann: The Scientific Method in Software Evaluation 31