Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Green Lab - [03 A] Experiment planning

1,528 views

Published on

This presentation is about a lecture I gave within the "Green Lab" course of the Computer Science master, Software Engineering and Green IT track of the Vrije Universiteit Amsterdam: http://masters.vu.nl/en/programmes/computer-science-software-engineering-green-it/index.aspx

http://www.ivanomalavolta.com

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Green Lab - [03 A] Experiment planning

  1. 1. 1 Het begint met een idee Experiment planning Ivano Malavolta
  2. 2. Vrije Universiteit Amsterdam Experiment planning Context selection Research questions and hypotheses formulation Variables selection Subjects selection 2 Ivano Malavolta / S2 group / Empirical software engineering Roadmap
  3. 3. Vrije Universiteit Amsterdam 3 Ivano Malavolta / S2 group / Empirical software engineering Recall
  4. 4. Vrije Universiteit Amsterdam 4 Ivano Malavolta / S2 group / Empirical software engineering Recall
  5. 5. Vrije Universiteit Amsterdam ● Experiment scoping describes WHY we run an experiment ● The planning determines HOW the experiment will be executed ○ Be careful here → the result of the experiment can be disturbed (or even destroyed) if not planned properly 5 Ivano Malavolta / S2 group / Empirical software engineering Scoping VS planning
  6. 6. Vrije Universiteit Amsterdam 6 Ivano Malavolta / S2 group / Empirical software engineering Planning phases Scope of this lecture
  7. 7. Vrije Universiteit Amsterdam 7 Ivano Malavolta / S2 group / Empirical software engineering Context selection
  8. 8. Vrije Universiteit Amsterdam 8 Ivano Malavolta / S2 group / Empirical software engineering We already heard about context...
  9. 9. Vrije Universiteit Amsterdam CONTEXT: the environment in which the experiment is performed Our goal here is to achieve the most general results → optimum: large real software projects, with practitioners Many risks involved…. 9 Ivano Malavolta / S2 group / Empirical software engineering Context selection
  10. 10. Vrije Universiteit Amsterdam 10 Ivano Malavolta / S2 group / Empirical software engineering Context selection dimensions RealityLab ● Off-line ● On-line ● Students ● Professionals ● Toy problem ● Real problem ● Specific ● General
  11. 11. Vrije Universiteit Amsterdam 11 Ivano Malavolta / S2 group / Empirical software engineering Quick exercise Think about the Progressive web app case study and formulate a potential context for an experiment
  12. 12. Vrije Universiteit Amsterdam 12 Ivano Malavolta / S2 group / Empirical software engineering Research questions and hypotheses formulation
  13. 13. Vrije Universiteit Amsterdam Research questions detail the specific objectives of the empirical study Incepted from the study definition Starting point to identify the variables of interest of your study 13 Ivano Malavolta / S2 group / Empirical software engineering Research questions formulation
  14. 14. Vrije Universiteit Amsterdam Research questions should be as clear as possible → they will guide the whole experiment Avoid questions you cannot answer ○ What is the best JavaScript framework in terms of performance? ○ What is the most productive programming language? ○ ... Golden words: to what extent..., what is the impact of X to Y, what are the traits of Xs, what are the characteristics of T... Remind that in your report you will come back to them and explicitly answer each of them in details 14 Ivano Malavolta / S2 group / Empirical software engineering Suggestions
  15. 15. Vrije Universiteit Amsterdam 15 Ivano Malavolta / S2 group / Empirical software engineering Example: CHABADA
  16. 16. Vrije Universiteit Amsterdam 16 Example: CHABADA
  17. 17. Vrije Universiteit Amsterdam 17 Example: CHABADA
  18. 18. Vrije Universiteit Amsterdam 18 Ivano Malavolta / S2 group / Empirical software engineering Example: CHABADA
  19. 19. Vrije Universiteit Amsterdam 19 Ivano Malavolta / S2 group / Empirical software engineering Example: CHABADA
  20. 20. Vrije Universiteit Amsterdam 20 Ivano Malavolta / S2 group / Empirical software engineering From research questions to hypotheses ● When a research question is going to be addressed by applying a statistical test, it would be helpful to formulate an hypothesis ● Very useful to select what kind of statistical procedure you need to use Not needed in all cases
  21. 21. Vrije Universiteit Amsterdam 21 Ivano Malavolta / S2 group / Empirical software engineering Hypotheses formulation ● Conjecture (P) ○ Administration of treatment has influence on some feature ● Consequence (Q) ○ We observe a difference in terms of some feature P → Q
  22. 22. Vrije Universiteit Amsterdam 22 Ivano Malavolta / S2 group / Empirical software engineering Hypotheses formulation Hypothesis: a formal statement about a phenomenon ● Null hypothesis H0 : no real trends or patterns in the experiment setting (aka ~Q) ● Alternative hypothesis Ha : there are real trends or patterns in the experiment setting (aka Q)
  23. 23. Vrije Universiteit Amsterdam 23 Ivano Malavolta / S2 group / Empirical software engineering Falsification (modus tollens) ● We aim at verifying the null hypothesis (~Q) ○ we test the null hypothesis H0 We can reject the null hypothesis → we can draw conclusions This comes from Popper (1959): any statement in a scientific field is true until anybody can contradict it ● Aiming at verifying Q is WRONG ○ Provides no insight on the conjecture
  24. 24. Vrije Universiteit Amsterdam 24 Ivano Malavolta / S2 group / Empirical software engineering Example ● Question: ○ Do green algorithms improve software energy efficiency? ● Consequence (Q): ○ (when applying green algorithms) we observe a reduction of energy consumption ● Conjecture (P): ○ applying green algorithms reduces energy consumption
  25. 25. Vrije Universiteit Amsterdam 25 Ivano Malavolta / S2 group / Empirical software engineering Example ● Null hypothesis (¬Q): there is no reduction in terms of energy consumption H0 : mean(Ega ) >= mean(Enormal ) ● Alternative hypothesis (Q): the energy consumption resulting from an application that uses green algorithms is lower Ha : mean(Ega ) < mean(Enormal )
  26. 26. Vrije Universiteit Amsterdam 26 Ivano Malavolta / S2 group / Empirical software engineering What can happen now? ● We confirm the null hypothesis (~Q) ○ ~Q → ~P ○ our conjecture P has been falsified → green algorithms do not reduce energy consumption (~P) ● We reject the null hypothesis (~Q) ○ Q = true ○ our conjecture P has been corroborated → we are more confident that it is likely that green algorithms reduce energy consumption (P)
  27. 27. Vrije Universiteit Amsterdam 27 Ivano Malavolta / S2 group / Empirical software engineering Example 2
  28. 28. Vrije Universiteit Amsterdam 28 Other examples of hypotheses
  29. 29. Vrije Universiteit Amsterdam 29 Ivano Malavolta / S2 group / Empirical software engineering Variables selection
  30. 30. Vrije Universiteit Amsterdam 30 Ivano Malavolta / S2 group / Empirical software engineering Recap
  31. 31. Vrije Universiteit Amsterdam 31 Ivano Malavolta / S2 group / Empirical software engineering Variables selection ● The choice of independent and dependent variables is usually done in parallel ● Some variables cannot be measured directly (e.g. productivity, code quality, effort...) ○ We use proxies to estimate them ■ proxies introduce a construct validity threat: is what we are measuring a good representation of our variable?
  32. 32. Vrije Universiteit Amsterdam 32 Ivano Malavolta / S2 group / Empirical software engineering Variables selection ● Independent variables should have some effect on the dependent ones → do not choose variables randomly, think about your RQs ● After choosing the variables you have to define their types, scales, ranges → this is part of measurement theory
  33. 33. Vrije Universiteit Amsterdam 33 Ivano Malavolta / S2 group / Empirical software engineering Hypotheses formulation ● Often there is only 1 dependent variable (the main factor) ○ e.g., power consumption ● Often one level for the control group ○ e.g. use of old/traditional technique/tool ● One or more levels for experimental groups ○ e.g. use of new technique(s) tool(s) Other independent variables are the co-factors
  34. 34. Vrije Universiteit Amsterdam Our main factor is not the only variable influencing the dependent variable(s) ○ e.g., network instability of your experimental environment, usage patterns of the analysed website, skills of subjects, analyzed system, experience of developers, ... We will never account for all possible co-factors Your best friend here is randomization In a good experiment: ● limit their effect through a good experimental design ● able to separate their effect from main factors ● analyze the interaction with main factor 34 Co-factors
  35. 35. Vrije Universiteit Amsterdam 35 Ivano Malavolta / S2 group / Empirical software engineering Example
  36. 36. Vrije Universiteit Amsterdam 36 Ivano Malavolta / S2 group / Empirical software engineering Subjects selection
  37. 37. Vrije Universiteit Amsterdam 37 Ivano Malavolta / S2 group / Empirical software engineering Subjects selection ● Population: the complete set of items of interest for our experiment ○ e.g. open-source software applications ○ e.g., all existing progressive web apps ● Sample: representative selection of individuals for that population ○ e.g. Apache, MySQL ○ e.g. progressive web apps mined from Alexa’s list
  38. 38. Vrije Universiteit Amsterdam 38 Ivano Malavolta / S2 group / Empirical software engineering Sampling techniques ● Probability Sampling: the probability of selecting each subject in the population is known ○ Simple Random Sampling: random selection from the population, probability is 1/total ○ Stratified random sampling: the population is divided into groups with a known distribution between the groups. Random sampling is then applied within each group ● Non-probability sampling: the probability of selecting each subject out of the population is unknown ○ Convenience: the most convenient (cost/distance/ complexity) subjects are selected [usually it is the only way to go] ○ Quota: you select samples from groups of subjects (e.g. male vs females, open-source vs closed source)
  39. 39. Vrije Universiteit Amsterdam 39 Ivano Malavolta / S2 group / Empirical software engineering How big should be a sample? ● Sample size: the larger, the better (more general results) ● If the population has a high variation a larger sample size is needed ● Data analysis may influence sample size ○ some statistical tests have meaning only on large samples
  40. 40. Vrije Universiteit Amsterdam You know how to: ● define the context of your experiment ● define research questions and hypotheses ● define independent and dependent variables ● strategies for selecting subjects Next step Measurement theory → how to define the “type” of variables 40 Ivano Malavolta / S2 group / Empirical software engineering What this lecture means to you?
  41. 41. Vrije Universiteit Amsterdam 41 Ivano Malavolta / S2 group / Empirical software engineering Readings Chapter 8
  42. 42. Vrije Universiteit Amsterdam 42 Ivano Malavolta / S2 group / Empirical software engineering Some contents of this part of lecture extracted from: ● Giuseppe Procaccianti’s lectures at VU ● Massimiliano Di Penta’s lectures at GSSI (Italy) Acknowledgements

×