Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Green Lab - [02 B] Experiment scoping

1,488 views

Published on

This presentation is about a lecture I gave within the "Green Lab" course of the Computer Science master, Software Engineering and Green IT track of the Vrije Universiteit Amsterdam: http://masters.vu.nl/en/programmes/computer-science-software-engineering-green-it/index.aspx

http://www.ivanomalavolta.com

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Green Lab - [02 B] Experiment scoping

  1. 1. 1 Het begint met een idee Experiment scoping Ivano Malavolta
  2. 2. Vrije Universiteit Amsterdam Experiment goal definition The Goal-Question-Metric approach A final Example 2 Ivano Malavolta / S2 group / Empirical software engineering Roadmap
  3. 3. Vrije Universiteit Amsterdam 3 Ivano Malavolta / S2 group / Empirical software engineering Experiment scoping [1] V.R. Basili, "Software Modeling and Measurement: The Goal Question Metric Paradigm," Computer Science Technical Report Series, CS-TR-2956 (UMIACS-TR-92-96), University of Maryland, College Park, MD, September 1992. ● Research approach: Goal-Question-Metric [1] ● Define the goal of the study
  4. 4. Vrije Universiteit Amsterdam 4 Ivano Malavolta / S2 group / Empirical software engineering Goal GoalConceptual level Operational level Quantitative level Question Question Question Metric Metric Metric Metric Overview
  5. 5. Vrije Universiteit Amsterdam 5 Ivano Malavolta / S2 group / Empirical software engineering Experiment goal definition
  6. 6. Vrije Universiteit Amsterdam 6 Ivano Malavolta / S2 group / Empirical software engineering How to define a goal? Analyze Object(s) of study for the purpose of Purpose with respect to their Quality focus from the point of view of Perspective in the context of Context
  7. 7. Vrije Universiteit Amsterdam 7 Ivano Malavolta / S2 group / Empirical software engineering Goal definition framework Object of study Purpose Quality focus Perspective Context Product Process Model Metric Theory Characterize Monitor Evaluate Predict Control Change Effectiveness Cost Reliability Maintainability Portability Performance Understandability Developer Maintainer Project manager Corporate manager Customer User Researcher Environmental conditions
  8. 8. Vrije Universiteit Amsterdam 8 Ivano Malavolta / S2 group / Empirical software engineering Image encoding example - goal Analyze for the purpose of with respect to their from the point of view of in the context of
  9. 9. Vrije Universiteit Amsterdam 9 Ivano Malavolta / S2 group / Empirical software engineering Image encoding example - goal Analyze Encoding algorithms for the purpose of Evaluation with respect to their Energy Efficiency from the point of view of Software Developer in the context of Mobile Software Applications
  10. 10. Vrije Universiteit Amsterdam 10 Ivano Malavolta / S2 group / Empirical software engineering Image encoding example - goal "Analyze image encoding algorithms for the purpose of evaluation with respect to their energy efficiency from the point of view of a software developer in the context of mobile software applications".
  11. 11. Vrije Universiteit Amsterdam 11 Ivano Malavolta / S2 group / Empirical software engineering From goals to questions ● Questions are at the operational level ● Answering a question should tell you if the goal has been attained ● Questions should be quantifiable
  12. 12. Vrije Universiteit Amsterdam 12 Ivano Malavolta / S2 group / Empirical software engineering Question examples Analyze Encoding algorithms for the purpose of Evaluation with respect to their Energy Efficiency from the point of view of Software Developer in the context of Mobile Software Applications ● Does the energy consumption of a mobile device vary, when using different image encoding algorithms? ● Which encoding algorithm is more efficient? ● Which algorithm provides the best quality of the image?
  13. 13. Vrije Universiteit Amsterdam 13 Ivano Malavolta / S2 group / Empirical software engineering Other examples... Analyze Encoding algorithms for the purpose of Evaluation with respect to their Energy Efficiency from the point of view of Software Developer in the context of Mobile Software Applications ● Which algorithm is most used in Android applications? ● What is the effort of changing algorithm implementation? ● How does the encoding algorithm affect energy consumption?
  14. 14. Vrije Universiteit Amsterdam 14 Ivano Malavolta / S2 group / Empirical software engineering From questions to metrics ● Metrics are at the quantitative level ● Metrics can be objective (e.g. energy consumption) or subjective (e.g. readability of code)
  15. 15. Vrije Universiteit Amsterdam 15 Ivano Malavolta / S2 group / Empirical software engineering Examples of metrics ● Does the energy consumption of a mobile device vary, when using different encoding algorithms? Metric: Energy Consumption (Joule or Watthour) ● What is the efficiency of the encoding algorithm? Metric: Number of rendered pixels over time (pixels/s) ● What is the effort of changing algorithm implementation? Metric: Effort (person/hours, money)
  16. 16. Vrije Universiteit Amsterdam 16 Ivano Malavolta / S2 group / Empirical software engineering A final example
  17. 17. Vrije Universiteit Amsterdam 17 Ivano Malavolta / S2 group / Empirical software engineering Goal Objects of study: Android apps Purpose: to understand how the APIs fault- and change-proneness could have impacted the apps’ success Quality focus: success of those apps in terms of ratings expressed by users on the market Perspective: app developers’ Context: 7,097 free apps from the Google Play Market
  18. 18. Vrije Universiteit Amsterdam 18 Ivano Malavolta / S2 group / Empirical software engineering Questions Q1: Does the fault-proneness of APIs affect the success of Android Apps? Q2: Does the change-proneness of APIs affect the success of Android Apps?
  19. 19. Vrije Universiteit Amsterdam 19 Ivano Malavolta / S2 group / Empirical software engineering Metrics Main goal Q1 (fault-proneness) Q2 (change-proneness) Average rating in the Google Play store #bugs fixed in the APIs used by the app #changes in the signatures of the methods used by the app
  20. 20. Vrije Universiteit Amsterdam You know how to: ● define the goal of an experiment ● precisely scope an experiment ○ … and have an hint about how to plan it ● You are able to work on assignment 1: ○ Describe your goal using the provided template ○ Develop a GQM for your experiment ○ Fill in Sections 1 and 2 of the report 20 Ivano Malavolta / S2 group / Empirical software engineering What this lecture means to you?
  21. 21. Vrije Universiteit Amsterdam 21 Ivano Malavolta / S2 group / Empirical software engineering Readings Chapter 7 + papers with examples (Koziolek, Basili) on Blackboard
  22. 22. Vrije Universiteit Amsterdam 22 Ivano Malavolta / S2 group / Empirical software engineering Some contents of this lecture extracted from: ● Giuseppe Procaccianti’s lectures at VU Acknowledgements

×