Successfully reported this slideshow.
Your SlideShare is downloading. ×

A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization [IVAPP 2014]

A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization [IVAPP 2014]

Download to read offline

In the field of software visualization controlled experiments are an important instrument to investigate the specific reasons, why some software visualizations excel the expectations on providing insights and ease task solving while others fail doing so. Despite this, controlled experiments in software visualization are rare. A reason for this is the fact that performing such evaluations in general, and particularly performing them in a way that minimizes the threats to validity, is hard to accomplish. In this paper, we present a structured approach on how to conduct a series of controlled experiments in order to give empirical evidence for advantages and disadvantages of software visualizations in general and of 2D vs. 3D software visualizations in particular.

In IVAPP'14: Proceedings of the 5th International Conference on Visualization Theory and Applications, 2014.

In the field of software visualization controlled experiments are an important instrument to investigate the specific reasons, why some software visualizations excel the expectations on providing insights and ease task solving while others fail doing so. Despite this, controlled experiments in software visualization are rare. A reason for this is the fact that performing such evaluations in general, and particularly performing them in a way that minimizes the threats to validity, is hard to accomplish. In this paper, we present a structured approach on how to conduct a series of controlled experiments in order to give empirical evidence for advantages and disadvantages of software visualizations in general and of 2D vs. 3D software visualizations in particular.

In IVAPP'14: Proceedings of the 5th International Conference on Visualization Theory and Applications, 2014.

Advertisement
Advertisement

More Related Content

Advertisement

A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization [IVAPP 2014]

  1. 1. Institut für Wirtschaftsinformatik/Informatik 5th International Conference on Information Visualization Theory and Applications Richard Müller1, Pascal Kovacs1, Jan Schilbach1, Ulrich Eisenecker1, Dirk Zeckzer2, Gerik Scheuermann2 1Information Systems Institute 2Institute of Computer Science University of Leipzig, Leipzig, Germany January 5, 2014 A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization
  2. 2. 2 Institut für Wirtschaftsinformatik/Informatik 2D or 3D: State of the Art © 2014 by R. Müller  Scientific drivers  Lack of empirical research in software visualization  Ongoing discourse: 2D vs. 3D  Five papers about 3D software visualizations at VISSOFT 2013  Technical drivers  Innovative methods to generate software visualizations  Increasing computing power  Emerging technology  High quality and low cost 3D environments 3D-Vision Glasses and Infrared Transmitter 3D-Vision Capable Monitor or Projector Quadro GPU
  3. 3. 3 Institut für Wirtschaftsinformatik/Informatik A Series of Controlled Experiments © 2014 by R. Müller  A series of controlled experiments is required to determine the circumstances when and why a specific software visualization is suitable for a certain software engineering task Software Visualization User Task Software Artifact Represen- tation Navigation & Interaction Implemen- tation
  4. 4. 4 Institut für Wirtschaftsinformatik/Informatik Munzner‘s Model and its Extension © 2014 by R. Müller [Munzner 2009; Meyer, Sedlmair, Munzner 2012] Block Guideline
  5. 5. 5 Institut für Wirtschaftsinformatik/Informatik Domain Specific Adaption of Munzner’s Extended Model for Software Visualization © 2014 by R. Müller
  6. 6. 6 Institut für Wirtschaftsinformatik/Informatik Possible Instantiations (1/3) Factor/Sub- Factor Examples for Possible Instantiations User Role Background Knowledge Circumstances Manager, Requirements Engineer, Architect, Developer, Tester, Maintainer, Reengineer, Documenter, Consultant, Team, Researcher Age, Gender, Color Blindness, Ability of Stereoscopic Viewing Education, Programming Experience, Domain Knowledge Occupation, Familiarity with Study Object/Tools Task Problem Development, Maintenance, Re-Engineering, Reverse Engineering, Software Process Management, Marketing, Test, Documentation Operation Retrieve Value, Filter, Compute Derived Value, Find Extremum, Sort, Determine Range, Characterize Distribution, Find Anomalies, Cluster, Correlate © 2014 by R. Müller
  7. 7. 7 Institut für Wirtschaftsinformatik/Informatik Possible Instantiations (2/3) Factor/Sub- Factor Examples for Possible Instantiations Software Artifact Type Size Aspect Requirements, Architecture, Source Code, Stack Trace, Revision History Small, Medium, Large Structure, Behavior, Evolution Representation Dimensionality Technique 2D, 2.5D, Augmented 2D, Adapted 2D, Inherent 3D Graph, Tree, Abstract/Real World Metaphor, Decorational/Representational Animation © 2014 by R. Müller
  8. 8. 8 Institut für Wirtschaftsinformatik/Informatik Possible Instantiations (3/3) Factor/Sub- Factor Examples for Possible Instantiations Navigation & Interaction Technique Input Output Overview, Zoom, Filter, Details-on-Demand, Relate, History, Extract Keyboard, Mouse, Gamepad, Flystick, Kinect, Touch Device, Leap Motion, Brain-Computer Interface Paper, Monitor, Projector, Virtual Reality Environment, Oculus Rift Implementation Algorithm Platform Dep. Automation Radial Layout, Balloon Layout, Treemap, Information Cube, Cone Tree Platform Platform Independent, Platform Dependent Full, Semi, Manual Data Famix, Dynamix, Hismo © 2014 by R. Müller
  9. 9. 9 Institut für Wirtschaftsinformatik/Informatik Example Experiment: Overview  Research question  Does an inherent 3D software visualization reduce time to solve software engineering tasks, compared to a 2D software visualization?  Dependent variable  time  Independent variable  dimensionality  Between-subjects design  control group (2D), experimental group (3D) © 2014 by R. Müller
  10. 10. 10 Institut für Wirtschaftsinformatik/Informatik Example Experiment: Detail © 2014 by R. Müller User Role Background Knowledge Developer Color Blindness, Ability of Stereoscopic Viewing Education, Programming Experience, Domain Knowledge (Virtual Reality, Touch Devices, 3D) Task Problem Find a Bug, Identify a Dominating Class Operation Retrieve Value, Filter, Find Extremum, Find Anomalies, Correlate Software Artifact Type Size Aspect Source Code Medium Structure Representation Dimensionality Technique 2D vs. Inherent 3D Graph (Nested Node-Link Technique) Navigation & Interaction Technique Input Output Overview, Zoom, Filter, Relate Touch Device Virtual Reality Environment Implementation Algorithm Platform Dep. Automation Force Directed Layout Plaform Independent Full Data Abstraction Famix Vary different factors in different experiments while keeping other relevant factors constant or measure their influence on the result
  11. 11. 11 Institut für Wirtschaftsinformatik/Informatik Summary and Conclusion  The structured approach to conduct a series of controlled experiments in software visualization is based on Munzner‘s model and its extension  The approach…  Provides a comprehensive overview of relevant influence factors and their possible interactions  Helps to plan new experiments, to categorize previous experiments and to assess their results  Supports researchers to identify research gaps © 2014 by R. Müller
  12. 12. 12 Institut für Wirtschaftsinformatik/Informatik Your questions… © 2014 by R. Müller
  13. 13. 13 Institut für Wirtschaftsinformatik/Informatik References  Meyer, M., Sedlmair, M. & Munzner, T., 2012. The four-level nested model revisited: blocks and guidelines. In Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV). pp. 1–6.  Munzner, T., 2009. A nested model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics, 15(6), pp.921–928. © 2014 by R. Müller

×