Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Interventionist-methods - Methods in user-technology studies

234 views

Published on

Where researcher sets up different conditions of action to see how it affects humans’ behaviour: Experimental (lab) research, Quasi-experimentation, Wizard-of-Oz simulations, Usability evaluations,
Field trials, Research through design approach

Published in: Science
  • Login to see the comments

Interventionist-methods - Methods in user-technology studies

  1. 1. Antti Salovaara Aalto University, School of Business 22 January 2015 Methods in User–Technology Studies Interventionist studies 16.15 – 17.00 Methods in User–Technology Studies / Interventionist studies by Antti Salovaara is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
  2. 2. Interventionist studies = where researcher sets up different conditions of action to see how it affects humans’ behaviour Experimental (lab) research Quasi-experimentation Wizard-of-Oz simulations Usability evaluations Field trials Research through design approach
  3. 3. Experimental (lab) research = studies where nuisance factors are eliminated or controlled Owes a lot to research methods in psychology and social psychology Needs: Predefined quantitiable research questions Precise measurements Sometimes (not always) special hardware (eye tracking, EEG, ..) Comparison between “treatment” and “control” (i.e., baseline) groups
  4. 4. Example: Experimental (lab) research Source: Lehtinen et al. (2012) Vibrotactile actuators (bzzz) © Lehtinen et al. Image credits: See last slide
  5. 5. Quasi-experiments and analogue experiments Quasi- experimentation: experimental research without the control group Shadish et al. (2002) Analogue experiments: Controlled studies that replicate simulate real-life situations Source: Oulasvirta et al. (2005) Image credit: See last slide
  6. 6. Wizard-of-Oz simulations Used in studies on futuristic technologies e.g., artificial intelligence Applies deception: Participant believes that the system is real while a part of its operation is controlled by the experimenter Setup is revealed after the study E.g., speech recognition studies Chess-playing automaton constructed by Wolfgang von Kempelen in 1770 Imagecredit:Seelastslide
  7. 7. Example: a Wizard-of-Oz study Nokia 9500 communicator HP PDA Suunto sports wrist- watch Plastic tray attached to pram’s handlebar ©Author © Author
  8. 8. Image credit: See last slide
  9. 9. Wizard-of-Oz setup Wizard Facilitator User Recorder Image credits: See last slide
  10. 10. Usability evaluations Typical method: 1.  List the representative tasks that technology is used for 2.  Write a realistic scenario around these tasks 3.  Create materials for these tasks 4.  Present the scenario for the participant and ask him/her carry out the tasks. 5.  Record with video 6.  Repeat with more participants until findings saturate (i.e., when new usability problems are not found) = studies on the quality of technology design with users Exist in many variations An effective method for identifying problems Rarely used in academic research Imagecredit:Seelastslide
  11. 11. Field trials = open-ended studies or quasi-experiments on research prototypes “in the wild” e.g., Muller et al’s study on a new cooperation technology Open-ended field trials’ differences to observational studies: Technology is provided to users by researchers Technology may also have been developed by the researchers RQ focuses on the change caused by the new technology The prototype’s design often embodies a research hypothesis e.g., Muller et al: “Users do not need to use so many systems if a system would support object-centric sharing, object-level awareness, user-created structured collections, and dynamic membership”
  12. 12. Research through design Is a conceptualization of the reason for organizing field trials in academic research Basic principles: “Consider information systems as research instruments that are realizations of your research hypotheses” “Research prototypes should be created and studied for the purposes of studying phenomena in IS use, not for finding out if they make a positive difference” ⇒ Research prototypes do not need to be “good” design Instead, they must help you answer to your research question
  13. 13. Example of a field trial: Comeks Comeks – An MMS-based mobile comic strip creator How the idea for Comeks originated: Discovery of live action role-players’ need for light-weight story-like game documentation Image credit: See last slide. Source: Salovaara (2007)
  14. 14. © Author
  15. 15. Image credit: See last slide.
  16. 16. How Comeks worked Phone with Comeks Message creation MMS Any phone Message reception © Author
  17. 17. Study on appropriation First encounter with a new technology Appropriation Integration into daily activities RQ: How do users appropriate Comeks and use its features?
  18. 18. Methods U U R 1. 2. 2. R 3. Phone with Comeks MMS Any phone Context Logger U U Cued recall interview every 2-3 weeks R
  19. 19. Using lots of methods together An observation-based field trial with logging, cued recall interviews, questionnaire, Wizard-of-Oz That followed the research through design approach Source: Jacucci et al., CHI 2007
  20. 20. Mobile multimedia group communication chat with a support for collocated interaction Motivation: mobile messages are often shared with collocated others RQ: How will a system with dedicated support for collocated interaction be used by groups? Rally visitors in Neste Rally 2005 Video > ©Theauthor © Author Source: Salovaara et al. (2006)
  21. 21. CoMedia user interface presence indicators Blue man icon: a nearby CoMedia user who is using CoMedia and is in Bluetooth range Green man icon: a remote CoMedia user who is using CoMedia but is not in Bluetooth range Gray man: an off-line CoMedia user who is not using CoMedia Violet man: the user him/herself Phone icon: a user has used the phone recently – be it CoMedia or any other application. ! ! ! ! ! People that are present in different chat spaces © Author
  22. 22. P “The Panopticon” Observation, logging, “The Panopticon”, and Wizard of Oz Phone with CoMedia Phone with CoMedia U U R SMS R “Rally news” & Wizard of Oz “The Panopticon”: Source: Salovaara et al. (2006)
  23. 23. Other methods Cued recall interviews: After the rally weekend, interviews about each message sent and received Questionnaire: Sense of presence scale
  24. 24. References Jacucci, G., Oulasvirta, A., Ilmonen, T., Evans, J., & Salovaara, A. (2007). CoMedia: mobile group media for active spectatorship. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2007) (pp. 1273--1282). New York, NY: ACM Press. Lehtinen, V., Oulasvirta, A., Salovaara, A., & Nurmi, P. (2012). Dynamic tactile guidance for visual search tasks. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST 2012) (pp. 445--452). New York, NY: ACM Press. Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005). Interaction in 4-second bursts: The fragmented nature of attentional resources in mobile HCI. In Proceedings of the SICCHI Conference on Human Factors in Computing Systems (CHI 2005) (pp. 919--928). New York, NY: ACM Press. Salovaara, A., Oulasvirta, A., & Jacucci, G. (2006). ``The panopticon'': a method for observing inter- group interactions. In Workshop on Reality Testing at CHI 2006. Montreal, Canada: Salovaara, A., Jacucci, G., Oulasvirta, A., Saari, T., Kanerva, P., Kurvinen, E., & Tiitta, S. (2006). Collective creation and sense-making of mobile media. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2006) (pp. 1211-1220). New York, NY: ACM Press. Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin.
  25. 25. Image credits Figure 1 from Lehtinen et al. (2012). © Paper authors. Figure 3 from Lehtinen et al. (2012). © Paper authors. Figure 3 from Oulasvirta et al. (2005). Used with permission. Figure 1 from Oulasvirta et al. (2005). Used with permission. Copper engraving from the book: Karl Gottlieb von Windisch, Briefe über den Schachspieler des Hrn. von Kempelen, nebst drei Kupferstichen die diese berühmte Maschine vorstellen. 1783. Public domain: Copyright expired. The Ruoholahti canal by Oghmoir. http://commons.wikimedia.org/wiki/ File:Ruoholahden_kanava.jpg. Licensed under Creative Commons Attribution-Share Alike 3.0 Unported Images © Author and HIIT Uix research team 2004 CodeSyntax usability lab by garaolaza, http://www.argazkiak.org/photo/ codesyntax-usability-lab/size/l/. Licensed under Creative Commons Attribution-Share Alike 3.0 Unported By unnamed study participant 2005. Used with permission.

×