Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluation of Prototypes and the Problem of Possible Futures

215 views

Published on

CHI2017 talk in Denver 9 May 2017.
PDF of the paper: http://bit.ly/2q1mJ3B

Published in: Science
  • Be the first to comment

  • Be the first to like this

Evaluation of Prototypes and the Problem of Possible Futures

  1. 1. Evaluation of Prototypes and the Problem of Possible Futures Antti Salovaara, University of Helsinki Antti Oulasvirta, Aalto University Giulio Jacucci, University of Helsinki UNIVERSITY OF HELSINKI AALTO UNIVERSITY
  2. 2. Evaluations are about what the prototype might become.
  3. 3. Picture from p. 518 in Abowd (1999) Classroom 2000: An Experiment with the Instrumentation of a Living Educational Environment. IBM Systems Journal 38(4), 508–530. Classroom of the future at Georgia Tech Benefit of note-taking support 1997 Wireless handheld devices
  4. 4. 2008 “Please Keep Your Laptops in an Upright and Unlocked Position” by Alan Levine. https://www.flickr.com/photos/cogdog/2871019373/ . CC0 1.0 Universal Social media Distracted attention The future as it became
  5. 5. Study context (1997) Actual context (future) Automated capture of learning material Wireless note sharingUpload after lecture Social media Overhead projectors Present– future gap
  6. 6. Study context Actual context Design brief, Requirement specification New way of work New software Functional prototype The old way of work Usability evaluation Present– future gap
  7. 7. A carelessly designed evaluation may inadvertently evaluate wrong futures, contexts, or user groups, thereby leading to false conclusions and expensive design failures.
  8. 8. Evaluation as a “time machine” Envisioning Concretisation Projection
  9. 9. The non-issue of the present–future gap Scenario-building, co-design, PD Future orientation, artifacts, empirical approach Experimental method Future orientation, artifacts, empirical approach Ethnographic methods, participant observation Future orientation, artifacts, empirical approach Engineering and programming Future orientation, artifacts, empirical approach Field trials, Usability evaluations
  10. 10. Envisioning Concretisation Projection ✔
  11. 11. Every methodological choice changes the enacted future Concretisation
  12. 12. CONTROL Removal of present-day features STAGING Creation of futuristic features Wireless note sharing Overhead projectors Every methodological choice changes the enacted future
  13. 13. CONTROL TECHNIQUES Narrowing Stabilising Removal Inhibition Gamification STAGING TECHNIQUES Propping Setting selection Feature promotion Repetition Recruiting
  14. 14. INHIBITION in an urban navigation study concretization Free walking Embedding information in urban environment for AR Limit use of AR to a small area
  15. 15. PROPPING in an in-the-wild AR study concretization Hi-fi prototype Human actors Embedding information in urban environment for AR
  16. 16. GAMIFICATION in an AR experiment concretization Excitement helps users forget their hesitation
  17. 17. Envisioning Concretisation Projection
  18. 18. Projective validity From imperfect data… …to a valid conclusion inferential leap Internal validity Construct validity Conclusion validity External validity Projective validity
  19. 19. Near future Far future Precise range of futures Singular futures Margin of tolerance
  20. 20. New mindset for evaluations 1.  Evaluate with the future in mind 2.  Consider the future-contingency of every methodological choice 3.  Consider how evaluation must be controlled & staged 4.  Report methodological choices transparently 5.  Define the margins for projective validity
  21. 21. Thank you! Acknowledgments: Academy of Finland (259281 and 298879), European Research Council (637991) David McGookin, Barry Brown, Duncan Brumby, Kasper Hornbæk, Mika Jokiniemi, Stuart Reeves, Baris Serim, John Zimmerman Evaluations are about what the prototype might become.

×