IHC2008 Final Keynote Bernhaupt


Published on

Final Keynote by Regina Bernhaupt at IHC 2008, Porto Alegre, Brasil, 24. October, 2008

Published in: Education, Technology, Design
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

IHC2008 Final Keynote Bernhaupt

  1. 1. Evaluating Usability and User Experience in Non-Traditional Environments IHC 2008 Porto Alegre, Brasil 21 to 24 October, 2008 Regina Bernhaupt
  2. 2. Think about it …. * How to evaluate the usability of this presentation … * And how to evalute the User Experience during this presentation …
  3. 3. Non-Traditional Environments * Stressed * Not being well * Critical Situations * Relax * Fun * Operational Situations
  4. 4. Evaluating Usability & UX in Non-Traditional Environments Overview * Usability and User Experience as Key Concepts * Current Evaluation Approaches * Three Case Studies: Adoption and Development of Evaluation Methods * How to evaluate in non-traditional environments
  5. 5. Usability and UX as Key Concepts
  6. 6. Definition of Usability * Usability is concerned with making systems easy to learn and use. * A usable system is one which enables users to perform their job effectively and efficiently. * Usability is the „effectiveness, efficiency and satisfaction with which a specified set of users can achieve a specified set of tasks in a particular environment“. ISO 9241-11
  7. 7. Usability Decomposed * Flexibility: Dialog Initiative, Multi-threading, Task migratability, Substitutivity, Customizability, … * Learnability: Predictability, Synthesizability, Familiarity, Generalizability, Consistency, …. * Robustness: Observability, Recoverability, Responsiveness, Task Conformance, …..
  8. 8. Fun as Factor of Usability (Caroll and Thomas, 1988) „We realize that many people will read this article as a joke. To this extend, we are the victims of our own analysis: there are risks in being serious about fun. Still though, we continue to see, without humor, the prospect of a decade of research analysis possibly failing to provide the leverage it could on designing systems people will really want to use by ignoring what could be a very potent determinant of subjective judgements of usability – FUN“ (p. 23)
  9. 9. Is User Experience different from Usability? For Playing the Game: Congratulationskey press any – you won!!
  10. 10. Defining UX * Experiences, Feelings and Needs of the user before, during and after interacting with the product in various contexts. * How do we evaluate?
  11. 11. Current Evaluation Approaches
  12. 12. Evaluation: Process Model
  13. 13. Limitations of Evaluation Methods * Real usage and user experiences differ from lab usage * Field studies: adoption of methods needed * UX is difficult to evalute: lots of factors * UX: when do we evaluate what and how?
  14. 14. Categories of Evaluation: Usability Heuristic Evaluation Model-based Evaluation Design Rationale Experiments User Studies User Testing
  15. 15. Adoption and Development of New Evaluation Methods
  16. 16. Case Study 1: Satellite Control Room * Usage of new interaction technique · Multimodal Interfaces · Multi-touch including voice · Usability Problems lead to enormous damage/loss
  17. 17. Case Study 1: Satellite Control Room * Problems: · You are not allowed to access the users · You can not test in real situations · Usability Problems lead to considerable damage/loss * Solution: · Full modeling of the interaction · Combine formal modeling state description of the model to investigate usability problems (*) · (*) Bernhaupt, Palanque, Winckler, Navarre (2007); Bernhaupt, Palanque (in preparation)
  18. 18. Adapting Usability Evaluation Methods * Combination of Methods: · Usability Studies with model-based evaluation (*) * Adapting Methods: · Heuristic Evaluation with usage scenarios (videos) (**) · Cognitive Walkthrough with usage scenarios (*) · Performing Heuristic Evaluation in the field (**) * Bringing the „Field“ into the „Lab“ · Sports equipment to simulate running (***) * In general: Combination and embedding of methods * (*) Bernhaupt et al. (2007) (**) Po, Howard, Vetere, Skov (2004) (***) Kjeldskov and Stage (2003)
  19. 19. Adopting Usability Evaluation Methods for Non-Traditional Environments Heuristic Evaluation Model-based Evaluation Experiments Design Rationale User Studies User Testing
  20. 20. User Experience Evaluation
  21. 21. Categories of Evaluation: User Experience Probing Questionnaire Focus Groups UX Sampling Interviews Experiments User Studies Play Testing
  22. 22. Case Study 2: Playful Probing
  23. 23. Cultural Probing and Variations * Probing Approach (Gaver et al., 1999) * Several Variations: Mobile Probes, Technology Probes, … * Insights into daily life e.g. to understand user experience factors
  24. 24. Playful Probing * Problem: · Privacy · Real usage situations: long-term usage · Adressing all users during study * Solution (*) : · Make Probing „Playful“ · Develop Games to extend the methods · Use creative/playful material/questions * (*) Bernhaupt, Weiss, Obrist, Tscheligi (2007),
  25. 25. Playful Probing: Material * Games, especially developed to address the research questions, to increase participation
  26. 26. Playful Probing: Material
  27. 27. Playful Probing: Results (*) * more involvement * more insights * more „material“ * helps investigating UX (*) e.g. Bernhaupt et al. 2007 ·
  28. 28. Case Study 3: Evaluating UX in Games * Developing/Using Games to evaluate aspects of UX * Effects of Emotion
  29. 29. Social and Ambient Factors related to UX * Emotional Flower can be played by a group of people * We use an ambient display to enable the players to see the progress of others * All flowers including the chosen nickname are displayed, the screen is in a public area
  30. 30. Investigation aspects of UX: Manipulating Emotions * Need for attentive user interfaces * Emotions are a major influence on peoples attitude * Emotional Interfaces are dealing with human emotions * Goal: make UX positive
  31. 31. Results (*) * Emotion can be influenced (but not on a longer term) * Social influences are important for UX in games * UX depends on usage situations, time is an important factor * UX design must take into account different characters of users (e.g. more/less play-oriented users)
  32. 32. Categories of Evaluation: User Experience Playful Probing Questionnaire Focus Groups Emotional Interviews Flowers Experiments User Studies
  33. 33. How to Evaluate in Non-Traditional Environments * Choose the right set of evaluation methods based on the context the final product will be used in * Evaluation should take place in real usage context with real users – but this is not realistic for several domains (security, airtraffic, cockpits, satellite control rooms, homes, cars, public places, …) * Modify and develop methods to allow the evaluation in non-traditional environments or adapt methods to reflect the real usage environment
  34. 34. Evaluation in Non-Traditional Environments * Real usage context vs. Lab/Field Study * Investigate the real usage problems and the user experiences * Add valuable information for the development of products that are usable and that are perceived with a positive user experience
  35. 35. Problems for Evaluation in Non-Traditional Environments * Users not accessible * Non-standardized technologies (e.g. Set-top box in living room) * Evaluation is typcially expensive * Sometimes not reproducable insights – especially for UX.
  36. 36. Solutions - Examples * Adaption of Methods: · Playful Probing · Creative cultural probing · User Experience Sampling · Field usability studies * Development of new methods and learning from „related“ disciplines: · Evaluating User Experience in Games [Bernhaupt, R., Springer Verlag, Mid 2009]
  37. 37. Future Work * Identification of factors influencing the evaluation in non-traditional environments * Development of new or adapted methodologies and their validation * Framework for comparing evaluation methods with respect to usage in non-traditional environments
  38. 38. Contacts * Regina Bernhaupt * IRIT ruwido * Groupe IHCS * 118, Route de Narbonne Köstendorferstr. 8 * 31062 Toulouse 5321 Neumarkt a. W. * Regina.Bernhaupt@irit.fr * Regina.Bernhaupt@ruwido.com
  39. 39. Literature * IF you have DIFFICULTIES in finding any of the publications I authored, please send me an e-mail to request the article! * Regina.Bernhaupt@sbg.ac.at
  40. 40. Literature: Playful Probing and Playful Evaluation Methods (Bernhaupt et al.) * PLAYFUL PROBING * Bernhaupt, R., Weiss, A., Obrist, M., Tscheligi, M.(2007). Playful Probing: Making Probing more Fun, INTERACT 2007, 10 -14 Sep 2007, Springer LNCS, 606-619. * Beck, E., Obrist, M., Bernhaupt, R., Tscheligi, M., (2008). Instant Card Technique: How and Why to apply in User-Centered Design. Conference on Participatory Design (PDC) 2008, Bloomington, Indiana, USA, to appear. * Weiss, A., Wurhofer. D., Bernhaupt, R., Beck, E., and Tscheligi, M. (2008) “This is a Flying Shopping Trolley”- A Case Study of Participatory Design with Children in a Shopping Context, Conference on Participatory Design (PDC) 2008, Bloomington, Indiana, USA, to appear. * STUDIES USING PLAYFUL PROBING * Bernhaupt, R., Obrist. M., Weiss, A., Beck, E. & Tscheligi, M. (2008). Trends in the Living Room and Beyond. Computers in Entertainment, Special Issue, Vol. 6, 1, online. * Obrist, M., Bernhaupt, R. and Tscheligi, M. (2008). Interactive TV for the Home: An ethnographic study on users’ requirements and experiences. International Journal of Human-Computer Interaction, Volume 24, Issue 2, 174-196. * Bernhaupt, R., Wilfinger, D., Weiss, A., and Tscheligi, M. (2008) An Ethnographic Study on Recommendations in the Living Room: Implications for Design of iTV Recommender Systems, Proc. of EuroiTV 2008, Springer LNCS 5066, 92-101.
  41. 41. Literature: Evaluation Methods – Introduction/Overviews * Bernhaupt, R., Mihalic, K. and Obrist M. (2008). Methods for Usability Evaluation of Mobile Applications. to appear in: Handbook of Research on User Interface Design and Evaluation for Mobile Technology, Lumsden, J. et al. (eds.), IGI Global, p. 742 - 755. * Bernhaupt, R, Palanque, P., Winckler, M. and Navarre, D. ( 2007) Usability Study of Multi-Modal Interfaces using Eye-Tracking, Interact 2007, 10 – 14 Sep 2007, Springer LNCS, p 412-424. * Bernhaupt, R., Palanque, P., Winkler, M. and Navarre, D. (2007). Supporting Usability Evaluation of Multimodal Safety Critical Interactive Applications using Dialogue and Interaction Models. In book: Maturing Usability: Quality in Software, Interaction and Value. Law, E., et al. (eds.) Springer 95-127.
  42. 42. Literature: User Experience * Hassenzahl, M. and Tractinsky, N. (2006) User Experience – a research agenda, Behaviour and Infomration Technology, Vol. 25, No. 2, March-April 2006, 91-97. * Blythe, M., Hassenzahl, M. and Wright, P. C. (2004), More Funology, Interactions, 11, 36-77. * Carroll, J.M. and Thomas, J. C. (1988) Fun. SIGCHI Bulletin, 19. 21-24. * Norman, D. (2004) Emotional Design: Why we love (or hate) everyday things (New York: Basic Books). * Blythe, M., Overbeeke, C., Monk, A.f., and Wright, P. C. (Eds), (2003) Funology: From Usability to Enjoyment (Dordrecht: Kluwer). * Tractinsky, N., Katz, A.S. and Ikar, D. (2000). What is beautiful is usable. Interacting with Computers, 13, 127-145.
  43. 43. Literature: User Experience - Emotion/Affect * Desmet, P.M.A., Overbeeke, C.j. and Tax, S.J.E.T (2001) Designing Products with added emotional value: development and application of an approach for research through design. The Design Journal, 4. 32- 47. * Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., Tscheligi, M. (2007). Emotional Flower: How Social Community Building is Influenced by an Emotional Interface Game; to AoiR 2007, online version at: http://conferences.aoir.org/papers.php?first_letter=B&cf=6. * Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., and Tscheligi, M. 2007. Using emotion in games: emotional flowers. In Proceedings of the international Conference on Advances in Computer Entertainment Technology (Salzburg, Austria, June 13 - 15, 2007). ACE '07, vol. 203. ACM Press, New York, NY, 41-48.
  44. 44. Literature: Evaluating Games * R. Bernhaupt (to appear) Evaluating User Experience in Games: Methods and Approaches, Springer Verlag, HCI- Series. * Bernhaupt, R., Ijsselsteijn, W., Mueller, F., Tscheligi, M. and Wixon, D. (2008). Evaluating User Experiences in Games, In CHI '08 Extended Abstracts on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 3905-3908. * Position Papers at: http://workshops.icts.sbg.ac.at/chi2008uxgames/
  45. 45. Literature: Evaluation in Non-Traditional Environments * Bernhaupt, R. (accepted) Application of Usability and User Experience Evaluation Methods for the Mass-Customization of Products and Services. (Eds).Mourlas, C., & Germanakos, P. Mass Customization for Personalized Communication Environments: Integrating Human Factors, IGI Global Publication. * Bernhaupt, R., Obrist, M., Tscheligi, M. (accepted) Methods for involving users in the development of social interactive TV: Enhancing usability and user experience in non-traditional environments, 15. pages, Ed. Caesar, P., Geerts, D., and Chorianopoulous, K. Social Interactive Television: Immersive Shared Experiences and Perspectives. IGI Global Publication.