The document outlines the importance of prototyping and evaluation in design, emphasizing the need to gather user feedback early to refine ideas and solutions. It details various aspects of evaluation, including what to measure, user involvement, and methods for capturing feedback. The document also provides guidelines for planning and executing evaluations effectively to inform future iterations of prototypes.

1. Gun Lee
IVE Winter School 2024
Evaluation

2. Why are you building a prototype?

3. Why are you building a prototype?
To evaluate your idea, early!

4. Why did Tom Chi prototype?
https://www.youtube.com/watch?v=d5_h1VuwD6g

5. Evaluation
§ What?
§ Why?
§ Who?
§ When?
§ Where?
§ How?

6. What is Evaluation?
§ Evaluation is gathering data about
the quality of a product/design/idea
§ User performance
§ Usability
§ User eXperience
§ …

7. Outcome of Evaluation
§ What worked ...
§ Things users liked or found notable
§ Useful, efficient, easy, fast ...
§ What could be improved ...
§ Failed, difficult, cumbersome, slow ...
§ Constructive criticism
§ Questions and Ideas ...
§ To continue investigation in the future

8. Outcome of Evaluation = Feedback
Identify needs,
problems &
requirements
(Re)Design
Evaluate /
Test
Implement /
Prototype
Final
A version of
solution

9. Why?
§ To validate our prototype/solution/design/idea.
§ Against the goal, requirement, and problem statement
§ To refine our prototype and design.
§ Informs the next iterations of prototypes.
§ Brings clarity when making decisions.
§ To learn more about user and the problem.
§ Often yields unexpected insights.
§ Test not only the solution, but also if the goal (the problem you
are trying to solve) is framed correctly.
§ To move forward to the next iteration

10. Why?
N-Gage, Nokia, 2003

11. Why?

12. Why?
https://sidetalkin.com

13. When?
§ “You can fix it now on the drafting board
with an eraser or you can fix it later on the
construction site with a sledge hammer.”
- unknown

14. When?
design implementation evaluation
redesign &
reimplementation
design implementation

15. Who?
§ Designer (yourself!)
§ Colleagues
§ Experts
§ Stake holders
§ (End) Users
§ The people for whom you are designing for!
§ Users are not like you (developer/designer)!
§ The user is ALWAYS right!
- Though they might misunderstand themselves …
Informal, Formative testing
Formal, Summative & Validation testing

16. How many?
§ 5 users will find approximately
80% of problems - Nielsen

17. Where?
§ Laboratory
§ In-the-Wild
§ Remote
§ Mechanical Turk
§ Prolific
VR Questionnaire Toolkit
https://www.youtube.com/watch?v=ctzbobrgcdE
https://github.com/MartinFk/VRQuestionnaireToolkit

18. How?
§ Let the user try it first, instead of showing how to.
§ Not a salesmen but an anthropologist
§ Not to pursue users about our solution,
but to learn more from the user and understand.
§ Do not defend your prototype;
instead, watch how the users use and misuse it.
§ Be open to unexpected outcomes and
misunderstandings.
§ Your prototype is not precious,
but the feedback and new insights it draws out are.

19. How?
§ But my prototype is not fully working… 😞
§ No worries, you can be a “Wizard of Oz”! 😉

20. How?

21. How?
§ Screen cast
https://www.meta.com/en-gb/help/quest/articles/in-vr-experiences/oculus-features/cast-with-quest/

22. How?
§ Screen cast is not working… 😞
§ Encourage the user to “Think Aloud” 😉
§ Or record the screen then review it with the user
(Retrospective Think Aloud)

23. How?
§ Feedback capture grid
§ Be systematic about
feedback
§ Be more intentional
about capturing
thoughts in the four
different areas.
Things one likes
or finds notable.
Constructive
criticism, wishes.
Questions that
the experience
raised.
Why?
Ideas that the
experience
or presentation
spurred.

24. How?
§ Got multiple design options?

25. How?
§ What to measure?
§ Presence
§ Co-Presence
§ Social Presence
§ User Experience
§ Usability
§ Cognitive Load
§ Emotion
§ Empathy
§ ...

26. § How to measure?
§ Presence
§ Co-Presence
§ Social Presence
§ User Experience
§ Usability
§ Cognitive Load
§ Emotion
§ Empathy
§ ...
How?
?
“Ask, and it shall be given to you.” - Jesus

27. § Questionnaires
§ Presence
§ Co-Presence
§ Social Presence
§ User Experience
§ Usability
§ Cognitive Load
§ Emotion
§ Empathy
§ ...
How?

28. How?
§ Presence Questionnaires
§ Slater, Usoh, Steed (SUS) (1994)
§ Witmer & Singer (1998)
§ Igroup Presence Questionnaire (IPQ) (Schubert et al. 2001)
§ Co-/Social Presence Questionnaires
§ Bailenson et al. (2001)
§ Networked Minds (Harms & Biocca 2004)
https://www8.informatik.umu.se/~jwworth/PresenceMeasurement.pdf

29. How?
§ User Experience Questionnaire
§ UEQ https://www.ueq-online.org
§ Usability
§ System Usability Scale (SUS) (Brooke 1996)
https://measuringu.com/sus/
§ Single Ease Questionnaire (Sauro & Dumas 2009)
https://measuringu.com/seq10/
https://measuringu.com/seq-prediction/

30. How?
§ Cognitive Load
§ NASA TLX (1986)
https://humansystems.arc.nasa.gov/groups/TLX/
§ Paas (1992)
https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2021.702616/full

31. How?
§ Emotion
§ Self Assessment Manikin
https://www.sciencedirect.com/science/article/pii/
0005791694900639
§ Emoji Grid
https://www.frontiersin.org/journals/psychology/ar
ticles/10.3389/fpsyg.2018.02396/full

32. How?
§ Emotion
§ Emoji Grid in VR
https://link.springer.com/chapter/10.1007/978-3-030-
31908-3_24
§ Real-time Continuous Emotion
Annotation
https://dl.acm.org/doi/abs/10.1145/3411764.3445487
https://www.youtube.com/watch?v=xZyyNTLjvlc

33. How?
§ Empathy?
§ Inclusion of Other in the Self scale (IOS)
https://sparqtools.org/mobility-measure/inclusion-of-
other-in-the-self-ios-scale/
§ Interpersonal Reactivity Index
https://www.eckerd.edu/psychology/iri/
§ Empathy Quotient
https://psychology-tools.com/test/empathy-quotient

34. How?
DECIDE framework to guide evaluation:
- Determine the goals the evaluation addresses.
- Explore the specific questions to be answered.
- Choose the evaluation methods and measures
- Identify the practical issues.
• User recruitment, equipment, space, schedule, budget
- Decide how to deal with the ethical issues.
- Evaluate, interpret and present the data.
Preece, J., Rogers, Y., Sharp, H. (2002), Interaction Design: Beyond Human-Computer Interaction

35. How?
Test your Test, too!

36. Iteration!
Identify needs,
problems &
requirements
(Re)Design
Evaluate /
Test
Implement /
Prototype
Final
A version of
solution

37. Summary
§ Evaluate your idea as early as possible
§ First, set your goal for evaluation!
§ Next, plan ...
§ How to present your prototype
§ What questions to answer
§ How to capture the feedback
§ Collect, summarize, and reflect on the findings
§ Apply what you’ve learnt to the next iteration