Games Design 2 - Lecture 9 - User Evaluation


Published on

Published in: Education, Technology, Design
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Games Design 2 - Lecture 9 - User Evaluation

  1. 1. [email_address] Game Design 2 Lecture 9: End-User Evaluation
  2. 2. Why End-User Evaluation? <ul><li>As discussed in Lecture 8, much evaluation can take place without end-user evaluation. </li></ul><ul><li>However, there is no substitute for the feedback that the target audience supplies. </li></ul><ul><li>Only by having players use your interface can you tell where the most important issues lie. </li></ul>
  3. 3. Two Approaches <ul><li>You can ask players what they think </li></ul><ul><ul><li>Identify unexpected issues </li></ul></ul><ul><ul><li>Can be relatively quick and simple way to gather data from many players </li></ul></ul><ul><ul><li>The only way to ‘get inside the head’ of a player. </li></ul></ul><ul><li>You can observe what players do </li></ul><ul><ul><li>It can be hard for players to voice issues </li></ul></ul><ul><ul><li>Designer can focus on players’ natural response to a game item without drawing attention </li></ul></ul><ul><ul><li>Can find issues that the designer doesn’t expect and the player doesn’t notice </li></ul></ul>
  4. 4. Asking Questions <ul><li>Oral or Written </li></ul><ul><li>2 Basic types of questions </li></ul><ul><ul><li>Open-ended questions </li></ul></ul><ul><ul><li>Closed-ended questions </li></ul></ul><ul><li>Questionnaires can contain one or both types of question </li></ul>
  5. 5. Open Ended Questionnaires <ul><li>Lead to larger variety of answers than closed-ended questionnaires </li></ul><ul><li>Answered in natural language. </li></ul><ul><li>Result in qualitative data which has to be analysed further before being considered quantitative. </li></ul>
  6. 6. Can you see any difficulties in receiving data like this?
  7. 7. Open Ended Questions
  8. 8. How to write open ended questions <ul><li>Avoid leading language </li></ul><ul><ul><li>“ Was the control scheme easy to use?” </li></ul></ul><ul><ul><li>Questions like this tend to guide the interviewee into answering the way the interviewer wishes. </li></ul></ul><ul><li>Avoid double barrelled questions that ask more than one thing at once. </li></ul><ul><ul><li>“ How challenging was the second level? Would you have preferred more or less ammo on this level?” </li></ul></ul>
  9. 9. Benefits of Open Ended Questions <ul><li>Allow respondents to include more information including feelings, attitudes and understanding of the subject. </li></ul><ul><li>Respondents can’t forget or miss the range of answers applicable (imagine a race survey and missing the race that most applies to you) </li></ul><ul><li>Respondents can’t lazily answer ‘no’ or ‘yes’ to every question (thus skewing data) </li></ul>
  10. 10. Disadvantages of Open-Ended <ul><li>More difficult to write than closed-ended questionnaires </li></ul><ul><li>May result in irrelevant information </li></ul><ul><li>Some interviewees don’t know how to answer them or feel on the spot </li></ul><ul><li>May result in too much information </li></ul><ul><li>May be (very) difficult to analyse </li></ul>
  11. 11. Closed Ended Questions <ul><li>User chooses from a list of predefined responses. </li></ul><ul><li>Lead directly to quantifiable data. </li></ul><ul><li>Although they naturally result in a narrower range of responses than open ended questions, it is still possible to determine much of the same kind of data. </li></ul>
  12. 12. How would you redesign the question as a closed ended one so as to avoid the problems identified earlier?
  13. 13. Closed Ended Questions
  14. 14. Choosing Answers <ul><li>Can be a list of predefined answers (such as with colours). </li></ul><ul><li>For simple questions, can use ‘yes’, ‘don’t know’, ‘no’ to determine the audience’s attitude towards a facet of your game. </li></ul><ul><li>To gauge more subtle responses, a Likert scale may be used. </li></ul>
  15. 15. Likert Scale <ul><li>Typically a 5 point scale </li></ul><ul><ul><li>Strongly disagree </li></ul></ul><ul><ul><li>Disagree </li></ul></ul><ul><ul><li>Neither agree nor disagree </li></ul></ul><ul><ul><li>Agree </li></ul></ul><ul><ul><li>Strongly Agree </li></ul></ul><ul><li>Allows gauging of ‘in between’ answers that are not boolean. </li></ul>
  16. 16. Benefits of Closed-Ended <ul><li>Easy to quantify since they directly provide quantitative data </li></ul><ul><ul><li>(you just count the number for each response) </li></ul></ul><ul><li>Simpler to write and to fill out </li></ul><ul><li>Can have a more semantic understanding of responses rather than the potential infinity of natural language </li></ul>
  17. 17. Observing Behaviour <ul><li>Playtests can be conducted in a focus group setting or individually. </li></ul><ul><li>Focus group settings allow for a quicker understanding of consensus </li></ul><ul><ul><li>Care is needed to avoid dominant personalities from steering group </li></ul></ul><ul><ul><li>Some people are too shy to speak up in public </li></ul></ul><ul><li>Individual observations allow for discovering a wider range of responses. </li></ul>
  18. 18. Usability Observation <ul><li>Ideally play is recorded for later analysis although it is possible to take notes during observation. </li></ul><ul><li>Video should be used to record facial expressions, body language and input devices in addition to game output. </li></ul><ul><li>The eMotion lab at Caledonian is fitted with additional equipment including an eye tracker and physiological sensors. </li></ul>
  19. 19. Observations <ul><li>Observations can result in meaningful data from a small number of participants. </li></ul><ul><li>Nielson says: “the best results come from testing no more than five users and running as many small tests as you can afford. As you add more and more users, you learn less and less because you keep seeing the same things again and again.” </li></ul>
  20. 20. <ul><li>Important to remember that the goal is to test the game (or game interface) and NOT to test the user. </li></ul><ul><li>If the player struggles to understand an interface, it is the designer’s fault, not the player’s. </li></ul><ul><li>The facilitator should provide no more information that the final end-user would receive. In other words, the facilitator should not try to help the participant in any way. </li></ul><ul><li>Observations are ideal places to use ‘Think Aloud’ methods in addition to analysis of what is recorded. </li></ul>
  21. 21. Iteration <ul><li>Audience feedback is used to modify designs which are then tested again until the design satisfies its requirements. </li></ul><ul><li>Often a wholly different set of users test the new design to avoid improved results due to familiarity. </li></ul>
  22. 22. Fixing Problems With Card Sorting <ul><li>If an interface proves difficult to design, a more structured approach may be necessary to create a design that is intuitive to the target audience. </li></ul><ul><li>Card Sorting is a methodology which enables non-expert end-users to help categorise items in a way which is useful for interface an user experience design. </li></ul>
  23. 23. Card Sorting <ul><li>Open sorting </li></ul><ul><ul><li>Often used early on in process </li></ul></ul><ul><ul><li>Users can define their own categories </li></ul></ul><ul><ul><li>Can also repeat the task dependent on a criteria of their choosing </li></ul></ul><ul><li>Closed sorting </li></ul><ul><ul><li>Used later in process </li></ul></ul><ul><ul><li>Categories are pre-defined </li></ul></ul>
  24. 24. Card Sorts Analysis <ul><li>Category/Criteria names </li></ul><ul><ul><li>verbatim agreement </li></ul></ul><ul><ul><li>gist agreement </li></ul></ul><ul><ul><ul><li>super-ordinate grouping </li></ul></ul></ul><ul><li>Groupings </li></ul><ul><ul><li>cluster analysis, tree diagram, co-occurrence matrices </li></ul></ul><ul><li>Number of criteria/categories </li></ul>
  25. 25. Co-occurrence matrix
  26. 26. Example superordinate grouping Super-ordinate grouping performed by an Independent Judge “ Your task is to interpret the criteria into super-ordinate constructs. You should try to identify where the criterion given by one respondent could be said to have meant the same as another but simply have chosen different wording.”
  27. 27. Further reading <ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.