Intro to Games User Research Methods - March 2013

2,671 views

Published on

An update to my Introduction to Games User Research lecture (http://www.slideshare.net/Gortag/an-introduction-to-games-user-research-methods). Due to a changing course design this version focuses a bit more on questionnaire design and interviews. A few other changes have been made and the aesthetics have also been changed.

Published in: Technology

Intro to Games User Research Methods - March 2013

  1. 1. Games User Research b.lewis.evans@rug.nl Ben Lewis-Evans
  2. 2. b.lewis.evans@rug.nl@ikbenben
  3. 3. XBL: GortagPSN: LaglGortagSteam: laglgortagb.lewis.evans@rug.nl@ikbenben
  4. 4. Today What is Games User Research? User Research & Evaluation Methodology
  5. 5. QA Testing the game (bugs, technology, etc)
  6. 6. GUR  Examining the player  Fun & user experiences
  7. 7. Fun  What is fun?  Easy to use  Challenging  Emotional impact  Engaging  Compelling  Relaxing  It is subjective!
  8. 8. Sometimes fun High Activation Often fun Unpleasant Pleasant Low ActivationAlmost never fun
  9. 9. General tips  Get (enough) representative users  The game is being tested, NOT the user  Work out what you want to know
  10. 10. General tips  Test as early as possible, fix problems, & then test again (RITE)  Listen to problems, not necessarily solutions  Not (primarily) for balance & bugs, but for fun!  Impact evaluation is different from general GUR
  11. 11. http://www.polygon.com/2012/10/24/3538296 /data-entry-risk-management-and-tacos- inside-halo-4s-playtest-labs
  12. 12. Research Methods
  13. 13. Focus groups  6-10 people  Lead by a facilitator  Specific questions  Try the game/discuss potential ideas  Talk about it
  14. 14. Focus Groups  Pros  More people can = more feedback  Gets everyone together in one place  Follow up questions  Good for discussing concepts  Cons  You need a good facilitator  Strong voices may take over  Too many “helpful” suggestions  What people say is not often what they do  Not for evaluation of impact
  15. 15. Heuristic Evaluation  Expert evaluation (very loosely like a game review) Are clear goals provided? Are players rewards meaningful? Does the player feel in control?  Is the game and the outcome Is the game balanced? fair? Is the first playthrough and first  Is the game replayable? impression good?  Is the AI Is there a good story? visible, consistent, yet Does the game continue to somewhat unpredictable? progress well?  Is the game too frustrating? Is the game consistent and  Is the learning curve too steep responsive? or too long? Is it clear why a player failed?  Emotional impact? Are their variable difficulty  Not too much boring levels? repetition?  Can players recognise important elements on screen? (Christina et al 2009)
  16. 16. Heuristic Evaluation  Pros  Smaller numbers  Experts are experts  Cons  You need experts  Which heuristics to pick?  Experts are experts
  17. 17. Questionnaires &surveys During gameplay (at or after set moments)  After gameplay  Ask for what interests you  Allow for some open ended answers
  18. 18. I can’t show the whole scale here. If you want the GEQ go to http://www.gamexplab.nl/Game Experience Questionnaire (GEQ)http://www.gamexplab.nl/
  19. 19. The MANIKIN scale
  20. 20. The Affect Grid
  21. 21. Custom Questionnaire
  22. 22. How to design aquestionnaire 1. Work out what you want to know 2. Design the questions/statements 3. Create the questionnaire 4. Test the questionnaire
  23. 23. 1. Work out what you want to know Brainstorm, make lists, look at your design documents Then cut it down  Only what you need to know  Aim for no more than 15 minutes
  24. 24. 2. Design the questions/statements  Questions vs StatementsQuestions are good for gaininginformation (age, ratings of fun,etc)How much fun did you have in the last quest? 1–2–3–4–5-6-7 None A lot
  25. 25. 2. Design the questions/statements  Questions vs StatementsStatements are good forassessing attitudes i.e. agreement with an ideaThe last quest was a lot of fun 1–2–3–4–5-6-7 Strongly Strongly agree disagree
  26. 26. 2. Design the questions/statements Use clear, everyday language I really felt like I identified with the motivations of the main protagonist in the game The main character in the game was just like me
  27. 27. Can players identifythese enemies byname?Consider visual aids
  28. 28. Without images would your playersknow a M4A1 from a G36C?
  29. 29. 2. Design the questions/statements  Closed or Open?Indicate your agreement with the followingstatementThis is one of the best research methodslectures I have ever attended: 1 2 3 4 5 Strongly – Agree – Neutral – Disagree – Strongly Agree Disagree
  30. 30. 2. Design the questions/statements  Closed or Open?What was your favourite weapon, and why?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
  31. 31.  Yes/No, Male/Female, True/False  Direct & precise, not very data rich2. Design the questions/statements Scale:  Dichotomous
  32. 32.  Place a mark along a continuum  Data rich, a bit vague2. Design the questions/statements Scale:  Continuous
  33. 33.  Interval: 1 – 2 – 3 – 4 – 5, etc  Good data, quite specific2. Design the questions/statements Scale:  Interval
  34. 34. 2. Design the questions/statements Scale  Be consistent  Problem with lazy responding?  Use a trick question/statement e.g. “Agree with this statement” - Use even spacing 1 2 3 4 5 1 2 3 4 5
  35. 35. 2. Design the questions/statements Both the question/statement and the scale should be clear, and exhaustiveWhat education do you have? Primary School Secondary School Bachelors Masters
  36. 36. 2. Design the questions/statementsWhat is your highest completed level of education? Primary School Secondary School Trade School Bachelors Masters PhD Other:______________
  37. 37. 2. Design the questions/statements Free vs Forced choice What game types do you enjoy (select as many as you like) vs What is your favourite game type (select one)
  38. 38. 2. Design the questions/statements Clear, and exhaustive  Only a single concept/question  Not leading  “This game was fun, how fun was it?”  Not loaded  “People with bad co- ordination typically do badly in our game, how did you do?”
  39. 39. 3. Create the questionnaire Paper vs digital Question order  Number the questions  Easy questions at the start  Don’t lead/cue/give away later answers
  40. 40. Qualtrics survey systemhttp://www.qualtrics.com/
  41. 41. 4. Test the questionnaire Give it to a few people to fill in  No plan survives first contact with the participants  Revise
  42. 42.  Yes/No, Male/Female, True/False  Direct & precise, not very data rich2. Design the questions/statements Scale:  Dichotomous
  43. 43. (sometimes?)Neutral/Neither?
  44. 44. Never Seldom Sometimes Often Always
  45. 45. Disagree Agree
  46. 46.  Questionnaires & Surveys  Pros  Consistent  Quantifiable  Fast  Cons  Lack follow up  Not objective  Need a large(ish) sample
  47. 47. Interviews 1. Setup 2. During an interview 3. After the interview
  48. 48. Conducting an Interview 1. Setup  Chose a good setting  Comfortable with few distractions  Only one or two people  Explain why you are interviewing them  Again, about the game, not them  Tell them how long it will take  Get permission to record the interview
  49. 49. Conducting an Interview1. Setup  Have some set questions  Ask follow up questions  Don’t be confrontational
  50. 50. Conducting an Interview 1. Setup  The same rules as for in a questionnaires  Not leading, neutral, not loaded, clear, only one meaning  Avoid questions that can be answered with yes or no  Start with easy questions
  51. 51. Conducting an Interview 2. During the interview  Only one question at a time  Encourage responding  Head nodding, “ah huh”  But don’t let them go off on a tangent for too long
  52. 52. Conducting an Interview 2. During the interview  Be as neutral as possible  “Act as if you have heard it all before”  Careful with note taking  People are suspicious of interviewers  End with an opportunity for interviewees to add anything they would like
  53. 53. Conducting an Interview 3. After the interview  If you have time make a transcript  Look for common threads in what people say  Group into themes  May point issues you may have never expected
  54. 54.  Interviews  Pros  Rich data  Can follow up  Good for scope  Cons  Less quantifiable  Time consuming  Not objective
  55. 55. Observation studies  Watch/Record  Either with a facilitator or without  Facilitator must be as hands off as possible  Watch faces/body for emotion  Only write down what you actually see!
  56. 56. >Test scripts  Like a recipe for a delicious cake, these lay out how the test will go  Order of events  What the tester should do and say, and when  Clear and precise  Could anyone follow the instructions and carry out a test?  Ensures consistency!
  57. 57. Contextual Inquiry (fieldstudies)  Similar to formal observational studies  In the wild  Observing real use in the real environment  Better once a game is released  Or similar games  Useful for evaluation
  58. 58. Think out loud Observe gameplay, and note down what they say & when they say it Don’t prompt them, and don’t correct them
  59. 59. YOU DON’T COME IN THE BOX WITH THE GAME
  60. 60. Observation studies  Pros  Objective data  i.e. You see what players actually DO, not what they say they would do  Facilitator can help if really needed  Cons  Time consuming to analyse video  Training required to get the best out of observation (especially for emotion)  Avoid Observer Bias
  61. 61. Gameplay Metrics  Observation via game data  Number of incidents  Where, when, and how they occurred and with who or what?
  62. 62. http://engineroom.ubi.com/game-telemetry-with-playtest-dna-on-assassins-creed/
  63. 63. http://engineroom.ubi.com/game-telemetry-with-playtest-dna-on-assassin%E2%80%99s-creed-part-2/
  64. 64. http://engineroom.ubi.com/game-telemetry-with-playtest-dna-on-assassins-creed-part-3/
  65. 65. Gameplay Metrics  Pros  Objective data  Good for evaluation  See trends  Cons  Time consuming  No subjective feedback/context  Needs larger sample sizes  Data overload
  66. 66. Biometrics/psychophysiology  Measuring body signals:  From the Brain (EEG), the Heart (EKG), the muscles (EMG), the eyes (eyetracking), the skin (EDA), etc  The body gives clues into cognition, and emotion
  67. 67. Biometrics/psychophysiology  Pros  Gives objective quantifiable data  Allows for continuous data recording  Cons  Invasive  Costs a lot of time & money to use & analyse  Problems with specificity, artefacts, inference and validity
  68. 68. Summary  Many options  In most cases I recommend:  Observation  Objective data & insights  After play (between level) questionnaires  Specific questions & subjective feedback  WARNING: People don’t often do what they say  Metrics are great for evaluation, tweaking, issues, & insights
  69. 69. Summary  Don’t wait until the game is almost finished for GUR  It is easier to change & plan things early in the process  Listen to what people say is wrong/right, don’t worry too much about what they suggest to do to fix it  You are the game designer
  70. 70. Credits All the copyright holders of the images I have used A rough primer to user research (parts 1 & 2)  http://www.gamasutra.com/view/fe ature/169069/  http://www.gamasutra.com/view/fe ature/170332/ Christina et al 2009  http://mi-lab.org/wp- content/blogs.dir/1/files/publicatio ns/uxInGames_Koeffel_et_al.pdf Special thanks to the GUR-SIG for their feedback and inspiration
  71. 71. Questions?XBL: GortagPSN: LaglGortagSteam: laglgortagb.lewis.evans@rug.nl@ikbenben

×