An Introduction to Games User Research Methods


Published on

An introductory lecture on Games User Research methods which was first given to students at Hanze University on the 9th of March 2011.

This presentation was later turned into two articles on Gamasutra that can be read here:
- Part 1:
- Part 2:

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This image from Dead Space could be said to have Low Valence, High Arousal = and therefore could be experienced as being scary by players
  • Whereas this image from the game desert bus on the other hand is somewhat low in Valence, but also low in Arousal - Producing boredom.
  • In contrast this is an High Valence, High Arousal moment from Halo Reach which, since it is my Spartan pictured, I can say was a moment of excitement in a firefight game.
  • And finally we have a High Valence, low Arousal image, from the game flower which aims to create a pleasant feeling of relaxation in the players.
  • Now, you will notice there is nowhere on this axis marked “ fun ” – again, fun is somewhat slippery concept. However what can be said is that fun is much more likely to be found associated with emotions on this side on the axis than the other – and is hardly ever associated with this quadrant. The final quadrant is a tricky one, because being scared can be fun – but it can also be overwhelming if the experience is too unpleasant or intense.
  • Closed questions on the other hand let you have much tighter control on the answers your participants give, and come in several flavours. There is the dichotomous scale where people are giving simple yes or no answers.
  • An interval scale uses set positions, and asks for the person filling in the questionnaire to select one of those set intervals. This lowers the resolution of the scale, but makes answers more easy to compare.
  • Whatever you chose to do though, remember it is important that you start doing this research as soon as you think you can manage. It really is much easier to change things early in the process rather than waiting until the end. AND then repeat once changes have been made. And again, do listen to the feedback you get, and observe what people do. But in the end you are the game designers, so you have to decide what is an important issue and what is not.
  • An Introduction to Games User Research Methods

    1. 1. Games User Research Ben Lewis-Evans
    2. 2. >Introduction  Games User Research - Fun & Awareness raising - Emotion - Awareness  Methods - Production - Post-Production
    3. 3. >Games User Research  Game testing traditionally done by the QA/Test department - QA/Test are (usually) experts at gaming - The audience may not be - QA/Test have an investment in the game - The audience does not - Mainly looking for bugs
    4. 4. >Games User Research  About the user experience & fun  What do you want to know? - Is the game fun? - Does it raise awareness?
    5. 5. >Fun  What is fun?  Well… - Easy to use - Challenging - Emotional impact - Engaging - Compelling - Relaxing  It is subjective!
    6. 6. >Emotions & Feelings High Activation Scared - x x – Excited Unpleasant Pleasant Bored - x x – Relaxed Low Activation
    7. 7. >Emotions & Feelings Sometimes fun High Activation Usually fun Scared - x x – Excited Unpleasant Pleasant Bored - x x – RelaxedAlmost never fun Low Activation
    8. 8. >Awareness Raising  What is this?
    9. 9. >Awareness Raising
    10. 10. >In production testing (aka Playtesting)  General points: - Get representative users (kids, 10-14) - Make it clear that the game is being tested, NOT the user - Work out what you want to know before you test
    11. 11. >In production testing (aka Playtesting)  General points cont.: - Test as early as possible, it is easier to fix problems that way (then test again) - Listen to problems, but not necessarily solutions - Not for balance, and bugs, but for fun! (& raised awareness)
    12. 12. >Methods  Focus Groups  Heuristic Evaluation  Questionnaires, Surveys and Interviews  Observational studies  Gameplay metrics  Biometrics/psychophysiology  Think out loud
    13. 13. >Focus groups  6-10 people  Lead by a facilitator - Specific questions  Try the game/discuss potential ideas  Talk about it
    14. 14. >Focus Groups  Pros - More people can = more feedback - Gets everyone together in one place - Follow up questions - Good for discussing concepts  Cons - You need a good facilitator - Strong voices may take over - Too many “helpful” suggestions - What people say is not what they do
    15. 15. >Heuristic Evaluation  Expert evaluation (somewhat like a game review)
    16. 16. >Heuristic Evaluation  List of Heuristics:• Are clear goals provided? • Is the game and the outcome fair?• Are players rewards meaningful? • Is the game replayable?• Does the player feel in control? • Is the AI visible, consistent, yet• Is the game balanced? somewhat unpredictable?• Is the first playthrough and first • Is the game too frustrating?impression good? • Is the learning curve too steep or• Is there a good story? too long?• Does the game continue to progress • Emotional impact?well? • Not too much boring repetition?• Is the game consistent and • Can players recognise importantresponsive? elements on screen?• Is it clear why a player failed? • etc…• Are their variable difficulty levels? From Christina et al 2009
    17. 17. >Heuristics Evaluation  Pros - Smaller numbers - Experts are experts  Cons - You need experts - Which heuristics to pick? - Experts are experts
    18. 18. >Questionnaires, Surveys & Interviews  During gameplay (at or after set moments)  After gameplay  Ask specifically for what interests you (don’t forget about the raising awareness part) - But also allow for some open ended answers
    19. 19. >Questionnaires, Surveys & Interviews  Some pre-existing questionnaires: - Game Experience Questionnaire (GEQ) - - The Computer System Usability Questionnaire (if modified) -  If you use them modify them to fit your particular game & what you want to know
    20. 20. >Questionnaires, Surveys & Interviews  For Emotion – The Affect Grid Russell, Weiss & Mendelsohn, 1989
    21. 21. >Questionnaires design  Order of questions  Use clear, concise everyday, simple language - Avoid jargon - Don’t be vague  Avoid asking duplicate questions - The same question in a different way  Avoid questions that are phrased negatively - “I don’t like the jumping” agree -> disagree - “I like the jumping” agree -> disagree
    22. 22. >Examples of bad questions  Leading ”Now that you have had fun playing our game, which was your favorite level?” Level 1 Level 2 Level 3 Level 4
    23. 23. >Examples of bad questions  Double(7)barrelled: "Should there be a reform of our justice system placing greater emphasis on the needs of victims, providing restitution and compensation for them and imposing minimum sentences and hard labour for all serious violent offences?” Yes No
    24. 24. >Examples of bad questions  Loaded, ill defined and misleading: "Should a smack as part of good parental correction be a criminal offence in New Zealand?" Yes No[The question] "could have been written by Dr Seuss – this isnt GreenEggs and Ham, this is yes means no and no means yes, but were allmeant to understand what the referendum means. I think its ridiculousmyself.” - PM John Key
    25. 25. >Questionnaires design  Type of question - Open - Closed
    26. 26. >Questionnaires design  Type of question - Open - “What did you like about level 1?” - “Why might a child not be able to come to school?” - “How old are you?” - Closed
    27. 27. >Questionnaires design  Closed questions - Dichotomous scale - Yes/No
    28. 28. >Questionnaires design  Closed questions: - Dichotomous scale - Continuous scale
    29. 29. >Questionnaires design  Closed questions: - Dichotomous scale - Continuous scale - Interval scale
    30. 30. >Questionnaires design  Interval scale - Numeric/categorical: 1 2 3 4 5 - Likert: 1 2 3 4 5 Disagree Agree - Semantic: 1 2 3 4 5 Poor Good
    31. 31. >Questionnaires design  Interval scale - Unipolar: 1 2 3 4 5 Not Exciting Very Exciting - Bipolar: 1 2 3 4 5 Very Boring Very Exciting
    32. 32. >Questionnaires, Surveys & Interviews  Questionnaires & Surveys - Pros - Consistent - Quantifiable - Fast - Good for testing raised awareness - Cons - Can lack follow up - Not objective - Need a large(ish) sample
    33. 33. >Questionnaires, Surveys & Interviews  Interviews - Pros - Rich data - Can follow up - Cons - Less quantifiable - Time consuming - Not objective
    34. 34. >Observation studies  Watch/Record through video  Either with a facilitator or without - Facilitator must be as hands off as possible  Watch faces/body for emotion  Only write what you actually see!
    35. 35. >Observation studies  Pros - Objective data - i.e. You see what players actually DO, not what they say they would do - Facilitator can help if really needed  Cons - Time consuming to analyse video - Training required to get the best out of observation (especially for emotion) - Avoid Observer Bias
    36. 36. >Gameplay Metrics  Observation via the game data - Number of incidents - Where and when they occurred and with who or what?
    37. 37. Heatmap of Kills
    38. 38. Heatmap of Deaths
    39. 39. >Gameplay Metrics Deaths  Pros - Objective data - See trends  Cons - Time consuming Kills - No subjective feedback/context - Needs larger sample sizes - Data overload
    40. 40. >Biometrics/psychophysiology  Measuring body signals: - From the Brain (EEG), the Heart (EKG), the muscles (EMG), the eyes (eyetracking), the skin (EDA), etc  The body gives clues into cognition, and emotion
    41. 41. >EDA Game Research  Dying is fun?  Ravaja et al (2008) - EDA increased for - Opponent Killed - Player Killed - EMG (Zygomatic and Orbicularis) - Increased for longer when player killed
    42. 42. >Biometrics/psychophysiology  Pros - Gives objective quantifiable data - Allows for continuous data recording  Cons - Invasive - Costs a lot of time & money to use & analyse - Problems with specificity, artefacts, inference and validity
    43. 43. >Think out loud  An expansion of observation  Players play the game and talk about what they are thinking as they go along - Observe gameplay, and note down what they say & when they say it - Do NOT prompt them, and do NOT correct them
    44. 44. >Think out loud  Pros - Gain an idea what players are thinking & feeling - May give unexpected insight - Could test raised awareness too  Cons - Unnatural - Subjective
    45. 45. >In production testing (aka Playtesting)  Many options  In your case I recommend: - Observation - Gives objective data, and may provide insights (think out loud?) - After play (between level) questionnaire - Allows you to ask specific questions and receive feedback - Collect gameplay metrics as if you can (Optional)
    46. 46. >In production testing (aka Playtesting)  Remember - Don’t wait until the game is almost finished - It is easier to change things early in the process - Listen to what people say is wrong/right, don’t worry too much about what they suggest to do to fix it - You are the game designer
    47. 47. >Post-production testing  Raised awareness, it is too late to be overly concerned about fun  Can’t observe, take biometrics, run focus groups, collect gameplay metrics (game runs offline), etc - On the ground observation? Costly.
    48. 48. >Questionnaires  Pen & Paper is one option  Better to build them into the game - Between levels or within levels - Multiple choice questions - Fill in the blanks
    49. 49. >Questionnaires  In game questions - Require completion to continue & provide rewards for correct answers - Try to make them fun too - i.e playtest them - Have to get the data back somehow though - Uploaded when connected to the web? - Teachers & students can access a report?
    50. 50. >In game behavioural modelling  Raised awareness is fine  Actually doing behaviour is better  So, if you can, build it into the game - Require the awareness you are raising to be used to progress in the game - Not just through answering questionaires - But by DOING
    51. 51.
    52. 52. >Going further than Awareness?  Is there any way you can monitor behaviour change? - New questions on later play throughs? - Teacher/adult report system? - Tied to in-game achievements/rewards?
    53. 53. >Summary  Playtest early, playtest as often as you can - With yourselves - But also with players - Behavioural observation allows for good data with small numbers for playtesting  Build in awareness testing - Require it to progress - Try and make it fun too
    54. 54. Thank you
    55. 55. All images used in this presentation belong to theirrespective copyright holders. If an image belongs to you and you wish it removed please notify me at