Games User Research is for Game Design!

2,404 views

Published on

Sarah Romoslawski and Marina Kobayashi presented this talk at the 1st Game Design Conference in San Francisco on September 17th, 2012. The talk defines Games User Research (GUR), and includes two perspectives on practicing GUR at different companies, using GUR with different platforms, genres, and with different target audiences.

Published in: Design, Entertainment & Humor
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,404
On SlideShare
0
From Embeds
0
Number of Embeds
94
Actions
Shares
0
Downloads
0
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide
  • ----- Meeting Notes (9/15/12 12:58) -----introduce names, say names, and say "today we are going to talk to you about WHY and HOW games user research is for game desing
  • Sarah RomoslawskiSifteoGame Designer (Producer) (Playtesting)MFA from ASU’s arts media and engineering program, 5+ years game and experience design for learning.Marina KobayashiElectronic ArtsSr. Games User Research @ EA Game LabMasters in Human-Computer Interaction (CMU HCII), expert with 7+ years experience in usability and user research.----- Meeting Notes (9/15/12 12:58) -----My name is Sarah, I am a game designer at Sifteo where I also help lead playtesting for all dev teams, I have over 5 years experience doing game design and experiential design for learning, a game I really like is TWEWY
  • Sarah RomoslawskiSifteoGame Designer (Producer) (Playtesting)MFA from ASU’s arts media and engineering program, 5+ years game and experience design for learning.Marina KobayashiElectronic ArtsSr. Games User Research @ EA Game LabMasters in Human-Computer Interaction (CMU HCII), expert with 7+ years experience in usability and user research.----- Meeting Notes (9/15/12 12:58) -----My name is Sarah, I am a game designer at Sifteo where I also help lead playtesting for all dev teams, I have over 5 years experience doing game design and experiential design for learning, a game I really like is TWEWY
  • ----- Meeting Notes (9/15/12 12:58) -----we will tell you a little bit about ourselves, we will define games user research, we will explain how GUR helps turn games into great games, and we will provide concrete examples of how GUR helps game design in the context of target user, platform and genre, then we will wrap it up and open the session for Q and A
  • We will provide you with an understanding that you can do GUR on any budget. The types and amount of GUR you can do obviously scales with the amount of support you have and the scope of the game, but every game can be playtested and should be by your target audience.----- Meeting Notes (9/15/12 12:58) -----but Marina, doesn't GUR cost lots of money? I am just a team of one!Marina: yes, it does in some cases, but by providing examples from Sifteo and EA we hope to show that GUR can be scalable depending on your budget, the size of your game, everyone can do GUR
  • Through careful planning, development, and analysis.Different types of research informs the development and strategy. Market researchGames user researchTechnical researchGames user research is a feedback loop for game designers to understand how users experience their games.
  • Design IntentHypotheses about how users will behave or respondExperiment design “how are we going to observe no one doing that thing you want them to do?”Reporting test results quickly and effectively – because its all for you!
  • Test early and often – the lab is a safe place to failIterative testing throughout the game development lifecycle – see if design changes improve the experienceTesting different facets of the game.----- Meeting Notes (9/15/12 12:58) -----its important to test at all stages of the dev cycle
  • Test EARLY and OFTENZoom in to core features and mechanicsThis true is because the basic mechanics will be used as building blocks in the game systems which will rely on these mechanics being solid. An example from Dante’s Inferno is the melee combat mechanic of using a scythe for short and long range attacks against mobs of enemies.
  • Which we are able to capture during a full playthrough. We do this through survey metrics collected at stops at regular intervals and/or after specific events. Usually these pop up when there’s a lull in the gameplay
  • EXP CHART. Zoom out to full game experience, understanding how long it takes to play the game, where are players having the most fun and most frustrationLooking at these metrics holistically we can see how the experience unfolds over the course of the entire game. This helps the dev team visualize the experience and alerts them to any times the player may be significantly derailed. However, if you test whitebox levels and mechanics earlier in the process there should be fewer of these as they should have already been identified. ----- Meeting Notes (9/15/12 12:58) -----to understand the full game experience, how long it takes to play, where users are getting stuck
  • Really we’re not evil – we care about our consumers!
  • Video clip 1: radio silenceJust let me shoot the enemies!
  • WHAT GAME IS THIS lol
  • Our research looked at what players expected to be able to say to control the game, what kinds of actions and controls would be available via this new platform.Then the question was how usable and accessible were the voice commands? Were players able to be as effective with the new controls? Could they get the characters to do what they wanted in close to the same number of attempts as they would with the controller? How would the players feel about the experience of using voice commands?
  • Player 1 tyler: “…”Player 2 chris: “liara?” Oh it fires off where I’m looking? Yeah that’s what its doing.
  • Player 1 tyler: “…”Player 2 chris: “liara?” Oh it fires off where I’m looking? Yeah that’s what its doing.
  • Player 1 tyler: “…”Player 2 chris: “liara?” Oh it fires off where I’m looking? Yeah that’s what its doing.
  • Our research looked at what players expected to be able to say to control the game, what kinds of actions and controls would be available via this new platform.Then the question was how usable and accessible were the voice commands? Were players able to be as effective with the new controls? Could they get the characters to do what they wanted in close to the same number of attempts as they would with the controller? How would the players feel about the experience of using voice commands?
  • Design intent: create a game that encourages players to do physical activity with fitness in mind.Measure the gamers exercises and movement and provide feedback and performance metrics.GUR seeks to address whether the UX is doing this, and where it is failing. When we test a user experience – this must encompass more than just the software – it also takes into account the context, hardware and platform, any related systems that the game will interact within. In Sports active – the game relies on sensors to capture the player’s movement and behavior. The test may look at whether the player understands how to use the hardware effectively, whether they are physically able to, and the emotional experience – do they enjoy the activity? If there are breakdowns in any of those 3 areas, identifying them is key – knowing what the problems are means they can be addressed or at least the team is aware of them. Testing this meant having real gamers come to the lab and workout! Boy did we get sweaty. As the moderator – I wore workout clothes too – to help make the participants feel comfortable participating in an exercise test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • We dedicated a lot of resources to testing Battlefield 3 – doing all these methods to get feedback on the game throughout the development lifecycle.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
  • What other methods are there?We’re looking at
  • What other methods are there?Diary study
  • What other methods are there?Diary study
  • What other methods are there?Diary study
  • What other methods are there?Diary study
  • What other methods are there?Diary study
  • What other methods are there?Diary study
  • What other methods are there?Diary study
  • Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
  • Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
  • Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
  • Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
  • Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
  • Tell me why you like Skyrim – well I like the characters and the graphics. Oh which characters did you like? Ummm the king? I can’t remember his name right now, but I thought he was cool. Ok, how about the graphics? What did you like about them?They were cool – they looked good. I liked the style.How about the gameplay? Yeah it was cool. I liked it.
  • So why does getting the right players matter?When we’re looking for players to test a co-op campaign
  • So why does getting the right players matter?When we’re looking for players to test a co-op campaign
  • Because we’re looking for qualified responses.
  • Targeted to ages 7-12, the sifteo cube gaming platform is hands on. Games are played by manipulating a set of cubes, putting them together (we call that neighboring), shaking, tilting, flipping – Since the cubes are a spatial and tangible interface, the platform really lends itself to puzzle games including word games, logic games, and maze adventure games. The system has audio as well that comes out of what we call the master cube. So now that you have a better understanding of the platform, we can dig a little deeper into what playtesting this platform is like.
  • Finally, the platform AND the games were being built at the same time. As our LEAD QA put it, it is like buiding an overpass and driving on it at the same time.
  • So what do we do at the PlayLab?
  • Cube buddies is a puzzle game. There are characters in the game and you engage with their faces by putting them back together after they have been scrambled. Faces are put back together by neighboring cubes and thus allowing the game to swap parts. So basically, if the blue/green guy and the orange guy were just neighbored, there eyes would have just swapped.
  • Learned through playtesting
  • Lots of things are wrong here, especially the eyes. Zorg’s eyes are on Suli’s head and Suli’s eyes are on Zorg’s head.
  • The player would start by putting the cubes together to swap the eyes.
  • Kids understood that the eyes were upside down, but making the leap to swap the same type of part was HUGE for kids. Why weren’t we teaching this? Kids could obviously tell that the eyes were wrong, they just had no idea what to do with it.
  • We had to single out this puzzle solving skill. This was tough to pinpoint just in level design but easy to pinpoint while playtesting. And playtesting resulted in an entire puzzle level around the core skill – swapping eyes.
  • Tilt the cubes to rearrange letters only on the cube, move the cubes around and neighbor to spell a target word
  • Why present word caravan in the context of platform?
  • Through playtesting we realized that word games are extra tricky. Players came to the game with models of other word games in their mind. They wanted to swipe the cubes like they were an iPad and they wanted to be able to swap out the letters, which they couldn’t. We had to find a way to teach OUR model of a word game.
  • SO how do we teach them our model and our rules, when they have something engrained in their mind of how word games work, AND how do we do that in a way that made the game too easy for older kids.
  • It helps a little, but if you are not an avid reader, you wont even know what patterns to look for, 7-9year olds may only just be learning compound words, phonemic patterns, prefixes and suffixes
  • This certainly helped in early levels of the game, but we were still finding that kids got stuck in later levels
  • We worked with our designers, creating that feedback loop – to carefully to make hypothesis on hinting, and initially, we still weren’t getting results
  • Developer watched each iteration Turn around within a day Sometimes reprogrammed hint mechanism on the fly after watching kids play, seeing first hand that the kids did not notice the icons encouraged the developer to take action on the critical items
  • Sandwich Kingdom is a
  • For this game, it was really important to work with the game designer to come up with a script of specific tasks we wanted to the player to do.
  • Are the users having fun playing as a girl character? Do they feel emersed in the world? What causes them not to be? Will the platform be a distraction?
  • But you have to start somewhere, the best way to start writing a script for your playtest is to play that version of the build
  • And figure out where you yourself are getting confused… this works great in early stages of the game, later one, you might know too much, so put pressure on the developer to discuss what the critical paths are to ensure the player makes it through their quest
  • One feature that is a great example are the dangerous pits. In the final implementation of the game, which you see here, we have a lot of modalities teaching us about the pits, we have an NPC telling us, and art that we will stumble upon if we have pearl walk through the temple a little further
  • So the purpose of the pit is that if you walk over it, it sends you back to the beginning, but early in the game…
  • We didn’t have NPCs to tell the player to be aware and the art for the pit was just some hash marks. Testing something like this early, even before the feature was done, allowed us to understand if the purpose feature was clear to the player or if it was just annoying. By testing this we learned that the player enjoyed avoiding the pits and saw it as a challenging mechanic rather than just a frustrating obstacle, so we continued with the feature and built it out.
  • And depending on your genre, you will need to ask different questions.
  • First off, the boring stuff – forms forms forms…
  • Make sure kids know you aren’t testing them, you are testing the game… you are only there to observe. Let them know that
  • Having siblings play together naturally is a great way to learn if your game is usable or not. It is especially helpful if you can see one child teach the other child how to play. If they are explaining a feature that doesn’t even exist or if they have trouble explaining a feature, then you know you may not be teaching core skills of the game properly
  • Its tough to ask kids “was this hard”
  • But how do we know how the entire game works???
  • We were developing the hw and the sw at the same time. Saving game metrics wasn’t a sensible priority for the team so we had to find another way to get quantitative data
  • Adults can be easier to come by, but you have to keep in mind that adults play differently than kids, so use your target audience as much as possible but recognize what value you can get from having a different set of eyes
  • Longitudinal testing allows you to see major issues, but sometimes it just isn’t possible to fix them
  • - Doing Games User Research is better than nothing! Get your feedback loop on!Think about what your design is about – what do you want the player to experience?Work closely with the Games User Researcher or persons working on the test to make sure they understand what feedback you need – what your priorities are? What information are you lacking?Games user research can help you improve the Game Designthe platform experience, Experience when used by your target audience, and collecting the most actionable feedback – regardless of development phase, gameplay, genre…
  • - Doing Games User Research is better than nothing! Get your feedback loop on!Think about what your design is about – what do you want the player to experience?Work closely with the Games User Researcher or persons working on the test to make sure they understand what feedback you need – what your priorities are? What information are you lacking?Games user research can help you improve the Game Designthe platform experience, Experience when used by your target audience, and collecting the most actionable feedback – regardless of development phase, gameplay, genre…
  • - Doing Games User Research is better than nothing! Get your feedback loop on!Think about what your design is about – what do you want the player to experience?Work closely with the Games User Researcher or persons working on the test to make sure they understand what feedback you need – what your priorities are? What information are you lacking?Games user research can help you improve the Game Designthe platform experience, Experience when used by your target audience, and collecting the most actionable feedback – regardless of development phase, gameplay, genre…
  • Games User Research is for Game Design!

    1. 1. Games User Researchis forGame Design!Marina Kobayashi, Electronic ArtsSarah Romoslawski, Sifteo
    2. 2. Who are we?SarahRomoslawski Marina Kobayashi Game Designer Lead Games User Researcher
    3. 3. Who are we?SarahRomoslawski Marina Kobayashi
    4. 4. OverviewIntrosWhat is games user research(GUR)?How does GUR turn game design into GREAT gamesApplying games user research to game design Platforms Genres Target users (age, lifestyle)Our conclusionsQ&A
    5. 5. From two perspectives: vs. What a mess!
    6. 6. Who are you?It is ok to raise your hand more than once! • Game Designer • Artist • UX/UI Designer • Programmer • Producer • Wearer of many hats • I’m just a unicorn
    7. 7. There is always a player in a game. Gamesare made for people by people.
    8. 8. How do gamesbecome greatgames?Answer:by doing GamesUser Research!
    9. 9. Games user research is a feedback loop forgame designers to understand how usersexperience their games.
    10. 10. GUR + Game Designer = CollabGenius Lather Rinse Repeat
    11. 11. When Should I do GUR? Concept Early Full Alpha Post Phase Production Production Phase ReleaseCollaging/ White Box Vertical Slice Level Tests E3/XBL Demo Full Play Benchmark DLC Tests Tests MP Tests Tests Through Tests Tests Tests All stages of the dev cycle!
    12. 12. From early stages…
    13. 13. Combat Mechanics Test Test EARLY and OFTEN: White Box Testing
    14. 14. To full playthroughs…
    15. 15. Full Playthrough Test Average Player Experience (Via Survey Stops) 10.0 9.0 8.0 7.0 6.0 5.0 4.0 3.0 2.0 1.0 0.0 Fun Difficulty Frustration Start of End of Game Game
    16. 16. What do we learn about? FUNImpact Difficulty factor Ramping Usability in context slate of Puzzle of the new platform Content Early Character and Story Development
    17. 17. Corporate PerspectivePlatformGenreTarget Audience
    18. 18. Platforms,Genres,Gamersand GUR at EA Game Lab“Why do we fall sir? So we might learn to pick ourselves up. ” - Alfred, Batman Begins
    19. 19. Know Your Platform
    20. 20. All Platforms Have Affordances
    21. 21. All Platforms Have Affordances The quality of the device that lets you perform an action
    22. 22. All Platforms Have Constraints
    23. 23. All Platforms Have Constraints Small screens, dual screens, touch screens
    24. 24. All Platforms Have Standards
    25. 25. All Platforms Have Standards
    26. 26. Testing Different Platformsat EAConcrete Examples: Kinect Voice for Mass Effect 3 and EA Sports Active: NFLTraining Camp
    27. 27. GUR on Kinect Voice inMass Effect 3+ 1:1 think-aloud (command aloud) + exploration & experimentation + usability
    28. 28. GUR on Kinect Voice for Mass Effect 3 Kinect voice ≠ natural language
    29. 29. GUR on Kinect Voice in Mass Effect 3 What would you say to control the game?
    30. 30. Video clips
    31. 31. Player 1 Ok, Mordin I’ll see what I can do… “…” Kill kill kill… Die die die…
    32. 32. Player 2 “Xbox Inventory?” Can she “Liara?” hear me? Hmm… I forgot the names of the powers “Liara, Lift Orb?”
    33. 33. GUR on Kinect Voice in Mass Effect 3 What would you say to control the game? ExpectedCommandsUsability of VoiceCommands
    34. 34. GUR on Sports Active Games
    35. 35. “I have to wear what?” Usability Ergonomics
    36. 36. So how do I use this thing?
    37. 37. So how do I use this thing? 8 Tutorials Opinion 8 8 7 7 7 7 7 6 6 6 6 5 5 5 4 4 4 3 3 3 2 2 2 2 2 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 Liked tutorial video Tutorial video needs improvement Players commented that they would like to see There were two mentions that holding the more of a step-by-step breakdown of the exercises. Wii-motes felt awkward, that they were not sure One player also mentioned not knowing they were how to hold them or that the exercise required supposed to observe the trainer doing the exercises. using them at all.
    38. 38. How’d you do?
    39. 39. GUR and Genre at EAConcrete Examples: different methods for different games
    40. 40. Why we used different methods forBF3 Multiplayer vs. Warp?
    41. 41. Why we did different methods for BF3Multiplayer vs. Warp?First-Person Shooter Puzzle Platformer
    42. 42. GUR for theFirst-Person Shooter+ 1:1 think-aloud usability tests for Single Player and Multiplayer+ Dyad usability tests for Co-op+ Multiplayer playtests (over 32 players in-game simultaneously)+ Full playthrough tests+ PPT screen walkthroughs+ Flash Prototype tests
    43. 43. Some observations are best made with live gameplay and think-aloud.
    44. 44. GUR for thePuzzle Platformer+ 1:1 think-aloud usability+ Full playthrough
    45. 45. Warp [Puzzle Platformer]
    46. 46. GUR for Puzzle PlatformersWarp
    47. 47. GUR for Puzzle PlatformersWarp
    48. 48. GUR for Puzzle PlatformersWarp
    49. 49. GUR for Puzzle PlatformersWarp
    50. 50. GUR for Puzzle PlatformersWarp
    51. 51. GUR for Puzzle PlatformersWarp
    52. 52. GUR for RPGsHow to test a game that has 160 hours of content?+ 1:1 think-aloud usability+ Extended Playthrough+ Combat Encounter Tests+ Diary Studies
    53. 53. Dragon Age II
    54. 54. Dragon Age II
    55. 55. Dragon Age II
    56. 56. Dragon Age II
    57. 57. GUR for Facebook RPGsHow to test a game that might be played in 5-10 minuteincrements?+ 1:1 think-aloud usability+ Diary Studies
    58. 58. Dragon Age Legends
    59. 59. Dragon Age Legends
    60. 60. Dragon Age Legends Dragon Age Legends
    61. 61. Dragon Age Legends Dragon Age Legends
    62. 62. Diary Study
    63. 63. GUR on targetaudience~OR~Figuring out who the right gamers are and how to get themin the door.
    64. 64. But who is Mr. Right - for this playtest?
    65. 65. Does Mr. Right check offall these boxes?
    66. 66. Wait you’re giving out incentives?
    67. 67. Phone Screening“Good answer.”
    68. 68. Why getting the right players matters?
    69. 69. Franchise Fans vs. Newcomers Shooting Target Sliding Shooting Shooting Weak Time #Deaths 1 (max # cycles ) (cyc) Spots (min)New PlayersPair 1 8 4 4cyc. 3rd cyc. 3New PlayersPair 2 10 2 5cyc. 1st cyc. 6Fan Pair 1 10 3 3cyc. 1st cyc. <1Fan Pair 2 5 1 4cyc. 1st cyc. <1Average 8.25 2.5 max 4cyc.
    70. 70. Why getting the right players matters?4-player co-op campaign
    71. 71. Why getting the right players matters?When testing a 4-player co-op campaign: “Be the best you can be!” “I will pwn you!”
    72. 72. Why getting the right players matters? Melee DPS Tank Healer / Tech Ranged DPS4-player co-op campaign
    73. 73. Why getting the right players matters? Melee DPS Tank Healer / Tech Ranged DPS “I really like the idea behind the mag-shield. I usually play tanks/defenders aside from assault classes and this “I really enjoyed allowed me to police the playing Izzy. I front lines while my frequently choose teammates flanked and healers in MMOs or sniped from cover. Definitely group/co-op games fits my play style.” and thought she was a great character to start off with.”
    74. 74. Grassroots PerspectivePlatformGenreTarget Audience
    75. 75. Wait, What is Sifteo?
    76. 76. Platforms,Game Design,and GUR at Sifteo(or in other words) “neighbor” isn’t a verb
    77. 77. The Need for the Sifteo PlayLab Original Sifteo Cubes Ok so now let’s find some people to play…
    78. 78. The Need for the Sifteo PlayLabLess Off Site to More On Site Testing! Ok so now let’s find some people to play…
    79. 79. Design in Parallel Wait, you want me to drive on that?
    80. 80. Creating Standard Visual Language Neighbor isn’t a verb
    81. 81. What we do at the PlayLab:MeasureFunthroughPlaytesting
    82. 82. How does Sifteo Measure Fun? Fun = Learning (Koster, A Theory of Fun) tinkering FUN of MASTERING new skill Frustration, the game doesn’t teach me!
    83. 83. All Platforms Have Affordances The quality of the device that lets you perform an action
    84. 84. Affordances – Spatial reasoning and CubeBuddiesSifteo Cubes AFFORD for spatial reasoning puzzles, but spatial skills arehoned throughout life.
    85. 85. Game Play
    86. 86. Zorg’s EYE Problem
    87. 87. Really Mixed Up Faces
    88. 88. Almost Done, But Eyes Wrong
    89. 89. The player would get to here andSTOP
    90. 90. What Skill Were We Forgetting to teach?
    91. 91. We didn’t teach aneye for an eye!
    92. 92. Solution: Teach How to Swap UpsideDown Eyes
    93. 93. All of the Above -- Sifteo Example Dealing with platform CONSTRAINTS, AFFORDANCES and STANDARDS on Word Caravan
    94. 94. WC Game Play
    95. 95. Why WC?• Must be fun for 7-12, the platform requires this as a STANDARD for all games
    96. 96. Why WC?• Must be fun for 7-12, the platform requires this as a STANDARD for all games• Must improve our original word game, Word Play, yet still be playable on 3 or more cubes, a design CONSTRAINT of the platform
    97. 97. Why WC?• Must be fun for 7-12, the platform requires this as a STANDARD for all games• Must improve our original word game, Word Play, yet still be playable on 3 or more cubes, a design CONSTRAINT of the platform• Shuffling letters to make words exploits the AFFORDANCES of the platform
    98. 98. Why WC?• Must be fun for 7-12, the platform requires this as a STANDARD for all games• Must improve our original word game, Word Play, yet still be playable on 3 or more cubes, a design CONSTRAINT of the platform• Shuffling letters to make words exploits the AFFORDANCES of the platform• ALL OF THE ABOVE due to the platform resulted in the creation of an appropriate hint mechanism imited inputs and screen space
    99. 99. Pre-existing Models ofWord Games
    100. 100. Problem(s)• Game gets hard too quickly• Players get stuck too early in the game• Players naturally want to apply models of other word games•BUT we didn’t want to make it too easy for the older age groups Everyone I know has a big but…
    101. 101. Too much frustration, too little fun FUN of MASTERING tinkering new skill Frustration, the game doesn’t teach me!
    102. 102. Will pattern recognition help?•Block, lock•Lock, unlock•Date, update•Make remake•Act, fact, facts, factor, artifact
    103. 103. We simplified the puzzles BAT FIX WORD
    104. 104. What do kids do when they can’t reada word?
    105. 105. Solution: Hint MechanismHave the game be the parent! SCAFFOLD the player.
    106. 106. Iterate, test, iterate, test, iterate, test…• Earned hint, tracked with icon• Automatic hint• Increase the size of the hint icon• Delayed hint Older version, Not noticeable
    107. 107. Hypotheses vs. Actual …well I guess not this time…
    108. 108. Final Implementation – InfiniteDelayed Hints, Large Icons
    109. 109. Developer On Site! Children often do not report seeing animated elements that appear on one or more of the screens, Even if the elements are high contrast and cover 1/64 of the total screen area. Increasing the size to 1/16 total screen area seemed to solve this problem. Chris March, Lead on Word Caravan
    110. 110. GUR and Genre at SifteoDesigning task oriented 1:1 sessions for the Sandwich Kingdom RPG
    111. 111. Sandwich Kingdom RPG
    112. 112. Task Oriented, 1 or 1:2, Think Aloud
    113. 113. Designing a Script: Quest or TaskFocused I am… vs. Pearl is… Yes, yes with help, or no
    114. 114. PLAY THE BUILD
    115. 115. WHERE DID I GET CONFUSED?
    116. 116. Early Feature Test: Dangerous Pits
    117. 117. Early Feature Test: Dangerous Pits
    118. 118. Feature Test: Dangerous PitsNo NPCsNo Final Art
    119. 119. So we use similar methods over allgames
    120. 120. But we ask different questions in theend
    121. 121. Acquiring Actionable Data fromKids~OR~when the chocolate covered pretzels are more interesting thanyour game, you might have a design problem
    122. 122. Informed Consent, NDA and PhotoRelease
    123. 123. The Researcher’s Role
    124. 124. Facilitate Play We want you to help us find problems! We want you to tinker!
    125. 125. Facilitate Communication Teach each other how to play.
    126. 126. Play with Parent
    127. 127. Everyone wants to feel smart, evenkids!
    128. 128. Sifteo: Longitudinal StudiesConcrete Example -- Collecting Data With Kids on the ~Entire Puzzle Game~
    129. 129. No Stats Saved on HW at this Point.• So how do we know how long it takes?• How many tries to solve the puzzle?• Where is the player successful?• Hrmmm…• We can do this!
    130. 130. Solution: Pizza and Games•IT’S SUMMER!•2-3 x 3hr sessions•Lunch included (pizza)•Gift card for the parent•Certificate of completion•Did I mention these were 3 - 4 hr sessions with kids 14 and under?
    131. 131. Testing Chroma Splash Puzzle Mode
    132. 132. Game Play
    133. 133. Goal:Puzzle Difficulty Ramp Up
    134. 134. A Chroma Splash Puzzle
    135. 135. What about adults?Sometimes you run out of kids…BUT you still need data!
    136. 136. Kids and Adults are Different My biggest surprise was how reticent older adults seemed to be to experiment with the game. That resulted in certain adults taking far longer to complete puzzles than kids did. Scott Kim, Chroma Splash
    137. 137. Actionable Data can be Tricky wanted a Some testers perfect solver that would fail the user in any situation where they could not proceed. This seemed dangerous and very difficult, so I did not attempt it. Eric Liao, Chroma Splash
    138. 138. Key Takeaways
    139. 139. GUR is your hydrant to feedback…enjoy it!
    140. 140. Thank you!Marina Kobayashi Sarah Romoslawskimkobayashi@ea.com sarah@sifteo.com

    ×