SlideShare a Scribd company logo
Running User Tests for
VR Games
Steve Bromley
@steve_bromley
My background
I’ve been at PlayStation for 5 years, working on their
hardware, software and system UI.
For VR, I have:
- Run research for ergonomics and comfort
- Run research in participants homes
- Run quantitative tests
- Run many, many qualitative tests on the headset and
the launch games
Onto the good stuff…
Room Setup
Running User Tests for VR Games
1:1 Games Testing
Getting the room right is very important for VR games
Watch out for the one way mirror
Beware moving chairs
Room scale VR
Room Setup
Make sure that the room setup is not going to cause technical or usability issues
during the session.
Keep the user safe!
Methodology
Running User Tests for VR Games
Qualitative Research
They cant see or hear you
Multi-Seat Tests
Methodology
Remember to adjust your study design to take into account that players can’t hear
you.
Participants
Running User Tests for VR Games
No-one has used VR before
Remove comfort issues
Participants
Help people with the set-up
Recruit resilient people!
Opinions
Running User Tests for VR Games
For usability, 5-8 users per audience will give a good balance between
comprehensiveness and practicality.
5-8 users is not good enough for reliable opinion data.
REPRESENTATIVE
It’s often tempting to ask ‘overall’ opinion questions anyway
- Development teams want you too.
- You want to be nice to your dev teams.
This is (usually) a waste of time for VR Games
Can I do anything to get better opinion data from qualitative sessions?
Palate Cleanser Getting experienced users
Opinions
Ultimately, you probably shouldn’t bother with overall scores or generic likes/dislikes.
Run a proper quantitative test if that's what you're after, and ask more nuanced
questions
Summary
Set the room up right.
Think about the methodology, it’s not the same as other games.
Don’t expect to talk to players.
Help users with the system, they’ll be new to it!
Don’t expect useful opinion data.
Thanks!
Any Questions?

More Related Content

What's hot

Design for results2
Design for results2Design for results2
Design for results2
Mirjam Eladhari
 
How to run 1-to-1 usability testing (with life scientists)
How to run 1-to-1 usability testing (with life scientists)How to run 1-to-1 usability testing (with life scientists)
How to run 1-to-1 usability testing (with life scientists)
Jennifer Cham
 
Prototyping: Making Questions More Tangible
Prototyping: Making Questions More TangiblePrototyping: Making Questions More Tangible
Prototyping: Making Questions More Tangible
4.0 Schools
 
Serious games cwltgm
Serious games   cwltgmSerious games   cwltgm
Serious games cwltgm
David Farrell
 
2011 1- the role of the game designer
2011  1- the role of the game designer2011  1- the role of the game designer
2011 1- the role of the game designer
Dublin Institute of Technology
 
Design for Results - Considerations for experimental prototyping and play tes...
Design for Results - Considerations for experimental prototyping and play tes...Design for Results - Considerations for experimental prototyping and play tes...
Design for Results - Considerations for experimental prototyping and play tes...
Mirjam Eladhari
 
2011 the role of the game designer
2011   the role of the game designer2011   the role of the game designer
2011 the role of the game designer
Dublin Institute of Technology
 
Lecture 11
Lecture 11Lecture 11
Lecture 11
Charles Palmer
 
Mission Possible Day 3.ping.pong2014
Mission Possible   Day 3.ping.pong2014Mission Possible   Day 3.ping.pong2014
Mission Possible Day 3.ping.pong2014
jmori
 
Informing SoTL using playtesting techniques
Informing SoTL using playtesting techniquesInforming SoTL using playtesting techniques
Informing SoTL using playtesting techniques
Katrin Becker
 
Serious games career quest
Serious games   career questSerious games   career quest
Serious games career quest
David Farrell
 
How to bring creativity in testing
How to bring creativity in testingHow to bring creativity in testing
How to bring creativity in testing
Tabăra de Testare
 
Guerrilla usability testing
Guerrilla usability testingGuerrilla usability testing
Guerrilla usability testing
Nexer Digital
 
Game design 2 (2013): Lecture 10 - Expert Evaluation Methods for Game UI
Game design 2 (2013): Lecture 10 - Expert Evaluation Methods for Game UIGame design 2 (2013): Lecture 10 - Expert Evaluation Methods for Game UI
Game design 2 (2013): Lecture 10 - Expert Evaluation Methods for Game UI
David Farrell
 
Development Game with Purpose - AGS
Development Game with Purpose - AGSDevelopment Game with Purpose - AGS
Development Game with Purpose - AGS
Paul Boos
 
Moderate This!
Moderate This! Moderate This!
Moderate This!
Lilly Smith
 
Adrian estoup beat the-test
Adrian estoup   beat the-testAdrian estoup   beat the-test
Adrian estoup beat the-test
elkako38
 
4YFN 2016 Guerrilla UX
4YFN 2016 Guerrilla UX4YFN 2016 Guerrilla UX
4YFN 2016 Guerrilla UX
Sarah Rink
 
Game Design, October 24th, 2013
Game Design, October 24th, 2013Game Design, October 24th, 2013
Game Design, October 24th, 2013
Miami University
 
10 questions
10 questions10 questions
10 questions
Robert Roth
 

What's hot (20)

Design for results2
Design for results2Design for results2
Design for results2
 
How to run 1-to-1 usability testing (with life scientists)
How to run 1-to-1 usability testing (with life scientists)How to run 1-to-1 usability testing (with life scientists)
How to run 1-to-1 usability testing (with life scientists)
 
Prototyping: Making Questions More Tangible
Prototyping: Making Questions More TangiblePrototyping: Making Questions More Tangible
Prototyping: Making Questions More Tangible
 
Serious games cwltgm
Serious games   cwltgmSerious games   cwltgm
Serious games cwltgm
 
2011 1- the role of the game designer
2011  1- the role of the game designer2011  1- the role of the game designer
2011 1- the role of the game designer
 
Design for Results - Considerations for experimental prototyping and play tes...
Design for Results - Considerations for experimental prototyping and play tes...Design for Results - Considerations for experimental prototyping and play tes...
Design for Results - Considerations for experimental prototyping and play tes...
 
2011 the role of the game designer
2011   the role of the game designer2011   the role of the game designer
2011 the role of the game designer
 
Lecture 11
Lecture 11Lecture 11
Lecture 11
 
Mission Possible Day 3.ping.pong2014
Mission Possible   Day 3.ping.pong2014Mission Possible   Day 3.ping.pong2014
Mission Possible Day 3.ping.pong2014
 
Informing SoTL using playtesting techniques
Informing SoTL using playtesting techniquesInforming SoTL using playtesting techniques
Informing SoTL using playtesting techniques
 
Serious games career quest
Serious games   career questSerious games   career quest
Serious games career quest
 
How to bring creativity in testing
How to bring creativity in testingHow to bring creativity in testing
How to bring creativity in testing
 
Guerrilla usability testing
Guerrilla usability testingGuerrilla usability testing
Guerrilla usability testing
 
Game design 2 (2013): Lecture 10 - Expert Evaluation Methods for Game UI
Game design 2 (2013): Lecture 10 - Expert Evaluation Methods for Game UIGame design 2 (2013): Lecture 10 - Expert Evaluation Methods for Game UI
Game design 2 (2013): Lecture 10 - Expert Evaluation Methods for Game UI
 
Development Game with Purpose - AGS
Development Game with Purpose - AGSDevelopment Game with Purpose - AGS
Development Game with Purpose - AGS
 
Moderate This!
Moderate This! Moderate This!
Moderate This!
 
Adrian estoup beat the-test
Adrian estoup   beat the-testAdrian estoup   beat the-test
Adrian estoup beat the-test
 
4YFN 2016 Guerrilla UX
4YFN 2016 Guerrilla UX4YFN 2016 Guerrilla UX
4YFN 2016 Guerrilla UX
 
Game Design, October 24th, 2013
Game Design, October 24th, 2013Game Design, October 24th, 2013
Game Design, October 24th, 2013
 
10 questions
10 questions10 questions
10 questions
 

Viewers also liked

Deborah Amar - Differences between mainstream UX and games UX
Deborah Amar - Differences between mainstream UX and games UXDeborah Amar - Differences between mainstream UX and games UX
Deborah Amar - Differences between mainstream UX and games UX
uxbri
 
Henry Ryder - Xd presentation
Henry Ryder - Xd presentation Henry Ryder - Xd presentation
Henry Ryder - Xd presentation
uxbri
 
TalkUX - UX in VR - UNIT9
TalkUX - UX in VR - UNIT9TalkUX - UX in VR - UNIT9
TalkUX - UX in VR - UNIT9
Laura Cortes
 
To Bot or Not to Bot: Using Dramaturgy to connect Research with Development
To Bot or Not to Bot: Using Dramaturgy to connect Research with DevelopmentTo Bot or Not to Bot: Using Dramaturgy to connect Research with Development
To Bot or Not to Bot: Using Dramaturgy to connect Research with Development
Mark McElhaw
 
Designing the future of Augmented Reality
Designing the future of Augmented RealityDesigning the future of Augmented Reality
Designing the future of Augmented Reality
Carina Ngai
 
The rise of VR & AR era. Why this time is different?
The rise of VR & AR era. Why this time is different?The rise of VR & AR era. Why this time is different?
The rise of VR & AR era. Why this time is different?
Vasily Ryzhonkov
 
Making The Source for Macmillan Cancer Support: Rob Pearson's talk at Product...
Making The Source for Macmillan Cancer Support: Rob Pearson's talk at Product...Making The Source for Macmillan Cancer Support: Rob Pearson's talk at Product...
Making The Source for Macmillan Cancer Support: Rob Pearson's talk at Product...
Rob Pearson
 
How to get better answers from asking better questions
How to get better answers from asking better questionsHow to get better answers from asking better questions
How to get better answers from asking better questions
Chris How
 
Narración Interactiva en Realidad Virtual: Vivir la Película
Narración Interactiva en Realidad Virtual: Vivir la PelículaNarración Interactiva en Realidad Virtual: Vivir la Película
Narración Interactiva en Realidad Virtual: Vivir la Película
Università degli Studi di Genova
 
Virtual Reality (VR) Lecture @ SAE Institute London
Virtual Reality (VR) Lecture @ SAE Institute LondonVirtual Reality (VR) Lecture @ SAE Institute London
Virtual Reality (VR) Lecture @ SAE Institute London
Richard Hoffmann
 
Workshop 15: Ionic framework
Workshop 15: Ionic frameworkWorkshop 15: Ionic framework
Workshop 15: Ionic framework
Visual Engineering
 
Cs 2. del cross al trans
Cs 2. del cross al transCs 2. del cross al trans
Cs 2. del cross al trans
Università degli Studi di Genova
 
Next Gen Input Modalities in VR/AR
Next Gen Input Modalities in VR/ARNext Gen Input Modalities in VR/AR
Next Gen Input Modalities in VR/AR
Anders Bo Pedersen
 
Certificaciones y empleabilidad julio2016
Certificaciones y empleabilidad julio2016Certificaciones y empleabilidad julio2016
Certificaciones y empleabilidad julio2016
CENEC MALAGA
 
Transmedia italiano
Transmedia italianoTransmedia italiano
Designing the User Experience Curve
Designing the User Experience CurveDesigning the User Experience Curve
Designing the User Experience Curve
Andy Budd
 
COMP 4010: Lecture11 AR Interaction
COMP 4010: Lecture11 AR InteractionCOMP 4010: Lecture11 AR Interaction
COMP 4010: Lecture11 AR Interaction
Mark Billinghurst
 
Marketing digital sin misterio.
Marketing digital sin misterio.Marketing digital sin misterio.
Marketing digital sin misterio.
Rodrigo Reyes Restrepo
 
Using AR for Vehicle Navigation
Using AR for Vehicle NavigationUsing AR for Vehicle Navigation
Using AR for Vehicle Navigation
Mark Billinghurst
 
2013 Lecture4: Designing AR Interfaces
2013 Lecture4: Designing AR Interfaces2013 Lecture4: Designing AR Interfaces
2013 Lecture4: Designing AR Interfaces
Mark Billinghurst
 

Viewers also liked (20)

Deborah Amar - Differences between mainstream UX and games UX
Deborah Amar - Differences between mainstream UX and games UXDeborah Amar - Differences between mainstream UX and games UX
Deborah Amar - Differences between mainstream UX and games UX
 
Henry Ryder - Xd presentation
Henry Ryder - Xd presentation Henry Ryder - Xd presentation
Henry Ryder - Xd presentation
 
TalkUX - UX in VR - UNIT9
TalkUX - UX in VR - UNIT9TalkUX - UX in VR - UNIT9
TalkUX - UX in VR - UNIT9
 
To Bot or Not to Bot: Using Dramaturgy to connect Research with Development
To Bot or Not to Bot: Using Dramaturgy to connect Research with DevelopmentTo Bot or Not to Bot: Using Dramaturgy to connect Research with Development
To Bot or Not to Bot: Using Dramaturgy to connect Research with Development
 
Designing the future of Augmented Reality
Designing the future of Augmented RealityDesigning the future of Augmented Reality
Designing the future of Augmented Reality
 
The rise of VR & AR era. Why this time is different?
The rise of VR & AR era. Why this time is different?The rise of VR & AR era. Why this time is different?
The rise of VR & AR era. Why this time is different?
 
Making The Source for Macmillan Cancer Support: Rob Pearson's talk at Product...
Making The Source for Macmillan Cancer Support: Rob Pearson's talk at Product...Making The Source for Macmillan Cancer Support: Rob Pearson's talk at Product...
Making The Source for Macmillan Cancer Support: Rob Pearson's talk at Product...
 
How to get better answers from asking better questions
How to get better answers from asking better questionsHow to get better answers from asking better questions
How to get better answers from asking better questions
 
Narración Interactiva en Realidad Virtual: Vivir la Película
Narración Interactiva en Realidad Virtual: Vivir la PelículaNarración Interactiva en Realidad Virtual: Vivir la Película
Narración Interactiva en Realidad Virtual: Vivir la Película
 
Virtual Reality (VR) Lecture @ SAE Institute London
Virtual Reality (VR) Lecture @ SAE Institute LondonVirtual Reality (VR) Lecture @ SAE Institute London
Virtual Reality (VR) Lecture @ SAE Institute London
 
Workshop 15: Ionic framework
Workshop 15: Ionic frameworkWorkshop 15: Ionic framework
Workshop 15: Ionic framework
 
Cs 2. del cross al trans
Cs 2. del cross al transCs 2. del cross al trans
Cs 2. del cross al trans
 
Next Gen Input Modalities in VR/AR
Next Gen Input Modalities in VR/ARNext Gen Input Modalities in VR/AR
Next Gen Input Modalities in VR/AR
 
Certificaciones y empleabilidad julio2016
Certificaciones y empleabilidad julio2016Certificaciones y empleabilidad julio2016
Certificaciones y empleabilidad julio2016
 
Transmedia italiano
Transmedia italianoTransmedia italiano
Transmedia italiano
 
Designing the User Experience Curve
Designing the User Experience CurveDesigning the User Experience Curve
Designing the User Experience Curve
 
COMP 4010: Lecture11 AR Interaction
COMP 4010: Lecture11 AR InteractionCOMP 4010: Lecture11 AR Interaction
COMP 4010: Lecture11 AR Interaction
 
Marketing digital sin misterio.
Marketing digital sin misterio.Marketing digital sin misterio.
Marketing digital sin misterio.
 
Using AR for Vehicle Navigation
Using AR for Vehicle NavigationUsing AR for Vehicle Navigation
Using AR for Vehicle Navigation
 
2013 Lecture4: Designing AR Interfaces
2013 Lecture4: Designing AR Interfaces2013 Lecture4: Designing AR Interfaces
2013 Lecture4: Designing AR Interfaces
 

Similar to Steve Bromley - Running User Tests for VR Games

LAFS Game Design 8 - Playtesting
LAFS Game Design 8 - PlaytestingLAFS Game Design 8 - Playtesting
LAFS Game Design 8 - Playtesting
David Mullich
 
User Experience 6: Qualitative Methods, Playtesting and Interviews
User Experience 6: Qualitative Methods, Playtesting and InterviewsUser Experience 6: Qualitative Methods, Playtesting and Interviews
User Experience 6: Qualitative Methods, Playtesting and Interviews
Marc Miquel
 
Making a game "Just Right" through testing and play balancing
Making a game "Just Right" through testing and play balancingMaking a game "Just Right" through testing and play balancing
Making a game "Just Right" through testing and play balancing
Julio Gorgé
 
Intro to Games User Research Methods - March 2013
Intro to Games User Research Methods - March 2013Intro to Games User Research Methods - March 2013
Intro to Games User Research Methods - March 2013
Ben Lewis-Evans
 
Initial thoughts on live user tests for games
Initial thoughts on live user tests for gamesInitial thoughts on live user tests for games
Initial thoughts on live user tests for games
Johan Hoberg
 
A Primer On Play: How to use Games for Learning and Results
A Primer On Play: How to use Games for Learning and ResultsA Primer On Play: How to use Games for Learning and Results
A Primer On Play: How to use Games for Learning and Results
Sharon Boller
 
Make Your Games Play, Teach, and Socialize Better: Usability & Playability Te...
Make Your Games Play, Teach, and Socialize Better: Usability & Playability Te...Make Your Games Play, Teach, and Socialize Better: Usability & Playability Te...
Make Your Games Play, Teach, and Socialize Better: Usability & Playability Te...
Jason Schklar Consults!
 
Primer on Play: Case Study for Knowledge Guru
Primer on Play: Case Study for Knowledge GuruPrimer on Play: Case Study for Knowledge Guru
Primer on Play: Case Study for Knowledge Guru
Marlo Gorelick
 
Lessons from the Trenches of Learning Game Design
Lessons from the Trenches of Learning Game DesignLessons from the Trenches of Learning Game Design
Lessons from the Trenches of Learning Game Design
Sharon Boller
 
Creative Engineering 101
Creative Engineering 101Creative Engineering 101
Creative Engineering 101
Deborah Mensah-Bonsu
 
Cohesive Game Experience
Cohesive Game ExperienceCohesive Game Experience
Cohesive Game Experience
DevGAMM Conference
 
Serco Usability Research, Ben Weedon, The challenge of measuring game play ex...
Serco Usability Research, Ben Weedon, The challenge of measuring game play ex...Serco Usability Research, Ben Weedon, The challenge of measuring game play ex...
Serco Usability Research, Ben Weedon, The challenge of measuring game play ex...
Use8.net
 
Digital Learning Game Design: Lessons from the Trenches
Digital Learning Game Design: Lessons from the TrenchesDigital Learning Game Design: Lessons from the Trenches
Digital Learning Game Design: Lessons from the Trenches
Sharon Boller
 
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
Marc Miquel
 
Comu346 lecture 7 - user evaluation
Comu346   lecture 7 - user evaluationComu346   lecture 7 - user evaluation
Comu346 lecture 7 - user evaluation
David Farrell
 
LAFS SVI Level 3 - Game Design and Analysis
LAFS SVI Level 3 - Game Design and AnalysisLAFS SVI Level 3 - Game Design and Analysis
LAFS SVI Level 3 - Game Design and Analysis
David Mullich
 
402 w2
402 w2402 w2
Finding The Fun
Finding The FunFinding The Fun
Finding The Fun
home, personal use
 
Figuring out the right metrics for your game
Figuring out the right metrics for your gameFiguring out the right metrics for your game
Figuring out the right metrics for your game
Saurav Sahu
 
Analytics - How and why you're making a mess of it and ruining your Analysts ...
Analytics - How and why you're making a mess of it and ruining your Analysts ...Analytics - How and why you're making a mess of it and ruining your Analysts ...
Analytics - How and why you're making a mess of it and ruining your Analysts ...
Dan Bergin-Holly
 

Similar to Steve Bromley - Running User Tests for VR Games (20)

LAFS Game Design 8 - Playtesting
LAFS Game Design 8 - PlaytestingLAFS Game Design 8 - Playtesting
LAFS Game Design 8 - Playtesting
 
User Experience 6: Qualitative Methods, Playtesting and Interviews
User Experience 6: Qualitative Methods, Playtesting and InterviewsUser Experience 6: Qualitative Methods, Playtesting and Interviews
User Experience 6: Qualitative Methods, Playtesting and Interviews
 
Making a game "Just Right" through testing and play balancing
Making a game "Just Right" through testing and play balancingMaking a game "Just Right" through testing and play balancing
Making a game "Just Right" through testing and play balancing
 
Intro to Games User Research Methods - March 2013
Intro to Games User Research Methods - March 2013Intro to Games User Research Methods - March 2013
Intro to Games User Research Methods - March 2013
 
Initial thoughts on live user tests for games
Initial thoughts on live user tests for gamesInitial thoughts on live user tests for games
Initial thoughts on live user tests for games
 
A Primer On Play: How to use Games for Learning and Results
A Primer On Play: How to use Games for Learning and ResultsA Primer On Play: How to use Games for Learning and Results
A Primer On Play: How to use Games for Learning and Results
 
Make Your Games Play, Teach, and Socialize Better: Usability & Playability Te...
Make Your Games Play, Teach, and Socialize Better: Usability & Playability Te...Make Your Games Play, Teach, and Socialize Better: Usability & Playability Te...
Make Your Games Play, Teach, and Socialize Better: Usability & Playability Te...
 
Primer on Play: Case Study for Knowledge Guru
Primer on Play: Case Study for Knowledge GuruPrimer on Play: Case Study for Knowledge Guru
Primer on Play: Case Study for Knowledge Guru
 
Lessons from the Trenches of Learning Game Design
Lessons from the Trenches of Learning Game DesignLessons from the Trenches of Learning Game Design
Lessons from the Trenches of Learning Game Design
 
Creative Engineering 101
Creative Engineering 101Creative Engineering 101
Creative Engineering 101
 
Cohesive Game Experience
Cohesive Game ExperienceCohesive Game Experience
Cohesive Game Experience
 
Serco Usability Research, Ben Weedon, The challenge of measuring game play ex...
Serco Usability Research, Ben Weedon, The challenge of measuring game play ex...Serco Usability Research, Ben Weedon, The challenge of measuring game play ex...
Serco Usability Research, Ben Weedon, The challenge of measuring game play ex...
 
Digital Learning Game Design: Lessons from the Trenches
Digital Learning Game Design: Lessons from the TrenchesDigital Learning Game Design: Lessons from the Trenches
Digital Learning Game Design: Lessons from the Trenches
 
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
 
Comu346 lecture 7 - user evaluation
Comu346   lecture 7 - user evaluationComu346   lecture 7 - user evaluation
Comu346 lecture 7 - user evaluation
 
LAFS SVI Level 3 - Game Design and Analysis
LAFS SVI Level 3 - Game Design and AnalysisLAFS SVI Level 3 - Game Design and Analysis
LAFS SVI Level 3 - Game Design and Analysis
 
402 w2
402 w2402 w2
402 w2
 
Finding The Fun
Finding The FunFinding The Fun
Finding The Fun
 
Figuring out the right metrics for your game
Figuring out the right metrics for your gameFiguring out the right metrics for your game
Figuring out the right metrics for your game
 
Analytics - How and why you're making a mess of it and ruining your Analysts ...
Analytics - How and why you're making a mess of it and ruining your Analysts ...Analytics - How and why you're making a mess of it and ruining your Analysts ...
Analytics - How and why you're making a mess of it and ruining your Analysts ...
 

More from uxbri

Jon Roobottom – Murder on the dancefloor: The death of disco
Jon Roobottom – Murder on the dancefloor: The death of disco Jon Roobottom – Murder on the dancefloor: The death of disco
Jon Roobottom – Murder on the dancefloor: The death of disco
uxbri
 
Olena Bulygina – Designing for Humanity: A UX Journey in Humanitarian Aid
Olena Bulygina – Designing for Humanity: A UX Journey in Humanitarian Aid Olena Bulygina – Designing for Humanity: A UX Journey in Humanitarian Aid
Olena Bulygina – Designing for Humanity: A UX Journey in Humanitarian Aid
uxbri
 
Paul Robert Lloyd – Time team: Documenting decisions and marking milestones
Paul Robert Lloyd – Time team: Documenting decisions and marking milestonesPaul Robert Lloyd – Time team: Documenting decisions and marking milestones
Paul Robert Lloyd – Time team: Documenting decisions and marking milestones
uxbri
 
cxpartners – Impact mapping: the Service Designer’s secret weapon
cxpartners – Impact mapping: the Service Designer’s secret weaponcxpartners – Impact mapping: the Service Designer’s secret weapon
cxpartners – Impact mapping: the Service Designer’s secret weapon
uxbri
 
Sharon Webster – Are you ready to mentor?
Sharon Webster – Are you ready to mentor?Sharon Webster – Are you ready to mentor?
Sharon Webster – Are you ready to mentor?
uxbri
 
Luke Hay – Making the Most of Mentoring
Luke Hay – Making the Most of MentoringLuke Hay – Making the Most of Mentoring
Luke Hay – Making the Most of Mentoring
uxbri
 
Lucy Spence - Trade-offs
Lucy Spence - Trade-offsLucy Spence - Trade-offs
Lucy Spence - Trade-offs
uxbri
 
Janna Bastow - Avoiding the Agency Trap
Janna Bastow - Avoiding the Agency TrapJanna Bastow - Avoiding the Agency Trap
Janna Bastow - Avoiding the Agency Trap
uxbri
 
Jonty Sharples - Arrogance & Confidence in ...Redux
Jonty Sharples - Arrogance & Confidence in ...Redux Jonty Sharples - Arrogance & Confidence in ...Redux
Jonty Sharples - Arrogance & Confidence in ...Redux
uxbri
 
Louise Bloom - T-shaped skills save lives (and products). How and why to lear...
Louise Bloom - T-shaped skills save lives (and products). How and why to lear...Louise Bloom - T-shaped skills save lives (and products). How and why to lear...
Louise Bloom - T-shaped skills save lives (and products). How and why to lear...
uxbri
 
Kate Flood - Speaking the same language
Kate Flood - Speaking the same languageKate Flood - Speaking the same language
Kate Flood - Speaking the same language
uxbri
 
Alison Rawlings - Is UX Strategy even a thing?
Alison Rawlings - Is UX Strategy even a thing?Alison Rawlings - Is UX Strategy even a thing?
Alison Rawlings - Is UX Strategy even a thing?
uxbri
 
Jonathan Smare - Leading culture change to increase customer centricity
Jonathan Smare - Leading culture change to increase customer centricityJonathan Smare - Leading culture change to increase customer centricity
Jonathan Smare - Leading culture change to increase customer centricity
uxbri
 
Matt LeMay - YOU DON'T "GET" ANYONE TO DO ANYTHING
Matt LeMay - YOU DON'T "GET" ANYONE TO DO ANYTHINGMatt LeMay - YOU DON'T "GET" ANYONE TO DO ANYTHING
Matt LeMay - YOU DON'T "GET" ANYONE TO DO ANYTHING
uxbri
 
Shrut Kirti Saksena - Holy Shift! Learnings in UX Research over the years
Shrut Kirti Saksena - Holy Shift! Learnings in UX Research over the yearsShrut Kirti Saksena - Holy Shift! Learnings in UX Research over the years
Shrut Kirti Saksena - Holy Shift! Learnings in UX Research over the years
uxbri
 
Lewis Nyman - Building effective mentoring relationships
Lewis Nyman  - Building effective mentoring relationshipsLewis Nyman  - Building effective mentoring relationships
Lewis Nyman - Building effective mentoring relationships
uxbri
 
Peter Winchester - Growing your career with (or without) a mentor
Peter Winchester - Growing your career with (or without) a mentorPeter Winchester - Growing your career with (or without) a mentor
Peter Winchester - Growing your career with (or without) a mentor
uxbri
 
Jessica Squires - Starting your mentorship journey, common ground and self-care
Jessica Squires - Starting your mentorship journey, common ground and self-careJessica Squires - Starting your mentorship journey, common ground and self-care
Jessica Squires - Starting your mentorship journey, common ground and self-care
uxbri
 
Martyn Reding - Connecting the Dots
Martyn Reding - Connecting the DotsMartyn Reding - Connecting the Dots
Martyn Reding - Connecting the Dots
uxbri
 
Nat Buckley - Good design is a team effort
Nat Buckley - Good design is a team effortNat Buckley - Good design is a team effort
Nat Buckley - Good design is a team effort
uxbri
 

More from uxbri (20)

Jon Roobottom – Murder on the dancefloor: The death of disco
Jon Roobottom – Murder on the dancefloor: The death of disco Jon Roobottom – Murder on the dancefloor: The death of disco
Jon Roobottom – Murder on the dancefloor: The death of disco
 
Olena Bulygina – Designing for Humanity: A UX Journey in Humanitarian Aid
Olena Bulygina – Designing for Humanity: A UX Journey in Humanitarian Aid Olena Bulygina – Designing for Humanity: A UX Journey in Humanitarian Aid
Olena Bulygina – Designing for Humanity: A UX Journey in Humanitarian Aid
 
Paul Robert Lloyd – Time team: Documenting decisions and marking milestones
Paul Robert Lloyd – Time team: Documenting decisions and marking milestonesPaul Robert Lloyd – Time team: Documenting decisions and marking milestones
Paul Robert Lloyd – Time team: Documenting decisions and marking milestones
 
cxpartners – Impact mapping: the Service Designer’s secret weapon
cxpartners – Impact mapping: the Service Designer’s secret weaponcxpartners – Impact mapping: the Service Designer’s secret weapon
cxpartners – Impact mapping: the Service Designer’s secret weapon
 
Sharon Webster – Are you ready to mentor?
Sharon Webster – Are you ready to mentor?Sharon Webster – Are you ready to mentor?
Sharon Webster – Are you ready to mentor?
 
Luke Hay – Making the Most of Mentoring
Luke Hay – Making the Most of MentoringLuke Hay – Making the Most of Mentoring
Luke Hay – Making the Most of Mentoring
 
Lucy Spence - Trade-offs
Lucy Spence - Trade-offsLucy Spence - Trade-offs
Lucy Spence - Trade-offs
 
Janna Bastow - Avoiding the Agency Trap
Janna Bastow - Avoiding the Agency TrapJanna Bastow - Avoiding the Agency Trap
Janna Bastow - Avoiding the Agency Trap
 
Jonty Sharples - Arrogance & Confidence in ...Redux
Jonty Sharples - Arrogance & Confidence in ...Redux Jonty Sharples - Arrogance & Confidence in ...Redux
Jonty Sharples - Arrogance & Confidence in ...Redux
 
Louise Bloom - T-shaped skills save lives (and products). How and why to lear...
Louise Bloom - T-shaped skills save lives (and products). How and why to lear...Louise Bloom - T-shaped skills save lives (and products). How and why to lear...
Louise Bloom - T-shaped skills save lives (and products). How and why to lear...
 
Kate Flood - Speaking the same language
Kate Flood - Speaking the same languageKate Flood - Speaking the same language
Kate Flood - Speaking the same language
 
Alison Rawlings - Is UX Strategy even a thing?
Alison Rawlings - Is UX Strategy even a thing?Alison Rawlings - Is UX Strategy even a thing?
Alison Rawlings - Is UX Strategy even a thing?
 
Jonathan Smare - Leading culture change to increase customer centricity
Jonathan Smare - Leading culture change to increase customer centricityJonathan Smare - Leading culture change to increase customer centricity
Jonathan Smare - Leading culture change to increase customer centricity
 
Matt LeMay - YOU DON'T "GET" ANYONE TO DO ANYTHING
Matt LeMay - YOU DON'T "GET" ANYONE TO DO ANYTHINGMatt LeMay - YOU DON'T "GET" ANYONE TO DO ANYTHING
Matt LeMay - YOU DON'T "GET" ANYONE TO DO ANYTHING
 
Shrut Kirti Saksena - Holy Shift! Learnings in UX Research over the years
Shrut Kirti Saksena - Holy Shift! Learnings in UX Research over the yearsShrut Kirti Saksena - Holy Shift! Learnings in UX Research over the years
Shrut Kirti Saksena - Holy Shift! Learnings in UX Research over the years
 
Lewis Nyman - Building effective mentoring relationships
Lewis Nyman  - Building effective mentoring relationshipsLewis Nyman  - Building effective mentoring relationships
Lewis Nyman - Building effective mentoring relationships
 
Peter Winchester - Growing your career with (or without) a mentor
Peter Winchester - Growing your career with (or without) a mentorPeter Winchester - Growing your career with (or without) a mentor
Peter Winchester - Growing your career with (or without) a mentor
 
Jessica Squires - Starting your mentorship journey, common ground and self-care
Jessica Squires - Starting your mentorship journey, common ground and self-careJessica Squires - Starting your mentorship journey, common ground and self-care
Jessica Squires - Starting your mentorship journey, common ground and self-care
 
Martyn Reding - Connecting the Dots
Martyn Reding - Connecting the DotsMartyn Reding - Connecting the Dots
Martyn Reding - Connecting the Dots
 
Nat Buckley - Good design is a team effort
Nat Buckley - Good design is a team effortNat Buckley - Good design is a team effort
Nat Buckley - Good design is a team effort
 

Recently uploaded

Fear and Faith (Slideshow by: Kal-el Go)
Fear and Faith (Slideshow by: Kal-el Go)Fear and Faith (Slideshow by: Kal-el Go)
Fear and Faith (Slideshow by: Kal-el Go)
Kal-el Shows
 
The Wise Way (Slideshow by: Kal-el Marcus)
The Wise Way (Slideshow by: Kal-el Marcus)The Wise Way (Slideshow by: Kal-el Marcus)
The Wise Way (Slideshow by: Kal-el Marcus)
Kal-el Shows
 
Enemy Encounter Standards for Hogwarts Legacy
Enemy Encounter Standards for Hogwarts LegacyEnemy Encounter Standards for Hogwarts Legacy
Enemy Encounter Standards for Hogwarts Legacy
AidanOKeefe2
 
Mixer design with example is given in this ppt
Mixer design with example is given in this pptMixer design with example is given in this ppt
Mixer design with example is given in this ppt
ssuserbaba88
 
2024_summer_my_dream_from_users_gnjp_20240708
2024_summer_my_dream_from_users_gnjp_202407082024_summer_my_dream_from_users_gnjp_20240708
2024_summer_my_dream_from_users_gnjp_20240708
kousato1
 
With Fear For Our Democracy I Dissent Shirt
With Fear For Our Democracy I Dissent ShirtWith Fear For Our Democracy I Dissent Shirt
With Fear For Our Democracy I Dissent Shirt
TeeFusion
 
Introduction to Vertical Cities Vertical cities are high-density, mixed-use ...
Introduction to Vertical Cities  Vertical cities are high-density, mixed-use ...Introduction to Vertical Cities  Vertical cities are high-density, mixed-use ...
Introduction to Vertical Cities Vertical cities are high-density, mixed-use ...
Mostafa Abd Elrahman
 
Girls Call Jogeshwari 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Jogeshwari 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9910780858 Provide Best And Top Girl Service And No1 in...
77sayre
 
Chinese Girls Call Indore 000XX00000 Provide Best And Top Girl Service And No...
Chinese Girls Call Indore 000XX00000 Provide Best And Top Girl Service And No...Chinese Girls Call Indore 000XX00000 Provide Best And Top Girl Service And No...
Chinese Girls Call Indore 000XX00000 Provide Best And Top Girl Service And No...
monikaservice00
 
The Stone (Slideshow by: Kal-el Marcus Renne Go)
The Stone (Slideshow by: Kal-el Marcus Renne Go)The Stone (Slideshow by: Kal-el Marcus Renne Go)
The Stone (Slideshow by: Kal-el Marcus Renne Go)
Kal-el Shows
 
A Journey Through Islamic Architecture.pdf
A Journey Through Islamic Architecture.pdfA Journey Through Islamic Architecture.pdf
A Journey Through Islamic Architecture.pdf
Mostafa Abd Elrahman
 
Unit-1 OOMD- Inthhro- class modeling.ppt
Unit-1 OOMD- Inthhro- class modeling.pptUnit-1 OOMD- Inthhro- class modeling.ppt
Unit-1 OOMD- Inthhro- class modeling.ppt
ChiragSuresh
 
Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Pune 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No1 in City
sexytaniya455
 
ITR Filing for the year of the 2023-24 .pdf
ITR Filing for the year of the 2023-24 .pdfITR Filing for the year of the 2023-24 .pdf
ITR Filing for the year of the 2023-24 .pdf
shyamraj39
 
SS26 Men's Fashion Trend Book Peclers Paris
SS26 Men's Fashion Trend Book Peclers ParisSS26 Men's Fashion Trend Book Peclers Paris
SS26 Men's Fashion Trend Book Peclers Paris
Peclers Paris
 
Introduction of Stone Display Stand.pptx
Introduction of Stone Display Stand.pptxIntroduction of Stone Display Stand.pptx
Introduction of Stone Display Stand.pptx
Xiamen Tsianfan Industrial & Trading Co.,Ltd.
 
A Green City is an urban area that prioritizes sustainability
A Green City is an urban area that prioritizes sustainabilityA Green City is an urban area that prioritizes sustainability
A Green City is an urban area that prioritizes sustainability
Mostafa Abd Elrahman
 
Pranay Mhatre Introduction cum Portfolio
Pranay Mhatre Introduction cum PortfolioPranay Mhatre Introduction cum Portfolio
Pranay Mhatre Introduction cum Portfolio
Pranay Мhatre
 
bedsores20-9-14-150627122143-lva1-app6892.pptx
bedsores20-9-14-150627122143-lva1-app6892.pptxbedsores20-9-14-150627122143-lva1-app6892.pptx
bedsores20-9-14-150627122143-lva1-app6892.pptx
gameransh001
 
Overview to the Oracle Fixed Assets Module
Overview to the Oracle Fixed Assets ModuleOverview to the Oracle Fixed Assets Module
Overview to the Oracle Fixed Assets Module
lakshmichitra031
 

Recently uploaded (20)

Fear and Faith (Slideshow by: Kal-el Go)
Fear and Faith (Slideshow by: Kal-el Go)Fear and Faith (Slideshow by: Kal-el Go)
Fear and Faith (Slideshow by: Kal-el Go)
 
The Wise Way (Slideshow by: Kal-el Marcus)
The Wise Way (Slideshow by: Kal-el Marcus)The Wise Way (Slideshow by: Kal-el Marcus)
The Wise Way (Slideshow by: Kal-el Marcus)
 
Enemy Encounter Standards for Hogwarts Legacy
Enemy Encounter Standards for Hogwarts LegacyEnemy Encounter Standards for Hogwarts Legacy
Enemy Encounter Standards for Hogwarts Legacy
 
Mixer design with example is given in this ppt
Mixer design with example is given in this pptMixer design with example is given in this ppt
Mixer design with example is given in this ppt
 
2024_summer_my_dream_from_users_gnjp_20240708
2024_summer_my_dream_from_users_gnjp_202407082024_summer_my_dream_from_users_gnjp_20240708
2024_summer_my_dream_from_users_gnjp_20240708
 
With Fear For Our Democracy I Dissent Shirt
With Fear For Our Democracy I Dissent ShirtWith Fear For Our Democracy I Dissent Shirt
With Fear For Our Democracy I Dissent Shirt
 
Introduction to Vertical Cities Vertical cities are high-density, mixed-use ...
Introduction to Vertical Cities  Vertical cities are high-density, mixed-use ...Introduction to Vertical Cities  Vertical cities are high-density, mixed-use ...
Introduction to Vertical Cities Vertical cities are high-density, mixed-use ...
 
Girls Call Jogeshwari 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Jogeshwari 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9910780858 Provide Best And Top Girl Service And No1 in...
 
Chinese Girls Call Indore 000XX00000 Provide Best And Top Girl Service And No...
Chinese Girls Call Indore 000XX00000 Provide Best And Top Girl Service And No...Chinese Girls Call Indore 000XX00000 Provide Best And Top Girl Service And No...
Chinese Girls Call Indore 000XX00000 Provide Best And Top Girl Service And No...
 
The Stone (Slideshow by: Kal-el Marcus Renne Go)
The Stone (Slideshow by: Kal-el Marcus Renne Go)The Stone (Slideshow by: Kal-el Marcus Renne Go)
The Stone (Slideshow by: Kal-el Marcus Renne Go)
 
A Journey Through Islamic Architecture.pdf
A Journey Through Islamic Architecture.pdfA Journey Through Islamic Architecture.pdf
A Journey Through Islamic Architecture.pdf
 
Unit-1 OOMD- Inthhro- class modeling.ppt
Unit-1 OOMD- Inthhro- class modeling.pptUnit-1 OOMD- Inthhro- class modeling.ppt
Unit-1 OOMD- Inthhro- class modeling.ppt
 
Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Pune 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No1 in City
 
ITR Filing for the year of the 2023-24 .pdf
ITR Filing for the year of the 2023-24 .pdfITR Filing for the year of the 2023-24 .pdf
ITR Filing for the year of the 2023-24 .pdf
 
SS26 Men's Fashion Trend Book Peclers Paris
SS26 Men's Fashion Trend Book Peclers ParisSS26 Men's Fashion Trend Book Peclers Paris
SS26 Men's Fashion Trend Book Peclers Paris
 
Introduction of Stone Display Stand.pptx
Introduction of Stone Display Stand.pptxIntroduction of Stone Display Stand.pptx
Introduction of Stone Display Stand.pptx
 
A Green City is an urban area that prioritizes sustainability
A Green City is an urban area that prioritizes sustainabilityA Green City is an urban area that prioritizes sustainability
A Green City is an urban area that prioritizes sustainability
 
Pranay Mhatre Introduction cum Portfolio
Pranay Mhatre Introduction cum PortfolioPranay Mhatre Introduction cum Portfolio
Pranay Mhatre Introduction cum Portfolio
 
bedsores20-9-14-150627122143-lva1-app6892.pptx
bedsores20-9-14-150627122143-lva1-app6892.pptxbedsores20-9-14-150627122143-lva1-app6892.pptx
bedsores20-9-14-150627122143-lva1-app6892.pptx
 
Overview to the Oracle Fixed Assets Module
Overview to the Oracle Fixed Assets ModuleOverview to the Oracle Fixed Assets Module
Overview to the Oracle Fixed Assets Module
 

Steve Bromley - Running User Tests for VR Games

Editor's Notes

  1. Hi everyone, I am Today, I wanted to share some of the things that I've learned running 1:1 user tests in VR
  2. This is my fifth year at Sony PlayStation, based in their Euopean HQ in London During that time I’ve worked on their system software, on hardware and on games made both by studios owned by PlayStation, and studios they’ve partnered with. I’ve been working on VR for the last 2 years, and during that time have run a lot of user research including ….
  3. So, from all of this research, I wanted to share some of the tips that I’ve learned along the way, in particular about how to get good quality results from qualitative research Some parts may seem obvious, but you don't want to waste participants discovering these things yourself, so hopefully it’ll help you get more useful results quicker.
  4. The first area I wanted to talk about was how to set up the room for VR, which needs some extra thought about it compared to ‘traditional’ games user research.
  5. Normally the set-up of the room isn’t particularly important. This is a picture of our social room, a set up that most games user research labs also have. It has a TV, some couches, and is just meant to represent a typical living room set-up. As long as you have the participant, and a TV they can play on, that’s good enough to get good quality usability feedback from.
  6. However getting the room right is much more important for VR tests, because getting it wrong can cause plenty of issues with the game. The first thing to think about is where to sit the player. Both Oculus and PSVR rely on line of sight between the camera and the headset – they use infrared so that the game can track the player’s position, and allow them to move their head and body forward/back in the virtual environment, something not possible on stand-alone systems like GearVR. Because of this, there’s an optimal place for the player to sit. If they are too far forward, or too far back, the system won’t be able to see them, and tracking will fail causing other issues in the game. Be especially aware if the game will require movement – some of the demos that PlayStation have publically shown require players to move between sitting and standing – so make sure that the player is sat somewhere the camera can see them both sitting and standing. If the player is in a poor position, you’ll have plenty of potential usability issues – e.g. whether players recognise that the tracking is failing, and whether they know how to deal with it. However this is a system level feature, and so not the responsibility of the game you are testing – so development teams who have put the time and expense into organising the test will not be particularly happy if this occurs during their test, and obscures other potential usability issues in the game. So, before starting, ensure the player is in the optimal position to play the game – which can be particularly an issue in multi-seat setups where many playtests are run simultaneously in the same room… but more of that soon.
  7. Another common issue with usability labs is that they often have 1 way mirrors. Returning to the test we saw earlier ,we can see from the other angle that there is a one-way mirror in the room, which allows a notetaker (and members of the development team) to observe the session. This is usually great – it gives a clear view of the context of the issue, and inviting the game team to come and see the session live allows the development team to come and engage with the session – paying more attention, and understanding the issues greater than watching later on a video. However because many VR headsets use the IR sensors, the one way mirror can be a potential issue. It’ll cause a duplication of the IR sources for the camera, which can cause the quality of positional tracking to degrade, again introducing interference and reducing the quality of the experience, which can impact usability. Luckily there’s an easy solution to this. On amazon they sell blackout blinds, which are bits of plastic that stick, like clingfilm, to glass. This takes seconds to attach and can blackout the window preventing the reflection of the IR lights. The development team will once again have to rely on the camera feed, but the players experience is more authentic.
  8. Another thing to be aware of is swivelling chairs. These are very common in an office environment, and so can be found in some playtest labs. However they are much less common in the home environment, particularly with home consoles, so are unrepreseitive. The issue with swivel chairs is that they cause players to lose their orientation, and ‘forget’ which way is forward (or which way the camera is). This can cause usability issues, but again they are unrealistic because players would be on a sofa at home, where the forward direction is much clearer just from the feel of it. They also have wheels on, which can cause them to move, and be a safety issue when working with players who cannot see!
  9. The last thing to mention is in regards to room scale VR. HTC Vive is built to allow players to walk around each experience physically. Oculus and PSVR have also both shown equivalent demos that allow players to physically move in a space. For research on room scale VR – or anything that will encourage the players to move around, you want to keep the room free of obstacles (because they are walking around blindfolded). Also remember not to move things – if you move a controller, or more importantly a chair, players will not be aware that it’s moved, which can cause safety issues.
  10. So, to summarise this section on the room setup – extra care needs to be taken to ensure that the room setup isn’t going to cause ‘unfair’ usability issues that are out of scope for your current research. Also remember to keep the user safe – don’t move things or let them walk into stuff!
  11. The next thing I want to talk about Is methodology – how the test needs to be adapted to run good quality user research sessions.
  12. So, my favourite type of user research is 1:1 qualitative sessions. We usually have a moderator in the room with the participant, who can help ensure the game is running correctly and interview them at the end. We also have a note-taker behind the glass who can observe the usability issues, and write down players behaviours and what they say. This all works fine for VR tests. One of my favourite parts of these sessions is probing players to understanding why players are acting in a specific way. For example, you can observe that the player walked straight past the next objsctive. But without probing, you cannot tell why – did they not see it? Did they want to collect something else first? Did they think they *needed* to collect something else first? The challenge is asking questions in the right way so that players are not lead to give the answer that they think you expect, so it’s like a puzzle, which is why I enjoy it. However, with VR there’s a problem wiith this…
  13. You can’t talk to the players as they play. With the headset on, they can’t see you. These headsets also have 3D audio (so they can tell the direction of sounds), which requires headphones to work. Meaning they can’t hear you either. This means that asking probing questions isn’t easy. There’s a couple of ways to address this. You can make sure your study design doesn’t rely on interruptions. Short play sessions to ask them questions after may be one way of doing this. You can watch the video back later with them, and ask them what their thoughts were. However this doubles the session length (which is bad in fast moving industries like game dev where teams start working on fixes immediately after seeing the issues). It also encourages players to attempt to justify their interactions by guessing, or trying to reason after the fact, which studies have shown humans are particularly bar at. One of the most encouraging things we’ve tried is to have the moderator talk into the player’s headphones. Instead of the audio signal going straight from the VR headset to the headphones, we’ve routed it through a mixing desk, which allows us to add other audio sources – such as a moderator microphone. This isn’t perfect as without visual cues players can be surprised by you talking to them, but with some priming can be a good work-around. It also stops you having to tap players on the shoulder to get their attention, which will break precense, and can be a big surprise!
  14. Mult-seat playtests work fine for VR, with a few adaptions. Because they are typically based on survey data and telemetry rather than observation, it means very little diference to moderation. However the room needs to be ready for VR - each player will need more space than usual as the games encourage movement. And again, any one way mirrors will need to be covered. Also, because people are new to VR, they’ll need a lot more help getting setup. To help with this in multi-seat playtests, we often recruit extra temporary assistants, (a few of which are here today ) to help with orientating players. We’ll talk more about that in a minute though.
  15. So, as we’ve seen, some changes need to made to your study design to account for VR. Making sure that you work around not being able to communicate with your participants is the biggest area to focus on. And ensuring that your multi-seat playtest area has enough room to run quant studies.
  16. The next area I want to talk about is recruitment, and working with the participants for user research sessions.
  17. The first thing to remember is that VR is a new medium, and most people have no experience with it. This means that there’s a lot of things you’ll have to help them with, starting with how to get the headset on. It’s important to remember that people new to VR do not know what it should look like, and so will not be able to assess if it’s in properly/in focus etc. And because you can’t see what they are seeing, you can’t tell them. Because this is not the focus of your test, and will cause usability issues, it’s something that the moderator needs to deal with. There are a couple of things you can do to handle this. First, prepare a script to help them get the headset on. Make sure that it includes instructions on how to fit the headset, a way of assessing whether it’s on properly, and steps to resolve if the picture isn’t good. A good way of assessing if the picture is in focus is finding a menu with sharp text, and then asking them to tell you whether the text appears sharp, or blurry/doubled. As a test, its simple enough that most people can understand whether they’re seeing the correct thing or not, and then whether you need to help them further with the headset.
  18. Another thing to consider is recruitment. Because VR is a very new tech, this impacts recruitment. If you recruit based on intent to buy VR, that will have an effect on what demographics you get. So, looking for casual players who intend to buy VR will be unrealistic and make recruitment very difficult to find legitimate users. Although, you and your development team should probably consider why you want these users, as they will not be your customers. There are certain cases when casual players may be relevant for VR testing, for example social/party games, but it’s worth a second thought. The other thing to be aware of is if you’re recruiting based on people who currently own VR, that introduces a very heavy bias on who your participants are. As VR is a niche product, and up until recently only development versions of the hardware has been available, anyone who owns a VR headset, (and most people who have tried them) are a-typical – typically very tech savvy and unrepresentative of a typical consumer. If you’re developing for a device intended for a home audience, such as consoles or mobile, care should be taken to ensure that they are representative.
  19. The last thing to be aware of is potential comfort issues. On launch, oculus rated the comfort of each of their experiences – the likelihood that some people would feel a form of motion sickness. When running usability tests, you don’t want to have to stop the session early due to people feeling ill. So, for usability tests you want to screen before hand for predisposition, through factors like “have you ever felt motion sickness when travelling” and “have you ever felt motion sickness when performing extreme activities”. This will help minimise comfort issues when playing, and ensure that you don’t miss out on getting usability feedback.
  20. So, as we’ve seen with participants we’ll want to help people set up, recruit realistically, and minnimise the potential for comfort issues.
  21. The last topic I wanted to cover, is how to deal with opinion data. First, of all, let’s start with the basics…
  22. So, as we all know, Nielsen’s research has shown that 5-8 users per audience is a good number of users to be ensure you are discovering most participants, and is a good balance between making sure the results are comprehensive, but also that you are not running too many sessions and wasting the team’s money. This is great for usability findings.
  23. But again, as we know, this isn’t a good number for opinion data (whether people like things). Your test was designed to discover things, but doesn’t give you any indication of how representative the data is – for usability or opinion data. If one person out of 5 had a usability issue, that could mean that 2 to 64% of the final audience may encounter it. And opinions are so much more diverse that the opinion data will not even begin to be representative. Also the test is unfair for opinion data – players will have encountered usability issues, and in the past we’ve found that this will heavily influence their opinion – if you ask them what they like least about the game, they’ll probably start talking about usability issues they encountered.
  24. Knowing all of that, there is always the temptation to ask opinion questions anyway in usability tests. Despite explaining that the data isn’t particularly actionable, teams still usually want to hear it - so it’s very tempting to be naughty and include those questions. This is usually a waste of time for VR games. People have no conception of what VR is like, and so there top opinion is almost always "its immersive" "i imagined it would be a big TV, but it's actuallya being inside the game“ If you’re even naughtier and ask for ratings, players will rate VR experiences much higher than equivalent non-VR experiences. However all off these opinions are opinions about the technology, not the game itself, and so are not particularly relevant, or useful, to the game.
  25. There are a couple of things that might help get better quality opinion data from qualitative sessions. The first thing to try would be a palate cleanser – let everyone try an initial VR experience before they play the real game, to get them over the “wow” factor, and bring them onto assessing the individual merits of this specific game. However this wastes a lot of time – it’ll add at least 30 minutes to each session. And as we know, this opinion data is unactionable anyhow, so it doesn’t make sense to extend the session to collect it. You could also look at hiring expert users – people who have used VR before and are familiar before. However again, as we’ve seen earlier this will add a huge bias to the recruitment. That may be fine, depending on what you’re trying to learn from your test, but it’s important to be aware of for caveating any findings.
  26. So, my recommendation – and best practise for all usability tests really – is to resist the temptation and ignore the opinion data. It’ll be unreliable and not worth compromising your study design to address. If you’re after reliable opinion data, plan a different test for that.
  27. So, here’s some of the things I hope I’ve covered about preparing to run tests in VR.