1.
A Practical Guide to
Doing Ethical Player
Testing
Mia Consalvo
Concordia University

2.
Who am I?
● Studying game players since 2000
● Use a variety of methods to better
understand play
● Wrote Cheating, Players and their Pets,
Atari to Zelda
● Run the mLab at Concordia University

3.
Outline for today’s talk
● A brief history of why ethical research became a
thing at universities
● Important concepts to keep in mind for testing
● Questions to ask yourself when prepping for
testing
● What about VR?
● Q&A

4.
What this talk is NOT about
● Improving your game design through
player testing

5.
What this talk IS about
● Ensuring play testers are getting the best
possible experience (and want to come
back again)
● Improving the quality of testing feedback

6.
A caveat
● “Following a code of ethics is not the
same as being ethical. A domain-specific
ethics code … can never function as a
substitute for ethical reasoning itself.”

7.
A brief history of human subjects research

8.
The Milgram experiment (1961)
● Studied obedience to authority figures via
willingness to give increasingly severe
‘shocks’ to others
● 65% of participants administered final
massive 450-volt shock
● Study criticized for use of deception,
other flaws

9.
Stanford Prison Experiment (1971)
● A study of the psychological effects of
becoming a prisoner or prison guard
● Conducted by Philip Zimbardo with a
group of college students
● Prisoners ‘arrested’ at their homes, driven
to prison, processed, given jumpsuit and
assigned an ID number

10.
Stanford Prison Experiment
● Experiment abandoned after six days –
some prisoners had breakdowns
● Subjects could not leave voluntarily
● Zimbardo was not a neutral observer
● No debriefing was done to assess
potential harms done

11.
The introduction of
protocols and guidelines
for doing federally funded
research

12.
Consent

13.
What is consent?
● Free, informed and ongoing consent
● Free = voluntary

14.
Voluntary Consent
● No Undue Influence or Coercion
● Is there a power relationship between the
Investigator and potential participants?
● Is the investigator relying on the trust or
dependency of particular people (ex: my
students)?
● Threat of harm or punishment for failure to
participate?

15.
Informed Consent
● Participants know what is expected of
them during the test
● How long the test will last
● How will they be observed/recorded
● It’s okay to change their mind

16.
Informed Consent
● Told of foreseeable risks and potential
benefits
● Under no obligation to participate, free to
withdraw at any time
● Whether participants will be identified
directly or indirectly

17.
Informed Consent
● Contact information if they have
questions after the fact
● Information about how data will be used,
who it (may be/will be) disclosed to
● Information about payments or incentives
● Time to consider all of this, ask questions

18.
Ongoing Consent
● Consent is not a ‘once and you’re good’
decision
● Consent can be negotiated if there are
different aspects to the play test

19.
Withdrawing Consent
● Testers should be told they can stop the
test at any time
● The environment should support this
possibility
● There should be multiple ways to
withdraw (verbal, nonverbal/written)

20.
Using incentives to participate
● Incentives can attract more potential
participants
● If incentives are very large or valuable this
may encourage participants to disregard risks
● Economic circumstances of participant pool,
age and capacity, customs and practices

21.
Use of deception
● Deception can be a part of ethical
research
● What is the risk involved in the
deception?
● Does the benefit of deception outweigh
potential risks?

22.
Harm and Risk

23.
Defining Harm and Risk
● Minimal risk: probability and magnitude
of possible harms implied by participation
in the study is no greater than those
encountered by participants in those
aspects of their everyday life that relate
to the research

24.
Beyond minimal risk

25.
Vulnerable Populations
● Children
● Diminished capacity
● Indigenous peoples

26.
Children
● Consent usually obtained through parents/
guardians
● Consider the age range you are including and
why
● Children who are 7, 10 and 13 are very different
in terms of what they understand as risks, as
feedback, as consent

27.
Children
● During testing, nonverbal communication
can be more important than what is being
said (for both testers and researchers)

28.
Children
● For a detailed examination of kids and
play testing, check out Gareth Griffiths’
2014 GDC talk “Child’s Play: Playtesting
with Children in the World of Skylanders”
● http://www.gdcvault.com/play/1020348/
Child-s-Play-Playtesting-with

29.
Privacy and Confidentiality

30.
Privacy and Confidentiality
● How are you collecting data?
● How are you storing data?
● How are participants identified?
● What are you doing with the data?
● What happens to the data when the study
is over?

31.
Privacy and Confidentiality
● Levels of disclosure of participant identity
● Anonymous
● Pseudonyms/Confidential
● Identifiable
● What personal information are you collecting and how
might it be linked to participants’ tests?

32.
Special Considerations for
VR

33.
“VR poses risks that are novel, that go
beyond the risks of traditional
psychological experiments in isolated
environments & go beyond risks of
existing media technology for the
general public” –Madary & Metzinger,
2016

34.
“the virtual pit”
● Subject with HMD stands on ‘ledge’ and asked to
lean over and drop a beanbag into a deep pit
● Subject stands on wooden platform 1.5” from
the ground
● Showed increased signs of stress through
increases in heart rate, skin conductance

35.
Remember the Milgram Experiment?
● Re-done in VR environment
● Asked participants to administer shocks
to a virtual human performing memory
tests [they knew she was virtual]
--Slater M, Antley A, Davison A, Swapp D, Guger C, et al. (2006) A Virtual
Reprise of the Stanley Milgram Obedience Experiments. PLoS ONE 1(1):
e39. doi:10.1371/journal.pone.0000039

36.
Virtual Reprise
● 2 participants emphasized correct answers while
reading words ‘in an attempt to help her’
● 8 repeated the question after receiving no
response
● Voices of some participants showed increasing
frustration with wrong answers

37.
Virtual Reprise
● Often behaved in a way that only made sense if
they were responding to the virtual character as
if she were real
● Humans respond realistically at subjective,
physiological, and behavioral levels in
interactions with virtual characters
notwithstanding their cognitive certainty that
they are not real

38.
Virtual Reprise
● Might be continued evidence that subjects will
be obedient to an authority figure
● May be a matter of participants being willing to
put up with their own discomfort for the sake of
honoring their agreement to be a participant in
the experiment

39.
Virtual Reprise
● Might be continued evidence that subjects will
be obedient to an authority figure
● May be a matter of participants being willing to
put up with their own discomfort for the sake of
honoring their agreement to be a participant in
the experiment

40.
Proteus Effect
● Subjects conform to the behavior that
they believe others would expect them to
have based on the appearance of their
avatar
● Behavior in VE can have lasting
psychological impact after subjects return
to the physical world

41.
Guidelines for VR research
● Do no harm
● No real history we can use as a source for
insight
● Tautology?

42.
Guidelines for VR Research
● Have an explicit statement (or explain) to
the effect that “immersive VR can have
lasting behavioral influences on subjects,
and some of these risks may presently be
unknown”

43.
Guidelines for VR Research
● Tell people that they may have powerful
emotional responses to game content
whether or not they ‘believe’ it’s real
● Torture in a virtual environment “is still
torture”

44.
Guidelines for VR Research
● Many new, additional kinds of data being
collected
● Eye-movements, emotions, real-time reactions, bodily
movements (mo-cap)
● One’s kinematics may be uniquely connected to one’s
identity
● Consider what data you really need to keep and
what might be erased to preserve privacy

45.
Broader considerations beyond testing
● We don’t know the psychological impact
of long term immersion in VR (addiction,
manipulation of agency, unnoticed
psychological change, mental illness)
● The potential for abuse of avatars that
look like their users

46.
Creating a protocol for play
testing – Questions to guide
you

47.
What am I testing for?
● Do players understand what to do in the
first level?
● Do my instructions make sense or are
players floundering around?
● How long do players take to get to the
end of the level?

48.
Recruitment
● Who is my player base? How can I recruit
players from that group without
exploiting trust/power relationship?
● Should I provide an incentive or reward
for play testing?
● Where am I asking them to play test?

49.
Informed Consent
● How will I handle obtaining consent?
● What do I say about withdrawing
consent?
● How can I ensure testers feel comfortable
enough to leave if they really want to quit
testing?

50.
Deception
● Is there anything in the game I want to be a
surprise to testers?
● Is there a way to let testers know there’s some
material they might be really bothered by
without spoiling the content for everyone?
● Is this ‘surprise’ really worth it?

51.
Harm and Risk – VR Edition
● Having a ‘kill switch’
● Testers’ avatars have protective bubble
● Protected populations issues
● Find information or agencies that testers might
want to contact if they are bothered by content
in your game

52.
Privacy and Confidentiality
● What should I record – gameplay, tester audio,
tester video, physiological responses?
● If I share data with anyone, am I keeping tester
identities anonymous or confidential?
● How am I storing data and keeping it secure?
● What am I doing with that data after testing is
done?

53.
Questions for multiplayer testing
● Do players understand how they can interact
with other players?
● Do instructions for that make sense?
● Are players using anti-harassment tools we
created?
● What sorts of toxicity (if any) am I seeing during
play?

54.
Harm and risk – an aside
● Be wary of defining what ‘harassment’
looks like/sounds like
● Ask what they might have seen done/said
about other players, not just this player
● ‘Abuse’ versus ‘Drama’

55.
Putting it all together –
Consent Form Language/
Guidelines for testers to
understand and agree to

56.
Example Consent Form Text
● “I understand that I have been asked to
participate in play testing of Eksa: Isle of
the Wisekind by Mia Consalvo of GAMBIT
Studios, contact information HERE.”

57.
Example Consent Form Text
● “I have been informed that the purpose
of the play testing is to determine [how
clear directions are in the game, and how
challenging puzzles are in the first few
levels.]”

58.
Example Consent Form Text
● “I understand that I am being asked to
play a game that isn’t finished, for about
30 minutes.”
● “I understand that the computer is video
recording my gameplay and I am being
observed by someone while I play, who is
taking written notes.”

59.
Example Consent Form Text
● “I understand that I am free to withdraw
my consent and discontinue my
participation at any time without negative
consequences.”

60.
Example Consent Form Text
● “There might be certain risks in
participating in this play test. These risks
include feeling frustration or anger while
playing, or hearing other players saying
negative things during gameplay which
might upset me.”

61.
Example Consent Form Text
● “I understand that my participation in this
study is…
● CONFIDENTIAL (the researcher will know, but
will not disclose my identity)
● NON-CONFIDENTIAL (my identity will be
revealed in study results)”

62.
Example Consent Form Text
● “I have carefully studied the above and
understand this agreement. I freely
consent and voluntarily agree to
participate in this study.”

63.
Additional Resources
● For US-based developers
● https://phrp.nihtraining.com/index.php
● The Belmont Report
● http://www.hhs.gov/ohrp/regulations-and-policy/belmont-
report/index.html

64.
Additional Resources
● For Canadian-based developers
● http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/
tcps2-eptc2/Default/
● For European-based developers
● http://www.eurecnet.org/index.html

65.
Additional Resources
● Madary M and Metzinger TK (2016) Real
Virtuality: A Code of Ethical Conduct.
Recommendations for Good Scientific
Practice and the Consumers of VR-
Technology. Frontiers in Robotics and AI.
3:3. doi: 10.3389/frobt.2016.00003

66.
Questions?