Slides from a talk that I gave for the Boston CHI meeting on April 9, 2013. Moderating usability studies and interviewing research participants is more than just reading questions off of a page - it's about having a conversation. In this talk I discuss why it's important to have a conversation with participants and how to do so in a way that encourages them to talk honestly. I also discuss how to maintain neutrality and minimize introducing bias into the conversation.

1. Boston
CHI
Monthly
Mee>ng
STRENGTHENING
DESIGN
RESEARCH
MODERATION
SKILLS
APRIL
9,
2013
Susan
Mercer
Senior
Experience
Researcher
smercer@madpow.com
@susanamercer

2. Strengthening Moderating and Interviewing Skills
§ Once I learned the basics
§ And ran a couple of studies,
§ Now what?
§ How do I strengthen my skills?
2

3. Story Time
Don’t play the notes on the page…
…play the music
This applies to moderating and
interviewing too.
Don’t just read the questions from the page…
…have a conversation
3

4. The Art of Moderating and Interviewing
§ Not just reading questions
§ It’s also…
§ Building rapport with the participant
§ Creating a smooth conversation
§ Encouraging the participant to talk
§ Remaining objective
§ Listening effectively
§ Being human
4

5. What Is Our Goal?
§ To elicit honest thoughts from participants in an unbiased manner.
§ “Would you like to try one of my chocolate chip oyster and herring
brownies? They’re really good!”
§ Sometimes it’s hard to be fully honest
5

6. The Rules of Politeness
Linguistic anthropologists have studied this phenomenon
The Rules of Politeness 1 “Human
beings
are
always
balancing
the
1. Don’t Impose
paradoxical
fact
that
2. Give Options they
are
simultaneously
3. Be Friendly individuals
and
social
creatures.”
-­‐
Deborah
Tannen
Sources: 1 Lakoff, 1976; 2 Tannen, 2005
6

7. How Do We Get Beyond Politeness?
§ In other words, we’re often polite rather than honest to strangers.
§ As human beings we have a need to be socially accepted
Social Acceptance > Honesty
§ We guess what is expected and answer to avoid offending
§ It’s second nature – we’re often not aware we’re doing it
7

8. How Do We Get Beyond Politeness?
Build their trust
Politeness Trust
Familiarity
Strangers Good Friend
8

9. Building Trust in an Interview
1. Be trustworthy upfront
2. Build rapport
3. Engage in comfortable conversation
4. Stay neutral and accepting
9

10. 1. Be Trustworthy Upfront
§ Confidentiality
§ How are you recording what is said?
§ Who is listening?
§ What will you do with the information you collect?
§ Neutrality
§ You’re not emotionally involved in the design/project
§ There are no right or wrong answers
§ Your job is to get honest opinions
§ Consent Form
§ Put it all in writing – using their language

11. 2. Build Rapport
A
person's
name
is
§ Greeting a Participant to
that
person
the
§ Smile sweetest
and
most
§ Use their name important
sound
in
any
language.
§ Handshake / Warm phone greeting
-­‐
Dale
Carnegie
§ Make them feel important - sincerely
§ Small Talk – Find Common Ground
§ Safe topics: travel to office, traffic, weather
§ Avoid asking direct questions
§ Listen and look for shared experiences
Source: Carnegie, 1936.

12. 2. Build Rapport
§ Be Empathetic
§ Apologize if they had trouble finding the office
§ Show you understand their point of view
§ “Oh, it’s raining there? It is here too. I hate rainy days.”
§ Inject Some Humor
§ Joke about yourself
§ Joke about the situation
§ Don’t joke about them
§ Short version – Show them that you are human too

13. 3. Engage in Comfortable Conversation
Two schools of usability testing moderating techniques
§ Moderator keeps silent and says “Keep talking” 1
§ Awkward feeling for participant
§ Reminds them that they are in a “study”
§ Moderator creates conversation where participant is primary
speaker 2
§ Still accomplishes goals of gathering information
§ Meets the participant’s expectations of a socially acceptable conversation
§ Moderator can still be neutral and minimize bias
Sources: 1 Ericsson and Simon, 1980, 2 Boren and Raney, 2000.
13

14. 3. Engage in Comfortable Conversation
What is a Comfortable Conversation?
§ Conversational cues and turn-taking are expected
§ Acknowledgement tokens – “Uh huh”, etc.
§ Encourage the continuation of the other speaker’s talk
§ Usually implies that the other speaker’s prior talk is incomplete
§ Some indicate change of speakership
§ “Yeah” – more than half the time indicates changing turns in speaking
§ “Yeah. I heard that the other day and…”
§ “Oh!” - may indicate noticing something, then transitioning to another topic
§ “Oh! That reminded me…”

15. 3. Engage in Comfortable Conversation
§ Some may introduce bias
§ “Oh!”, “Interesting” – indicating unexpected answer
§ “Yes”, “Perfect”, “Great” – indicating agreement
§ “Hmmm.”, “Really?” – indicating disagreement
§ Notice that tone is key
§ Neutral is best
§ “Mhmm”, “Uh huh”, “Continue”, “Tell me more”, “OK”
§ “Mhmm” or “Uh huh” vs. silence à interviewees saying 31% more
phrases. 1
Source: 1 Matarazzo et. al., 1964

16. 4. Stay Neutral and Accepting
§ Ask Open-ended Questions
§ Start with Who, What, When, Where, Why, How?
§ Really Listen
§ Pay attention – stay in the moment
§ Look at the participant
§ Take notes if you can
§ Be quiet - give them time to say what they need to

17. 4. Stay Neutral and Accepting
§ Watch Your Reactions
§ Don’t show surprise
§ May make them think that they are giving a wrong answer
§ Don’t overly agree
§ May make them think that they are giving the right answer
§ Don’t be negative
§ Watch your tone – stay neutral and accepting
§ Try not to laugh

18. 4. Stay Neutral and Accepting
§ Be yourself
§ No one is perfectly neutral
§ Recover gracefully and move on
§ “Perfect” – “That’s the level of detailed feedback we’re looking for.”
§ “Interesting!” – “I haven’t heard that perspective yet, tell me more.”
§ (something surprising) – “I can understand that.”
§ Interject some Rapport-building comments when needed
§ Quiet or uncomfortable participants
§ “I hate it when that happens.”, “I can imagine that was challenging”, etc.
§ Again, showing that you are human like them

19. 5. Encourage them to talk
§ Be Quiet!
§ Most agreements happen immediately. Most people delay before
disagreeing.1
§ If you don’t respond to their answer, it encourages them to talk more
§ People often delay speaking before disagreeing – give them time
§ Some people are uncomfortable with silence, so they will keep talking
§ The best way to stay neutral J
Source: 1 Goodwin and Heritage, 1990.

20. Building Trust in an Interview
1. Be trustworthy upfront
2. Build rapport
3. Engage in comfortable conversation
4. Stay neutral and accepting
5. Be Quiet
20

21. KEEPING
YOUR
SKILLS
SHARP
21

22. 1. Identify your improvement areas
§ Watch your videos
§ Have colleagues give you feedback
22

23. 2. Learn From Others
§ Watch others moderate
§ Be a participant
§ Listen to talk radio interviews
23

24. 3. Practice These Skills in Everyday Life
§ Be quiet and listen in everyday conversations
§ Ask open-ended questions more frequently
24

25. The most important thing is…
PRACTICE!
25

26. References
Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional
Communication, 43 (3), 261-278.
Carnegie, D. (1936) How to win friends and influence people. Simon & Shuster.
Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.
Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.
Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3), 215-251.
Goodwin, C., & Heritage, J. (1990). “Conversation analysis.” Annual review of anthropology 19 (1990): 283-307.
Matarazzo, J.D.., Wiens, A. N., Saslow, G., Allen, B. V., & Weitman, M. (1064). Interviewer Mm-Hmm and interviewee
speech durations. Psychotherapy: Theory, Research & Practice, 1(3), 109.
Tannen, D. (2005). Conversational Style: Analyzing Talk among Friends. Oxford University Press, USA.
26

27. Coming Soon!
New book on Moderating
• Local Authors
• Coming this Fall
• Follow @ModSurvivalUX
27

28. Questions?
28

29. Thank You!
Susan
Mercer
Senior
Experience
Researcher
smercer@madpow.com
@susanamercer
29