1. Susan Mercer – Senior Experience Researcher
@susanamercer
May 29, 2013
UXPA Boston 2013 Conference
Moderating to the Max
REFINING YOUR MODERATING &
INTERVIEWING SKILLS

2. Hello, I’m Susan Mercer
 BA and MSc in Geophysics
 19 years in software and web UI and UX design
 Developer
 Designer
 Web Producer
 Product Manager
 Researcher
 MS Human Factors, Bentley University
 Twitter: @susanAmercer
2

3. Interviewing and Moderating
3
Research & Strategy
Interaction
Design
Visual Design Development
Interviews
Formative
Usability
Summative
Usability

4. Moderating and Interviewing Skills
 On paper, they seem straightforward
 Write a screener
 Recruit
 Write a script
 Conduct the study
 Once I ran a couple easy ones, I thought I had it down.
4
Interviews
Formative
Usability

5. But Then…
 I got the participants who…
And I learned an important lesson:
Interviews and usability tests are
more than just a series of questions.
5

6. My Epiphany
Don’t play the notes on the page…
…play the music
This applies to moderating and
interviewing too.
Don’t just read the questions from the page…
…have a conversation
6

7. What Is Our Goal?
 To elicit honest thoughts from participants in an unbiased manner.
Sometimes it’s hard to be fully honest.
7

8. The Rules of Politeness
We’re often polite rather than honest to strangers
Social Acceptance > Honesty
8

9. How Do We Get Beyond Politeness?
Build their trust
9

10. 10
Leading
Questions
Introduce
Bias

11. Participant Comfort
Two schools of usability testing moderating techniques
 Moderator keeps silent and says “Keep talking” 1
 Awkward feeling for participant
 Reminds them that they are in a “study”
 Moderator creates conversation where participant is primary
speaker 2
 Still accomplishes goals of gathering information
 Meets the participant’s expectations of a socially acceptable conversation
 Moderator can still be neutral and minimize bias
11
Sources: 1 Ericsson and Simon, 1980, 2 Boren and Ramey, 2000.

12. Conversations
• We carry on conversations all of the time
• We convey a lot of subtle meaning in our conversations
• We have rules for conversation
• We know what is acceptable in which situations
12

13. What Do We Do in Usability tests and interviews?
We ask participants to:
 Come into a usability lab / meet us in a strange location
 Talk to a stranger about important topics
 Use a product or website and make mistakes
 While being observed
 Without knowing the right social norms on how to behave
13

14. To Do List
• Elicit honest feedback
• In an unbiased manner
• Instill trust in us
• Make the participant comfortable
• …and get them to answer all of your questions
14

15. HOW TO DO IT – STEP BY STEP
15

16. Structure of an Interview/Usability Session
16
Trust
Comfort
Interview
Build Rapport
Neutral
Observation
Define Social
Interaction Rules
Be Transparent
Comfortable
Conversation
Be Accepting
Thanks
Manage the Flow
Greeting Introduction
Interview /
Evaluation
Wrap-Up

17. 1. Greeting – Building Rapport
 Be a Friendly Person
 Smile
 Use their name
 Be a good listener
 Make the other person feel important – and do so sincerely
A person's name is to that person the
sweetest and most important sound in
any language.
- Dale Carnegie

18. 1. Greeting – Building Rapport
 Small Talk – Find Common Ground
 Safe topics: travel to office, traffic, weather, local sports
 Avoid asking direct questions
 Listen and look for shared experiences

19. 1. Greeting – Building Rapport
 Be Empathetic
 Apologize if they had trouble finding the office
 “Oh, it’s raining there? It is here too. I hate rainy days.”
 Show you understand their point of view
 “I can understand that…”
 “I can see that…”
 “That does sound very frustrating…”

20. 1. Greeting – Building Rapport
 Inject Some Humor
 Joke about yourself
 Joke about the situation
 Don’t joke about them
 Rapport is about showing them that you are human too

21. 2. Introduction – Be Transparent
 Confidentiality
 How are you recording what is said?
 Who is listening?
 What will you do with the information you collect?
 Consent Form
 Put it all in writing – using their language

22. 2. Introduction – Define Social Interaction Rules
 Describe the Session
 List the activities
 Describe the roles
 Social Niceties Do Not Apply
 You’re not emotionally involved in the design/project
 There are no right or wrong answers
 Your job is to get honest opinions

23. 3. Interview/Evaluation – Be Accepting
 Watch Your Reactions
 Don’t show surprise
 May make them think that they are giving a wrong answer
 Don’t overly agree
 May make them think that they are giving the right answer
 Don’t be negative
 Watch your tone – stay neutral and accepting
 Try not to laugh

24. 3. Interview/Evaluation – Be Accepting
 Be yourself
 No one is perfectly neutral
 Recover gracefully and move on
 “Perfect” – “That’s the level of detailed feedback we’re looking for.”
 “Interesting!” – “I haven’t heard that perspective yet, tell me more.”
 (something surprising) – “I can understand that.”
 Interject some Rapport-building comments when needed
 Quiet or uncomfortable participants
 “I hate it when that happens.”, “I can imagine that was challenging”, etc.
 Again, showing that you are human like them

25. 3. Interview/Evaluation – Comfortable Conversation
What is a Comfortable Conversation?
 Conversational cues and turn-taking are expected
 Acknowledgement tokens – “Uh huh”, etc.
 Encourage the continuation of the other speaker’s talk
 Usually implies that the other speaker’s prior talk is incomplete
Source: 1 Drummond and Hopper, 1993.

26. 3. Interview/Evaluation – Comfortable Conversation
 Some acknowledgement tokens indicate other things
 “Yeah” – half the time indicates changing turns in speaking 1
 “Yeah. I heard that the other day and…”
 “Oh!” - may indicate noticing something, then transitioning to another
topic
 “Oh! That reminded me…”
 “Okay” – may indicate that the listener wants the speaker to stop
speaking1
Source: 1 Drummond and Hopper, 1993.

27. 3. Interview/Evaluation – Neutral Observation
 Some may introduce bias
 “Oh!”, “Interesting” – indicating unexpected answer
 “Yes”, “Perfect”, “Great” – indicating agreement
 “Hmmm.”, “Really?” – indicating disagreement
 Notice that tone is key
 Neutral is best
 “Mhmm”, “Uh huh”, “Continue”, “Tell me more”, “OK”
 “Mhmm” or “Uh huh” vs. silence  interviewees saying 31% more
phrases. 1
• Body Language
• Head nodding while participant is speaking  interviewees speak 50%
longer. 2
Source: [1] Matarazzo et. al., 1964, [2] Matarazzo et al., 1963

28. 3. Interview/Evaluation – Neutral Observation
 Ask Open-ended Questions
 Start with Who, What, When, Where, Why, How?

29. 3. Interview/Evaluation – Neutral Observation
 Really Listen
 Pay attention – stay in the moment
 Look at the participant
 Take notes if you can
 Be quiet - give them time to say what they need to

30. 3. Interview/Evaluation – Neutral Observation
 Be Quiet!
 Most agreements happen immediately. Most people delay before
disagreeing.1
 If you don’t respond to their answer, it encourages them to talk more
 People often delay speaking before disagreeing – give them time
 Some people are uncomfortable with silence, so they will keep talking
 “People speak in paragraphs.” (Steve Portugal)
 The best way to stay neutral 
Source: 1 Goodwin and Heritage, 1990.

31. 3. Interview/Evaluation – Manage the Flow
• Wait for topics to come up organically
• If the participant mentions something that you have later in your question
list, ask it now.
• Transitions
• Don’t just switch topics – provide context.
• Summarize last topic, then introduce new one.
• “That’s great about XYZ, but now I’d like to talk about ABC”. Question.
• “You mentioned XYZ, tell me a bit more about how you blah blah..”

32. Structure of an Interview/Usability Session
32
Trust
Comfort
Interview
Build Rapport
Neutral
Observation
Define Social
Interaction Rules
Be Transparent
Comfortable
Conversation
Be Accepting
Thanks
Manage the Flow
Greeting Introduction
Interview /
Evaluation
Wrap-Up

33. The most important thing is…
33
PRACTICE!

34. KEEPING SHARP
34

35. Keep Improving
1. Identify your improvement areas
 Watch your videos
 Have colleagues give you feedback
35

36. Keep Improving
2. Learn from others
 Watch others moderate
 Be a participant
 Listen to talk radio interviews
Terri Gross, Fresh Air, Ira Glass, This American Life
36

37. Keep Improving
3. Practice skills in everyday life
 People watch (observe) in public
 Be quiet and listen in everyday conversations
37

38. MORE INFO
38

39. Classic Books on Moderating
39

40. Coming Soon!
New book on Moderating
• Donna Tedesco, Fiona Tranquada
• Book coming this Fall
• Follow @ModSurvivalUX
See them today:
• “Expecting the Unexpected: Preparing for Successful User Research
Sessions”
• 3:00 PM, Back Bay C
40

41. Interviewing
41

42. 42
Questions?

43. References
Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional
Communication, 43 (3), 261-278.
Carnegie, D. (1936) How to win friends and influence people. Simon & Shuster.
Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.
Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.
Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3), 215-251.
Goodwin, C., & Heritage, J. (1990). “Conversation analysis.” Annual review of anthropology 19 (1990): 283-307.
Matarazzo, J.D.., Saslow, G., Wiens, A. N., Weitman, M. & Allen, B. V. (1963). Interviewer head nodding and interviewee
speech durations. Psychotherapy, 1, 54-63.
Matarazzo, J.D.., Wiens, A. N., Saslow, G., Allen, B. V., & Weitman, M. (1964). Interviewer Mm-Hmm and interviewee
speech durations. Psychotherapy: Theory, Research & Practice, 1(3), 109.
Tannen, D. (2005). Conversational Style: Analyzing Talk among Friends. Oxford University Press, USA.
http://www.uxbooth.com/articles/5-useful-lies-to-tell-user-research-participants/
http://www.skillsyouneed.com/ips/rapport.html
43

44. 44
Thank You!
Susan Mercer
Senior Experience Researcher
smercer@madpow.com
@susanamercer