Strengthening Design Research Moderating Skills


Published on

Slides from a talk that I gave for the Boston CHI meeting on April 9, 2013.

Moderating usability studies and interviewing research participants is more than just reading questions off of a page - it's about having a conversation.

In this talk I discuss why it's important to have a conversation with participants and how to do so in a way that encourages them to talk honestly.

I also discuss how to maintain neutrality and minimize introducing bias into the conversation.

Published in: Design, Technology, Business

Strengthening Design Research Moderating Skills

  1. 1. Boston  CHI  Monthly  Mee>ng  STRENGTHENING  DESIGN  RESEARCH    MODERATION  SKILLS      APRIL  9,  2013        Susan  Mercer  Senior  Experience  Researcher  @susanamercer  
  2. 2. Strengthening Moderating and Interviewing Skills§  Once I learned the basics§  And ran a couple of studies,§  Now what?§  How do I strengthen my skills? 2  
  3. 3. Story TimeDon’t play the notes on the page……play the musicThis applies to moderating andinterviewing too.Don’t just read the questions from the page……have a conversation 3  
  4. 4. The Art of Moderating and Interviewing§  Not just reading questions§  It’s also… §  Building rapport with the participant §  Creating a smooth conversation §  Encouraging the participant to talk §  Remaining objective §  Listening effectively §  Being human 4  
  5. 5. What Is Our Goal?§  To elicit honest thoughts from participants in an unbiased manner.§  “Would you like to try one of my chocolate chip oyster and herring brownies? They’re really good!”§  Sometimes it’s hard to be fully honest 5  
  6. 6. The Rules of PolitenessLinguistic anthropologists have studied this phenomenonThe Rules of Politeness 1 “Human  beings  are   always  balancing  the  1.  Don’t Impose paradoxical  fact  that  2.  Give Options they  are  simultaneously  3.  Be Friendly individuals  and  social   creatures.”   -­‐  Deborah  Tannen   Sources: 1 Lakoff, 1976; 2 Tannen, 2005 6  
  7. 7. How Do We Get Beyond Politeness?§  In other words, we’re often polite rather than honest to strangers.§  As human beings we have a need to be socially accepted Social Acceptance > Honesty§  We guess what is expected and answer to avoid offending§  It’s second nature – we’re often not aware we’re doing it 7  
  8. 8. How Do We Get Beyond Politeness?Build their trust Politeness Trust Familiarity Strangers Good Friend 8  
  9. 9. Building Trust in an Interview1.  Be trustworthy upfront2.  Build rapport3.  Engage in comfortable conversation4.  Stay neutral and accepting 9  
  10. 10. 1. Be Trustworthy Upfront§  Confidentiality §  How are you recording what is said? §  Who is listening? §  What will you do with the information you collect?§  Neutrality §  You’re not emotionally involved in the design/project §  There are no right or wrong answers §  Your job is to get honest opinions§  Consent Form §  Put it all in writing – using their language
  11. 11. 2. Build Rapport A  persons  name  is  §  Greeting a Participant to  that  person  the   §  Smile sweetest  and  most   §  Use their name important  sound  in   any  language.   §  Handshake / Warm phone greeting -­‐  Dale  Carnegie   §  Make them feel important - sincerely§  Small Talk – Find Common Ground §  Safe topics: travel to office, traffic, weather §  Avoid asking direct questions §  Listen and look for shared experiences Source: Carnegie, 1936.
  12. 12. 2. Build Rapport§  Be Empathetic §  Apologize if they had trouble finding the office §  Show you understand their point of view §  “Oh, it’s raining there? It is here too. I hate rainy days.”§  Inject Some Humor §  Joke about yourself §  Joke about the situation §  Don’t joke about them§  Short version – Show them that you are human too
  13. 13. 3. Engage in Comfortable ConversationTwo schools of usability testing moderating techniques§  Moderator keeps silent and says “Keep talking” 1 §  Awkward feeling for participant §  Reminds them that they are in a “study”§  Moderator creates conversation where participant is primary speaker 2 §  Still accomplishes goals of gathering information §  Meets the participant’s expectations of a socially acceptable conversation §  Moderator can still be neutral and minimize bias Sources: 1 Ericsson and Simon, 1980, 2 Boren and Raney, 2000. 13  
  14. 14. 3. Engage in Comfortable ConversationWhat is a Comfortable Conversation?§  Conversational cues and turn-taking are expected§  Acknowledgement tokens – “Uh huh”, etc. §  Encourage the continuation of the other speaker’s talk §  Usually implies that the other speaker’s prior talk is incomplete§  Some indicate change of speakership §  “Yeah” – more than half the time indicates changing turns in speaking §  “Yeah. I heard that the other day and…” §  “Oh!” - may indicate noticing something, then transitioning to another topic §  “Oh! That reminded me…”
  15. 15. 3. Engage in Comfortable Conversation§  Some may introduce bias §  “Oh!”, “Interesting” – indicating unexpected answer §  “Yes”, “Perfect”, “Great” – indicating agreement §  “Hmmm.”, “Really?” – indicating disagreement §  Notice that tone is key§  Neutral is best §  “Mhmm”, “Uh huh”, “Continue”, “Tell me more”, “OK” §  “Mhmm” or “Uh huh” vs. silence à interviewees saying 31% more phrases. 1 Source: 1 Matarazzo et. al., 1964
  16. 16. 4. Stay Neutral and Accepting§  Ask Open-ended Questions §  Start with Who, What, When, Where, Why, How?§  Really Listen §  Pay attention – stay in the moment §  Look at the participant §  Take notes if you can §  Be quiet - give them time to say what they need to
  17. 17. 4. Stay Neutral and Accepting§  Watch Your Reactions §  Don’t show surprise §  May make them think that they are giving a wrong answer §  Don’t overly agree §  May make them think that they are giving the right answer §  Don’t be negative §  Watch your tone – stay neutral and accepting §  Try not to laugh
  18. 18. 4. Stay Neutral and Accepting§  Be yourself§  No one is perfectly neutral§  Recover gracefully and move on §  “Perfect” – “That’s the level of detailed feedback we’re looking for.” §  “Interesting!” – “I haven’t heard that perspective yet, tell me more.” §  (something surprising) – “I can understand that.”§  Interject some Rapport-building comments when needed §  Quiet or uncomfortable participants §  “I hate it when that happens.”, “I can imagine that was challenging”, etc. §  Again, showing that you are human like them
  19. 19. 5. Encourage them to talk§  Be Quiet! §  Most agreements happen immediately. Most people delay before disagreeing.1 §  If you don’t respond to their answer, it encourages them to talk more §  People often delay speaking before disagreeing – give them time §  Some people are uncomfortable with silence, so they will keep talking §  The best way to stay neutral J Source: 1 Goodwin and Heritage, 1990.
  20. 20. Building Trust in an Interview1.  Be trustworthy upfront2.  Build rapport3.  Engage in comfortable conversation4.  Stay neutral and accepting5.  Be Quiet 20  
  22. 22. 1. Identify your improvement areas§  Watch your videos§  Have colleagues give you feedback 22  
  23. 23. 2. Learn From Others§  Watch others moderate§  Be a participant§  Listen to talk radio interviews 23  
  24. 24. 3. Practice These Skills in Everyday Life§  Be quiet and listen in everyday conversations§  Ask open-ended questions more frequently 24  
  25. 25. The most important thing is…PRACTICE! 25  
  26. 26. ReferencesBoren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication, 43 (3), 261-278.Carnegie, D. (1936) How to win friends and influence people. Simon & Shuster.Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3), 215-251.Goodwin, C., & Heritage, J. (1990). “Conversation analysis.” Annual review of anthropology 19 (1990): 283-307.Matarazzo, J.D.., Wiens, A. N., Saslow, G., Allen, B. V., & Weitman, M. (1064). Interviewer Mm-Hmm and interviewee speech durations. Psychotherapy: Theory, Research & Practice, 1(3), 109.Tannen, D. (2005). Conversational Style: Analyzing Talk among Friends. Oxford University Press, USA. 26  
  27. 27. Coming Soon!New book on Moderating•  Local Authors•  Coming this Fall•  Follow @ModSurvivalUX 27  
  28. 28. Questions? 28  
  29. 29. Thank You!        Susan  Mercer  Senior  Experience  Researcher  @susanamercer   29