Here to talk about moderating and interviewingInterviews and Usability tests are often done in different parts of the project lifecycle, but they are quite similar in the skills requiredFormative done during the design process, to get qualitative feedback used to iterate on the designSummative done at the end of development. More structured, quantitative, to measure specific aspects of the softwareInterviews and Formative tests are similar in structure and skillsets needed to effectively moderate them. That’s what I’ll be focusing on today.
There are lots of soft skills involved.
“Would you like to try one of my chocolate chip oyster and herring brownies? They’re really good!”
Rules of Politeness govern social conversations – and are stronger with people we do not know wellDon’t Impose“Would you like something to drink?”“Yes, that would be nice.”VS“Yes, I’ll have an Orange Juice mixed with Raspberry Lime seltzer.”Give Options“Would you like something to drink?”“Water, tea, or whatever you have – don’t go to any trouble.”Be Friendly“What do you think of this website?”“Well, it’s not my favorite…”VS.“This is crap.”
A warm smile is the universal language of kindness – William Arthur Ward
Moderating to the Max: Refining Your Interviewing and Moderating Skills
Susan Mercer – Senior Experience Researcher@susanamercerMay 29, 2013UXPA Boston 2013 ConferenceModerating to the MaxREFINING YOUR MODERATING &INTERVIEWING SKILLS
Hello, I’m Susan Mercer BA and MSc in Geophysics 19 years in software and web UI and UX design Developer Designer Web Producer Product Manager Researcher MS Human Factors, Bentley University Twitter: @susanAmercer2
Interviewing and Moderating3Research & StrategyInteractionDesignVisual Design DevelopmentInterviewsFormativeUsabilitySummativeUsability
Moderating and Interviewing Skills On paper, they seem straightforward Write a screener Recruit Write a script Conduct the study Once I ran a couple easy ones, I thought I had it down.4InterviewsFormativeUsability
But Then… I got the participants who…And I learned an important lesson:Interviews and usability tests aremore than just a series of questions.5
My EpiphanyDon’t play the notes on the page……play the musicThis applies to moderating andinterviewing too.Don’t just read the questions from the page……have a conversation6
What Is Our Goal? To elicit honest thoughts from participants in an unbiased manner.Sometimes it’s hard to be fully honest.7
The Rules of PolitenessWe’re often polite rather than honest to strangersSocial Acceptance > Honesty8
How Do We Get Beyond Politeness?Build their trust9
Participant ComfortTwo schools of usability testing moderating techniques Moderator keeps silent and says “Keep talking” 1 Awkward feeling for participant Reminds them that they are in a “study” Moderator creates conversation where participant is primaryspeaker 2 Still accomplishes goals of gathering information Meets the participant’s expectations of a socially acceptable conversation Moderator can still be neutral and minimize bias11Sources: 1 Ericsson and Simon, 1980, 2 Boren and Ramey, 2000.
Conversations• We carry on conversations all of the time• We convey a lot of subtle meaning in our conversations• We have rules for conversation• We know what is acceptable in which situations12
What Do We Do in Usability tests and interviews?We ask participants to: Come into a usability lab / meet us in a strange location Talk to a stranger about important topics Use a product or website and make mistakes While being observed Without knowing the right social norms on how to behave13
To Do List• Elicit honest feedback• In an unbiased manner• Instill trust in us• Make the participant comfortable• …and get them to answer all of your questions14
Structure of an Interview/Usability Session16TrustComfortInterviewBuild RapportNeutralObservationDefine SocialInteraction RulesBe TransparentComfortableConversationBe AcceptingThanksManage the FlowGreeting IntroductionInterview /EvaluationWrap-Up
1. Greeting – Building Rapport Be a Friendly Person Smile Use their name Be a good listener Make the other person feel important – and do so sincerelyA persons name is to that person thesweetest and most important sound inany language.- Dale Carnegie
1. Greeting – Building Rapport Small Talk – Find Common Ground Safe topics: travel to office, traffic, weather, local sports Avoid asking direct questions Listen and look for shared experiences
1. Greeting – Building Rapport Be Empathetic Apologize if they had trouble finding the office “Oh, it’s raining there? It is here too. I hate rainy days.” Show you understand their point of view “I can understand that…” “I can see that…” “That does sound very frustrating…”
1. Greeting – Building Rapport Inject Some Humor Joke about yourself Joke about the situation Don’t joke about them Rapport is about showing them that you are human too
2. Introduction – Be Transparent Confidentiality How are you recording what is said? Who is listening? What will you do with the information you collect? Consent Form Put it all in writing – using their language
2. Introduction – Define Social Interaction Rules Describe the Session List the activities Describe the roles Social Niceties Do Not Apply You’re not emotionally involved in the design/project There are no right or wrong answers Your job is to get honest opinions
3. Interview/Evaluation – Be Accepting Watch Your Reactions Don’t show surprise May make them think that they are giving a wrong answer Don’t overly agree May make them think that they are giving the right answer Don’t be negative Watch your tone – stay neutral and accepting Try not to laugh
3. Interview/Evaluation – Be Accepting Be yourself No one is perfectly neutral Recover gracefully and move on “Perfect” – “That’s the level of detailed feedback we’re looking for.” “Interesting!” – “I haven’t heard that perspective yet, tell me more.” (something surprising) – “I can understand that.” Interject some Rapport-building comments when needed Quiet or uncomfortable participants “I hate it when that happens.”, “I can imagine that was challenging”, etc. Again, showing that you are human like them
3. Interview/Evaluation – Comfortable ConversationWhat is a Comfortable Conversation? Conversational cues and turn-taking are expected Acknowledgement tokens – “Uh huh”, etc. Encourage the continuation of the other speaker’s talk Usually implies that the other speaker’s prior talk is incompleteSource: 1 Drummond and Hopper, 1993.
3. Interview/Evaluation – Comfortable Conversation Some acknowledgement tokens indicate other things “Yeah” – half the time indicates changing turns in speaking 1 “Yeah. I heard that the other day and…” “Oh!” - may indicate noticing something, then transitioning to anothertopic “Oh! That reminded me…” “Okay” – may indicate that the listener wants the speaker to stopspeaking1Source: 1 Drummond and Hopper, 1993.
3. Interview/Evaluation – Neutral Observation Some may introduce bias “Oh!”, “Interesting” – indicating unexpected answer “Yes”, “Perfect”, “Great” – indicating agreement “Hmmm.”, “Really?” – indicating disagreement Notice that tone is key Neutral is best “Mhmm”, “Uh huh”, “Continue”, “Tell me more”, “OK” “Mhmm” or “Uh huh” vs. silence interviewees saying 31% morephrases. 1• Body Language• Head nodding while participant is speaking interviewees speak 50%longer. 2Source:  Matarazzo et. al., 1964,  Matarazzo et al., 1963
3. Interview/Evaluation – Neutral Observation Really Listen Pay attention – stay in the moment Look at the participant Take notes if you can Be quiet - give them time to say what they need to
3. Interview/Evaluation – Neutral Observation Be Quiet! Most agreements happen immediately. Most people delay beforedisagreeing.1 If you don’t respond to their answer, it encourages them to talk more People often delay speaking before disagreeing – give them time Some people are uncomfortable with silence, so they will keep talking “People speak in paragraphs.” (Steve Portugal) The best way to stay neutral Source: 1 Goodwin and Heritage, 1990.
3. Interview/Evaluation – Manage the Flow• Wait for topics to come up organically• If the participant mentions something that you have later in your questionlist, ask it now.• Transitions• Don’t just switch topics – provide context.• Summarize last topic, then introduce new one.• “That’s great about XYZ, but now I’d like to talk about ABC”. Question.• “You mentioned XYZ, tell me a bit more about how you blah blah..”
Structure of an Interview/Usability Session32TrustComfortInterviewBuild RapportNeutralObservationDefine SocialInteraction RulesBe TransparentComfortableConversationBe AcceptingThanksManage the FlowGreeting IntroductionInterview /EvaluationWrap-Up
Coming Soon!New book on Moderating• Donna Tedesco, Fiona Tranquada• Book coming this Fall• Follow @ModSurvivalUXSee them today:• “Expecting the Unexpected: Preparing for Successful User ResearchSessions”• 3:00 PM, Back Bay C40
ReferencesBoren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on ProfessionalCommunication, 43 (3), 261-278.Carnegie, D. (1936) How to win friends and influence people. Simon & Shuster.Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3), 215-251.Goodwin, C., & Heritage, J. (1990). “Conversation analysis.” Annual review of anthropology 19 (1990): 283-307.Matarazzo, J.D.., Saslow, G., Wiens, A. N., Weitman, M. & Allen, B. V. (1963). Interviewer head nodding and intervieweespeech durations. Psychotherapy, 1, 54-63.Matarazzo, J.D.., Wiens, A. N., Saslow, G., Allen, B. V., & Weitman, M. (1964). Interviewer Mm-Hmm and intervieweespeech durations. Psychotherapy: Theory, Research & Practice, 1(3), 109.Tannen, D. (2005). Conversational Style: Analyzing Talk among Friends. Oxford University Press, USA.http://www.uxbooth.com/articles/5-useful-lies-to-tell-user-research-participants/http://www.skillsyouneed.com/ips/rapport.html43