Welcome everyone. What we want to talk about today is preparing for successful user research sessions. This came up as we were writing a book – a “survival guide” - on how to handle tricky situations that come up while moderating. There are steps that you can take ahead of time that will help prevent – or mitigate – those tricky situations. Specifically, how to prepare keeping in mind that the “unexpected” situations that we talk to each other about when sharing horror stories aren’t really as unexpected or infrequent as we think. It’s not enough to think that you can handle complications on the fly, which experienced and inexperienced moderators are prone to do. You need to anticipate what might happen and plan accordingly. Think of it like preparing for a trip. You create an itinerary but you should also make sure your insurance covers helicopter evacuations.
Hope for the best, but prepare for the worst!All types of user research sessions – different methods, different goals (qualvs quant), remote or in-person, at your location or on-site
Very simple to build on your regular study preparation so you can minimize the effect of infrequent (unexpected) events.
Remember – we’re building on established good practices for each of these with a specific focus on avoiding problems. Consider this a call to arms to treat all of these more seriously – no more complacency!
What you’ll avoid: participants who are not qualified (who will waste your time), participants who you can’t understandTalk with your stakeholders ahead of time about the criteria and dealbreakersA good way to identify behavior that you need – in addition to talking to stakeholders – is talking to a target usersTalk with your recruiter, especially if recruiting internal employees! (e.g., representative users not top performers). Understand how they work and the process they use to find participants – you need to trust them. Giveaway questions – instead of multiple choice or yes/no, try to use free form (“tell me about the last item you purchased on Amazon” instead of “Was the last item you purchased on Amazon a book?”)
What you’ll avoid: participants who expect a different kind of session (like focus group) and may not show up because of that, or who bring someone, or are unprepared Just like you are inviting someone to attend an event (you’d tell them time, location, dress code, etc.)If internal employee, their direct team will not be observingEXAMPLE: Double-jacking w/call center, need to ask them ahead of time to have the special jacks availableOr remote participant, have a hands free device
What you’ll avoid: empty timeslots, not enough data, unhappy stakeholdersYou can put together an activity or survey for these backup/floater participants so you’re able to get some feedback
Study plan -> moderator’s guide, interview script – any level of formality
What you’ll avoid: (list above) + technical/environmental difficultiesHow will you adjust your study plan, based on your research goals, if you encounter…Time constraints (Late participant; Slow/through participant)Misrecruit (Unqualified participant; Unprepared participant)Personality challenges (Reluctant or obstinate participant)Technical difficulties (Uncooperative technology)Adjustments should be based on your research goals. E.g., quantitative. Work with your stakeholders to prioritize. Know which tasks/questions can be skipped, or moved around. Don’t be afraid to adjust or even end the session early depending on what’s happening and your participant’s comfort level
What you’ll avoid/limit: confusion over your behavior, frustration about limited prototype functionalityAdjust this per study – don’t just use the same template over and over againVerify participant qualifications early in the session – you can fit it naturally into the briefing so it comes across as less of an info dump and more of a conversationYou can refer back to this throughout the session
Help avoid: Unexpected prototype/product changes; participant frustration with a prototype’s limitations; technical difficultiesPrototype – we all have limited time (especially Agile), so illusion can be important.Also set expectations at beginning of session that everything may not be working. Updates to prototype – even if Agile/rapid, avoid changes in the middle of a session (will confuse participant and you)Live Products:Try to avoid updates during your research (if you are in-house)A/B tests – either turn off, or find way to always see same version if you need consistency
Avoid: Wasted time spent troubleshooting in the middle of a sessionHelpful if anything goes wrong in the middle of a session, or if multiple people are moderatingBackup equipment – e.g., if you’re going on-site and need to be prepared for different types of setupsIf you have time, ask someone on your team to “break” something in your setup as a troubleshooting exercise
Environmental, e.g., smell levels at animal testing facility, or wearing closed-toe hard soled shoes at manufacturing facility
Observers – whether stakeholders, team members, etc. Anyone who will be watching the session!
Avoid: interrupting/misbehaving observersCONTINUES on next slide with examples of ground Hold observers accountable: Remember your obligations to the participant and to the protection of your organization
These will vary depending on your study goalsIdeally discuss these in person with your observers/stakeholders ahead of time, in personAvoid: Confusion over your behavior, interruptions
Person observing can be stakeholder or team member, as long as they can be paying attentionLiability issues!Avoid running research on your own!
….Because sometimes, no matter what preparation you do, things will go wrong
Participant comfort comes first!Your attitudeBe neutral but kind and respectfulNot their buddy!Take responsibilityYour body languagePay attentionBe consistentWhat you sayLet participant talkAvoid leading questionsYour toneCurious and interestedNot defensive
Expect certain behaviors – e.g., if you need a participant to attempt every task but expect to run short of time, you might write down a way to smoothly transition them to the next taskTrying to adjust your style – e.g., if you’re trying to train yourself to avoid/include moderating behaviors such as giving the participant more time to answer a questionTONE is importantDepending on your goals, you may want different kinds of things jotted down (e.g., if you suspect that you’ll need to provide assists, you may want to write down wording for how you’d do this)EXAMPLES on next page
TONE is importantDepending on your goals, you may want different kinds of things jotted down (e.g., if you suspect that you’ll need to provide assists, you may want to write down wording for how you’d do this)
Expecting the Unexpected: Preparing for Successful User Research Sessions (Donna Tedesco & Fiona Tranquada, UXPA Boston 2013)
Donna TedescoFiona TranquadaExpecting the Unexpected:Preparing for SuccessfulUser Research SessionsUXPA Boston 2013
Identify what you can dobefore your study to: Reduce the chances ofencounteringannoying, tricky, and stickysituations – especially theones you don’t usually expect– during a session Set yourself up tosuccessfully handle thosesituations when they occurOur Goal
Frequent Participant is late or a no-show Participant not thinkingaloud Participant is unable tocomplete a necessary taskInfrequent Participant makes you feeluncomfortable Participant frustrated byprototype limitations Observer interrupts thesession Natural disasterWhat do we mean by “unexpected”?Being prepared will help you with both types of situations!
Recruit Study Plan Product/Space/Technology Observers YourselfWhat We’ll Cover
Identify your“dealbreakers” Look for behavior inaddition to role Avoid “giveaway”questionsBe specific with your criteria
Confirmationletter/email, includingdirections to facility orremote session 1 on 1 Recording and observers If you need them to bring anything to do anything during thesession (e.g., watch themprocess payroll) to prepareSet expectations about the session
Backup – 2 participantsscheduled for 1 timeslot Floater – 1 participantscheduled to cover multipletimeslots Incremental cost but maybe worth it if you have high-profile observers If you don’t have backup orfloaters, plan for no shows!Schedule backup or floater participants
Establishingpriorities andmakingbackup plansStudy Plan
Think about how you canadjust your plan if youencounter: Time constraints Misrecruits Personality challenges Technical difficulties Create a backup such as apaper survey orquestionnairePrioritize your tasks/questions
Reiterate the points coveredin your consent form: Recording/observers How data will be used Goals for the session Take a break or end the sessionat any time Set clear expectations for: How the session time will beused What they’ll be using (e.g., aprototype) Your behavior (e.g., may notanswer questions) Verify qualificationsPrepare the participant briefing
Prototype “Illusion of functionality” Agree on if/when any updateswill be made to the prototypeActual “live” product Be aware of the productschedule Research any A/B tests Have a backup if necessary(e.g., screenshots) or setappropriate expectationsPrepare the Product
Walk through your full setup If participants are remote, do awalkthrough with them as well Document everything!Include screenshots! Order of operations Lab/space setup Recording Screen sharing / Audioconferencing Controls for observers (Mute All) “Kick out” options Prototype/Product URLs Login/Password Paths Any known issues (e.g., browserincompatibilities) Any backup equipment Technical support contactDocument your setup
Emergency exits Restrooms Kitchen / Vending areas Security / Emergencyprocedures Printers / Copiers Climate controls If going off-site, Equipment prohibitions Seating (e.g., if you haveobservers) Internet access Noise levels Other environmental factorsExplore your research space
Establish a code of conduct Remember – you’re responsible for theparticipant’s well-being! Share with observers ahead of time Via email Poster on the wall of research space Printouts Hold observers accountableProvide ground rules
Core ground rules to include: Keep quiet; no laughing at or making fun of participant! Keep participant’s identity confidential Avoid slamming doors If observers are remote, Keep phone muted Avoid in-meeting chat feature If observers are in the room with you, Avoid multitasking Be careful with body language What to do if they have questions or want to communicate withyou/participantProvide ground rules cont.
May not exactly follow study plan May let participant follow tangents Assists and diversions When you might use them How to tell if that’s what you’re doing May pretend to not know an answerCommunicate your approach
Make sure you always haveat least 1 person observing Avoid going to someone’sspace without 1 otherperson “Safe word” to indicatewhen you need assistanceEstablish safety precautions
Ethical obligations Your attitude Your body language What you say Your toneReview good moderating behavior
Helpful if you’re: New to moderating Running a new kind ofresearch Expect certain behaviors Trying to adjust your style Include in your study plan –don’t be afraid to put it onevery pageJot down key phrases or reminders
Examples of what you might jot down: Wait 3 seconds before responding to a participant Neutral acknowledgements (“Mmhmm”, “Uh-huh”) Turn question around “If you were doing this at home, and I wasn’t sitting there next toyou, what would you do?” Prompt non-responsive participant “What are you trying to do right now? Reassure "This is just the kind of feedback we want to hear..." Redirect or cut short "Im sorry, I just want to interrupt you here for a second. For the sake oftime I’d like you to return to the task/question, and we can revisit thistopic at the end of the session if time permits.Jot down key phrases or reminders cont.
Watch recordings of yourself Practice with a colleague Have your colleague “role-play” problematic situations Ask colleagues to watch you moderate and provide feedback Watch others moderatePractice!
Set expectations With participant With observers Dive deep Your space Your technology Your study plan Create backups For your technology For your participants For your product Practice Your technical setup Your moderatingPreparation Takeaways