Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Questioning Picture-in-Picture: Why Showing the Participant May Not Be Such a Great Idea After All

1,010 views

Published on

People doing usability testing often display video of the test participant in a small picture-in-picture window in the corner of the screen. Theoretically this increases observer empathy and helps convince even the most stubborn executive that their product makes users miserable—and needs fixing.

After years of teaching people to do usability testing, I’ve come to feel that showing observers the participant's face may not always be a good thing. In this presentation, I spell out the pros and cons of using picture-in-picture video.

Slides by Steve Krug, author of Don’t Make Me Think. Presented at the UXPA Boston Conference, May 19, 2017.

Published in: Design
  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Questioning Picture-in-Picture: Why Showing the Participant May Not Be Such a Great Idea After All

  1. 1. Steve Krug UxPA Boston 2017 Questioning Picture-in-Picture: Why Showing the Participant May Not Be Such a Great Idea After All
  2. 2. The backstory  Participant Picture-in-Picture (PPIP) means displaying the usability test participant’s face in a small picture-in-picture window.  It’s long been thought of as a good thing  Increases empathy with the user  Helps convince the team (esp. management) that the product makes users miserable and needs fixing  I call it “Squirmcam” because historically it’s been about highlighting user discomfort, pain, and failure  Once difficult and expensive to do, it’s now easy  Has become more or less standard practice
  3. 3. Backstory (cont’d.)  I’ve come to feel that it may not necessarily be a good thing  I think it's often an unnecessary distraction  I’ve kept this opinion to myself except when asked  Always prefaced it by saying “I know I may be wrong about this, because I seem to be the only person who feels this way”  When I decided to do this presentation, I was surprised to find that some people whose opinions I respect felt the same way I did  As one of them said:
  4. 4. “My main issue about showing the participant is that for me it really takes away from being able to concentrate on the interface and what the user is DOING and SAYING in relation to it. It’s hard to keep concentrating anyway and easy to get distracted, so why furnish a distraction that’s NOT the main concern of the test?”
  5. 5. We’ve only got 45 40 minutes  What this is about (done)  A very brief history of PPIP  My original list of pros and cons  Survey results  The questions that interest me  My recommendations  Q&A
  6. 6. A very brief history of PPIP video  For a long time, PPIP was very difficult and expensive  Had to be done in a “lab” with dedicated video cameras, tape decks, video mixer, etc.
  7. 7. One-Way Glass!
  8. 8. A very brief history (cont’d.)  Adding PPIP required a video operator and editor  Very time-consuming  Show of hands: Anyone ever edited analog video?
  9. 9. A very brief history (cont’d.)  Around the turn of the century, things got dramatically better, cheaper, easier  Cameras > Webcams  Scan converter > Morae, Camtasia  Tape decks > Digital editing  Boxes of videotapes > 1Tb pocket drive  Video cables > Wi-fi, screen sharing software  Thousands of $$ of hardware > Hundreds of $$ of software  Do it if ROI warrants it > Capture it even if you may never use it
  10. 10. Typical ways PPIP has been used  To create highlight reels to demonstrate participant “suffering”  To create clips to use in presentations  Part of a video record  For viewing by people who couldn’t attend live sessions  For possible future reference  In the live view shown to observers
  11. 11. Two demo clips  According to my proposal, at this point:  “I’ll be showing two brief video clips with and without PPIP so session attendees can experience the difference”  Ultimately deemed infeasible for many reasons, like limited time and your distance from the screen
  12. 12. Existing research  Tried to do due diligence  Apparently there isn’t much  Here’s the most relevant study I could find  Thanks to Chauncey Wilson, and others
  13. 13. BTW: Why don’t research papers have dates? (It’s a rhetorical question.) http://download.techsmith.com/morae/docs/casestudies/pip-usafa.pdf
  14. 14. They took the obvious approach  Showed two groups the same test videos  One group with PPIP, the other without  Asked them to identify usability problems  Studied what problems each group reported and how seriously they rated them  Did a pilot test with experts, then final test with 24 human factors students
  15. 15. Unfortunately…  Main finding was that there was little agreement on problems between and within groups  e.g., “These results clearly show that expert agreement was less than what would be expected by chance.”
  16. 16. Strong sense of déjà vu  Studies done this way almost always seem to show/”prove” that observers (even experts) don’t agree very much  Rolf Molich’s Comparative Usability Evaluation (CUE) studies are an amazing source of insights into what works
  17. 17. http://www.dialogdesign.dk/CUE.html
  18. 18. Strong sense of deja vue  But they’ve “proven” over and over that if you show the same usability test to experts, there won’t be much overlap in the problems they report  The “evaluator effect”  I think it’s a flawed approach  For reasons too complicated to fit in the margin here (Thanks, Fermat!)  My conclusion:  This one research study doesn’t tell us much
  19. 19. My initial pros and cons  My original list, which I thought was pretty exhaustive  …until I made the classic mistake of actually listening to users
  20. 20. Pros of using PPIP  Adds sizzle  The same way eye tracking can  May get team members more excited about the process  May get more bodies in the observation room  May keep observers more engaged  Keep them watching the screen, instead of their phones  Can be persuasive  It can help drive home the misery the UI/UX is creating  Good way to get attention/support of higher-ups  Improves chances of actually implementing changes
  21. 21. Cons  Can distract observers from think aloud and screen activity  Doesn’t reveal much that wouldn’t be otherwise obvious  Complicates privacy issues  Faces are recognizable  How do you maintain control over clips?  Will clients take same care you do?  Best privacy solution is not to identify users at all; this is the opposite  Adds more complexity to setup  One more thing to break/keep track of
  22. 22. Cons (cont’d.)  Can add to the participant’s stress  I haven’t observed this myself  (But I don’t do PPIP)  My experience: they make a joke about recording at the beginning, then forget about it  Picture-in-picture obscures part of the screen
  23. 23. The survey  Four questions plus a Comments field  Ran for six days  200 responses  Mean time to complete: 3 minutes  Deceptively advertised as 90 seconds  But the questions were easy to answer  Response data is available at www.sensible.com/downloads/PPIP-survey-results.pdf  Show of hands: Took the survey?
  24. 24. Caroline Jarrett!  My forms and surveys hero  She’s writing a book (Surveys that Work) for Rosenfeld Media  “…just what you need to know about surveys: the least you need to know to do a survey well, with the smallest amount of effort.”  Helped pilot test and revise  Analysed results
  25. 25. Caroline says: Given our quite small sample size, we’ve got an almost even number of people across all the categories of PPIP use
  26. 26. What do you think are the three most important reasons *for* using PPIP? (choose up to three) 0 20 40 60 80 100 120 140 160 Nothing in favor Other It makes observers more likely to attend live tests Adds a dramatic “wow” factor Good to use in clips to convince people who can’t/won’t take time to observe whole tests It keeps observers more engaged Emphasizes user misery in a way that is more convincing and likely to produce changes It makes participants’ problems more visible and vivid Encourages identification with and empathy for the user Always / most of the time Sometimes Rarely / never
  27. 27. What do you think are the three most important reasons *for* using PPIP? (choose up to three) 0 20 40 60 80 100 120 140 160 Nothing in favor Other It makes observers more likely to attend live tests Adds a dramatic “wow” factor Good to use in clips to convince people who can’t/won’t take time to observe whole tests It keeps observers more engaged Emphasizes user misery in a way that is more convincing and likely to produce changes It makes participants’ problems more visible and vivid Encourages identification with and empathy for the user Always / most of the time Sometimes Rarely / never Top three are all about misery, empathy
  28. 28. What do you think are the three most important reasons *for* using PPIP? (choose up to three) 0 20 40 60 80 100 120 140 160 Nothing in favor Other It makes observers more likely to attend live tests Adds a dramatic “wow” factor Good to use in clips to convince people who can’t/won’t take time to observe whole tests It keeps observers more engaged Emphasizes user misery in a way that is more convincing and likely to produce changes It makes participants’ problems more visible and vivid Encourages identification with and empathy for the user Always / most of the time Sometimes Rarely / never My “sizzle” notion was a bust
  29. 29. What do you think are the three most important reasons *for* using PPIP? (choose up to three) 0 20 40 60 80 100 120 140 160 Nothing in favor Other It makes observers more likely to attend live tests Adds a dramatic “wow” factor Good to use in clips to convince people who can’t/won’t take time to observe whole tests It keeps observers more engaged Emphasizes user misery in a way that is more convincing and likely to produce changes It makes participants’ problems more visible and vivid Encourages identification with and empathy for the user Always / most of the time Sometimes Rarely / never Caroline adds: There’s quite a lot of agreement about the good things and bad things about PPIP, no matter how much they use it
  30. 30. What do you think are the three most important reasons *for* using PPIP? (choose up to three) 0 20 40 60 80 100 120 140 160 Nothing in favor Other It makes observers more likely to attend live tests Adds a dramatic “wow” factor Good to use in clips to convince people who can’t/won’t take time to observe whole tests It keeps observers more engaged Emphasizes user misery in a way that is more convincing and likely to produce changes It makes participants’ problems more visible and vivid Encourages identification with and empathy for the user Always / most of the time Sometimes Rarely / never Caroline: Respondents who don’t use PPIP saw slightly more bad things (but not that much more) Respondents who do use PPIP saw slightly more good things (but not that much more)
  31. 31. What do you think are the three most important reasons for *not* using PPIP? (choose up to three) 0 20 40 60 80 100 120 Did not choose any Against options Other (please specify) It makes life more complicated It doesn’t produce better results For some observers, it can be a distraction from… Some people may refuse to take part if they’re… Hard to do when testing mobile devices Some observers may be biased based on the… Showing people’s faces raises additional privacy… It may make some participants uncomfortable or… All / most of the time Sometimes Rarely / never
  32. 32. What do you think are the three most important reasons for *not* using PPIP? (choose up to three) 0 20 40 60 80 100 120 Did not choose any Against options Other (please specify) It makes life more complicated It doesn’t produce better results For some observers, it can be a distraction from… Some people may refuse to take part if they’re… Hard to do when testing mobile devices Some observers may be biased based on the… Showing people’s faces raises additional privacy… It may make some participants uncomfortable or… All / most of the time Sometimes Rarely / never I was glad to see so much concern about privacy
  33. 33. What do you think are the three most important reasons for *not* using PPIP? (choose up to three) 0 20 40 60 80 100 120 Did not choose any Against options Other (please specify) It makes life more complicated It doesn’t produce better results For some observers, it can be a distraction from… Some people may refuse to take part if they’re… Hard to do when testing mobile devices Some observers may be biased based on the… Showing people’s faces raises additional privacy… It may make some participants uncomfortable or… All / most of the time Sometimes Rarely / never Not much support for my distraction theory
  34. 34. 50 people added comments  “The %age of my recommendations that were adopted went up by 60% when I started using videos”  “It's also invaluable to teach stakeholders who their users really are (breaking assumptions - for example, an early adopter can't be an older woman... actually, yes they can! See for yourself!)  “Creating video highlights is a time luxury that many (most?) researchers don't have.”  “I just do it because it's default behaviour in my recording software.”  “My teacher, umm you, said it wasn't that necessary and after these years I agree.”
  35. 35. Questions I find interesting  Is it possible that showing observers picture-in-picture video is actually a distraction?  There’s a lot to focus on when watching a usability test  What’s happening on the screen (clicking, scrolling, text entry, etc.)  What the user is saying about what they’re thinking, and how they’re saying it  Your own mental model of what the participant is doing and thinking
  36. 36. Audio trumps1 video  You can usually infer emotional state from voice  Clear audio reduces effort, people can focus longer  My priorities, in order  High quality audio  Large enough screen image  Participant face video (distant third) 1 Pardon the expression
  37. 37. “But won’t you miss non-verbal clues?”  I think you get the same information from audio  Facial expression and body language clues are almost always accompanied by tone-of-voice clues and/or pregnant pauses  A good facilitator will notice and call out important aspects of nonverbal behavior  “What are you thinking?”
  38. 38. We’ve got to consider privacy  It worries me that some people don’t seem to take privacy all that seriously  Is it possible to distribute clips and still maintain privacy?  Perv bars  No names, please (should be rule in any case)
  39. 39. What are you in it for?  I consider myself a bit of an outlier  I’m in it for insights  My charmed consultant life  I can just peddle insights and good ideas  I live with the Eloi and play in the sun
  40. 40. Members of the gentle surface-dwelling Eloi race - The Time Machine (1960)
  41. 41. Members of the ruthless subterranean Morlock race - The Time Machine (1960)
  42. 42. Eloi Yvette Mimieux - The Time Machine (1960)
  43. 43. Yvette Mimieux – “Tyger, Tyger” episode of Doctor Kildare (1964)
  44. 44. What are you in it for?  But many people are in it for persuasion  You may live in a Morloch world, where powers of persuasion are required  PPIP may be a much-needed, powerful tool for you
  45. 45. Anyone looking for a research project?  Comparative study of observer findings of the same test with and without PPIP  Eye tracking of some observers to determine time spent watching PPIP  I’d be happy to help  But you’d have to tolerate my bias for qualitative research
  46. 46. In conclusion…  Sometimes I wish I hadn't pulled on this thread  Who knew recording participants could be so complicated?
  47. 47. What I recommend  Do whatever works in your context  The same way I wouldn’t tell you not to do eye tracking, if it furthers the cause of usability in your organization  But do it consciously, with awareness of the tradeoffs  Be judicious about privacy issues  Seeing the participant does give you an additional useful sense of who they are  Possible alternative: Show participant to observers briefly at beginning of session  Check out these two books:
  48. 48. One final show of hands, please  Have you changed your mind at all about PPIP?
  49. 49. Photo © Jeff Jeffords www.divegallery.com
  50. 50. Thanks for all the fish  Send any questions, feedback, gripes to skrug@sensible.com @skrug or visit www.sensible.com
  51. 51. Q&A
  52. 52. © 2017 Steve Krug

×