Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Making Research Count

Originally delivered at An Event Apart 2019, a talk about how to make sure that the good user research you do is actually listened to and acted on.

  • Be the first to comment

  • Be the first to like this

Making Research Count

  1. 1. Making Research Count Cyd Harrell, AEA 2019.
  2. 2. “Full many a flower is born to blush unseen, And waste its sweetness on the desert air.” ― Thomas Gray AnElegyWrittenInACountryChurchyard insights studyso intranet. go unread ― Cyd
  3. 3. We know what we need to do research - Find participants - Obtain consent - Provide the prompt - Collect the response - Offer compensation - Bring home evidence - Take care of ourselves - Be good citizens
  4. 4. What do we need to make research matter? - Relationships - Budget - Participation - Tangible outcomes (maybe) count
  5. 5. If you’re - doing real research on a regular cadence - sharing it with people who matter - & they’re excited to hear it - because they know it helps them You’ve won 👏 If you’re - doing real research - they know it helps them
  6. 6. Research counts when It answers a question people care about.
  7. 7. You can explain - oddities in analytics - weird feedback from customers - unexpected uses of products - strange hunches (not just your own)
  8. 8. Curious people with power are the most useful ones to influence.
  9. 9. Open up your practice. - Don’t mystify about the work. - Put as much effort into communicating as into practicing. - Create opportunities for people to ask questions and learn.
  10. 10. Other people who make design decisions Communities of research practice Designers, researchers, product managers Team Discipline mates People affected by design decisions
  11. 11. Open up the research practice via - office hours - open Slack channel - brown bag program - newsletter - wall work
  12. 12. Research counts when People can see it for themselves.
  13. 13. Who in your organization is disconnected from their user?
  14. 14. “When someone has been in the field with you, the data doesn’t have to be explained.” ―Meena Kothandaraman, Twig+Fish
  15. 15. Invite your team along - Show them what you’re doing. - Make the rules clear. - Send them home with gifts.
  16. 16. Elements of an invitation - Who is doing the inviting? - What is the event? (What will happen there?) - When & where is it? - What will be provided? - What do guests need to bring? (Dress code? Potluck dish?)
  17. 17. Hey, we’re going to fire up the grill on Friday night, would you & Chris like to come by? Say, 6pm - we’ll have burgers & veggie options, bring a side or some beer if you can! Board games later, Jay’s going to bring Pandemic.
  18. 18. Make sure they want to come back - Design the experience of observing research. - Make sure everyone has tools. - Give everyone a responsibility.
  19. 19. People who get to ask a question buy in to the results.
  20. 20. Research counts when Methods fit the question
  21. 21. Mixing your methods - choosing interviews, diary methods, card sorting, shadowing, etc. - grounding the user research in landscape analysis, review, or comparison.
  22. 22. 2008: How do people really use phones in their cars?
  23. 23. Research methods & skills: - Ride-along interviews - Broadcasting - Observer management - Photography - Video editing
  24. 24. 2016: Best Practices in Digital Transformation
  25. 25. Research question: What are the best practices for government digital transformation? Method hypothesis: - Interviews?? - ¯_(ツ)_/¯
  26. 26. Better research question: What makes modern digital practices stick within a government entity, beyond a single innovation project?
  27. 27. Literature review: how did this work when companies were going online? New York Times, 1999
  28. 28. Interview goals: - What does digital transformation mean? - How do you know when you’re done? - What are the biggest obstacles to this work? - How do you make the changes last? (Or not) - What resources would be valuable for other agencies doing this work?
  29. 29. Cluster recruiting: “Who else in the organization had an important role in this moving forward or not?” 3 involved people at different levelsoffer a much more complete picture.
  30. 30. Research methods & skills: - moderated interviews - literature review - policy & regulation review - cluster recruiting - tight consent & confidentiality
  31. 31. Cutting the right corners at the right time Lets you be fast and cheap
  32. 32. Bare Bones (plus $20) Consent Notes Photos Prototype Incentives
  33. 33. Bare Bones Script & Score Sheet
  34. 34. Label your assumptions these are the only two categories They think this is at least somewhat hard 3 is a frustration threshold
  35. 35. Tracking it all for analysis After every interview or shadow - 1-sentence description of the person (photo if you can) - #1 surprise - Summary of 2-3 things they said (can be your scoresheet)
  36. 36. Research counts when Researchers tell the story well.
  37. 37. “Research is our product & the rest of our organization are our users.” ―Laurissa Wolfram-Hvass, MailChimp
  38. 38. Synthesis works best as a conversational practice.
  39. 39. Miller’s Law “In order to understand what another person is saying, you must assume that it is true, and try to imagine what it could be true of.”
  40. 40. You know the 5 whys If people talk about something five different ways, it’s virtually certain at least one of them will be an apt metaphor. “Can you say that another way?” Use the 5 ways as well
  41. 41. “Spend as much time on communicating outcomes as you did on executing the work.” ―Paul André, Facebook
  42. 42. Remember about tracking? After every interview or shadow - 1-sentence description of the person (photo if you can) - #1 surprise - Summary of 2-3 things they said (can be your scoresheet)
  43. 43. After every day of research (by email or Slack) - we talked with X customers - the most interesting thingwe saw/heard was… - themes that are emerging (or fading) - how we’ll pursue those tomorrow
  44. 44. After every study (by any means that works) - we talked with X customers - this is the questionwe all wanted to answer - this is the storyof what we learned together - here’s what we need to do
  45. 45. Ways to tell a story: - wall work - giant PDF reports (sometimes these really win) - edited video - backlog tickets - design artifacts - prototypes
  46. 46. 2014: walls at Government Digital Service in London
  47. 47. The single best case for research is an illuminating story.
  48. 48. “I don’t know technology but it doesn’t scare me. I’m shaking because this paperwork just gets to me - it’s terrifying.” ―Manuel at the Bakersfield courthouse
  49. 49. Sometimes you do have to fight (nicely).
  50. 50. BUT NO ONE HAS TIME FOR THAT It doesn’t fit the sprint.
  51. 51. Your 3-day research plan - Day 1: create testbed, write scripts & scoresheets, buy incentives - Day 2: go where the users are & intercept - Day 3 morning:synthesis session - Day 3 afternoon: host show-&-tell
  52. 52. BUT You can’t contact my list.
  53. 53. Research Participants - team members. - friends, not on your team. - target customers. - real users with recent experience. - real users with current needs. - real users with a need this hour. MoreReliableResults
  54. 54. Can’t openly contact customers? - Focus on your company’s sector more generally. - What groups of employees can stand in? - Where do people who need your product go? - Get topic-related stickers & biz cards - Consider using personal social media
  55. 55. BUT What’s the exact ROI?
  56. 56. Your $200 research budget - $60 incentives (Starbucks cards) - $40 equipment (clipboards, backup chargers, thumb drives) - $30 stickers or Moo cards - $70 parking, taxis, meals, etc.
  57. 57. Starting out cheap enough refocuses people from ROI to positive value.
  58. 58. BUT It’s not statistically significant
  59. 59. Remember: you can explain - oddities in analytics - weird feedback from customers - unexpected uses of products - strange hunches (not just your own) Statistics can’t do all that.🔥
  60. 60. BUT Couldn’t the intern do this?
  61. 61. If research is about - understanding the organization - making space for participation - refining questions - leading conversations 🚫 It’s unfair to ask the intern to do all that.
  62. 62. BUT You can always do researchHOW
  63. 63. No one seems interested in research - Make allies in Customer Service. - Find the most curious engineer on the team. - Put your questions on a wall. - Record a pain point & send a video to executives.
  64. 64. No budget for research - Patch the gaps with personal equipment. - Consider buying minor refreshments as incentives, with personal funds. - Go without incentives. - Try to get approval to use swag & business cards.
  65. 65. “Once an idea is out and about, it can’t be called back, silenced or erased. You can’t contain it, any more than you could put the head of a dandelion back together after the wind has scattered its seeds.” ―P.W. Catanese “Once an idea is out and about, you can’t contain it, any more than you could put the head of a dandelion back together after the wind has scattered its seeds.” ―P research Cyd
  66. 66. Questions?
  67. 67. Thank you for coming! @cydharrell any time you want to talk about this