1. Making
Research
Count
Cyd Harrell, AEA 2019.

2. “Full many a flower is born to blush unseen,
And waste its sweetness on the desert air.”
― Thomas Gray
AnElegyWrittenInACountryChurchyard
insights
studyso
intranet.
go unread
― Cyd

3. We know what we need to do research
- Find participants
- Obtain consent
- Provide the prompt
- Collect the response
- Offer compensation
- Bring home evidence
- Take care of ourselves
- Be good citizens

4. What do we need to make research matter?
- Relationships
- Budget
- Participation
- Tangible outcomes
(maybe)
count

6. If you’re
- doing real research on a regular cadence
- sharing it with people who matter
- & they’re excited to hear it
- because they know it helps them
You’ve won 👏
If you’re
- doing real research
- they know it helps them

7. Research counts when
It answers a question people care about.

8. You can explain
- oddities in analytics
- weird feedback from customers
- unexpected uses of products
- strange hunches (not just your own)

9. Curious people with power
are the most useful ones
to influence.

10. Open up your practice.
- Don’t mystify about the work.
- Put as much effort into communicating as into
practicing.
- Create opportunities for people to ask questions and
learn.

11. Other people who make
design decisions
Communities of research practice
Designers, researchers,
product managers
Team
Discipline
mates
People affected by
design decisions

12. Open up the research practice via
- office hours
- open Slack channel
- brown bag program
- newsletter
- wall work

13. Research counts when
People can see it for themselves.

14. Who in your organization
is disconnected
from their user?

15. “When someone has been in the field with
you, the data doesn’t have to be explained.”
―Meena Kothandaraman, Twig+Fish

16. Invite your team along
- Show them what you’re doing.
- Make the rules clear.
- Send them home with gifts.

17. Elements of an invitation
- Who is doing the inviting?
- What is the event? (What will happen there?)
- When & where is it?
- What will be provided?
- What do guests need to bring? (Dress code? Potluck dish?)

19. Hey, we’re going to fire up the grill on Friday
night, would you & Chris like to come by? Say,
6pm - we’ll have burgers & veggie options, bring
a side or some beer if you can! Board games
later, Jay’s going to bring Pandemic.

20. Make sure they want to come back
- Design the experience of observing research.
- Make sure everyone has tools.
- Give everyone a responsibility.

21. People who get to ask a question
buy in to the results.

22. Research counts when
Methods fit the question

23. Mixing your methods
- choosing interviews, diary
methods, card sorting,
shadowing, etc.
- grounding the user research
in landscape analysis,
review, or comparison.

24. 2008: How do people really use
phones in their cars?

25. Research methods & skills:
- Ride-along interviews
- Broadcasting
- Observer management
- Photography
- Video editing

26. 2016: Best Practices in Digital
Transformation

27. Research question:
What are the best practices for government digital
transformation?
Method hypothesis:
- Interviews??
- ¯_(ツ)_/¯

28. Better research question:
What makes modern digital practices stick within a
government entity, beyond a single innovation
project?

29. Literature review:
how did this work when
companies were going
online?
New York Times, 1999 https://www.nytimes.com/1999/12/23/business/toys-r-us-falls-behind-on-shipping.html

30. Interview goals:
- What does digital transformation mean?
- How do you know when you’re done?
- What are the biggest obstacles to this work?
- How do you make the changes last? (Or not)
- What resources would be valuable for other
agencies doing this work?

31. Cluster recruiting:
“Who else in the organization had an important
role in this moving forward or not?”
3 involved people at different levelsoffer a
much more complete picture.

32. Research methods & skills:
- moderated interviews
- literature review
- policy & regulation review
- cluster recruiting
- tight consent & confidentiality

33. Cutting the right corners
at the right time
Lets you be fast and cheap

34. Bare Bones
(plus $20)
Consent
Notes
Photos
Prototype
Incentives

35. Bare Bones Script & Score Sheet

36. Label your assumptions
these are the only two categories
They think this is at least
somewhat hard
3 is a frustration threshold

37. Tracking it all for analysis
After every interview or shadow
- 1-sentence description of the person
(photo if you can)
- #1 surprise
- Summary of 2-3 things they said
(can be your scoresheet)

38. Research counts when
Researchers tell the story well.

39. “Research is our product & the rest of our
organization are our users.”
―Laurissa Wolfram-Hvass, MailChimp

40. Synthesis works best as
a conversational practice.

41. Miller’s Law
“In order to understand what another person is saying,
you must assume that it is true, and try to imagine what
it could be true of.”

42. You know the 5 whys
If people talk about something five different ways, it’s
virtually certain at least one of them will be an apt
metaphor.
“Can you say that another way?”
Use the 5 ways as well

43. “Spend as much time on communicating
outcomes as you did on executing the work.”
―Paul André, Facebook

44. Remember about tracking?
After every interview or shadow
- 1-sentence description of the person
(photo if you can)
- #1 surprise
- Summary of 2-3 things they said
(can be your scoresheet)

45. After every day of research (by email or Slack)
- we talked with X customers
- the most interesting thingwe saw/heard was…
- themes that are emerging (or fading)
- how we’ll pursue those tomorrow

46. After every study (by any means that works)
- we talked with X customers
- this is the questionwe all wanted to answer
- this is the storyof what we learned together
- here’s what we need to do

47. Ways to tell a story:
- wall work
- giant PDF reports (sometimes these really win)
- edited video
- backlog tickets
- design artifacts
- prototypes

48. 2014: walls at Government Digital Service
in London
https://gds.blog.gov.uk/2014/09/03/vertical-campfires-our-user-research-walls/

49. The single best case for research
is an illuminating story.

50. “I don’t know technology but it doesn’t scare
me. I’m shaking because this paperwork just
gets to me - it’s terrifying.”
―Manuel at the Bakersfield courthouse

51. Sometimes you do have to fight
(nicely).

52. BUT
NO ONE
HAS TIME
FOR THAT
It doesn’t
fit the
sprint.

53. Your 3-day research plan
- Day 1: create testbed, write scripts &
scoresheets, buy incentives
- Day 2: go where the users are & intercept
- Day 3 morning:synthesis session
- Day 3 afternoon: host show-&-tell

54. BUT
You can’t
contact
my list.

55. Research Participants
- team members.
- friends, not on your team.
- target customers.
- real users with recent experience.
- real users with current needs.
- real users with a need this hour.
MoreReliableResults

56. Can’t openly contact customers?
- Focus on your company’s sector more generally.
- What groups of employees can stand in?
- Where do people who need your product go?
- Get topic-related stickers & biz cards
- Consider using personal social media

57. BUT
What’s the
exact ROI?

58. Your $200 research budget
- $60 incentives (Starbucks cards)
- $40 equipment
(clipboards, backup chargers, thumb drives)
- $30 stickers or Moo cards
- $70 parking, taxis, meals, etc.

59. Starting out cheap enough
refocuses people from ROI
to positive value.

60. BUT
It’s not
statistically
significant

61. Remember: you can explain
- oddities in analytics
- weird feedback from customers
- unexpected uses of products
- strange hunches (not just your own)
Statistics can’t do all that.🔥

62. BUT
Couldn’t
the intern
do this?

63. If research is about
- understanding the organization
- making space for participation
- refining questions
- leading conversations
🚫 It’s unfair to ask the intern to do all that.

64. BUT
You can
always do
researchHOW

65. No one seems interested in research
- Make allies in Customer Service.
- Find the most curious engineer on the team.
- Put your questions on a wall.
- Record a pain point & send a video to
executives.

66. No budget for research
- Patch the gaps with personal equipment.
- Consider buying minor refreshments as incentives,
with personal funds.
- Go without incentives.
- Try to get approval to use swag & business cards.

68. “Once an idea is out and about, it can’t be
called back, silenced or erased. You can’t
contain it, any more than you could put the
head of a dandelion back together after the
wind has scattered its seeds.”
―P.W. Catanese
“Once an idea is out and about,
you can’t contain it, any more than you could
put the head of a dandelion back together
after the wind has scattered its seeds.”
―P
research
Cyd

69. Questions?

70. Thank you
for coming!
@cydharrell any time you want
to talk about this