Slides from a 5/10/2017 talk at the Nasdaq Entrepreneurial Center (@theCenter) about a lean research mindset, the mechanics of learning from users, and the structure of a research prototype test session.

1. Lean Learning
and Prototyping
Dave Hora
Research Lead, PlanGrid
dave.hora@plangrid.com
10 May 2017
Nasdaq Entrepreneurial Center Center
@theCenter

4. There are specific things we must
learn to push our work forward
but Joe & Josephine won’t help…

5. Research Mindset
Talking with Users
Risks & Assumptions
Targeting Users
Prototype
Test Script
A Test Set
Intros (me, you)
Questions (you)
1
2
3
4
Guiding Principles
Plan
Prepare
Execute

6. Intros

7. 1
2 3
4
5

10. You: what stage?

12. section
1
Guiding Principles
• Research Mindset (Applied)
• How to Talk With Users

13. Learn From People, Not Failure
• Start with our users—understand needs & context
• Ask focused questions that lead to smart choices
• Prioritize how you learn based on impact
• Never ask “do you like it?” (for research purposes)

14. Impact through Applied Outcomes
• Red flag: “doing research”
• Red flag: “learning a lot”

15. Behavior is the Strong Signal
• Don’t trust what they say
• Watch for visceral reactions
• See what they do—or have done recently

17. Cautionary Tales
• The Usability Report
• Project Gopher
• Product Manager Survey

19. Build to Learn
• Building creates alignment
• “to learn” allows you to let go
• Fast cycles when appropriate — “research spike”

21. Speaking with Users
• Target a representative audience
—behaviors, not demographics

22. 1. Ask open-ended questions
2. Ask for specific, past examples
3. Continue to ask why
4. Let the participant lead
5. Allow for uncomfortable silence
6. Never, ever, ask “do you like it?”

23. PRACTICE — 8 minutes
• You are developing the next
generation of breakfast foods
• …and starting a discovery process
• “How do people decide on breakfast?”
• “What factors are important to them?”

24. Past Specifics
• Red flags: “typically” and “generally” and “usually”
• It takes work to keep on track
• You have the right to direct the conversation

25. Asking why
• Force yourself to go 2 layers deeper
• Get at root motivations and surface larger context
• Allow yourself to seem stupid, or awkward

26. 1. Ask open-ended questions
2. Ask for specific, past examples
3. Continue to ask why
4. Let the participant lead
5. Allow for uncomfortable silence
6. Never, ever, ask “do you like it?”

27. section
2
Plan
• Risk / Assumptions
• Targeting Users

28. Objectives > Outcomes > R&A
• Our objectives stand on risks & assumptions
• We want to mitigate that risk, or plan around it
• As fast as possible, with the least amount of effort

29. Let’s talk about risks…
• Policy
• Technology
• Business
• Product
“...we have agreed to cease
operations immediately, ...
and hand over ownership of
our website, our mobile apps,
and intellectual property...”
https://arstechnica.com/tech-policy/2015/04/grooveshark-shuts-down-in-settlement-with-record-companies/

30. Let’s talk about risks…
• Policy
• Technology
• Business
• Product
“Something went wrong with
the launch,”NASA tweeted.
NASA astronaut Scott Kelly,
who is spending a year in space,
tweeted from orbit a universal
truth:“Space is hard.”
https://www.washingtonpost.com/national/health-science/failed-spacex-rocket-launch-another-setback-

31. Let’s talk about risks…
• Policy
• Technology
• Business
• Product
“One of its biggest problems
was the crippling cost of
customer acquisition.”
https://backchannel.com/why-homejoy-failed-bb0ab39d901a

32. Let’s talk about risks…
• Policy
• Technology
• Business
• Product

33. Product Risks
• People don’t see any benefit
• People won’t take time to learn how to use it
• People are confused, can’t use it
• It doesn’t meet needs in the real world

34. Look for Impact: ignore the fuzz
• Objectives
• Questions
• Risks
• Assumptions
• Hypotheses
• Confidence
What is going to push our
work forward the most?

35. Minimum Viable
• Given an objective (outcome)
• And hypothesis of value
• What is the smallest thing we can
build to validate that assumption?
“…the minimum viable
product is that version
of a new product which
allows a team to collect
the maximum amount
of validated learning
about customers with
the least effort.”
— Eric Ries
http://www.startuplessonslearned.com/2009/08/minimum-viable-product-guide.html

36. Focus:
• Find the right people who will exhibit the
behaviors you need to evaluate
• A rough match is better than nobody
• Highly specialized roles—can you proxy?

37. Finding the Right People
• Use friends & family & networks
• userinterviewing.com for gen pop
• respondent.io for professionals
• Remote is good, and OK!

39. section
3
Prepare
• Prototype
• Test Script

40. Plan Your Prototype
• “What do we need to see, to know X?”
• Risk-based scenario of use
• (hitting the right level takes a few tries)

44. Building Your Prototype
• Click-through:
• invisionapp.com
• marvelapp.com
• proto.io
• Paper
• Experience

46. A hybrid research session
• Learn about your user
• Learn about their problems and needs
• Test your concept / prototype
• Respect everyone’s time
http://customerdevlabs.com/2013/11/05/how-i-interview-customers/

47. Customer Development questions
• What’s the hardest part about [problem context]?
• Can you tell me about the last time that happened?
• Why was that hard?
• What, if anything, have you done to solve that problem?
• What don’t you love about the solutions you’ve tried?
http://customerdevlabs.com/2013/11/05/how-i-interview-customers/

48. Task-based Analysis
• What are things people try to accomplish in
your scenario of use?
• Can they accomplish this with your prototype?
• How do they react? Understanding? Value?

49. Think Aloud Protocol
• You’re going to ask them to try and do
something while thinking out loud about it
• (the envelope example)
• “Please keep speaking out loud.”

50. A session with users
• Get your user comfortable (background)
• Learn about their experience with problem scenario
• Deliver the tasks-on-prototype with Think Aloud
http://customerdevlabs.com/2013/11/05/how-i-interview-customers/

51. section
4
Execute
• Run a set of test sessions

52. Structure for Individual Session
• Short version of the CustDev interview
• User to accomplish a task with your prototype
• Reflect on past behavior change
• Short “topline” debrief with your team

53. Test your prototype + script
~30m individual session

54. Bring your team along
• Broadcast if possible
• Record if at all possible
• Minimum—1 other team member for notes
• Nothing matters without shared alignment

55. Debrief after every session
~30m individual session
• Intro
• CustDev question set
• Task on prototype
• Reflect on past
• Close
+ topline debrief
“If we forget everything and
come back in a month, what are
the interesting ideas and
questions to remember?”

56. WAIT a sec to change everything
• Adjust low-hanging fruit
• Don’t miss the orchard for the apples
• Your “test set” will surface deeper patterns

57. Structure for a Test Set
• Plan test: key questions, timing, recruiting
• Conduct research sessions—start with 4x
• run session according to plan
• MUST save a few minutes for topline debrief
• Run affinity debrief after running a set
• affinity, action items, next steps

58. Go until pattern convergence
plan on a minimum of 3-4 sessions
run an affinity debrief
session to “cut” your
insights and signal
learning + next steps

60. Structure for a Wrap-up Debrief
• Team crawl notes; pull topline + “bites” onto sticky notes
• Affinity mapping process:
• one at a time / out-loud
• groups as you see fit
• no names until reasonably formed
• re-arrange when groups too big
• Address each group; decide per-group action if necessary
zoom out & decide overall

61. Know when to cut
• Pattern convergence
• Don’t “learn a lot” — find what you need
• Make slightly provocative decisions and move on

62. Research Mindset
Talking with Users
Risks & Assumptions
Targeting Users
Prototype
Test Script
A Test Set
Intros (me, you)
Questions (you)
1
2
3
4
Guiding Principles
Plan
Prepare
Execute

63. Q&A

64. Lean Learning
and Prototyping
Dave Hora
Research Lead, PlanGrid
dave.hora@plangrid.com
10 May 2017
Nasdaq Entrepreneurial Center Center
@theCenter