In this talk from Bunnyfoot founder Dr Jon Dodd at Unboxed 2019 (held at the Curzon cinema on the 16th of October), he explained how to determine if your customer research was giving you good data. Jon showed how accurate customer testing is a vital part of user-centred design and how the data from it reduces knowledge gap between customer and management expectations. It can help any marketing campaign - but only if we understand our limitations and what good data looks like.

2. HOW TO SPOT GOOD FROM BAD RESEARCH
16th October 2019
FOR MARKETING, UX, &
ANY CUSTOMER INTERACTION

3. 3
1999-2019 CEO Bunnyfoot
Hi I’M JON
1990-1999 Neuroscience researcher (and Guinea pig)

4. 4
A SUCCESSFUL FRAMEWORK: USER CENTRED DESIGN
An established process from the software
industry – the underpinning of UXD, Service
design etc.
THE IMPORTANT THING IS:
Consideration of, and engagement with, your
audience at each stage of the development
Essentially driving design decisions based on
evidence rather than assumption
aka ‘Evidence based design’

5. 5
IN USER CENTRED DESIGN
learn about your
Business
¡ Business requirements/
stakeholder workshop
¡ Participatory
design/concept testing
¡ Strategy and goal
workshops
¡ Requirement analysis
planning
¡ Business to customer
uncovery
¡ Desk research
learn about your
Competitors
¡ Customer research/
benchmarking
¡ Best practice
analysis/feature analysis
learn about your
Customers
¡ Behaviour observation
§ Observed browsing
§ Customer Inquiry
§ Ethnography
§ Competitor testing
¡ Diary studies
¡ Audience surveys
¡ Depth interviews
¡ User/Customer requirement
workshop (focus groups)
define the
Information
architecture
¡ Information architecture
development
¡ Content strategy
develop the
Interaction design
¡ Paper prototyping
¡ Iterative wireframe
development
¡ Clickable prototype
development
document with
Blueprints /instructions
¡ Information architecture
blueprints
¡ Interaction design
documentation
§ Wireframes
§ Templates
§ Elements
long term
Support
¡ ‘Rent-a-guru’ ongoing ‘’on
call’ immediate access to
our experts
¡ Workshop, training and
seminars
¡ Policy documentation
regular
Monitoring
¡ Advanced visitor analysis
¡ Benchmark quantitative
testing
¡ Competitor
monitoring/trend spotting
¡ Social media monitoring
¡ Satisfaction surveys and
interviews
ongoing
Optimisation
¡ A/B testing
¡ Multi-variate testing
¡ Updated customer
intelligence
¡ Design Modification
updates
¡ Persuasion auditing
define
Customer Models
¡ Persona development
¡ Scenario and use case
setting
explore
Customer interaction
¡ Content audit and analysis
¡ Ideation/concepting
¡ Participatory design
workshop
¡ Mental modeling
¡ Mental model/content
model
develop
Testable concepts
¡ Concept rationalisation
¡ Information architecture
concept generation
¡ Storyboarding
¡ Lo-fi prototype sketching
research
Creative concepts
¡ Creative research
exploration
¡ Creative workshop
evolve and
Develop the creative
¡ Creative concept
generation
¡ Creative application
§ Templates
§ Elements
develop
Design guides/assets
¡ Design guide production
¡ Design asset production
§ Layered psd, png, etc,
§ Presentation layer code
RESEARCH MODEL ARCHITECT DESIGN OPTIMISE
¡ Expert usability eval
¡ User testing
¡ Accessibility audit
¡ Prototype testing
¡ Eye tracking
¡ Card sorting
¡ Navigation testing
¡ Look and feel testing
¡ Remote testing
¡ Analytics
…and more tailor made for you
TESTING AND EVALUATION

6. 6
4D OR DOUBLE DIAMOND

7. 7
DESIGN THINKING…

8. 8
IN SERVICE DESIGN
Management
expectations
Customer
expectations
Knowledge gap Do your research to reduce this
gap
Management
expectations
Service
specification
Try to set spec to meet customer
expectations
Policy gap
Service
Specification
Service
delivery
Attempt to deliver the spec
(resources etc.)
Delivery gap
Service
delivery
Customer
coms
Communication realistically and
truthfully
– help set expectations
Communication gap
Note: this is a model we use called the gap model to help frame service design
Customer
perception
Overall goal:
Reduce this gap!
but be careful with
exceeding
expectations
Reduce the knowledge gap
this is the difference between what the org thinks its users/customers want and need – and what they actually need.

9. 9
IN DESIGN SPRINTS
UXBoost
Define the
problem &
start the UX
Canvas
Learn from
others &
develop ideas
Rapidly
evolve ideas
& produce
visualisation(s)
Build suitable
prototype(s)
Test &
determine
next steps
Monday Tuesday Wednesday Thursday Friday

10. 10

11. 11
TO SOME RESEARCH SEEMS SIMPLE….
…but that is where some of the problems lay…

12. 12
BIG DATA PEOPLE BEWARE OF LOCAL MAXIMA

13. 13
JUNK IN – JUNK OUT

14. 14
ANYONE RECOGNISE THIS?

15. 15
THE BIG PICTURE
Kruger & Dunning 1999
…people who lack the knowledge or wisdom to
perform well are often unaware of this fact.
We attribute this lack of awareness to a deficit in metacognitive
skill. That is, the same incompetence that leads them to make
wrong choices also deprives them of the savvy necessary to
recognize competence, be it their own or anyone else’s.
“

16. 16
PERHAPS THE POSTER CHILD OF THE MOMENT…

17. 17

18. 18
A NICE ONE

19. 19
…EASIER TO REMEMBER AND MORE POINTED

20. 20
AND HAS BEEN APPLIED TO MANY INDUSTRIES

21. 21
IT’S ACTUALLYA BIT MORE COMPLICATED THAN THAT…
…which is a bit ironic…

22. 22
BUTA USEFUL TOOL…
Essentially all models are
wrong – but some are useful
George E P Box
“

23. 23
ASK YOURSELF REGULARLY – WHERE ARE YOU?
UX and research etc.
I hope I am here – every
day a school day
I did stand-up comedy
(4 years later I have
booked on a new
comedy writing course)
I’m about to
do my fourth
marathon

24. 24
This proves this
RESEARCH: SOME THINGS I HEAR REGULARLY
We have loads of
[survey] data
ask them this…
We need more
people (a bigger
sample)
I want [eyetracking, focus group,
interviews…]
You’re an expert –
why do we need
research
That wasn’t a good test…
(moderator didn’t say enough)
(a child kept running in)
But we know our
customers
Just ask them what
they think about
this

25. 25
…AND IN DESIGN (PARTICULARLY INTERACTION & SERVICE DESIGN)

26. 26
RESEARCH: WHAT WE WOULD LIKE TO SEE MORE OF
What’s the best
approach(s)…
These are our
research
objectives
How can we
improve…
This is our problem
Every day is a
school day

27. 27
You are not your audience/user/customer *
A FUNDAMENTAL TRUTH
You do not…
¡ See things like they do
¡ Know what they know
¡ Want what they want
¡ Work how they work
This is critical information when designing a product or service
…and btw your bosses have even less clue than you do…
* delete as appropriate!
…that you will often have to remind yourself and your colleagues about

28. 28

29. 29
MORE FROM WHAT KIDS CAN TEACH US…

30. 30
DON’T BASE DECISIONS ON ASSUMPTIONS
Don’t Make Me Think (Revisited), 2014
It’s only natural to assume that everyone uses
the Web the same way we do, and - like everyone
else – we tend to think that our own behaviour is
much more orderly and sensible than it really is
Steve Krug, 2014
“
Give this book a read, or buy
it for your boss

31. 31
GARBAGE IN – GARBAGE OUT
It is essential to choose the
right methodology !

32. 32
THE STORY OF THE HEALTHY HAMBURGER?

33. 33
THE STORY OF THE HEALTHY HAMBURGER!
Q?
So what would
be the correct
approach?

34. 34
ANOTHER EXAMPLE
If I asked you what you looked
at – would you lie?
Be truthful!

35. 35
YOUR CHALLENGE
Which is from 30 Men?
Which is from 30 women?

36. 36
ITS JUST PAST MIDDAY...

37. 37
FILL IN THE BLANKS

38. 38

39. 39
ULTIMATELY...
People make decisions in the unconscious...
People don’t remember...
People lie (confabulate)...
People post rationalise...
People hate to appear stupid...
People are bad at introspection...
People will be kind to you and tell you what they think you want to
hear

40. 40
and…most of our behaviour including ‘complex’
behaviour is non-conscious (sub-conscious)
- So we need to design (and design our
research) with this in mind

41. 41
THIS OLD CHESTNUT...
If I had asked people what they wanted...
...they would have said faster horses
Henry Ford

42. 42
OR EVEN WORSE...

43. 43
YOU DON’T WANT TO HEAR FROM THE PRESS OFFICE (CONSCIOUSNESS, SPIN…)

44. 44
YOU WANT TO ACCESS THE OVAL OFFICE… WHERE DECISIONS ARE REALLY MADE

45. 45
…OR MAYBE NOT

46. 46
A MORE UP TO DATE ANALOGY

47. 47
INTERACTIVE EXERCISE: RESEARCH FRAMEWORK/SPECTRUM…
Shout out some research techniques that you use, have used, or have at least heard of….

48. 48
HERE IS ONE THEY PREPARED EARLIER…

49. 49
MORE WAYS TO LOOK AT THEM…

50. 50
ACROSS THE LIFECYCLE…

51. 51
LETS LEARN FROM SOME BLOOPERS & HORROR STORIES
Every day is a school day…

52. 52
RESEARCHING THE WRONG PEOPLE…
Under-specification
Over-specification
Bias
What about the ‘unknown unknowns’?

53. 53
RESEARCHING THE WRONG PEOPLE…
‘Professional’ research participants
Fakes
Dunning Kruger sufferers (usage, expertise)
The lovelorn
(the smelly)

54. 54
RESEARCHING THE RIGHT PEOPLE…
Take recruitment very seriously
Test screeners and modify appropriately
Know the flex points
Go where they flock

55. 55
FALLING IN LOVE WITH A METHOD – IF YOU HAVE A HAMMER EVERYTHING IS A NAIL…
The tyranny of eyetracking
The ease of the survey deployment
The fame of the focus group

56. 56
= BE PROMISCUOUS WITH METHODS
Start with the research objectives
Use a mix of methods
Be innovative

57. 57
TALKING TOO MUCH
Asking too many questions
Interrupting flow
(leading Qs, closed Qs etc.)

58. 58
= WATCH, LISTEN & LEARN
Think very carefully before speaking
Power of silence
Practice, checklists etc.

59. 59
ROBOTIC ADHERENCE TO A SCRIPT
Soulless Q & A
No empathy
No Engagement
No discovery

60. 60
= BE FLEXIBLE, BE INTERESTED
Be prepared (and confident) to explore
Be interested (and look like you are)
Have empathy
Respond appropriately

61. 61
THE WRONG NUMBER
Often misinterpreted
Too many = wasted ROI = too much to
analyse (paralysis through analysis)
Scientific significance v trend v
confidence

62. 62
THE RIGHT NUMBER…
Rules of thumb for qual
Calculators for quant
Repeats

63. 63
AND LOTS MORE TO WATCH OUT FOR
Confirmation bias (selective nature for research particularly important)
Expectation (experimentors/observer) bias (along with confirmation bias dangerous for the truth)
Availability heuristic (take notes, independent observers, group analysis)
Courtesy bias (look out for people doing this – occurs a lot in focus groups, at the end of user testing)
Curse of knowledge (especially important in research – domain experts lack empathy for users)
Group think (focus groups especially)
Interoceptive bias (well fed etc. changes outlook of researcher/judge)
Mere exposure effect (familiarity breeds… in this case positivity – important in ratings etc. e.g. satisfaction)
False consensus effect (you are not your user… and neither is your husband, wife, mother etc…)

64. 64
CONCLUSIONS
Assess where you are on the curve
Every day is a school day
Observe behaviour where possible
Use mix methods and be flexible
Avoid and learn from bloopers

65. 65
CONCLUSION
Communicating and interacting with customers and ‘converting them’ is a complex proposition
Only by understanding them and their needs can we hope to support their goals and influence their decision
making
Question your existing methods and existing suppliers…
Using a mix of methods is important
¡ choose from the matrix of techniques (and both big and little data) and apply appropriately at the
appropriate time
¡ Don’t fall in love with the method or an output
¡ Focus on research objectives and outcomes (outcomes not outputs!)