Slides from Learn Inbound 2016 in Dublin, exploring the problems with bad measurement and how this affects strategy.

1. @philnottingham

3. Boston, Massachusetts is
a lovely place, but it has a
number of problems.

4. Chief among these
problems is…

6. The mayor’s office had
a bright idea to fix this

8. And it worked
incredibly well!

10. Except there was one
problem…

12. The app was used primarily by
Boston techies, who have a lot
more money than the majority
of the population

13. And so the data suggested
that the potholes in the rich
parts of the city were causing
the most problems

14. GARY KING, HARVARD UNIVERSITY
“This was an example of really bad analytics
and it’s even worse because it’s the kind of
thing that feels like it should work and does
work a little bit.”

15. What they think they’re solving for
The most problematic potholes

16. What they’re actually solving for
Potholes near rich people

17. What they should be measuring
Potholes in areas with highest density of traffic

19. When we don’t know how to
properly read and interpret
data, gathering it often causes
more problems that it solves

20. I think this is an endemic
issue in digital marketing

21. 1 Valuing everything by ROI
3 Problems

22. 1 Valuing everything by ROI
2 Treating all metrics as KPIs
3 Problems

23. 1 Valuing everything by ROI
2 Treating all metrics as KPIs
3 Misaligning goals, channels and content
3 Problems

24. 1
Valuing everything by ROI

25. (Gain from investment - Cost of investment)
Cost of investment
ROI =

26. What you think you’re solving for
Efficiency & efficacy of spend

27. So, you try to put a monetary
value on all activities

28. The most common, and
egregious way of doing this is
with last touch attribution

29. Session 1 Session 2 Session 3 Session 4
Last Touch Attribution

30. When you use last-touch
attribution, this is how you
end up valuing the different
parts of your website…

31. Product Support Blog Pricing Contact
Pretty
good!
Meh?
Rubbish
Great!
Amazing!!!

32. MATTHEW ZOOK, UNIVERSITY OF KENTUCKY
“Once something gets measured and put into
a model then everything starts optimising
around that metric.”

34. No-one in their right minds would read this
blog post and then buy a car insurance
policy, but when you value your content by
how it contributes to conversions… then you
end up trying to make that happen.

35. A slightly better way to
measure value is a linear or
time decay model

36. Session 1 Session 2 Session 3 Session 4
Linear Attribution

37. What you think you’re solving for
Efficiency & efficacy of spend

38. What you’re actually solving for
Conversions

39. But this still has issues with
any activity designed for
brand building.

40. Lets take the example of PR

42. Earlier this year, my boss Chris
Savage was on BBC news

43. Gut feel tells you that this is
really good PR coverage

44. Here’s what the return looked
like in Google Analytics

46. 500 visits
0 Conversions
~$10,000

47. Even with linear or even first
touch attribution, this looks like
incredibly poor return

49. But… maybe we can measure
the value of this TV spot instead
as if it were a bit of TV coverage

51. The way TV was historically
measured was via impressions
and how it contributed to brand
salience (as determined by
recall surveys)

52. This was an ok way to measure
TV 30 years ago, when there
wasn’t many other marketing or
advertising channels available

53. But today… there’s just too
much noise to make such data
meaningful. Brands who still
measure in this way
dramatically over value TV

54. AN ALL TOO COMMON RESPONSE TO THIS PROBLEM
“Lets temporarily stop all other brand activities so
we can properly measure PR…”

55. But we know that PR, or advertising,
or other brand building activity never
works in isolation. Stopping
everything necessarily reduces the
efficacy of the activity you’re trying
to measure.

56. So we’re left with the conclusion
that…

57. ROI is just a bad way to measure PR

58. Think of it like a soccer team.

59. #WistiaFest / @philnottingham

60. “ROI” in a soccer analogy is the
number of goals scored.

61. However… you wouldn’t measure
every player on how many goals they
score, or indeed even how they
contribute in terms of assists.

62. Goal keepers are measured by saves,
defenders by tackles, Midfielders by
passes up the field, and strikers by
goals and assists.

63. Yet you need a well rounded team,
with all the different players, to be
successful.

64. If you measure everything by “ROI”,
you’re building a team of strikers.

65. THE WORST CLIENTS OF ANY SEO AGENCY
“How much is this link worth?”

66. Anyone in SEO who’s had this
question knows how problematic
and widespread this mentality is.

67. What you think you’re solving for
Efficiency & efficacy of spend

68. What you’re actually solving for
Conversions

69. What you should be measuring
A sensible proxy for ‘value’

70. Effectiveness
Sum (DA^2)
Efficiency
KPIs for PR
Sum (DA^2)
Cost

71. 2
Treating all metrics as KPIs

74. Everything needs to go up. We need
to get more conversions, more
traffic, more links, lower bounce-
rate…longer time on page etc…

75. What you think you’re solving for
Optimising breadth of value

76. Here’s an example from a brand I
was working with recently.

78. They appreciated the potential value
YouTube could bring to their
marketing, and so tried to optimise
their activity for all possible value

79. Views
Retention
Shares
Subscribes
Click-Throughs
Conversions
TrueView Advertising
Shorten Video
Include Share CTA
Include Subscribe CTA
Include Link to Website
Track Click-Through Conversions

80. What you think you’re solving for
Optimising breadth of value

81. This created a frankenstein’s
monster of a video that had no value.

82. What you’re actually solving for
Nothing

83. What you should be measuring
One specific, qualified variable

84. Here’s how we’ve tried to do this at
Wistia

87. Effectiveness
View-through Conversions
Efficiency
KPIs for YouTube Advertising
View-through Conversions
Cost

88. 3
Misaligning goals, channels and content

89. Here’s a strategy I was recently
handed by a global insurance
business

90. Strategy
Industry Finance
Audience Millenials
Content Explainer Videos
Channel LinkedIn
Goal Sign-ups
Distribution Budget £1M

91. What you think you’re solving for
Integrated Strategy

92. Channel
ContentAudience

93. The problem is that the strategy had
no overlap in the channel, the
audience or the content

94. Strategy
Industry Finance
Audience Millenials
Content Explainer Videos
Channel LinkedIn
Goal Sign-ups
Distribution Budget £1M

95. No amount of budget in the world
can make this strategy work.

96. What you’re actually solving for
Discrete, unconnected and fragmented problems

97. What you should be doing
Creating content on channels appropriate for
the goal and audience

98. Goal
Audience
Channel
Content
Distribution
Metrics
KPI
What are you trying to achieve?
Who are you trying to reach?
Where can you reach them?
How can you meaningfully engage on this channel?
How can you get your content to the right people?
What can you accurately measure?
What’s a good proxy to measure the goal?
Strategy Questionnaire

99. Here’s an example of how we
recently applied this plan to
Facebook video

101. Goal
Audience
Channel
Content
Distribution
Metrics
KPI
Strategy Questionnaire
Brand Awareness
People working on video for social media
Facebook
Competition

102. We created a test video to see if
there was any latent demand for this
kind of content.

104. It worked!

105. Goal
Audience
Channel
Content
Distribution
Metrics
KPI
Strategy Questionnaire
Brand Awareness
People working on video for social media
Facebook
Competition
Promoted Posts, Word of Mouth

106. We then encouraged further
distribution with a bit of light
influencer marketing

108. Goal
Audience
Channel
Content
Distribution
Metrics
KPI
Strategy Questionnaire
Brand Awareness
People working on video for social media
Facebook
Competition
Promoted Posts, Word of Mouth
Views, Audio, Retention, Likes, Comments, Shares
Page Likes

109. Effectiveness
Page Likes
Efficiency
KPIs for Facebook Competitions
Page Likes
Cost

110. This lead to a 30% increase in
overall page likes over the duration
of the campaign, at a cost of about
$2 per like.

111. • Bad Data < No Data
• Measure everything, but always optimise for just one KPI
• Only use ROI for direct response & conversion activities
• Ensure alignment of content with channel and audience
• Tie everything back to the goal
Takeaways

112. Thanks!
Please feel free to follow
me on the Twitters!
@philnottingham