1. Rand Fishkin, Wizard of Moz | @randfish | rand@moz.com
Why Great Marketers
Must Be Great Skeptics

2. This Presentation Is Online Here:
bit.ly/mozskeptics

3. Great Skepticism
Defining

4. I have some depressing news…

7. Does anyone in this room believe that the Earth
doesn’t revolve around the Sun?

9. The Earth (and everything
in the solar system,
including the Sun) revolves
around our system’s
gravitational Barycenter,
which is only sometimes
near the center of the Sun.

10. Let’s try a more
marketing-centric example...

11. In 2009, Conversion
Rate Experts built us a
new landing page, and
increased our
subscribers by nearly
25%. What did they do?
Via CRE’s Case Study

12. One of the most
commonly cited facts
about CRE’s work is the
“long landing page.”

13. The Crap Skeptic The Good Skeptic The Great Skeptic
Let’s change our
landing page to
be a long one
right now!
We should A/B
test a long
landing page in
our conversion
funnel.
How do we know
page length was
responsible? What
else changed?

14. The Crap Skeptic The Good Skeptic The Great Skeptic
“I do believe sadly it’s
going to take some
diseases coming back to
realize that we need to
change and develop
vaccines that are safe.”
“Listen, all magic is
scientific principals
presented like "mystical
hoodoo" which is fun,
but it's sort of
irresponsible.”
"The good thing about
science is that it's true
whether or not you
believe in it."

15. In fact, we’ve changed our
landing pages numerous
times to shorter versions and
seen equal success. Length, it
would seem, was not the
primary factor in this page’s
success.

16. What separates the crap, good, & great?

17. Assumes one belief-reinforcing
data point is evidence enough
Doesn’t question what’s truly
causal vs. merely correlated
Doesn’t seek to validate

18. Doesn’t make assumptions about
why a result occurred
Knows that correlation isn’t
necessarily causal
Validates assumptions w/ data

19. Seeks to discover the reasons
underlying the results
Knows that correlation
doesn’t imply causality
Thoroughly validates, but doesn’t
let imperfect knowledge stop
progress

20. Will more conversion tests
lead to better results?
Testing

21. Obviously the more tests we
run, the better we can
optimize our pages. We
need to build a “culture of
testing” around here.

22. Via Wordstream’s What is a Good Conversion Rate?

23. Via Wordstream’s What is a Good Conversion Rate?
Do Those Who Test More Really Perform Better?

24. Hmm… There’s no correlation
between those who run more
tests across more pages and
those who have higher
conversion rates. Maybe the
number of tests isn’t the right
goal.

25. Via Factors That Drive How Quickly You Can Run New Online Tests

26. Trust
Word of Mouth
Likability
Design
Associations
Word of Mouth
Amount of Pain
CTAs
UX
Effort Required
Process
Historical Experiences
Social Proof
Copywriting
CONVERSION DECISION
Timing
Discovery Path
Branding
Price
(it’s a complex process)

27. How do we know where our
conversion problems lie?

28. Ask Smart Questions to the Right People
Potential Customers
Who Didn’t Buy
Those Who Tried/Bought But
Didn’t Love It
Customers Who
Bought & Loved It
Professional,
demographic, &
psychographic
characteristics
Professional, demographic, &
psychographic characteristics
Professional,
demographic, &
psychographic
characteristics
What objections did you
have to buying?
What objections did you have;
how did you overcome them?
What objections did
you overcome; how?What would have made you
stay/love the product?
What would have made
you overcome them?
What do you love
most? Can we share?

29. We can start by targeting
the right kinds of
customers. Trying to please
everyone is a recipe for
disaster.

30. Our tests should be
focused around
overcoming the
objections of the people
who best match our
customer profiles

31. Testing button colors

32. Testing headlines, copy,
visuals, & form fields

33. Designing for how customers
think about their problems &
your solution

34. THIS!

35. Does telling users we
encrypt data scare them?
Security

36. Via Visual Website Optimizer
Could this
actually HURT
conversion?

37. Via Visual Website Optimizer

38. Via Visual Website Optimizer
A/B Test Results
They found that without the secure icon had over
400% improvement on conversions as compared
to having the image.
[Note: results ARE statistically significant]

39. We need to remove the
security messages on our
site ASAP!

40. We should test this.

41. Is this the most meaningful test we
can perform right now?
(I’m not saying it isn’t, just that we
should prioritize intelligently)

42. Via Kayak’s Most Interesting A/B Test
vs.

43. Via Kayak’s Most Interesting A/B Test
A/B Test Results
“So we decided to do our own experiment about this
and we actually found the opposite that when we
removed the messaging, people tended to book less.”
- Vinayak Ranade, Director of Engineering for Mobile, KAYAK

44. Good thing we
tested!
Good thing we
tested!
Your evidence is
no match for my
ignorance!

45. What should we expect
from sharing our content
on social media?
Social CTR

46. Just find the average social
CTRs and then try to match
them or do better. No
brainer.

47. Via Signup.to’s Analysis of CTR on Twitter

48. Via Signup.to’s Analysis of CTR on Twitter

51. 306/701 = 43.6%... WTF??

53. Phew! We’re not alone.
Via Chartbeat

54. Assuming social metrics and
engagement correlate was a
flawed assumption. We need
to find a better way to measure
and improve social sharing.

58. OK. We can create some
benchmarks based on these
numbers and their averages,
then work to improve them over
time.

59. That is an insane amount
of variability!

60. There are other factors at work
here. We need to understand
them before we can create
smart metrics or useful
expectations

61. Timing
Source
Audience Affinity
Formatting
Network-Created
Limitations to Visibility
Brand
Reach
Traffic
Engagement

62. Let’s start by examining the data
and impacts of timing.

63. Via Facebook Insights

64. Via Followerwonk

65. Via Google Analytics

66. There’s a lot of nuance, but we
can certainly see how
messages sent at certain times
reach different sizes and
populations of our audience.

67. Comparing a tweet or share sent
at 9am Pacific against tweets
and shares sent at 11pm Pacific
will give us misleading data.

68. But, we now know three things:
#1 - When our audience is online
#2 – Sharing just once is suboptimal
#3 – To be a great skeptic (and
marketer), we should attempt to
understand each of these inputs with
similar rigorousness

69. Do they work? Can we make
them more effective?
Share Buttons

70. After relentless testing, OKTrends found that the
following share buttons worked best:

73. OKTrends found that removing all but a
single button (the “like” on Facebook) had
the most positive effect.

74. And that waiting until the visitor had scrolled
to the bottom of the article produced the
highest number of actions

75. We should remove all our
social sharing buttons and
replace them with a single
slide-over social CTA for
Facebook likes!

76. Buzzfeed has also done a tremendous amount of social
button testing & optimization…

77. And sometimes they do this…

78. And sometimes this…

79. Is Buzzfeed still in testing mode?

80. Nope.
They’ve found it’s best to show different buttons
based on both the type of content and how you
reached the site.

81. OK… Well, then let’s do
that… Do it now!

82. Testing a small number of the
most impactful social button
changes should produce
enough evidence to give us a
direction to pursue.

83. Buzzfeed & OKTrends share several
unique qualities:
1) They have huge amounts of
social traffic
2) Social shares are integral to their
business model
3) The content they create is
optimized for social sharing

84. Unless we also fit a number of
these criteria, I have to ask again:
Is this the most meaningful test
we can perform right now?

85. BTW – it is true that testing social
buttons can coincide with a lot of
other tests (since it’s on content
vs. the funnel), but dev resources
and marketing bandwidth
probably are not infinite 

86. Does it still work better
than standard link text?
Anchor Text

87. Psh. Anchor text links
obviously work. Otherwise
Google wouldn’t be
penalizing all these sites for
getting them.

88. It has been a while since we’ve
seen a public test of anchor text.
And there’s no way to know for
sure how powerful it still is.

89. Testing in Google is very, very hard.
There’s so many confounding
variables – we’d have to choose our
criteria carefully and repeat the test
multiple times to feel confident of any
result.

90. 1) Three word, informational keyword phrase with relatively light
competition and stable rankings
Test Conditions:
2) We selected two results (“A” and “B”), ranking #13 (“A”) and
#20 ( “B”) in logged-out, non-personalized results
3) We pointed links from 20 pages on 20 unique, high-DA, high-
tr