How Analytics Improve Marketing Experiments

Applied Marketing Analytics
Paramore University 2013

About MarketingSherpa
• Established 13 years ago

• Independent research and
publishing organization
focused on the marketing
community

• From 2008 to 2012:
• 36,980 companies and
marketers surveyed
(cumulative)
• 3831 charts and tables
• 4,847 pages of insights and
analysis
• 1,857 pages of research
supported tactics and
recommended actions

About MECLABS
• Parent organization of
MarketingSherpa and other
research brands

• World’s largest independent
research lab focused
exclusively on marketing and
sales
• More than 15 years of research
partnership with our clients
• 1,300 experiments
• Over 1 billion emails tested
• 10,000 landing pages tested
• 5 million telephone calls
• 500,000 decision maker
conversations

Questions?

#ParamoreU

Let’s talk about…

Messaging

How would your team improve the messaging?

What’s going into our marketing decisions…
Instead of analytics data to make marketing decisions, we rely on:

Gut instincts Our own Intuition
Historical spending The status quo
Testing What works
Brand awareness The need to be known
Brand perception The need to be loved
Purchase intention What’s been decided

Willingness to What’s recommended
recommend

HIPPO (Highest paid What’s commanded
person’s opinion)
Other

Source: ©2012 MarketingSherpa Marketing Analytics Benchmark Survey
Methodology: Fielded November 2012, N= 682

How it changes based on role
Instead of analytics data to make marketing decisions, we rely on:

The Troops My What
Nonmanagement marketing
personnel
Need to be known
Gut Works

The Middle Mgr Be My What What we
Marketing manager or
supervisor
Known Gut Works already did

The Chief Need to My What What we
Chief Marketing Officer or
Senior Executive
be known Gut Works Already did

Brand awareness Gut instincts Testing Historical Spending


How would your team improve the messaging?

Improving the
message
should be no
big deal,
right?

Case Study: Background

 Case Study ID: Pier 1 Imports
Protocol Number: A-TP1002

Research Notes:
• Background: B2C home products company with a significant online and
retail presence

• Goal: To increase the click-through rate from the email to the landing page

• Primary research question: Which email design will generate the most
click-throughs?

• Approach: A/B/C/D split test (variable cluster)

Case Study: Campaign Designs
Team A

Original

Team B

Original

Team C

Original

Case Study: Which team would you vote for?

Team A Team B Team C

Original

Case Study: Results

52% Decrease in Clickthrough
Team A’s design decreased clicks by 51.8%

Email Designs CTR Rel. Diff.
Original 36.70% -
Team A 17.68% -51.83%
Team B 29.91% -18.50%
Team C 24.07% -34.41%


 Case Study ID: Protected
Protocol Number: TP1457

Research Notes:
• Background: B2C, B2B tax services brand with both online and offline
products

• Goal: To increase online product purchases

• Primary research question: Which e-commerce product detail page will
produce more purchases of the product being showcased?

• Approach: A/B split test (variable cluster)

Case Study: Challenge
About the original:

• Standard e-commerce
style product page

• Call to action and product
imagery above the fold

• Supporting information
tabbed and organized

Case Study: Campaign
About the new design:

• Completely broken mold
with heavy design
elements

• Call to action BELOW the
fold

• Product imagery totally
eliminated

Case Study: Results

83% Increase in Purchases
The new design increased purchases by 83.79%

Product Page Version Product Conv. Rate

Control 8.27%
Double Control 9.93%
Treatment 1 18.25%
Relative Difference: 83.79%

And the moral of the study is…

83%
52% 19% 34%

And the moral of the study is…

“To know what people really think, pay regard
to what they do, rather than what they say.”
- René Descartes
83%
52% 19% 34%

Key Point
• There are no expert marketers, there are just experienced marketers and
expert testers

Key Point
expert testers

• There is a critical element in the testing and optimization process that you
can access now and use without 10 hours of teaching and 10 weeks of
systems changes

To run a good test…

Research
Question Treatments Metrics Validity
(Hypothesis)

You need a premise…

Research
Question Treatments Metrics Validity
(Hypothesis)


“
Which sources of information do you

”
actively use to better understand your
prospects and customers?

Research
Question
(Hypothesis)



“
Which sources of information do you

”
actively use to better understand your
prospects and customers?

Research
The backbone of expert testing
Question
(Hypothesis)
is analytics examination


Case Study: Observation
Experiments designed with the strategic use of analytics examination on average
produced more valuable results compared to those that did not

• 2010 Homepage Tests
Test 2 (questionnaire style, -63%) vs. Test 1 (images, 0%)
Test 5 (reducing process friction, +22%) vs. Test 4 (flash treatment, -80%)

• 2010 Online Product Page Tests
Test 3 (product qualifiers emphasis, +9%) vs. Test 1 (page arrangement, 0%)

• 2010 PPC Landing Page Tests
Retail 2 (reducing process friction, +533%) vs. Retail 1 (above/below fold, 8%)
Shared 3 (reducing process friction, +34%) vs. Shared 1 (page arrangement, 0%)

• 2010 Banner Ad Landing Page Tests
Test 2 (product specific value, +28%) vs. Test 1 (page alignment, -62%)

How helpful are analytics, anyway?

Faster growth of our business (40%) Competitive advantage (37%)

Common basis for decision making (47%)
Better marketing Better utilization
of resources (43%)
messages (67%) Better risk management (19%)

More accurate and precise response to
More complete understanding of market customer needs (44%)
conditions and trends (40%)
Predicting customer behavior (38%)
Complete understanding of the marketing purchase cycle (37%)


How helpful are analytics, anyway?

Faster growth of our business (40%) Competitive advantage (37%)

Common basis for decision making (47%)
Better don’t need a test toutilization
You
marketing Better
of resources (43%)
messages (67%)
leverage your analytics Better risk management (19%)

More accurate and precise response to
More complete understanding of market customer needs (44%)
conditions and trends (40%)
Predicting customer behavior (38%)
Complete understanding of the marketing purchase cycle (37%)


How much analytics does your org collect?

14%
Vast quantities

17%
Limited

25%
40%
Average
79%
Average or more
Significant

How much analytics does your org collect?

14%
Vast quantities

Limited

25%
40%
Average
79%
Analytics are available in the
17% majority of organizations
Average or more
Significant

So what’s the problem, then?

52% 19% 34%

Are you able to leverage your org’s analytics?

9% 2%access
No
No tools,
6% skills
Rarely 37%
Routinely
& Effectively
46%
Occasionally

The only exception is Marketing Agencies
or Consultancies

Maybe it’s just too much to process…

Maybe you’ve never been the data type…

If only this
calculator had
excel built into
it…

Maybe you feel like its all or nothing…

Key Point
expert testers

• There is a critical aspect of testing and optimization that you can access
without 10 hours of teaching and 10 weeks of systems changes

• To take advantage of today’s data, all you need to do is see it with a
different perspective

• Today, we’re going to discuss four key principles that will help you see
today’s data with new eyes

POINT 1: Stop focusing on the “how many”

Stop focusing on the “how many”
1. The goal of any kind of customer research is to enable the marketer to
anticipate customer response to a particular message or approach.

2. Therefore, the primary usefulness of examining analytics, or even testing,
is not in answering “how many?” but rather in answering, “why so?”



Research Notes:
Background: A medical provider specializing in treating chronic pain.
They are the sole providers of an innovative procedure and pain
management plan.

Goal: To plan a content marketing strategy based on the copy focus that
generates the most appeal in condition-based searchers.

Primary research question: Which subject matter focus (copy) will achieve
a higher click-through rate?

Approach: A/B Multifactor Split Test


[Condition] Sufferer? [Condition] Sufferer?
Free access to [part]pain resources Compare available treatments,
from the experts in [part] health. from the experts in [part] health.
Company.com/[condition] Company.com/[condition]

Learn about the causes & solutions, How to recognize the symptoms,

Case Study: Results

Free access to [part] pain resources Compare available treatments,

Learn about the causes & solutions, How to recognize the symptoms,
73% more 99% more

*
What you need to understand: Customers will more likely engage with this
company trying to understand the problem as opposed to immediately
looking for a solution

What we have discovered
1. The goal of any kind of customer research is to enable the marketer to
anticipate customer response to a particular message or approach.

2. Therefore, the primary usefulness of examining analytics, or even testing,
is not in answering “how many?” but rather in answering, “why so?”

3. Ultimately, analytics from observation and experimentation can enable
the marketer to see cognitive trails left by the visitor’s mind.

4. These cognitive trails give us clues for how they will respond, even when
tracking isn’t always available in another medium

Remember this page?
About this page:

• Call to action

• Hero image

• copy

Not this…
• Analytics shouldn’t be points in a game that you can only watch from a
distance

But this…

Marketing Intuition
So-so

Opinion Research
Ok

Analytics should be about
Behavioral gathering business
Winner Observation & intelligence BEFORE a
Experimentation major online (or offline)
campaign.

Example: Hidden insights in web analytics

Web Analytic Cognitive Clues
Time on page Are visitors engaged with the content?
Are they confused with the process?

What are visitors interested in?
Click tracking Are they confused with something we are saying?

Do we have the wrong focus?
Bounce rate Are there too many distractions?
Is there too much (or little) information?

Segment-level data What motivates individual visitor types?
Where are the deeper optimization opportunities?

Form event tracking What form fields cause anxiety or confusion?
How much friction will your visitor put up with?

Traffic patterns Who is coming and where are they coming from?
Can we be more relevant to the visitor?

Example: Hidden insights in tests conducted

Customer Behavior Customer Theory

Which headline will What does my customer
generate a higher want the most?
response?
Which testimonial will What makes my customer
generate the most especially anxious?
response?

Which call to action will What is my customer comfortable
generate a higher with at this stage of the buying
response? cycle?

POINT 2: Simplify your perspective of analytics
so it gets used

Need a moment?

Massive information presented
equally assaults the mind

Simplify your perspective
1. When you focus on the “why so”, all The Who
analytics can be organized into four Source
categories
The What
2. Each analytics category reveals a different Result
aspect of the visitor’s story
The Where and When
3. Different perspectives (categories) can be Amount
combined to create a single understanding
of the person that encounters our The Why
messaging Nature

Like a Bento Box…
The Where Amount Source The Who
and when

The Why Nature Results The What

The who (source)

• If you want to know where people are Common Metrics
coming from
Referrers

• These analytics often give clues to Search Terms
the motivation of your visitors and allow Countries and Languages
you to understand how many different
types of visitors are viewing the same Top Landing Pages
message
• i.e. the kind of experience or content
the visitor is expecting.

The what (result)

• If you want to know what people Common Metrics
do once they get to a page
Conversions/Purchases

• These analytics are like mile markers Clicks
on your highway to conversion Next Pages
• What markers must people take Downloads
to get to the end of the road?
• At what markers do people get
off the highway and get off
track?

The where and when (amount)

• If you want to know the Common Metrics
amount of in each part of your Pageviews, Visits
process (including the purchase
category) Visitors
Impressions
Total Revenue

The why (nature)

• If you want to know what people
Common Metrics
are experiencing (or selecting)
while viewing your messaging Event, eye tracking
Clicks/page
• Use this group of analytics to find
Time on page
big problems/disconnects people
may be experiencing in your Transaction Details
messaging or experience.

Best sources of “why so” information
Medium: Organic/SEO
Where Amount Source Who
and when % of Total Traffic (64%) Top Website Referrers (47%)
Keyword clicks (45%) Unique Search terms (46%)
Inbound links (36%)
Branded vs. Non-branded
(25%)

Nature Results
Keywords triggering search Keyword Rankings (63%)
(51%) CTR (47%)
Keyword movement Term Conv. Rate (38%)
(37%) ROI (33%)

Why What

Best sources of “why so” information
Medium: Video Marketing
and when Placements on share sites Video SEO (21%)
(21%) Embeds on non-video
Most viewed video sharing sites (14%)
segments (33%)

Nature Results
Comments, Likes, +1s Conv Rate (27%)
(33%) Video Ad Clicks (23%)
Video ratings (20%) Play rate (40%)
Play-through rate (21%)
ROI (15%)
Video shares (29%)
Why What

Better sources of “why so” information
Medium: Email
and when Clicks per email (55%)
List Size (48%)
Deliv. rate (55%)
Inbox placement rate
(16%)

Nature Results
Clicks per link in email (49%) Open Rate (78%)
Complaint Rate (25%) CTR (78%)
Unsub rate (65%)
Social Sharing rate (21%) Conv Rate (55%)
ROI (28%)
Why What

Medium: Social Marketing
and when Views (55%) Top Influencers (26%)
RSS (23%)

Nature Results
Brand Sentiment (23%) Social Reach (61%)
Engagement/Post/Tw (33%) Traffic from social
(49%)
Sales (23%)
ROI (20%)
Why Conv Rate (27%) What

Medium: Paid Search
and when Clicks (66%)
Avg Ad Pos.(41%)
Impr. Share (28%)

Nature Results
Quality Score (36%) CTR (66%)
CPCl (65%) CPConv(44%)
Conv Rate (54%)
CPLd(43%) ROI (40%)
Profit per click (18%)
Why What

Profit per Impr.(10%)

Average sources of “why so” information
Medium: Display Advertising
and when Clicks (61%)
Reach (27%)
Frequency (27%)
Impr. Share (21%)

Nature Results
CTR (62%)
CPM (33%) CPConv(40%)
Conv Rate (45%)
ROI (33%)
Lost Impr. Share (6%)
Why What

Average sources of “why so” information
Medium: Content Marketing
and when Views (55%)
RSS (23%)

Nature Results
Comments/Post (29%) Leads (48%)
Downloads (41%)
Conv. Rate (40%)
ROI (23%)
Likes, Tweets, +1s
Why Shares (45%) What

POINT 3: Start with the minimum (not max)
effective dose

As for the rest of us…
• Not everyone has a “Data” on board to do the most daring of analyses

Forget the analysis paralysis
• You don’t need to throw everything (including the kitchen sink) at
something to get messaging the performs positively


 Case Study ID: RegOnline Homepage Test

Research Notes:
Background: Event management software company that lets users create
online registration forms and event websites to manage their events.

Goal: To increase number of completed leads on home page.

Primary research question: Which process will yield a higher conversion
rate?

Approach: A/B Multifactor Split Test

• Already tested and optimized by the
local design/dev team over the past
year

• Two easily accessible pieces of data (nav summary, time on page)

From here… To here… To here…
~2 minutes ~2 minutes <1 minute

And back again…

• With that data, the
team created a
messaging experience
that FORCED visitors to 1 Essential
Product
read and see certain and
Company
piece of information Overview
before others and Details
WITHOUT negatively
effecting SEO

2 Pricing
Info

Case Study: Results

89.8% increase in conversion
The treatment generated 89.8% more completed leads

Versions Conversion Rate Rel. diff

Control 0.3% -

Treatment 0.5% 89.8%

* What you need to understand: The team was able to achieve a
substantial lift by understanding how the customer responds to
information when presented in a certain sequence

Forget the analysis paralysis
• You don’t need to throw everything (including the kitchen sink) at
something to get messaging the performs positively

• To get the effect of an analysis for a minimal amount of effort, transform
your analytics bento box and into a pyramid

Analytics Pyramid
Not all analytics categories are created equal…

What we have discovered…
• The analytics (or clues) that are
more telling to create effective
messaging are at the bottom
and the metrics that need less
are at the top

Like the old food
Pyramid…

Analytics usage for message creation
Which of the following do you routinely use to create different message types?

Keyword
SOURCE
Website activity
NATURE
Performance or previous message
RESULT
New vs. Returning visitor
SOURCE
Purchase history
NATURE
Referral channel
SOURCE
Location
SOURCE
Date of last visit RESULT
Device SOURCE
Comprehensive testing strategy

Other


Analytics Pyramid

• The key: The more you combine and utilize source and nature based
analytics, the better performance potential you’ll have with your messaging

Analytics Pyramid Examples
• Example 1: Messaging that
doesn’t take Source analytics
into consideration is a message
that has no clear target.

• Example 2: If you see a
great Amount of visitors that
show a common Result (like
leaving the critical path in a
certain direction), then you may
have found a major disconnect
with the messaging
• but you still need more to
know what causes it

POINT 4: Stop focusing on tools and start
focusing on visibility

Tools seem to be top of mind…

In which areas are you planning additional investments? Please select all that apply.

Purchase of analytics
tools/platforms/software

Training

Data integration with other
applications

Staffing of in-house
personnel/analysts

Hiring of external
analysts/consultants

Other


Though few come close to using the same set
• 17 Experts, Digital Marketing
(not just SEO)

• 40+ tools referenced

• Very few consistent results

With no clear decision on free vs. paid
Are you satisfied with the PRECISION of your analytics systems? Paid Tools

Competitive intelligence tools

CRM systems

Live chat tracking tools

Attribution management software

Social media monitoring tools

Offline call management and tracking systems

Marketing automation software

SEO management tools

Email marketing analytics software

PPC bid management tools

Web-integrated call management and tracking…

Web (clickstream) analytics tools

Dissatisfied Neutral Satisfied
Methodology: Fielded November 2012, N≤79

With no clear decision on free vs. paid
Are you satisfied with the PRECISION of your analytics systems? Free Tools

Competitive intelligence tools

CRM systems

Live chat tracking tools

Attribution management software

Social media monitoring tools

Offline call management and tracking systems

Marketing automation software

SEO management tools

Email marketing analytics software

PPC bid management tools
Web-integrated call management and tracking
systems
Web (clickstream) analytics tools

Dissatisfied Neutral Satisfied

Methodology: Fielded November 2012, N≤409

What marketers really want from Oz…
If I only had __________, my marketing efforts would be substantially more effective

Advanced customer behavior analysis (37%)
Complete quantitative understanding of the entire
marketing and purchase cycle (36%)
Predictive analytics (33%)
Competitive trends insights (30%)
Integration of online and offline data (29%)
Customer sentiment/Voice of customer (27%)

Visibility info pipeline (funnel) performance (26%)
Cross-channel view of results (24%)

Lifetime value measurement (24%)

Social media and Web 2.0 measurement (24%)
A/B and multivariate testing (21%)

Real-time reporting (18%)

Custom report creation (16%)

What marketers want from Oz…
If I only had __________, my marketing efforts would be substantially more effective

Advanced customer behavior analysis (37%)
Complete quantitative understanding of the entire
marketing and purchase cycle (36%)
Predictive analytics (33%)
Competitive trends insights (30%)
Integration of online and offline data (29%)
Customer sentiment/Voice of customer (27%)

Visibility
Visibility info pipeline (funnel) performance (26%)
Cross-channel view of results (24%)

Lifetime value measurement (24%)

Social media and Web 2.0 measurement (24%)
A/B and multivariate testing (21%)

Real-time reporting (18%)

Custom report creation (16%)



Research Notes:
Background: A company that sells retail and wholesale collector items
primarily online

Goal: To increase conversion rate, specifically from new customers.

Primary research question: Which version of second step in the
conversion funnel will produce the highest conversion rate?

Approach: A/B variable cluster split test

• Their checkout’s messaging came to
queue, and Google Analytics was
only showing 1 page for a 6 page
process
Sample of code change

• When doing the research, they
discovered that getting the details
would require some extensive code
changes and risks to the current
tracking
• Legacy
• Simulated page tracking

Revenue drop offs
• Seeing the challenge, the team used
their creativity to do two alternatives
that would grant them the sight they
needed:
• Install a limited amount of code
from a new tool (mitigate risk,
faster turnaround)
• Utilize existing data already
being captured by other systems

• The result was two sources of
information that pointed to one
particular messaging problem for
new customers

• Can you see a potential
Control red flag (from the
customer’s point of view)
in the messaging?


• The emphasis on the
Treatment detailed terms and
conditions was refocused
to the satisfaction
guarantee that was
already in place

New focal point

Case Study: Results

$548,000 Increase in profit per year
The new checkout page increased conversion by 4.51%

Design Conversion Rate
Control 82.33%
Treatment 86.04%
Relative Difference 4.51%

*
What you need to understand: While the conversion increase is
small, optimizing messaging in this specific step in the sales
funnel resulted in a projected $500,000+ increase in profit per
year.

Visibility is a big problem with big payoff potential
Email Content Marketing Social

Organic/SEO Display PPC/SEM

Visibility isn’t just plugging holes either…
Which sources of information do you actively use to better understand your prospects and
customers? Please select all that apply.
Website activity

Customer service
feedback
Industry blogs,
professional journals

Transaction data

Social media
conversations

Demographic data

Third-party market
research
Competitive
benchmarking

Reviews and rankings

Focus groups/Customer
surveys
Brand performance
analysis

Other


Key Principles
expert testers

• Analytics examination is a fundamental aspect of testing that you can access
without 10 hours of teaching and 10 weeks of systems changes

• Take advantage of today’s analytics by changing your team’s perspective:
• POINT 1: Stop focusing on the “how many”, start focusing on the “why so”

• POINT 2: Adopt a simplified perspective of analytics to make it usable

• POINT 3: Focus on the minimum (not maximum) effective dose

• POINT 4: Stop focusing on tools and start focusing on visibility

How Analytics Improve Marketing Experiments

Recommended

Recommended

More Related Content

What's hot

What's hot (10)

Similar to How Analytics Improve Marketing Experiments

Similar to How Analytics Improve Marketing Experiments (20)

More from Paramore | the digital agency

More from Paramore | the digital agency (20)

How Analytics Improve Marketing Experiments