Delivering Relevance with Analytics & Optimization
Jonathan Mendez, OTTO Digital

Offermatica is an on-demand solution that changes
content on remote sites to influence and measure visitor
behavior.
•Automated A/B Testing
•Multivariate Testing
•Content Targeting
•Personalization
•Automatic Optimization
•Dynamic Merchandising/Content Delivery

OTTO Digital is the strategic services arm of Offermatica
that leverages the learning of hundreds of tests across
verticals and media to inform our methodology of Velocity
Marketing ™ (Vmx)
http://www.optimizeandprophesize.com
Blog:
http://www.ottodigital.com
Site:

A/B Testing
Issue: What layouts and designs work best?
A B C
187% CONVERSION
LIFT
OFFERMATICA
CASE STUDY

A/B test simple messaging changes

Measure results in real-time on real users
B
11% CONVERSION LIFT

Multivariate Testing
Test more, faster. GET AGILE! 4 Elements, 3 Variations
Landing Page Multivariate Test Matrix
Call to
action
Headline
Text
Picture

Test more, faster. GET AGILE! 4 Elements, 3 Variations
but only 9 Page Recipes need to be tested
81 Possible Combinations..

Multivariate Testing
Individual element contribution
Recipe 8 verified by testing
+ 17% Aggregate Lift
versus Control
Winner… Recipe 8
Control Recipe
+ 9% + 5%
+ 1%
+ 2%

Recipe B: Increased
registrations by 61.01%
86.9% confidence

The Basic Truth
Relevance and engagement are delivered by intelligently
changing content to suit what we know about the user.
More truths…

Truth #1 - You Need A Strategy
User Focused
Data driven (Qualitative & Quantitative)
Designs are emergent, not defined
Parallel tracks
Iterative

Truth #2 - No Risk, No Reward
“Our CMS can’t change”
“The sign-up is owned by IT and they will not let us change it”
“It is too expensive to create dynamic experiences”
“Our CEO will not let us change the homepage”
“Changing the navigation confuses people”

Truth #3 - The Data Tells The Story
Search Data
Media Data
Site Data
Focus Data
Sales Data
CRM Data
Optimization Data

Truth #4 - Search Changes Everything
User Control - Intent Driven Media
Fractures Content - Micro Relevance
Redefines User Expectations - Macro Relevance

Truth #5 - Social Media Changes Everything
Commoditization of Content - Many-to-Many Exchange
A Higher Authority - Blogosphere
Strength in Numbers - Building Interesting Spaces

Analytics - Measuring Relevance

Analytics
 What we know is happening
 Only useful if actionable

Mining User Data Means Leading By Listening…
 Traditional Usability/UX
Listening Labs
Focus groups
Surveys
Eye-tracking
 Metrics
Site side data
Search data
CRM data
 User Generated Content
Blogs
Reviews
Message Boards
 Testing and Optimization
A/B
MVT (Multivariate)
Targeting Content

Who Can Act on Data?
Marketing?
Analytics?
IT?

“Why do gut when you can go with objective information?
Hope is not a strategy. And gut instincts are always
less effective than objective analysis.”
- Ken Mehlman
NY Times Magazine, September 24, 2006

Delivering Relevance

Users are in control of the experience!
Effective engagement in this new paradigm means
your content must be contextually relevant
The Right Audience
The Right Message
The Right Time

Delivering Contextual Relevance
Segmentation + Targeting = Engagement
Engagement = Relevance

Relevance starts with understanding your audience

Your Audience
Every click has a goal in mind

User behavior is determined by the goal

Anticipate and message to goals meeting and exceeding

expectations in order to engage
Users have “tunnel vision” engaging


What is user “tunnel vision”?
 Users don't think in terms of sites, features, sections or
other elements that professionals spend their time
focused on
 Users engage in what they are interested in and little
else
 If site elements are not in context of the user focus, they
are likely invisible
 Users don’t pay attention to detail

The most relevant experiences are designed with the
user’s goals and interests in mind…
Not your goals
Not your corporate hierarchy
Users care about “What can YOU do for ME?”

Understanding the way your users think about content
This page was
clicked from an ad
under the kw
“telescopes” by
someone looking for
a gift for his son.
Landing on this
category page the
customer looked at
the left navigation
and was confused
by the options of
refractor, cassegrain,
dobsonian, etc. He
left.

Publishers need to speak user’s language
This category page
immediately
messages its
products in
categories like
“beginner scopes”
that are easy to
understand.
It also makes good
use of redundant
navigation and
categories popular
with most customers
like “bestsellers.”

Understanding the way your users think about content
NYT.com Users

Publishers need to speak user’s language
31 Navigation Messages Top 50 Internal Search Queries

ESPN understands the sports fan

ESPN - Relevance Delivered: 60 relevant links!

washingtonpost.com - Relevance is Not Delivered
Relevant navigation
is essential in the
fractured environs
created by natural
search results and
links

Category to Content: What can we learn from Retailers?
Global Navigation is important but
more important is delivering the
ability for the user to discover and
engage in what we know interests
them.
•Give users control
•Refine content based on preference
•Deliver obvious relevance
•Users understand “brands”
•Easy self segmentation

Category to Content: What can we learn from Retailers?
Global Navigation is important
but more important is delivering
the ability for the user to
discover and engage in what
we know interests them.
•No user control of delivery.
•No way to refine content
based on preference.
•Deliver obvious relevance.
Why is the “Talk to the
Newsroom” relevant? TV, Film,
Internet are relevant.
•Users understand “brands.”
Where is Google, Viacom,
Disney?
•No easy self-segmentation like
Publishing, Advertising, Internet.

Relevance Through Segmentation

Tools to assist in segment discovery
Site Analytic Data
 Search Channel Data
 Customer/Sales Data
 Research
39

High-Impact Segmentation
Action based, not persona
Divide based on behavior
Divide based on source

High-Impact Segmentation
Action based, not persona
Divide based on behavior
Divide based on source
Analytic Impact:
Thick slices will provide the greatest lift!
Thick slices provide the most confidence!

High Impact Segmentation
4 Key Segment Groups
 Keywords
 Source
 Geo
 Action
42

Segmentation Methodology
Keyword- As close as we can get to the goal
43

Keyword Relevance
CBS does great job ABC leaves lots on the table
44

Segmentation Methodology
Source (Referrer)- A window on user interests
45

Source
Reinforcement of users interest on landing page
(Religion, Pope, World, ABC) can double conversion
rates*
* OTTO/Offermatica test data
August, 2006
46

Segmentation Methodology
Geo- Few things create contextual relevance better
than location.
47

Geo Targeting is key for Ad Effectiveness…
48

…and users love the relevance geo brings
49

What Happens When Geo Relevance is Ignored
50

Reinforcement- KW/Source
51

Reinforcement - Testing the KW/Source Idea

KW/Source Reinforcement - Genre Page

KW/Source Reinforcement - Side by Side
Impact?

KW/Source Reinforcement - Test Results
Recipe Visitors CR Lift RPV Lift
SRP - Control 37,987 1.27 -- --
SRP - Reinforce 43,973 1.99 48.35% 71.38%
Genre - 221,227 1.34 -- --
Control
Genre - 220,736 1.37 1.93% 2.99%
Reinforce

Bridging Recognition with Targeted Relevance

Bridging Recognition with Targeted Relevance

Relevance Leads to Engagement

Engagement
Reliant on relevance
Interesting over time
Unusual ease of use / helpful
Entertaining in an ongoing, useful way
Solves primary goal +

Engagement: Group Behavior
Create Interesting Spaces!
More Relevant Content
150% More Views!

Engagement:
Unusual Ease of Use, Unlocks New Value

Engagement
Make it More Interesting Over Time
•Getting more
relevant over
time
•Tags
•Saved
•Viewed
•Searched

Engaging the user to achieve your goals
 Subscription - How much content do you give away?
 Registrations/RSS – How, when and where to ask?
 Page Views - Creating a trail of relevant content
 Video - Where and when to get users watching?
 Branding - User experiences define your brand

How to Handle the Truth
Embrace the New Paradigm
Tear down walls to content
Deliver relevance
Enable user control of delivery
Form a Strategy to (Optimize) Change
Layouts
Messaging
Dynamic and Targeted Content Delivery
Multivariate Testing & Optimization
Measure and Act on What Your Audience Tells You
Listen
Analyze
Act

Thank You!
http://www.optimizeandprophesize.com
Blog: