Creating an ad testing framework final

BRIGHT IDEAS
START HERE.
2015 SUMMIT MONTREAL | JULY
29+30
Creating an Ad Testing Framework

BRIGHT IDEAS
START HERE.
©2015 All rights reserved 2ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
MEET THE SPEAKERS
Stewart Sullivan
Campaign Manager
Display, Acquisio
Brad Geddes
Co-founder,
AdAlysis
Sandra Jimenez
Campaign Manager
Display, Acquisio
Ryan Saurer
Director of Strategic
Partnerships, Boost
Media

BRIGHT IDEAS
START HERE.
29+30
Creative (Testing) Matters

BRIGHT IDEAS
START HERE.
The Current State of Search Messaging
ADCOPY
KEYWORDS
KEYWORDS BIDS AD COPY

BRIGHT IDEAS
START HERE.
Examining an Account
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Healthier Account
High Volume,
High Opportunity
Low Volume,
Limited Opportunity
% of Ad Groups
%ofSpend

BRIGHT IDEAS
START HERE.
A Strong Testing Culture
Gather
information
DATA
ANALYSIS
INSIGHTS
Interpret
findings
Apply
analyses

BRIGHT IDEAS
START HERE.
Gather
information
DATA
ANALYSIS
INSIGHTS
Interpret
findings
Apply
analyses

BRIGHT IDEAS
START HERE.
{Keyword: Optimize Your Creative}
Increased Profitability, Time Savings,
& Better Customer Insights. Try Now !
BoostMedia.com/CreativeMatters
Value Proposition
Risk
Mitigation
D1
Punctuation
Call to Action
URL Suffix
Features
URL
Benefits
Capitalization
Headline
Body Lines
Display URL
DKI

BRIGHT IDEAS
START HERE.
Specifics: pricing, # of styles
Key differences
Benefits of product or service
Keyword alignment
Calls to action
Special offers
Risk mitigation
Synonyms within the ad
Punctuation
Display URL
10 Elements to Test
1
2
3
4
5
6
7
8
9
1
0

BRIGHT IDEAS
START HERE.
The Boost Methodology
1. Run a test
2. Challenger outperforms incumbent
3. Declare challenger winner
1. Realized incremental value
Primary Success
Metrics
Secondary Success
Metrics
• Clicks (CTR)
• Conversions (CPI)
• Revenue (RPI)
• CPA
• CPC
• Positioning
• Impression Share
• MPI

BRIGHT IDEAS
START HERE.
Lexical Analysis
Top Performing Words (by CTR)
always
money
range
orlando
lincoln
tampa
straight
moment
mobile
suit
spur
huge
group
site
saving
redeem
one
early
try
three
heartbest
earth
dads
largeofficial
orland
savings
small
thanks
tree
way
mom
make
CTR
Having insight on exactly which
words and phrases resonate
with your target audience is
crucial to optimizing ad creative.
2.00% 15.00%

BRIGHT IDEAS
START HERE.
Case Study: Digital Marketing Agency
Challenge: Like many marketing agencies, this agency is often under pressure to quickly produce hundreds
or even thousands of ad creative ideas to support client promotions.
Outcomes: This agency drove major increases in CPI, ROAS, and RPI. Additionally, they were able to
augment their search business and add a travel client, with which they garnered a 49% CTR.
RESULTS: +379%
RPI
+399%
ROAS
+112%
CPI
Retail Client

BRIGHT IDEAS
START HERE.
Contact Me:
Ryan.Saurer@BoostMedia.com
Download our Whitepaper on Creative Optimization:
http://bit.ly/UnlockingPaidSearch

BRIGHT IDEAS
START HERE.
29+30
Thank you!

BRIGHT IDEAS
START HERE.
Optimization & Testing - Display Advertising
Diving Into the Granularity of a Display
Campaign

BRIGHT IDEAS
START HERE.
Overview
 Variables to Test
 Ad Frequency, Recency & Day Parting
 Placement, Fold, Ad Format & Ad Size
 Technology & Mobile
 Video
 Summary

BRIGHT IDEAS
START HERE.
Ad Format
Fold
Placement
Creative
Operating
System
Browser
Device Type
Operating
System
User Day of
Week
User Hour of
Day
Recency
Sales Cycle
Frequency
Supply
Source
Site
Category
Site
Country
State/Region
DMA
City/Town
Variables to Test

BRIGHT IDEAS
START HERE.
Ad Frequency, Recency & Day Parting
 Changing Ad Frequency
 Adjusting Ad Serving Times based on Recency Window
 User Time of Day & Day of Week

BRIGHT IDEAS
START HERE.
Example – Recency Data

BRIGHT IDEAS
START HERE.
Placement, Fold, Ad Format & Ad Size
 Verticals & Blacklisting: Changing, Adding or Removing inventory
 Flash ads VS Static ads
 160x600, 300x250 and 728x90 represent approx. 65% of inventory on the exchanges
 Above the Fold Vs Below the Fold

BRIGHT IDEAS
START HERE.
Technology & Mobile
 Optimize by:
 Device Type
 Operating System
 Browser
 Model
 +900 testing combinations!
 Mobile Channel Targeting: In-App, Mobile-Optimized Web and Mobile Web

BRIGHT IDEAS
START HERE.
Example – OS Data
OS Family
Impressions Bid
On
Impressions
Won
Win Rate Cost Avg Bid eCPM Clicks CTR eCPC CTC-1 VTC-1 TC-1 eCPA TC-1
Android 212,615 17,609 8.28% 14.69$ 0.64$ 0.83$ 22 0.125% 0.67$ 0 0 0
iOS 108,475 6,987 6.44% 7.44$ 0.74$ 1.07$ 12 0.172% 0.62$ 1 0 1 7.44$
Linux 147,786 9,587 6.49% 12.99$ 0.95$ 1.35$ 12 0.125% 1.08$ 3 1 4 3.25$
OSX 3,840,302 133,686 3.48% 215.30$ 0.97$ 1.61$ 113 0.085% 1.91$ 28 21 49 4.39$
Other 22,091 1,466 6.64% 1.73$ 1.06$ 1.18$ 4 0.273% 0.43$ 0 0 0
Windows 9,574,927 422,654 4.41% 646.36$ 0.99$ 1.53$ 489 0.116% 1.32$ 88 46 134 4.82$
WindowsPhone 6,246 570 9.13% 0.40$ 0.49$ 0.70$ 2 0.351% 0.20$ 0 0 0
(blank) 23,217 345 1.49% 0.74$ 0.82$ 2.13$ 0 0.000% 0 0 0
Grand Total 13,935,659 592,904 4.25% 899.65$ 0.97$ 1.52$ 654 0.110% 1.38$ 120 68 188 4.79$

BRIGHT IDEAS
START HERE.
Video
 Small player VS Large player
 Video length – 15s, 30s & 60s inventory
 Skippable VS Non-Skippable
 Quality –User generated, Professional or Prosumer
 Video Completion Rate

BRIGHT IDEAS
START HERE.
Summary
 More levers to test than other mediums (Search, Email, Social)
 Small Changes, Big Differences
 Ongoing process –Don’t set it and forget it!
 Make sure you have access to granular levers
 Testing a Managed Service

BRIGHT IDEAS
START HERE.
29+30
Creating Scientific Ad Tests

BRIGHT IDEAS
START HERE.
About Me
Let’s Connect
@bgTheory
in/ewhisper
Founder, Certified Knowledge
Co-Founder, AdAlysis
Author, Advanced Google AdWords
Brad Geddes

BRIGHT IDEAS
START HERE.
Ad Testing is Improvement and/or Insights

BRIGHT IDEAS
START HERE.
A/B Testing – Known as Single Ad Group Testing

BRIGHT IDEAS
START HERE.
Multi-Ad Group Testing
Idea 1
Idea 2
Idea 3
Idea 4
Idea 5

BRIGHT IDEAS
START HERE.
Use Cases
Single Ad Group Testing
Insights within an ad group
-Best for high traffic ad groups
-Best for Brand ad groups
-Good for other ad groups after
you’ve done some multi-ad
group testing for global
insights
Multi-Ad Group Testing
Insights across a segment
-Best for template created
ads
-Best for small data accounts
-Best for market research

BRIGHT IDEAS
START HERE.
Quick Market Research Example

BRIGHT IDEAS
START HERE.
Rush Shipping Vs Fast Shipping
Ad CTR Conv Rate CPI Avg Order
Fast & Free
Shipping
4.96% 4.22% 0.209% $71.32
Rush Shipping 5.13% 4.32% 0.225% $357.91

BRIGHT IDEAS
START HERE.
Segmenting by Consumer Type: B2B vs B2C
Type Ad CTR Conv Rate RPI
B2B Fast & Free
Shipping
2.31% 3.46% 0.286328
B2B Rush Shipping 5.47% 5.30% 1.037939
B2C Fast & Free
Shipping
4.02% 4.73% 0.137408
B2C Rush Shipping 1.96% 3.05% 0.043392

BRIGHT IDEAS
START HERE.
Apply Insights to Ad Writing
 Segment by B2B vs B2C
 For B2B – use ‘rush shipping’
 For B2C – use ‘free shipping’
 Next Steps:
 Examine the branded terms, RLSAs and remarketing lists and see if there’s a good way to match the
shipping type to the ads.
 Can we change the landing page to match the shipping?

BRIGHT IDEAS
START HERE.
How to Start Testing

BRIGHT IDEAS
START HERE.
Determine Your Segments
Basic Segments:
• Your brand
• 3rd party brands
• Products
• Informational
• Long tail
• By buying cycle entrance
• Etc…
Ad Type:
• Text ads
• DSAs
• Image ads
• Call only ads
• HSAs
• etc…
Devices:
• Mobile
• Desktops
• All

BRIGHT IDEAS
START HERE.
Define Your Testing Metrics
Metric Best Usage
CTR: Click Through Rate Max Traffic (note: CPI is usually still better)
Conversion rate Most conversions from landing page
CPI: Conversion per Impression Most conversions possible
CPA: Cost per Conversion Lowering cost per conversion
ROAS: Return on Ad Spend Hitting ROAS goals
RPI: Revenue per Impression Maximizing revenue

BRIGHT IDEAS
START HERE.
What Minimum Data To Define
Metric Impressions Clicks Conversions Timeframe
CTR Yes Optional Yes
CPA Yes Yes
Conversion rate Optional Yes Yes
CPI Yes Optional Yes
ROAS Yes Yes
RPI Yes Optional Yes

BRIGHT IDEAS
START HERE.
Define Minimum Data
Impressions Clicks Conversions
Low Traffic 350 300 7
Mid Traffic 750 500 13
High Traffic 1000 1000 20
Well known brand terms 100,000 10,000 100 - 1000
The MINIMUM Data – It’s OK to have HIGHER Minimums

BRIGHT IDEAS
START HERE.
Define Minimum Confidence Levels
Term Type Minimum Confidence
Long Tail Keywords 90%
Mid data terms 90% – 95%
3rd Party Brands you Sell 90% (small brands) to 95% (large brands)
Top Keywords (the ones you watch daily) 95% - 99%
Your Brad Terms 99%

BRIGHT IDEAS
START HERE.
Define Maximum Data
 Option 1:
 3 months
 Option 2:
 Define maximums for each metric
 Define timeframes

BRIGHT IDEAS
START HERE.
Test by Device (if appropriate)
Count of Ad
Row Labels Mobile All Effective Device
Ad Group 1 2 All
Ad Group 2 None
Ad Group 3 1 2 Mobile/Desktop
Ad Group 5 10 All
Ad Group 9 7 All

BRIGHT IDEAS
START HERE.
Note: Google Changes Ad Formats on Mobile Devices

BRIGHT IDEAS
START HERE.
Actions
 If below minimum data, wait.
 If above minimum, calculate confidence factors
 If below minimum confidence – wait
 If above minimum confidence – take action
 If above maximum data and a test has no results, take action
 Action = Pausing loser OR pausing loser & creating new challenger
 In the case of multi-ad group testing: Replace entire string with a new pattern to test
 (optional): Use insight in other areas, such as email, landing pages, etc

BRIGHT IDEAS
START HERE.
See Ad Testing Results in Acquisio

BRIGHT IDEAS
START HERE.
At AdAlysis…
We want everyone to do what they do best!
People = Creativity
Computers = Math, statistics, and pattern recognition

BRIGHT IDEAS
START HERE.
AdAlysis
• Single ad group tests automatically created
• (optional) Define minimum data
• (optional) Define confidence factors
• Test up to 6 metrics
• Get alerts when there are no results
• Test patterns or labels across ad groups
• See global insights
• Support for AdWords & Bing Ads
• Works with text, rich media, DSAs, customizers, etc
• Only work on actionable data

BRIGHT IDEAS
START HERE.
Recap
 Determine what you want to test:
 Single ad group
 Multi-ad group
 Define:
 Testing metrics
 Minimum data
 Maximum data
 Segments
 Confidence levels per segment
 Run Your test
 When you reach minimum data:
 Run confidence calculations
 If winners & losers – take action
 If no winners – wait
 When you reach max data
 Pause test and reset
 When you have winners in Multi-Ad
Group testing:
 Apply that data to other channels

BRIGHT IDEAS
START HERE.
If You’re Interested in Learning More About AdAlysis
 Contact your Acquisio Rep
 The AdAlysis connector is Free
 Learn More
 AdAlysis.com
 FREE 30 Day Trial
 Contact Brad Geddes
 http://adalysis.com/contact/
 Twitter: @bgTheory
 LinkedIN: /in/ewhisper

BRIGHT IDEAS
START HERE.
29+30
Thank you! Questions?

Creating an ad testing framework final

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Creating an ad testing framework final

Similar to Creating an ad testing framework final (20)

More from Acquisio

More from Acquisio (20)

Recently uploaded

Recently uploaded (20)

Creating an ad testing framework final

Editor's Notes