2. What are the Keys to Successful Testing?
• Design your test around a single question
• Have a large differentiation b/w tested elements
• Account for changes in temporal behavior
• Collect enough data to reduce margin of error
• Segment results to provide added insight
• Act on the results!
2
3. Test 1: Is Running Ads on Brand Queries a
Waste of Money?
Example SERP w/o Ad Example SERP w/ Ad
3
4. Test Design
• 1 week ads on 1 week off
• No issues of seasonality or promotion
• 30 Branded Keywords
• 1 version of ad
4
5. Did the Ads Drive Traffic?
NO
• Google -1.3%
• Yahoo -8.7%
• Live -7.3%
5
6. Did the Ads Increase Conversion Rate?
YES
• Google +17.9%
• Yahoo +32.1%
• Live +11.7%
6
7. Did the Ads Increase RPV?
YES
• Google +15.4%
• Yahoo +47.0%
• Live +27.5%
7
8. New Questions
• Right Time? Marketing to Intent?
• Power of Persuasion?
• Power of Brand in Search?
8
9. Test 2: What is the Best Ad to Increase
Conversion?
Ad Testing Element 1
Multivariate Testing Works Element 2
Increase Conversion Rates Element 3
ottodigital.com/ads Element 4
4x3 L9 Orthogonal Array
4 Elements 3 Variations
9
10. Test Design
• 9 Ads Created and Tagged w/ URL Parameter
• Landing and Thank You Page Tagged
• Ad Serving Optimization Turned OFF
10
11. What does an L9 Array Really Look Like?
Recipe (9) Title (3) D1 (3) D2 (3) URL (3)
A - Control {keyword} Search {AdGroup} Compare Products www.brand.com
suppliers Today
B {keyword} Find {keyword} suppliers Search Thousands of www.brand.com/
Products {keyword}
C {keyword} Find Parts fast at (brand) {keyword} catalog www.brand.com/
search catalogs
D {keyword} suppliers Search {AdGroup} Compare Products www.brand.com/
suppliers Today catalogs
E {keyword} suppliers Find {keyword} suppliers Search Thousands of www.brand.com
Products
F {keyword} suppliers Find Parts fast at (brand) {keyword} catalog www.brand.com/
search {keyword}
G {keyword} catalog Search {AdGroup} Compare Products www.brand.com/
suppliers Today {keyword}
H {keyword} catalog Find {keyword} suppliers Search Thousands of www.brand.com/
Products catalogs
I {keyword} catalog Find Parts fast at (brand) {keyword} catalog www.brand.com
search
11
12. What Were the Results?
Recipe Visitors Conversion Lift
A - Control 11.80% 5.88% ---
B 14.30% 4.29% -27.05%
C 14.04% 6.08% 3.42%
D 9.34% 2.57% -56.29%
E 13.00% 6.16% 4.72%
F 9.42% 5.38% -8.50%
G 10.06% 5.04% -14.32%
H 10.41% 5.13% -12.82%
I 7.63% 8.04% 36.71%
12
13. What is the breakdown?
• Other Ads the Used Winning URL
– Control
– +4.72%
• Other Ads the Used Winning D1
– +3.42%
– -8.5%
• Other Ads the Used Winning Title
– -14%
– -12%
13
14. What Ad Elements Influenced Conversion?
Element Winning Alternative Influence on
Conversion
Title {keyword} suppliers 13.54%
D1 Find {keyword} suppliers 53.53%
D2 Search Thousands of Products 23.56%
URL 9.37%
www.brand.com/{keyword}
14
15. What’s Better?
Winning Recipe Best Predicted
{keyword} catalog {keyword} suppliers
Find parts fast at (brand) Find {keyword} suppliers
{keyword} catalog search Search Thousands of Products
www.brand.com www.brand.com/{keyword}
15
16. What’s Better?
Winning Recipe Best Predicted
{keyword} catalog {keyword} suppliers
Find parts fast at (brand) Find {keyword} suppliers
{keyword} catalog search Search Thousands of Products
www.brand.com www.brand.com/{keyword}
Best Predicted +52% Lift in Conversion Rate
15
17. Test 3: What are the effects of Quality Score?
16
18. Who & What Determines Relevance?
Recipe A-C
Title: “{keyword}”
Recipe Impressions Conversion Rate
Recipe A 265,909 5.72%
Recipe B 265,045 5.09%
Recipe C 264,784 5.49%
17
19. Who & What Determines Relevance?
Recipe D-F
Title: “ Find {keyword}”
Recipe Impressions Conversion Rate
Recipe D 47,093 8.59%
Recipe E 47,289 5.24%
Recipe F 47,066 7.76%
18
20. Who & What Determines Relevance?
Recipe G-I
Title: “The Wait is Over”
Recipe Impressions Conversion Rate
Recipe G 46,518 9.84%
Recipe H 46,795 8.11%
Recipe I 46,579 11.76%
19
21. Who Decides Relevance?
Group G-I had
• 17% of the Impressions of A-C
• 32% of the Conversions of A-C
• 82% higher Conversion Rate than A-C
20
22. Jonathan Mendez, Chief Strategy Officer
Thanks!
• optimizeandprophesize.com - My Blog
• ottodigital.com/blog - OTTO Blog
• offermatica.com - Technology & Parent
21
23. Jonathan Mendez, Chief Strategy Officer
Thanks!
• optimizeandprophesize.com - My Blog
• ottodigital.com/blog - OTTO Blog
• offermatica.com - Technology & Parent
21
