Ad Testing: Research & Findings Case Studies: Impact of Ad Elements on Conversion Jonathan Mendez   December 4th, 2006
<ul><li>4 Elements </li></ul><ul><li>3 Variations. </li></ul><ul><li>9 Recipes </li></ul><ul><li>In Google: added a parame...
Ad Test Plan Summary Who:  Major Product Aggregator/Vertical Search Engine Objective:  Increase User Registrations from Go...
Multivariate Test Array I H G F E D C B A - Control Recipe (9) www.brand.com {keyword} catalog search Find Parts fast at (...
Test Results 36.71% 8.04% 7.63% I -12.82% 5.13% 10.41% H -14.32% 5.04% 10.06% G -8.50% 5.38% 9.42% F 4.72% 6.16% 13.00% E ...
Winning Title, D1, D2 and URL : 36.71% Lift Other Ads that used that Title -14.32% -12.82% Other Ads that used that D1 3.4...
Test Element Influence 9.37% www.brand.com/{keyword} URL 23.56% Search Thousands of Products D2 53.53% Find {keyword} supp...
Next Steps A - Winning Recipe  B - Best Predicted {keyword} suppliers Find {keyword} suppliers Search Thousands of Product...
Next Steps A - Winning Recipe  B - Best Predicted {keyword} suppliers Find {keyword} suppliers Search Thousands of Product...
Performance of Ads that used {keyword} as the title Control -27.05% 3.42% Top 2 ads with highest CTR had {keyword} as the ...
Quality Score:  CTR Major Factor so… Title Major Factor in CTR All things being equal ads with kw insertion in title usual...
But what is Relevant? Used to be…Google Ad Optimizer Turned Off, Traffic distributed evenly across recipes Last few months...
Ad Test Plan Summary Who:  Major Product Aggregator/Vertical Search Engine Objective:  Increase User Registrations from Go...
Recipe A-C Title:  “{keyword}” Titles, Impressions & Relevance 5.72% 264,784 Recipe C 5.72% 265,045 Recipe B 5.72% 265,909...
Recipe D-F Title:  “Find {keyword}” Titles, Impressions & Relevance 7.76% 47,066 Recipe F 5.24% 47,289 Recipe E 8.59% 47,0...
Recipe G-I Title:  “The Wait is Over” Titles, Impressions & Relevance 11.76% 46,579 Recipe I 8.11% 46,795 Recipe H 9.84% 4...
<ul><li>Who Decides Relevance? </li></ul><ul><li>Group G-I had  17.58% of the impressions  of Group A-C </li></ul><ul><li>...
Ad Test Plan Summary Who:  Major Services Brand Objective:  Increase Service   Subscriptions  Conversion Metric:  Purchase...
3 x 2 MVT Test Array (Service) (Service) (Brand) (Service) (Brand) (Service) Title Protect your ___ from ___ using (Servic...
Overall Results Both Segments  -- RPV 16.57% 15.24% 2.15% 14.27% D 71.25% 160.50% 4.87% 11.26% C 11.04% 53.68% 2.87% 29.19...
Trial Subscriber Results Segment 1 135.71% 0.59% 14.27% D 582.82% 1.70% 11.26% C 245.79% 0.86% 29.19% B -- 0.25% 45.29% A ...
Regular Subscriber Results Segment 2 -- RPV 17.53% -3.30% 1.57% 14.27% D 61.32% 95.38% 3.16% 11.26% C 12.77% 24.11% 2.01% ...
Overall Element Contribution Both Segments 64.75% www.brand.com/service URL 11.97% (Brand) offers the strongest (Service) ...
Trial Sub Element Contribution Segment 1 64.53% www.brand.com/service URL 5.46% (Brand) offers the strongest (Service) Des...
Regular Sub Element Contribution Segment 2 59.71% www.brand.com/service URL 22.00% (Brand) offers the strongest (Service) ...
Brand and Service Combo Titles had  highest CTR  but  poorest RPV Service Titles had  poorest CTR  but  highest RPV Results
Titles matter more for CTR than for Conversion Description Lines Matter Especially when everyone is doing kw insertion! UR...
Thank You!   Blog:  http://www.optimizeandprophesize.com
Upcoming SlideShare
Loading in …5
×

Ad Testing: Research & Findings

1,018 views

Published on

Case Studies: Impact of Ad Elements on Conversion

Published in: Business, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,018
On SlideShare
0
From Embeds
0
Number of Embeds
56
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Ad Testing: Research & Findings

  1. 1. Ad Testing: Research & Findings Case Studies: Impact of Ad Elements on Conversion Jonathan Mendez December 4th, 2006
  2. 2. <ul><li>4 Elements </li></ul><ul><li>3 Variations. </li></ul><ul><li>9 Recipes </li></ul><ul><li>In Google: added a parameter to click-through URL: &recipe= N </li></ul><ul><li>Assigned users to test recipes based on click-through URL parameter “recipe” </li></ul><ul><li>No changes made on site—just tracking .js files </li></ul>Element 1 (Title) Element 2 (D1) Element 3 (D2) Element 4 (URL) How We Set-Up Search Ad MVT
  3. 3. Ad Test Plan Summary Who: Major Product Aggregator/Vertical Search Engine Objective: Increase User Registrations from Google Ads Conversion Metric: Free user registration sign-up Campaign Population: Multiple Google AdGroups Campaign Length: 1 month Test Type: 4x3 Multivariate Case Study 1
  4. 4. Multivariate Test Array I H G F E D C B A - Control Recipe (9) www.brand.com {keyword} catalog search Find Parts fast at (brand) {keyword} catalog www.brand.com/catalogs Search Thousands of Products Find {keyword} suppliers {keyword} catalog www.brand.com/{keyword} Compare Products Today Search {AdGroup} suppliers {keyword} catalog www.brand.com/{keyword} {keyword} catalog search Find Parts fast at (brand) {keyword} suppliers www.brand.com Search Thousands of Products Find {keyword} suppliers {keyword} suppliers www.brand.com/catalogs Compare Products Today Search {AdGroup} suppliers {keyword} suppliers www.brand.com/catalogs {keyword} catalog search Find Parts fast at (brand) {keyword} www.brand.com/{keyword} Search Thousands of Products Find {keyword} suppliers {keyword} www.brand.com Compare Products Today Search {AdGroup} suppliers {keyword} URL (3) D2 (3) D1 (3) Title (3)
  5. 5. Test Results 36.71% 8.04% 7.63% I -12.82% 5.13% 10.41% H -14.32% 5.04% 10.06% G -8.50% 5.38% 9.42% F 4.72% 6.16% 13.00% E -56.29% 2.57% 9.34% D 3.42% 6.08% 14.04% C -27.05% 4.29% 14.30% B --- 5.88% 11.80% A - Control Lift Conversion Visitors Recipe
  6. 6. Winning Title, D1, D2 and URL : 36.71% Lift Other Ads that used that Title -14.32% -12.82% Other Ads that used that D1 3.42% -8.50% Other Ads that used that D2 3.42% -8.50% Other Ads that used that URL Control 4.72%
  7. 7. Test Element Influence 9.37% www.brand.com/{keyword} URL 23.56% Search Thousands of Products D2 53.53% Find {keyword} suppliers D1 13.54% {keyword} suppliers Title Influence on Conversion Winning Alternative Element
  8. 8. Next Steps A - Winning Recipe B - Best Predicted {keyword} suppliers Find {keyword} suppliers Search Thousands of Products www.brand.com/{keyword} {keyword} catalog Find parts fast at (brand) {keyword} catalog search www.brand.com A/B Test
  9. 9. Next Steps A - Winning Recipe B - Best Predicted {keyword} suppliers Find {keyword} suppliers Search Thousands of Products www.brand.com/{keyword} {keyword} catalog Find parts fast at (brand) {keyword} catalog search www.brand.com Best Predicted Won with 52% Lift vs. Winning Recipe! A/B Test
  10. 10. Performance of Ads that used {keyword} as the title Control -27.05% 3.42% Top 2 ads with highest CTR had {keyword} as the title 54% higher CTR than winning recipe 94% lower conversion rate than winning recipe Influence of the KW (only) in the Title
  11. 11. Quality Score: CTR Major Factor so… Title Major Factor in CTR All things being equal ads with kw insertion in title usually get higher CTR. But why does quantity equal quality? Quality Score Factors
  12. 12. But what is Relevant? Used to be…Google Ad Optimizer Turned Off, Traffic distributed evenly across recipes Last few months…not the case: What’s the Effect of this change by Google on Relevance & ROI? KW (only) in the Title Gets the Clicks!
  13. 13. Ad Test Plan Summary Who: Major Product Aggregator/Vertical Search Engine Objective: Increase User Registrations from Google Ads Conversion Metric: Free user registration sign-up Campaign Population: Multiple Google AdGroups Campaign Length: 1 month Case Study 2
  14. 14. Recipe A-C Title: “{keyword}” Titles, Impressions & Relevance 5.72% 264,784 Recipe C 5.72% 265,045 Recipe B 5.72% 265,909 Recipe A Conversion Rate Impressions Recipe
  15. 15. Recipe D-F Title: “Find {keyword}” Titles, Impressions & Relevance 7.76% 47,066 Recipe F 5.24% 47,289 Recipe E 8.59% 47,093 Recipe D Conversion Rate Impressions Recipe
  16. 16. Recipe G-I Title: “The Wait is Over” Titles, Impressions & Relevance 11.76% 46,579 Recipe I 8.11% 46,795 Recipe H 9.84% 46,518 Recipe G Conversion Rate Impressions Recipe
  17. 17. <ul><li>Who Decides Relevance? </li></ul><ul><li>Group G-I had 17.58% of the impressions of Group A-C </li></ul><ul><li>Group G-I had 32% of the conversions of Group A-C </li></ul><ul><li>Group G-I had 82.3% higher conversion rate than Group A-C </li></ul>
  18. 18. Ad Test Plan Summary Who: Major Services Brand Objective: Increase Service Subscriptions Conversion Metric: Purchase/Trial Campaign Population: Google AdGroups (Brand & Services terms) Campaign Length: 1 month Test Type: 3x2 Multivariate Case Study 3
  19. 19. 3 x 2 MVT Test Array (Service) (Service) (Brand) (Service) (Brand) (Service) Title Protect your ___ from ___ using (Service) (Brand) offers the strongest (Service) Protect your ___ from ___ using (Service) (Brand) offers the strongest (Service) Description www.brand.com www.brand.com/(service) www.brand.com/(service) www.brand.com URL D C B A Recipe
  20. 20. Overall Results Both Segments -- RPV 16.57% 15.24% 2.15% 14.27% D 71.25% 160.50% 4.87% 11.26% C 11.04% 53.68% 2.87% 29.19% B -- -- 1.87% 45.29% A Lift Lift Conversion Rate Visitors Recipe
  21. 21. Trial Subscriber Results Segment 1 135.71% 0.59% 14.27% D 582.82% 1.70% 11.26% C 245.79% 0.86% 29.19% B -- 0.25% 45.29% A Lift Conversion Rate Visitors Recipe
  22. 22. Regular Subscriber Results Segment 2 -- RPV 17.53% -3.30% 1.57% 14.27% D 61.32% 95.38% 3.16% 11.26% C 12.77% 24.11% 2.01% 29.19% B -- -- 1.62% 45.29% A Lift Lift Conversion Rate Visitors Recipe
  23. 23. Overall Element Contribution Both Segments 64.75% www.brand.com/service URL 11.97% (Brand) offers the strongest (Service) Description 23.28% (Service) Title Influence on Conversion Winning Alternative Element
  24. 24. Trial Sub Element Contribution Segment 1 64.53% www.brand.com/service URL 5.46% (Brand) offers the strongest (Service) Description 30.01% (Service) Title Influence on Conversion Winning Alternative Element
  25. 25. Regular Sub Element Contribution Segment 2 59.71% www.brand.com/service URL 22.00% (Brand) offers the strongest (Service) Description 18.28% (Service) Title Influence on Conversion Winning Alternative Element
  26. 26. Brand and Service Combo Titles had highest CTR but poorest RPV Service Titles had poorest CTR but highest RPV Results
  27. 27. Titles matter more for CTR than for Conversion Description Lines Matter Especially when everyone is doing kw insertion! URL Matters Especially when you have brand awareness Summary
  28. 28. Thank You! Blog: http://www.optimizeandprophesize.com

×