Search Engine Marketing is a highly competitive field. To remain effective in the space, advertisers need to be smart about how they approach reaching customers via paid search. One strategy that helps ensure the success of paid search campaigns is regular A/B testing.

1. The Ins and Outs of Paid Search A/B Testing | 1
The Ins and Outs of Paid
Search A/B Testing
Written By: Dawn Busa, SEM Specialist II
Edited By: Tim Lavelle, Director SEO/Social Media

2. The Ins and Outs of Paid Search A/B Testing | 2
Introduction
Search Engine Marketing is a highly competitive field. To remain effective in the space, advertisers need
to be smart about how they approach reaching customers via paid search. One strategy that helps
ensure the success of paid search campaigns is regular A/B testing.
What is A/B Testing?
A/B testing compares a single variable to figure out which version of that variable provides the best ad
performance.
These side-by-side comparison tests enable paid search marketers to fine-tune their message in order to
achieve goals in a more efficient way, whether they’re optimizing for impressions and brand awareness,
or tangible conversions like purchases, form submissions, phone inquiries or leads.
A/B testing is typically conducted across various components of the paid search campaign, including
such elements as landing page design, calls to action, page content, and ad copy.
Why Conduct A/B Testing?
From a purely statistical standpoint, improving ad performance via A/B testing can improve click-
through rates, conversion rates, ad ranks and Google Quality Score, reducing click prices, optimizing
advertising spend and leading to ROI improvements.
Process: Brainstorm and Research
Before conducting A/B testing, it is important to first consider the following factors:
 Defining goals and success metrics
 Reviewing competitor behavior
 Analyzing ad performance
 Selecting an interface
Defining Goals & Success Metrics
To ensure a successful experiment, advertisers must first determine what they’re seeking to accomplish
– is the goal of this campaign solely focused on brand awareness? If so, a high click-volume and click-
through rate are strong key performance indicators.
Is the goal to generate leads? Look at conversions and cost-per-lead to provide a better understanding
of how the ad is performing.
Every goal has a different indicator, so goals must first be defined before performance can be reviewed,
and advertisers may apply the results of their A/B tests towards achieving those goals.

3. The Ins and Outs of Paid Search A/B Testing | 3
Reviewing Competitor Behavior
To ensure competitiveness, advertisers must review the landscape of their particular industry. Who are
the main competitors, and how are they utilizing Paid Search to conduct their own business? By
researching their behavior, advertisers can generate ideas on how to position their own ads in such a
way that they’re differentiated from the competition, improving their ability to capture attention.
A simple Google search for keyword phrases that you’re considering bidding on provides great insight on
the competitive landscape. For example, if you were considering running a campaign for a Mortgage
Broker, the first step in your competitive research would be to search “mortgage rates” on Google,
which returns the following results:
In this space, it’s obvious that listing APR rates in either the headline or ad copy is the conventional
approach, and most ads also include the terms “Mortgage” and “Rates” in their ad copy as well. With
this basic information, you can begin testing these elements in your own ad copy, seeing how they’ll
affect campaign performance.
In addition to simple searches, leveraging third party data aggregation tools, such as SpyFu,
KeywordSpy, or others, will provide a high-level overview of what competitors are doing, and how
they’re getting it done. SpyFu, in particular, offers an AdHistory feature that enables you to see the
progression of a competitors ads, including which elements were tweaked, and how well the ads were
performing during the time period that they were running.
Such an analysis allows marketers to leverage competing campaigns, and competitors’ tactical tests, as
A/B tests of their own, helping to determine which calls-to-action, offers, product features and
keywords appear to be the most successful within their particular niche.

4. The Ins and Outs of Paid Search A/B Testing | 4
Analyzing Ad Performance
Before launching any A/B tests, marketers must first determine whether or not they’ve collected enough
data to work with. When launching the tests, use the top performing ad as the control ad when
introducing new ads to test, then before making any changes to the campaign, ensure that there is a
sound statistical difference behind the collected data.
Selecting an Interface
Determine which interface will be used to compare the ad performance. When conducting A/B testing
on Google Adwords, ensure the proper ad rotation settings are utilized. The screenshot below shows
four options to choose from: Optimize for clicks, Optimize for conversions, Rotate Evenly and then
Optimize, and Rotate Indefinitely. This configuration is set at the campaign-level.
Optimize for Clicks
Under this ad rotation setting, Google will automatically show preference to the ad that is expected to
receive the most clicks, based on previous click-through rates. This setting will be ideal for advertisers
who are solely focused on getting click volume and using CTR as their success metric.
Optimize for Conversions
This option permits Google to favor the ad expected to bring most of the conversions to the campaign.
This setting is ideal for advertisers who are testing different landing pages with the same ad copy.
Should there be insufficient conversion data for the ad group, Google will automatically default testing
to optimize for clicks.

5. The Ins and Outs of Paid Search A/B Testing | 5
Rotate Evenly
This setting will allow both ads to run evenly across a time span of 90 days. If the settings remain
unchanged, Google will automatically optimize the ad serving depending on the bid setting. If the
campaigns are set to Enhanced CPC, Google will optimize for conversions, otherwise, Google will
automatically optimize for clicks.
Rotate Indefinitely
Although Google warns that this setting is “Not recommended for most advertisers”, this enables
advertisers to properly test both ad variations without having to worry about Google automatically
optimizing their ads. Advertisers should use this setting for A/B testing, as they will be able to properly
determine the best performing ad with a high level of confidence.
When advertising on Bing Ads, their ad rotation settings are found on the Ad Group level, rather than
the campaign settings. Currently, advertisers have two choices: Optimize for clicks or Rotate Ads Evenly.
Similar to Google’s settings, the former setting prioritizes the ads expected to bring the most click
volume, while the latter is best used for Optimizing conversions, or for Testing Ads Evenly.
The A/B Testing Process:
To optimize the results of A/B testing, advertisers should ensure that they follow these three simple
steps.
1. Test Only One Element at a Time
When testing ads against each other, it’s necessary that only one variable is changed at a time,
otherwise it becomes extremely difficult to gauge how individual changes impact performance.
Where multiple changes are made simultaneously, it’s not at all easy to pinpoint which component
change contributed to performance variations.
Advertisers can keep busy testing the following variables, followed by some examples:
 Headlines
o Dynamic Keyword Insertions vs. Static ads
 Body
o Calls-to-action
 Buy Now vs. Order Today
o Special offers and discounts
 10% Off vs. Free Shipping
o Punctuation/Capitalization
 Ad Extensions
o Sitelinks, Location, App, Call Extensions, Consumer Ratings, Reviews, Seller Ratings, and
Social Annotations.

6. The Ins and Outs of Paid Search A/B Testing | 6
 Display URLs
o Note that these do not have to match Destination URLs. Marketers should use them as
an opportunity to add more keywords into their copy.
2. Regularly Analyze Performance
Once enough data has been collected, analyze it to determine whether or not changes should be made
to the general campaign.
Note that when A/B testing, simply optimizing for the highest performing click-through rate, or the
highest conversion rate doesn’t always generate the best results. Instead, advertisers should optimize
ads in order to find the sweet spot that balances out the two metrics to return the best ROI.
Basing decisions solely on optimizing click-through rates may guarantee more traffic, but it doesn’t
necessarily mean that those additional clicks will lead to actual conversions.
Inversely, placing the entire emphasis on conversion rates could reduce inbound traffic by such a
substantial amount that campaign ROI decreases, leading to performance declines.
Attention to detail will be required to ensure a continuing proper balance between click-through rates,
Quality Score, cost-per-click and, most importantly, cost-per-acquisition.
3. Never Stop Testing
Once marketers have determined which ad is performing the best, they should pause the
underperforming version and continue testing by running a new variation against the current optimal
ad.
To maximize campaign performance over an extended period of time, make sure to continue running
A/B tests on a regular basis.
Conclusion
A/B testing is crucial to achieving search engine marketing success. The U.S. Interactive Media Search
Engine Marketing team is well-versed in running rigorous A/B tests, with extensive experience
leveraging this vital strategy to optimize client spends and help reach goals in a more efficient manner.

7. The Ins and Outs of Paid Search A/B Testing | 7
References Used:
Ad rotation settings (- AdWords Help)
https://support.google.com/adwords/answer/112876?hl=en
Conversion Rate Optimization in Paid Search: Why Click Through Rate Matters (Search Engine Land)
http://searchengineland.com/conversion-rate-optimization-in-paid-search-why-click-through-rate-
matters-118991
Bing Adcenter Ad rotation settings
http://advertise.bingads.microsoft.com/en-us/help-topic/how-to/51002/ad-rotation-explained