Price testing can be the most profitable a/b tests your company will run. Yet optimizing pricing, packaging, and the product line-up can be tricky. Here's an outline of what to consider, based on years of a/b and MVT experiments. Every CRO roadmap should include pricing strategy tests.

2. Pricing Challenge in Summary
Nichols & May “$65 funeral”

3. About Me
 I have run over 500 marketing experiments
 Many of the most successful tests have been related
to pricing & packaging
 I teach a/b testing courses around topics such as
statistics and design of experiments
Have run experiments for...

4. Intro
Price testing isn’t just the price. It encompasses:
 Price levels
 Price gaps
 Price anchors
 Discount strategy
 Line up strategy
 Feature composition
 Cannibalization
 Pricing by Marketing Channel
 Pricing by Customer Segment
 Pricing by Vertical
 and more

5. Broader Impact
Pricing decisions don’t just impact immediate
conversion. It also impacts...
 customer satisfaction and loyalty
 ability to sell other things in the short-term
 refund rates
 future revenue stream possibilities
 brand perception
 customer expectations

6. Can you? Should you?
There’s some debate around the legitimacy of price testing
against live customers.
 It is done all the time.
Be careful not to discriminate.
 Randomization is not discrimination.
Be careful around consistency of experience
 Test targeting can be tricky in particular for logged-out
experiences
 Test quickly and have a back-pocket offer.

7. Toolset
 Tougher to do with off-the-shelf testing tools
 Often requires building own testing and reporting
capabilities

8. Goals
As with any experiment, what are the key success metrics? Just
because it’s a “price test” doesn’t mean everyone is necessarily on
the same page at the start.
KPI options
 Top line revenue
 Price charged at checkout
 Bottom line profit short-term
 Bottom line profit long-term
 New customers
 NPS / WTR / WOM etc.
 etc.

9. Effort vs Reward
 Price testing is at the harder end of the optimization
spectrum
 The payoff is well worth the investment however
 I have seen > 40% lifts from price testing
 Price testing has the broadest impact: not a page or
segment usually but an overlay on top of the whole
business – huge leverage

10. Lab vs Live
 Ideal is to start testing in the lab (surveys, focus
groups, conjoints, etc.)
 Once you have pretty good confidence in certain
directions, then move it to live tests
 Does not have to be done in that order however.

11. Controlling Variables
 Simplest approach is to test in phases:
 Packages (e.g., components of the offer)
 Lineup (e.g., how many packages there are)
 Messaging (e.g., how to sell each package)
 Prices for Core Offering (the actual price levels)
 Price levels for Add-On Offerings
 Timing or Sequence of Sales Cycle
 Discounts
Be sure that you are either testing just one of these factors at a
time, or else establish an appropriate experimental design to tease
out the effects of the different factors.

12. Continuous vs Discrete Variables
 Price is continuous, can test more points on the
curve without additional sampling overhead
 Number of packages is not quite continuous
 1 means no choice
 Odd numbers are best
 Limited to single digits

13. When to Price Test
 You have new products
 Your products have changed
 Your competition has changed their pricing
 You’re at a new part of the market adoption curve
 You changed strategic directions

14. Subscriptions
 SaaS? Have to use cohorts
 Trial period? Have to track free to paid
 Monthly subscription? Get a few points on the curve
and estimate
 Annual subscription? Harder to estimate

15. Choosing Your Customers
 In some cases, the prices you choose determines the
kinds of customers you’ll get.
 The are downstream implications of acquiring different
types of customers.
 Lower list prices or frequently discounted prices bring in
more more conscious customers. The may be:
 less loyal
 less likely to buy future offerings
 more likely to cancel
 more likely to use customer support

16. Discounting
 You should study the
 frequency of discounts
 steepness of discounts
 duration of discounted period
 predictability of discounts
 impact on brand perception
 impact on loyalty
 impact on margins
 discounting wars with competitors

17. Scarcity
Scarcity is closely related to price
 The price will go up soon
 The discount will end soon
 There are a limited number
 This offer only available for customers from... /
who...

18. Fake Door Testing
 Possible in a modified manner with price tests, but
can be tricky. Can pair with limited quantity
messaging when done authentically.
 More easily done with product

19. Free
 A free version can bring in lots of potential customers
 Offering a free version can be viewed as an advertising expense
 Free can be important or even necessary from a competitive standpoint
 Free can also impact brand perception and ability to sell very high end products
 Free can also incur support costs
 Free can be a way to get valuable customer feedback to improve the product
 Free can be a way to build a network effect for certain offerings
 Free has a cannibalization impact that can be measured

20. Free Trial
To test:
 Credit card required vs not
 Trial period
 Auto-charged or not
 Feature limits
 What happens if customer doesn’t end up paying. Do
any features remain? Any data saved?

21. Refer a Friend as Pricing
 “Refer 3 friends to get an additional 1 GB of space”
is a form of price... instead of paying you money,
they are paying you in-kind with social-based
marketing
 How else can you “charge” without asking for
dollars?
 content uploads
 P2P disk usage
 data or other ways to help build/grow the product

22. Price Display
 Actual Price
 Total amount
 Per Month, Per Day etc.
 Perceived Price
 Valued at
 Compare at
 Price before of after discounts?
 Component Value
 Price Display
 Strike though
 List
 Discount Display
 % off
 $ off
 You save
 Badges
 Best value
 Most popular

23. Magic numbers
 Endings .99 .95 etc
 Round numbers
 Trick specificity

24. Package Hiding
 Companies tend to ever expanding product flavors and
package sets. Over time what started out as a rational
for the business becomes a drag on conversion
 Simplest way to approach price testing is to trim back the
lines offered to a more management set (1 to 3)
 Supplementary packages can be hidden completely or
pushed down to deeper levels or segment-based
marketing – excluded from general pop messaging
 This approach will give some idea of loss due to hazards
of excessive choice

25. Loyal Customer Pricing
 Upgrader discounts
 1st year premium
 Minimum period

26. Quantity Pricing
 Buy more save more
 Usage based

27. Indecision Risk
 Too many price value trade-offs and prospects bail
out
 Must also find right level of product detail to not
overwhelm, but still sufficiently inform

28. Component Values
 Hard to market all the features or benefits
 Should quantify relative value of each and present
as such
 Each package must have one sufficiently high value
component for important customer segments

29. Negative Example

30. Negative Example
 Strike through only on one package
 Huge jump between low and mid package
 Highlight (“best value”) on most expensive package
 Package/Pricing but no CTA based on that info
 State fees buried elsewhere
 Inconsistent messaging about differences between
packages
 Combination of transaction, subscription/trial, and lead
gen not easily parsed

31. Positive Example

32. Positive Example
 Odd number of packages
 Center package is recommended
 Pricing displayed at the lower-seeming “per month” amount
 Original strikethrough price, % off highlighted & amount saved
 Dead-simple benefits list; Clutter free design
 Visual hierarchy of CTAs
 Risk-free messaging
 Minimal asterisking
This page was tested extensively:

33. Game Changing Examples
 Amazon Prime
 Amazon partner / affiliate store
 Apple App store
 Southwest Airlines
 Netflix
 Adobe Creative Cloud

34. Conclusion
 Price testing encompasses a huge array of testing
possibilities
 Price testing requires a serious commitment and
some finesse to get right
 Price testing is likely the biggest lever you have