This document provides guidance on designing experiments to test interventions aimed at changing user behavior. It recommends identifying a key behavior to measure, barriers preventing that behavior, and testing approaches to remove barriers based on behavioral economic principles. As an example, it suggests testing the impact of adding a sense of urgency to adopt negative keywords by having the control condition simply mention adopting keywords for a reward, while the test condition adds language saying it must be done within 7 days. The hypothesis is that urgency will increase actions by providing a clear timeline. It stresses the importance of a proper control condition that differs from the test condition in only one aspect being tested.

1. Designing
xperiment
an

2. This is a brief
guide to designing
experiments

3. This is a brief
guide to designing
experiments
WARNING!
Be sure to fill in all blanks before
proceeding to the next step. Skipping
this step will prevent you from getting
the most out of this guide.

4. Now, with loss aversion in mind,
let’s proceed to the next step...
This is a brief
guide to designing
experiments
WARNING!
Be sure to fill in all blanks before
proceeding to the next step. Skipping
this step will prevent you from getting
the most out of this guide.

5. Identify Key
Behavior1

6. What is the key behavior you
want your users to do?
Identify Key
Behavior1
e.g. sign up for workshop

7. Identify Key
Behavior1
Key Behavior = Measured Variable
(what you will measure to see the effect of your test)
?
EXPERT TIPAsk around, does everyone
involved with your project
agree that this is the most
important behavior to
measure?

8. Identify Behavioral
Barriers 2

9. Identify Behavioral
Barriers 2
What are the barriers preventing
people from doing the key behavior
that you want them to do?
not enough time
read e-mail and forget about it
put off until later
(list as many as you can think of)

10. Choose a
Barrier
3
A B C D

11. Choose a
Barrier
3
Choose the behavioral barrier (1) that
you think contributes most to people
not doing the key behavior, and
explain it in the following format:
A B C D

12. Choose a
Barrier
3
The reason people are not currently partaking in
adopting negative keywords is because they put it off until later
key behavior behavioral barrier
A B C D

13. Removing
Barriers

14. Think about behavioral economic
principles. What are some ways
that you can remove this barrier?
(list as many as you can*)
Removing
Barriers

15. Precommitment: Put it on their calendar for
them
In-your way instructions: Have a pop-up
when they log into account
Urgency: Provide a deadline for the action to
be completed
Removing
Barriers

16. Removing
Barriers
*If you don’t know
principles, check out
our workbooks

17. Choose the approach that you think
is most likely to remove the barrier.
What is the behavioral economic
principle that you are using?
Choose a
Theory

18. By using urgency, we
can increase the
number of people
who adopt negative
keywords.
Choose a
Theory

19. If we do this...
then this will happen...
because...
If we add a sense of urgency, then people will be
more likely to take action to add negative
keywords, because urgency gives people a clear
timeline in which they must complete the action in
order to get the incentive
rite the Hypothesis

20. Creating Conditions
(Control & Test)
How would you test this? Let’s start off with just
two conditions (your most basic A/B test). Design
your test below. What will your each group of
participant see?
• Write out the exact text they will see?
• Will you use images? If so, add the exact images they will see?
Control Test

21. Must have at least two conditions -
control condition and test condition
The control condition must be exactly
the same as the test condition, except for
the ONE key difference you are testing.
Creating Conditions
Control Test

22. You must have a
control
Make your ad more relevant. Adopt
keywords and receive $50 in
AdWords Credit
Make your ad more relevant. Adopt keywords
in the next 7 days and receive $50 in
AdWords Credit
Control Test
•The difference between the control and the test is the one
thing you are testing
• Urgency: “In the next 7 days”
•Restricting to only one difference between any two condition
allows you to learn something from results.
• Imagine that you include 2 difference in the test (e.g., urgency and amount of incentive), and let’s imagine that
the test works twice as well as the control. Why did it happen? You don’t know - it could have been urgency it could
have been the amount of the incentive, or a combination of both)

23. Midterm Review

24. Midterm Review
Do I have a control
condition?
Do I have only one
difference between
the two conditions?
Checklist

25. Midterm Review
Given the information I
provided,
Does the customer
understand the key behavior
I want them to do?
Does the customer
understand how to get to
that key behavior? Are the
my instructions clear?