We present a new “cheap and fast” methodology for measuring the persuasiveness of communication. This methodology allows conducting experiments with thousands of subjects for a few dollars in a few hours, by tweaking and using existing commercial tools for advertising on the web, such as Google AdWords. Further details in: Guerini M., Strapparava C. & Stock O. “Evaluation Metrics for Persuasive NLP with Google AdWords”. In Proceedings of LREC2010.

1. Persuasive Message Evaluation
with Google AdWords
Marco Guerini
Carlo Strapparava
Oliviero Stock
1

2. Introduction
2

3. Problems
• Evaluating systems and theories about persuasion is a
bottleneck: expensive and time consuming experiments.
designing the experiment, inventing cover stories,
recruiting a sufficient number of subjects, making
them take part to the experiment, dispense
questionnaire (pre-post), gather data, analyze them.
• In lab settings results are biased by subjects awareness
of being tested.
3

4. Crowdsourcing - Mechanical Turk?
Mechanical Turk used for behavioral studies
(Mason and Suri, 2010) to overcome part of
these limitations - like subject recruitment.
We believe that it is not the right instrument
to assess behavioral changes, and, more
generally, persuasive effects…
4

5. Requirements
1. Studies as ecological as possible (real,
even if controlled, scenarios).
2. Subjects must not be aware of being under
observation.
3. Subjects must not be biased by external
rewards.
5

6. 6

7. Guerini M., Strapparava C. & Stock O. “Evaluation Metrics for
Persuasive NLP with Google AdWords”. In Proceedings of
LREC2010, May 2010.
Proposed Approach
7

8. Approach
• Goal: develop “cheap and fast” methodologies to
measure the persuasiveness of communication.
• Solution: tweaking and using existing commercial tools
for advertising on the web, such as Google AdWords.
!
Conducting field experiments with thousands
of subjects for a few dollars in a few hours
8

9. Adwords Description
• Google AdWords is Google advertising
program. It lets advertisers display their ads
only to relevant audiences by means of
keyword-based contextualization on the
Google network, divided into:
• Search network: Google search pages,
search sites etc.
• Content network: news pages, topic specific
websites, blogs and other
9

10. Put your test
messages on the Web.
10

11. AdWords Metrics
Identifying the performance (persuasiveness) of
each ad from various perspectives:
• CTR: the number of clicks divided by the number of times
an ad has been displayed. Indicates the initial impact.
• Web Analytics: insights on landing site: number of visited
pages, time spent on the site, etc. Indicates interest/
attitude generated.
• Conversion Rate: how many user turned into conversions
(e.g. buying a product, download a brochure). Indicates
message complete success.
• ROI: The more relevant (from a research pov) the action
the user performs, the higher the value.
11

12. AdWords Targeting
Targeting options for different evaluation
scenarios:
• Language message impact in different
languages.
• Location message impact in different cultures
sharing the same language.
• Keyword matching message impact with
users having different interests.
• Placements message impact on people with
different values –democrats/republican sites.
12

13. AdWords Tuning
To facilitate statistically meaningful exp. (subj
randomness, etc.) some tuning needed:
• Ad serving (optimize vs rotate)
• Ad delivery (standard vs. accelerated)
• …
No longer necessary.
Since our first experiments some proposal have
been implemented in AdWords, under the tool
AdWords Campaign Experiments (ACE)
13

14. Experiments
14

15. Experimental Setup
• Pilot experiments to test irony, using a real
promotion campaign.
• Experiments based on a between-subject
design with two conditions, one ad for
each condition:
• Control condition, with a neutral message.
• Experimental condition with ironical message.
• χ2 test used on CTR.
15

16. Experimental Design
Two experiments with text based ads on the
search network (promoting a book).
Two experiments with img
based ads on the content
network (promoting the
book series).
16

17. Stimuli Material
• “Diversamente Occupati” (tr. “differently
employed”) mocks the Italian “diversamente
abile” (tr. “differently able”), a politically
correct word for handicapped persons.
• “Diversamente Occupati” ironically refers to
temporary employed (i.e. “differently
employed”) people, generally considered a
less than ideal state of employment.
17

18. Dura%on
Cost
Subjects
24
h
1,2
€
-­‐
CPC
0,04
€
~
3.000
Dura%on
Cost
Subjects
24
h
1,5
€
-­‐
CPC
0,04
€
~
4.000
Experiments with
text based ads
Context: Keyword “Precario”
18

19. Results
1st Experiment - CTR (1A) = 1,7% CTR (1B) = 0,5%
• Test significant  condition 1A “Sei Precario?” (tr. “Are you
temporarily employed?”) performs three times better than 1B “Sei
Diversamenteoccupato?”.
• Possible bias: given keyword matching the word “Precario” in condition
1A is automatically highlighted by Google.
2nd Experiment - CTR (2A) = 1,2% CTR (2B) = 0,8%
• Test not significant  first experiment biased by highlighting.
• 2B “Sei DiversamenteOccupato?” outperformed other conditions (1A,
1B, 2A) w.r.t. permanence of users on landing site (~3 min vs ~1 min).
• Capitalizing letter “O” in “DiversamenteOccupati” helped users spelling
the word and get the irony - that induced more interest in the subjects.
19

20. Dura%on
Cost
Subjects
12
h
7
€
-­‐
CPC
0,60
€
~
4.200
Dura%on
Cost
Subjects
1
week
2
€
-­‐
CPC
0,20
€
~
12.000
Experiments with
image based ads
Context 1: a site of job offers
Context 2: the whole network
20

21. Results
3rd Experiment - CTR (C1) = 0,5% CTR (C2) = 0,1%
• Test significant, condition C1 performs five times better than C2.
• Should we conclude that the img has a negative impact on the
message?
4th Experiment - CTR (C1) = 0,16% CTR (C2) = 0,08%
• Test not significant  In the previous experiment the img did not
have a negative impact by itself, rather it had a “positive” impact
in cutting uninterested users, by helping them in “disambiguating”
the message (users were looking for a job, not for amusement).
21

22. Conclusion
• AdWords is an appropriate context for
evaluating persuasive messages.
• AdWords proved to be very accurate,
flexible and fast, far exceeding our
expectations.
• The ACE tool bring us a step further.
22

23. Further details in:
Guerini M., Strapparava C. & Stock O. “Evaluation Metrics
for Persuasive NLP with Google AdWords”. In Proceedings
of LREC2010.
23

24. 24