Persuasiveness and Audience Reactions in Political Speeches


Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Persuasiveness and Audience Reactions in Political Speeches

  1. 1. Persuasiveness and AudienceReactions in Political Speeches Marco Guerini
  2. 2. Contributors Marco Guerini Carlo Strapparava Oliviero Stock Danilo Giampiccolo Rachele Sprugnoli Giovanni Moretti
  4. 4. Persuasive NLP•  Persuasion is becoming a hot topic in Natural Language Processing.•  Automatic analysis and recognition of the persuasive impact of communication.•  Address the various effects which persuasive communication can have in different contexts on different audiences.
  5. 5. Approaches•  Knowledge-based: Starting from theory.•  Data-driven: Starting from linguistic data. Linguistic data should be possibly augmented with annotation of various audience reactions.
  6. 6. Resources Examples•  Long texts: Political Speeches•  Short texts: Posts on Social networks•  Short sentences: Advertising Slogans•  Single words: Evocative Brand Names
  8. 8. Characteristics & Collection•  CORPS: CORpus of tagged Political Speeches•  Hypothesis: tags about audience reaction, such as APPLAUSE, are indicators of hot- spots, where persuasion attempts succeeded•  Collection: Annotated speeches from various web sources•  Normalization: Metadata insertion (speaker, date, title, etc.) and Semi-automatic conversion of tags names to make them homogeneous
  9. 9. Characteristics & Collection CORPS - Main StatisticsTotal number of speeches: ~ 3,600Total number of speakers: ~ 200Total number of words: ~8MTotal number of tags: ~ 66,000Temporal range: from 1917 to 2010
  10. 10. CorpsFormatConverter•  Four annotators have been trained.•  Annotation supported by an ad-hoc standalone application.•  The tool facilitates the extraction of the speech text and metadata from the Web sources.•  The tool automatically converts the most frequent tags.
  11. 11. Example of Tags and Conversion
  12. 12. Document Structure ex. - JFK {title} Ich bin ein Berliner {/title}{event} ----- {/event}{speaker} John F. Kennedy {/speaker}{date} 26 June 1963 {/date}{source} {/source}{description} ----- {/description}{speech} … {/speech}
  13. 13. Speech Fragment ex. - JFKFreedom has many difficulties and democracy isnot perfect. But we have never had to put a wall upto keep our people in, to prevent them from leavingus. {APPLAUSE ; CHEERS}I want to say on behalf of my countrymen who livemany miles away on the other side of the Atlantic,who are far distant from you, that they take thegreatest pride, that they have been able to sharewith you, even from a distance, the story of the last18 years. I know of no town, no city, that has beenbesieged for 18 years that still lives with the vitalityand the force, and the hope, and the determinationof the city of West Berlin. {APPLAUSE ; CHEERS}
  14. 14. Speeches Distribution Number of speeches per Speaker.
  15. 15. Speeches Distribution Temporal distribution of the Speeches
  17. 17. Audience Tags - CountTag Count{AUDIENCE} Yes! {/AUDIENCE} 482{AUDIENCE} No! {/AUDIENCE} 390{AUDIENCE} Four more years! Four more years! {/AUDIENCE} 346{AUDIENCE} Yes, sir {/AUDIENCE} 87{AUDIENCE} U.S.A.! U.S.A.! U.S.A.! {/AUDIENCE} 41{AUDIENCE} All right {/AUDIENCE} 39{AUDIENCE} Flip-flop! Flip-flop! Flip-flop! {/AUDIENCE} 39{AUDIENCE} Hooah. {/AUDIENCE} 38{AUDIENCE} Reagan! Reagan! Reagan! {/AUDIENCE} 37… …{AUDIENCE} Hooah! {/AUDIENCE} 24{AUDIENCE} Tell it {/AUDIENCE} 23… …
  18. 18. Comment Tags - CountTag Frequencies{COMMENT="Inaudible"} 257{COMMENT="A toast is offered"} 30{COMMENT="The bill is signed"} 30{COMMENT="The medal was presented"} 26{COMMENT="The medal was awarded"} 24{COMMENT="Recording interrupted"} 18{COMMENT="The citation is read"} 18{COMMENT="The citation was read"} 16{COMMENT="Interruption"} 9{COMMENT="A moment of silence was observed"} 8… …
  19. 19. Audience Reactions Typologies•  Positive-Focus: a persuasive attempt that sets a positive focus in the audience. Tags considered: {APPLAUSE} , {STANDING-OVATION} , {SUSTAINED-APPLAUSE} , {CHEERING} , etc.•  Negative-Focus: a persuasive attempt that sets a negative focus in the audience. Negative focus set towards the object of the speech not on the speaker. {BOOING} , {AUDIENCE} No! {/AUDIENCE}•  Ironical: Indicate the use of ironical devices in persuasion. Tags considered: {LAUGHTER} and multiple tags containing laughter.
  20. 20. Audience Reactions Typologies•  These 3 groups represents different effects which political communication can have in different contexts on different audiences. Reaction Typology Count Percentage POSITIVE-FOCUS TAGS 49275 0.74 IRONICAL TAGS 15660 0.24 NEGATIVE-FOCUS TAGS 1147 0.02
  22. 22. Tag Density•  How much “persuasive” is, on average, a speech or group of speeches?•  Compute how many audience reaction tags are present in a speech (normalize according to speech length).
  23. 23. Tag Density•  Given a set of speeches - e.g. Democrats’ speeches -, tag density can be computed in two different ways: –  Micro-averaged tag density (µ) - counting all tag occurrences in the set and dividing the result for the total number of words. –  Macro-averaged tag density (M) - computing the tag density for each category (e.g. each Democrat speaker) and then averaging over the results of each speaker.•  µ gives the “real” tag density of the dataset, while M avoids over-representation of unbalanced classes (e.g. a vast majority of Bill Clinton’s speeches).
  24. 24. set of n speeches S, where aasingle speech is is repre- set of n speeches S, where single speech repre- |tii|| represents the number of tags inin a given speech |t represents the number of tags a given speech Tag Densitywe can define µnumber of words in the same speech; we can define µ umber words in the same speech; A set of n speeches, n n i=1 |tii | i=1 |t | |ti| represents the number of µ = n n tags in a given speech/ (1) (1) i=1 |wii | i=1 |w | categorybe defined as: be defined as: |wi| represents the number of |C| |ti | |C| |ti | words in the speech/category i=1 |wi | M= i=1 |wi | |C| represent the number (2) of M= |C| categories (speakers) in the(2) |C|number of categories (speakers) speeches.of speeches, set of in the setnumber of categories (speakers) in the set of speeches,the total number of tags and words for the category. he total number of tags and words for the category.
  25. 25. Tags Density - CorpusOverall Tag density (μ): 0.0084PF-density (μ): 0.0062I-density (μ): 0.0020NF-density (μ): 0.0002
  26. 26. Tags Density – Main SpeakersSpeaker Speeches Tag-Density PF-density I-density NF-densityBill Clinton 889 0.007 0.005 0.002 0.00001George W. Bush 427 0.015 0.012 0.002 0.00005Ronald Reagan 388 0.004 0.001 0.003 0.00044Dick Cheney 356 0.011 0.008 0.002 0.00061Barack Obama 347 0.01 0.008 0.003 0.00007John F. Kennedy 316 0.009 0.008 0.001 0Michelle Obama 107 0.009 0.005 0.003 0.00001Margaret Thatcher 102 0.005 0.004 0.001 0.00001Laura Bush 93 0.015 0.014 0.001 0Richard M. Nixon 61 0.006 0.005 0 0.00008Al Gore 53 0.007 0.005 0.002 0.00004Alan Keyes 51 0.004 0.003 0.001 0.00007
  27. 27. Tags Density – Party and Gender Party Corpus-Cover. Tag-Density PF-density I-density NF-densityDemocrats 0.45 0.0075 0,0055 0,0019 0,000027Conservatives 0.55 0.0097 0,0072 0,0022 0,000309 Gender Corpus-Cover. Tag-Density PF-density I-density NF-densityFemales 0.11 0.0085 0.0067 0.0018 0.000007Males 0.89 0.0083 0.0062 0.0020 0.000158 Micro-averaged densities (μ)
  28. 28. Tags Density – Party and Gender Party Corpus-Cover. Tag-Density PF-density I-density NF-densityDemocrats 0.45 0.0076 0.0056 0.0019 0.000036Conservatives 0.55 0.0094 0.0076 0.0017 0.000199 Gender Corpus-Cover. Tag-Density PF-density I-density NF-densityFemales 0.11 0.0068 0.0055 0.0013 0.0000007Males 0.89 0.0070 0.0052 0.0017 0.0000444 Macro-averaged densities (M)
  29. 29. Tags Density – Party and Gender•  While the Democrats/Conservatives partition is well balanced (0.45 vs. 0.55), the Males/Females partition is unbalanced (0.89 vs. 0.11).•  Tag density is slightly higher for Conservative speakers (the same holds for positive-focus tags), while the ironical- focus tags have almost the same density in both groups.•  Analysis ex. Negative-focus tags (representing a more “aggressive” kind of rhetoric): density in the Conservative group is 11 times higher than the in Democrats. A similar consideration for the male/female distinction: while other tag densities are almost the same, for the negative-focus tags we have a density 60 times higher for male speakers.
  30. 30. Tag Density - Temporal Distribution
  31. 31. Language andMicro Analysis
  32. 32. Language Persuasiveness•  Are there words, linguistic expressions that are more “persuasive” than others?•  In a speech not all text fragments have the same importance. Consider audience reaction tags.
  33. 33. Possible Uses•  Persuasive expression mining. recognition and classification of phenomena such as audience reactions, speaker vocal effort can improve information retrieval (Bertoldi et al. 2002; Hu et al., 2008). New approaches for extracting relevant linguistic material, e.g. words persuasive impact (pi), see (Guerini et al., 2008).•  Automatic analysis of political communication. Computational linguistics to automatize analysis on politicians’ rhetoric. Considering audience’s reactions new rhetorical phenomena discovered (vs. traditional approaches based on words counting).•  Prediction of text impact. Machine learning for predicting the persuasive impact of novel speeches (Strapparava et al., 2008).•  Persuasive natural language generation. Eg. lexical choice: on the basis of lemma impact rather than lemma use.
  34. 34. Approach•  In analyzing CORPS, we focused on the lexical level.•  We considered: –  Windows of different width wn of terms preceding audience reactions tags. –  The typology of audience reaction.
  35. 35. Approach ex. Fragment from JFKFreedom has many difficulties and democracy is notperfect. But we have never had to put a wall up tokeep our people in, to prevent them from leaving us.{APPLAUSE ; CHEERS}I want to say on behalf of my countrymen who live positive-focusmany miles away on the other side of the Atlantic,who are far distant from you, that they take the wn = 15greatest pride, that they have been able to sharewith you, even from a distance, the story of the last18 years. I know of no town, no city, that has beenbesieged for 18 years that still lives with the vitalityand the force, and the hope, and the determinationof the city of West Berlin. {APPLAUSE ; CHEERS}
  36. 36. Valence and PersuasionThe phase that leads - to audience reaction,if it presents valencedynamics, ischaracterized by avalence crescendo
  37. 37. Words persuasive impact•  Basic idea: a word is more persuasive if at the same time its occurrences appear close to audience reactions tags and they do not appear far from them.•  We extracted “persuasive words” by using a coefficient of persuasive impact (pi) based on a weighted tf-idf (pi = tf × idf).
  38. 38. Words persuasive impact (cont’d)•  We created a “virtual document” by collecting terms inside windows preceding audience reactions tags (wn = 15).•  |D| = number of speeches in the corpus (included the virtual document)•  n = number of times the term (word) ti appears in the i virtual document•  Σn s = sum of word scores (closer to the tag, higher score) i i•  Σ n = number of occurrences of all words in the virtual k k document = wn × |tags number|•  |{d : d ∋ t }| = number of documents where the term t i i appears (we made a hypothesis of equidistribution)
  39. 39. Corpus Pre-processing•  POS-tagged all the speeches to reduce data sparseness, e.g. –  win, won, wins  win#v –  war, wars  war#n
  40. 40. Topmost Persuasive Words
  41. 41. AdvantagesFor persuasive political communicationthe approach using the persuasive impact(pi) of words is much more effective thansimple word count.
  42. 42. Examples of Use - ReaganMany qualitative researches on Reagan’s (aka “the greatcommunicator”) rhetorics: conversational style, irony, etc.•  Great Communicator? 32 Reagan’s speeches, mean tag density 1/2 of the whole corpus (t-test; α 0.001). Being a “great communicator” not bound to “firing up” rate.•  Reagan’s style: “simple and conversational”. Hp: conversational style more polysemic than a “cultured” style (richer in technical, less polysemic, terms). No statistical diff. between mean polysemy of Reagan’s words and whole corpus. But mean polysemy of Reagan persuasive words is double of the whole corpus (t-test; α 0.001).•  Use of irony: Density of ironical tags in Reagan’s speeches almost double as compared to the whole corpus (t-test; α 0.001). In Reagan’s speeches the mean ironical-tags ratio (mtri) is about 7.5 times greater than the mtri of the whole corpus (t- test; α 0.001).
  43. 43. Examples of Use – Bush and 9/11•  How do political speeches change after key historical events? Bush’s speeches before and after 9/11 (70 + 70 speeches) –  While words positive valence remains unvaried, the negative increases by 15% (t-test; α 0.001). –  Words counts only partially reflects word impact…
  44. 44. Lemma pi before pi after Count before Count afterwin#v 112 7 27 52justice#n x 9 15 111military#n 197 36 23 29defeat#v x 16 1 44right#r x 25 94 55victory#n 826 65 9 26evil#a - 129 0 44death#n 4 450 65 32war#n 36 x 80 258soldier#n 70 296 20 47tax#n x 93 702 81drug-free#a 87 x 9 3leadership#n 81 261 40 75future#n 83 394 54 51dream#n 99 321 77 30 Notes. In the second and third column, the number represents the rank in the list of persuasive words; an “x” indicates a pi = 0; an “–” indicates the word is not present in the corpus at all. In the fourth and fifth columns the total number of occurrences.
  45. 45. Bush and 9/11- Analysis Example•  For every word, we can record an increase or decrease of use (word count) compared with an increase or decrease of persuasiveness (pi).•  Let us consider the words military#n or treat#v. Both words are used almost the same number of times before and after 9/11. So their informativeness, based on number of occurrences, is null. But considering the persuasiveness score, we see that their impact varies (respectively from 197 to 36 and from 54 to 473).•  Let us also consider the word war#n; if we consider only the number of occurrences, we could infer that after 9/11 this topic was much more “felt” (mentioned three times more after 9/11), but if we look at persuasiveness we see that before 9/11 the word war#n was very “popular” (position 36) while after it never got audiences’ reactions.
  47. 47. Experiments•  Using machine learning for predicting the persuasive impact of novel discourses. –  Distinguishing Democrats from Republicans –  Predicting the passages that trigger a positive audience reaction –  Cross classification (training made on adverse party speeches, and test on the others) –  Experimenting the classifiers on plain and typical non-persuasive texts taken from British National Corpus and on speeches from the Obama-McCain political campaign.
  48. 48. Framework and Dataset•  We used the Support Vector Machines (SVM) framework.•  Dataset preprocessing: to reduce sparseness, used lemma#pos instead of tokens.•  We did not make any frequency cutoff or feature selection.•  All the speeches divided into fragments of about four sentences (if a tag is present in the fragment the chunk ends at that point).•  Obtained chunks are then labeled as Neutral (i.e., no tag), and Positive-ironical (i.e., all positive-focus and ironical tags). We did not consider the negative-focus tags, since they are only a few.•  A total of ~38000 four-sentence chunks, roughly equally partitioned into the two considered labels.•  This accounts for a baseline of 0.5 in distinguishing between Neutral and Positive-ironical chunks. In all the experiments we randomly split the corpus in 80% training and 20% test.
  49. 49. Democrats vs. Republican Precision Recall F1 Democrats 0.842 0.756 0.797 Republicans 0.773 0.854 0.811 Average (μ) 0.804 0.804 0.804
  50. 50. Positive vs. Neutral •  Whole Corpus Precision Recall F1 Positive-Ironical 0.646 0.683 0.664 Neutral 0.676 0.641 0.658 Average (μ) 0.660 0.660 0.660
  51. 51. Positive vs. Neutral •  Republican only Precision Recall F1 Positive-Ironical 0.660 0.766 0.709 Neutral 0.663 0.549 0.601 Average (μ) 0.661 0.661 0.661•  Democrat Only Precision Recall F1 Positive-Ironical 0.666 0.674 0.670 Neutral 0.686 0.680 0.683 Average (μ) 0.676 0.676 0.676
  52. 52. Cross Classification•  Training on Democrats, Test on Republicans Precision Recall F1 Positive-Ironical 0.642 0.632 0.637 Neutral 0.579 0.599 0.589 Average (μ) 0.612 0.612 0.612•  Training on Republicans, Test on Democrats Precision Recall F1 Positive-Ironical 0.625 0.660 0.642 Neutral 0.658 0.626 0.641 Average (μ) 0.641 0.641 0.641
  53. 53. Untagged texts Classification•  Typical non- Total chunks 7243 persuasive texts from Positive-Ironical 784 BNC (A00 to A0H)  Neutral 6459 Supposing all chunks Prec/Rec/F1 0.892 are neutral•  Typical persuasive Obama McCain texts from the last Positive-Ironical 2372 2360 Obama-McCain Neutral 68 80 presidential campaign Total chunks 2440 2440
  54. 54. Conclusions •  We have presented a resource and some approaches for persuasive NLP: –  a Corpus of tagged Political Speeches (CORPS) and a method for extracting persuasive words. –  a measure of persuasive impacts of words
  55. 55. Future Work •  Consider also persuasive rhetorical pattern extraction from CORPS. •  Consider windows width (wn) based on sentences rather than tokens. •  …
  56. 56. Some References•  Marco Guerini, Danilo Giampiccolo, Rachele Sprugnoli, Giovanni Moretti and Carlo Strapparava. The New Release of CORPS: Tagged Political Speeches for Persuasive Communication Processing, to appear.•  Marco Guerini, Carlo Strapparava and Oliviero Stock. CORPS: A corpus of tagged political speeches for persuasive communication processing. Journal of Information Technology Politics 5 (1), 19-32, 2008.•  Marco Guerini, Carlo Strapparava and Oliviero Stock. Audience Reactions for information extraction about persuasive language in political communication. In M. Maybury (ed.) Multimodal Information Extraction, to appear.•  Carlo Strapparava, Marco Guerini and Oliviero Stock. Predicting Persuasiveness in Political Discourses. In Proceedings of LREC2010.