TorfsBot or Not? Evaluating User Perception on Imitative Text Generation (CLIN33)
TorfsBot or Not? Evaluating User Perception on Imitative Text Generation (CLIN33)
This talk was presented by Thomas Winters at the 33rd Computational Linguistics in the Netherlands conference. Abstract: Mimicking an individual's writing style using automated text generation approaches presents a challenge in determining the optimal level of imitation versus exaggeration. This study investigates the believability and user interaction of a text generator employing Markov models and dynamic templates in comparison to source data. Using the TorfsBot Twitterbot, which emulates the writing style of the Belgian professor Rik Torfs, we developed a secondary Twitterbot, \"TorfsBot or Not?”, that conducted daily polls asking users to identify whether a tweet originated from Rik Torfs or TorfsBot. The study collected 43K votes from approximately 500 polls. The findings reveal that participants correctly identified the source of the tweets 71% of the time, with majority votes inaccurately attributing the source in 14% of the polls. Furthermore, we observed a positive correlation between the number of interactions on the source tweet and the perceived believability that the tweet originated from Rik Torfs. Our results suggest that even relatively simple text generation models can approximately replicate a target's writing style, and that a closer approximation to the source material may positively influence user engagement.
Recommended
More Related Content
More from Thomas Winters
More from Thomas Winters(17)
Recently uploaded
Recently uploaded(20)
TorfsBot or Not? Evaluating User Perception on Imitative Text Generation (CLIN33)
- 2. Professor Canon Law & Former Rector KU Leuven
- 3. Fully-automated Twitterbot imitating Rik Torfs Trained on his tweets & columns Launched in 2016
- 4. Algorithm 1: Markov Chain (Interpolated 35-gram) 1. Counts frequency of word after previous words in tweets & columns van Rik Torfs “gevolgd door” 4: een 2: zijn 1: iemand 1: acht Beste, 2. Takes start words and repeatedly predicts
- 5. Algorithm 2: Dynamic template 1. Take random tweet and a random column lines 2. Change tweet key words with words from column with same part-of-speech
- 6. Markov chain 35% more interactions than dynamic template (17K interactions on 8K tweets over from June 2017 to June 2023) Previous work: Winters, T. (2019).Generating PhilosophicalStatements using InterpolatedMarkovModels and Dynamic Templates. 31st EuropeanSummer Schoolin Logic, Language and InformationStudent Session Proceedings, 181–189.
- 8. Tweets daily poll with random tweet from either Rik Torfs or TorfsBot
- 9. 602 polls 47K votes total (avg 79/poll) 71% correct votes 87% correct majority 68 TorfsBot success 12 Rik Torfs fails
- 12. Slightly positively correlated (0.11 for log(interactions) with %votes) Little difference between algorithms
- 13. More interactions ~ more believable Weirdly more so for Rik than bot
- 15. #1
- 17. #2
- 19. #3
- 22. #4
- 24. De laatste TorfsBot Or Not! #5
Editor's Notes
- Voor de mensen die Rik Torfs niet kennen: Professor Kerkelijk Recht Ex-rector KU Leuven Maarook: fervent Twitteraar Op Twitter sinds 2010 Speaks in algemeenheden en boutades “Vlaams Orakel” “Koning van de boutade”
- Volautomatische Twitterbot Heeft leren tweeten zoals Rik Torfs door zijn tweets & columns te analyseren Simpel Markov model & kernwoorden vervanger Tweet 5x per dag en antwoordt op iedereen
- automatische kernwoord vervangingen
- Deze tweet is met het dynamische sjabloon op Rik’s originele tweet