Presented at Digikomm 2019 in Berlin. Abstract We present a study on bot detection and its interpretation by assessing the different types of automation that one of the most popular methods for bot detection, Botometer (https://botometer.iuni.iu.edu/), detects. The study is based on the first project to assess the prevalence, influence, and roles of automated accounts in a Twitter follow network on a national scale: the German-speaking Twittersphere. This work in progress allows us to analyse the long-term structural role, impact, and possible audience of bots beyond the context of single events and topics.

1. What makes a bot a bot? Exploring benign
automation on Twitter
Dr. Felix Victor Münch1
; Ben Thies1
, B.A.; Dr. Cornelius Puschmann1
; Dr. Axel Bruns2
1
Leibniz Institute for Media Research | Hans-Bredow-Institut (HBI), Germany
2
Digital Media Research Centre (DMRC), Queensland University of Technology (QUT), Australia

2. http://bit.ly/whatabot

3. Background

4. Background
● Most Twitter-related bot research usually focuses on activity by examining
Tweets, Hashtags, Keywords, @-mentions, etc.*
● Most research focuses on malign automation, sometimes even assumes
that most automation is malign, and often cannot estimate its
impact
This research is focused on the Twitter follow network, takes a nuanced
approach to automation and assesses bot impact via the Twitter follow
network
*
https://www.pewinternet.org/2018/04/09/bots-in-the-twittersphere/

5. Research Goals
● provide an overview of bot
prevalence, influence, and
roles of automated accounts in
the German-speaking
Twittersphere
● develop and test
tools/methods that enable
such analyses for other
languages as well
● do some groundwork for a
theory of social
automation by cataloguing
suspected bots

6. Method
1. Create a language-based (German) Twitter follow network sample of most
influential accounts (method preprint: https://bit.ly/twitterwalk)
2. Use Botometer to estimate automation probabilities and apply probability
threshold of 0.75 to mark a suspected bot
3. Manually check 100 suspected bots prioritised by Page Rank and code them
inductively

7. Step 1: Drawing a Sample

8. Our adaptation of the ‘rank degree’ method
More details in our preprint: http://bit.ly/twitterwalk
Bottom: Original graph without walked edges. Starting nodes (seeds) are drawn randomly (1) and
walker move to their friend with the highest in-degree (2-6). Walked edges get removed/‘burned’.
Top: Current sample at each step. Walked (and symmetric) edges are added to sample.
1 2 3 4 5 6

9. Coverage
Distribution of public accounts with > 1 friend in the test sample over the percentage of their friends that
can be found in the influencer sample (left, filtered for in-degree >= 1, leaving 199,180 accounts) / baseline
sample (right, same size, randomly drawn from German accounts in global dataset)

10. (A Sample of) The German Twittersphere
● ~ 1 Mio Accounts, 1.6
Mio. Edges
● 6 Months of collection
using API authentication
keys of 10 collaborators
● Figure: 3-core of sample
network, coloured by
communities detected
with infomap. Node size
indicates Page Rank

11. Keyword extraction with chi-squared criterion
active
accounts keywords top accounts tag
2015
e3, stream, xd, e32019, nintendo, twitch, game, crossing, pc, zelda, animal, gameplay, cyberpunk2077,
games, switch, xbox, trailer, cyberpunk, gaming, xboxe3, uff, nice, awesome, keanu, pk, live, nen, lol,
mega
unge, dagibee, Gronkh, MelinaSophie, LeFloid,
iBlali, Taddl, rewinside, HandIOfIBlood, PietSmiet
YouTubers &
Gaming
1855
berlin, innen (female suffix), spd (German party), berliner, study, companies, discuss, cdu (party),
demand, topics, important, federal government, german, digitisation, topic, has been, annefrank,
climate protection, more, june, shows, germany, interview, politics, brandenburg
tazgezwitscher, Die_Gruenen, Tagesspiegel,
c_lindner, gutjahr, dunjahayali, sigmargabriel,
sixtus, HeikoMaas, spdde German politics
1414
women’s strike, switzerland, swiss people, glarner, svp (Swiss party), bern, women’s strike2019, zurich,
canton, grand, basel, national council, women
NZZ, 20min, viktorgiacobbo, Blickch,
tagesanzeiger, srfnews, MikeMuellerLate,
watson_news, migros, srf3
Swiss politics /
women's strike
1044
season, trainer, bundesliga, new arrival, player, dfb, fc, gerest, em, exchanges, estonia, exchange,
transfer, goal, victory, team, wm, cup, liga
DFB_Team, FCBayern, ToniKroos, MarioGoetze,
esmuellert_, Podolski10, Manuel_Neuer,
Bundesliga_DE, ZDFsport, JB17Official German football
767
övp (Austrian party), spö (Austrian party), fpö (Austrian party), vienna, austria, bierlein, oenr,
austria‘s, ibiza, viennese, strache, kickl, turquoise, hofer, parliament, national council, zib2, abdullah,
mandate, election campaign, centre, chancellor (female form), heinz, austrian, glyphosate, proposal,
blue
florianklenk, sebastiankurz, IngridThurnher,
kesslermichael, HannoSettele, vanderbellen,
Gawhary, HBrandstaetter, HHumorlos,
michelreimon Austrian politics
376
dessau, roßlau, afd (German (far-)right-wing party), görlitz, islam, raped, migrants, asylum seeker,
dangerous person, rejected person, sed, wippel, patriots, greta, niger, hosni, strongest, green,
fridayforfuture, african, rosslau, left, gretathunberg, crime, old parties, merkel, radical left, greens,
habeck, keep silent, tear apart, vote, girl, refugee, saxony, rape, maas, citizen, islamistic, sexual
DonJoschi, AfD, MSF_austria, Alice_Weidel,
SteinbachErika, Joerg_Meuthen, Beatrix_vStorch,
GrumpyMerkel, krone_at
hard right /
xenophobia /
migration /
refugees

12. Tagged Community graph Community graph of communities
in the 3-core of our sample with
over 300 accounts, at least 80 active
accounts during the examined time
frame, and edges with a weight of at
least 150; edge width represents
weight; edge direction follows
clockwise curvature; edges coloured
by source node; node size represents
the number of accounts in each
community

13. Step 2: Use Botometer1
to identify suspected Bots
1
https://botometer.iuni.iu.edu

15. Suspected bots cluster mostly outside of topical communities

16. Suspected bots cluster mostly outside of topical communities

17. Suspected bots cluster mostly outside of topical communities

18. Step 3: Rate suspected Bots with a
human rater

19. Inductive Coding of Suspected Bots

20. Only 58% of the accounts in the “75% bot” sample
seem to be automated

21. The majority of automated accounts is benign

22. Communities of automated accounts

23. Automation types

24. Automation Purposes

25. Preliminary results
● Prevalence of (suspected) automated accounts in line with prior analyses 1
● Low centrality of automated accounts (usually outside central clusters)
● Automated accounts are primarily located outside of politics and news clusters
● Most automated accounts are benign
● Botometer: false-positive rate higher than selected threshold (58% vs. 75% set
threshold) according to human ratings.
● But: good approach to inductively explore account-categories and test detection
methods
Overall, the negative impact of automated accounts seems rather low in our
sample.
1
Varol, O., Ferrara, E., Davis, C. A., Menczer, F., & Flammini, A. (2017). Online Human-Bot Interactions: Detection,
Estimation, and Characterization. In International AAAI Conference on Web and Social Media. Retrieved from
https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15587/14817

26. (Selected) Limitations / What to do next? / Outlook
This study Other studies In general
Sampling ● Method still
experimental
● Missing peripheral
auto accounts with
low impact (‘bot
swarms’)
Activity/keyword based or
random
● Impact/activity
assessment
problematic
● ‘SEO problem’: bots
optimise on
keywords
● Population unknown
● No concept for
representativity in
networks/media
environments with highly
skewed distributions of
degree/attention
Bot
detection
● Needs triangulation
with other
automated tools
● Needs follow-up
with multiple coders
● ‘Bad Bot’ stigma:
Quiet assumption
that automation is
malign
● Too much trust in
automated tools
● Automated tools often
developed and trained
without a working theory
of social automation
● Binary concept of ‘botness’
instead of
multidimensional
approach to
(semi-)automation

27. So what makes a bot a bot?

28. Advertising
+ =
https://github.com/FlxVctr/RADICES
(Continuous Development)

29. Thanks!
@FlxVctr, @BenAThies, @cbpuschmann, @snurb_dot_info