1. Understanding Online Socials Harm:
Examples of Harassment and Radicalization
Prof. Amit Sheth
Founding Director, AI Institute
University of South Carolina
AI @
UofSC
33rd Annual IFIP WG 11.3 Conference on Data and Applications Security
and Privacy (DBSec'19)
Charleston, SC, USA
July 15 -17, 2019
Icons by thenounproject
Slides by SlideModel

2. 2
The youngest adults stand out in their social media consumption
88% of 18- to 29-year-olds indicate that they use any form of social media.
By Pew Research Center “Social Media Use Report 2018”

3. 3
Social Good and Social Harm on Social Media
A spectrum to demonstrate the variety of social good to social harm
Adapted from : Purohit, Hemant & Pandey, Rahul. (2019). Intent Mining for the Good, Bad, and Ugly Use of Social Web: Concepts, Methods, and
Challenges. 10.1007/978-3-319-94105-9_1.
Zika Virus
Monitoring
Help
Fighting
depression
Disaster
Relief
Opioid Usage
Monitoring
Joking
Marketing
Sensationalizing
Harassment
Accusing
Rumouring
Deceiving
Fake News
Radicalization
Illicit Drugs
Social HarmSocial Good
Positive Effects Negative Effects

4. 4
Fake-porn videos are being weaponized to harass and
humiliate women: ‘Everybody is a potential target’
‘Deepfake’ disturbingly realistic, computer-generated videos with photos taken from
the Web, and ordinary women are suffering the damage

5. 5
Different meanings of
diagnostic terms
Ambiguity
Different perceptions
of same concepts.
Subjectivity
Low prevalence of
relevant content
Sparsity
Nature of content with
more than one
context.
Multi
Dimensionality
Significant implications
in a big scale
application.
False Alarms
Knowledge Graphs and Knowledge Networks: The Story in Brief - IEEE Internet
Computing Magazine 2019
Multimodal
Content
Different modalities of
data
Challenges --Complex Problems

6. 6
1.
Online Harassment
Context in an interaction determines bad behavior.

7. 7
Severity of online harm can differ
based on several criteria
It can span for more than a decade in
one’s life
Or it can lead to teenage suicides
Police accuse two students, age
12, of cyberbullying in suicide
By Jamiel Lynch, CNN
Teenage cyber bullying victims are
Use-cases of Online Harassment

8. 9
Existing Approaches
Tweet Content
Binary
Classifier
Harassing Non-Harassing
People
Network
Our Approach (Incorporating context)
Tweet Content
Multiclass
Classifier
Sexual
harassment Appearance
related
Harassment
Traditional Approach vs Recent Advances

9. 10
Problem Definition & Sparsity
Dataset # of Tweets Classes (%)
Waseem et al.
(2016)
16093
Racism (12%),
Sexism (19.6%),
Neither (68.4%)
Davidson et al.
(2017)
24,802
Hate (5%), Non-
hate (95%)
Zhang et al.
(2018)
2,435
Hate (17%),
Non-hate (83%)
Mostly binary classifications
Datasets have small percentages of positive
(harassing) instances
A Quality Type-aware Annotated Corpus And
Lexicon For Harassment Research [Rezvan et
al.]
This paper provides both a quality annotated
corpus and an offensive words lexicon
capturing different types of harassment
content:
(i) sexual (7.4%)
(ii) racial (22.5%)
(iii) appearance-related (21.8%)
(iv) intellectual (26%)
(v) political (22.4%)
Harassing (12.9%)
Non-Harassing
(87.1%)
Challenges in Online Harassment

10. 12
According to Pew research center (2017)
Subjectivity
“pEoPlE dO iT tO tHeMsElVeS”
shut the f**k up
you know what im not gonna
argue anymore u guys are all so
f**king ignorant when it comes to
addiction so PLEASe stop f**king
speaking on it
all the girls who i have beef w are
little ass girls who think they can
say the n word but get scared
when black guys come around 🤔 🤔
on that note, gn twit
An interaction example from
Highschool students’ tweet corpus
Challenges in Online Harassment

11. “Language used to express hatred
towards a targeted individual or group, or
is intended to be derogatory, to
humiliate, or to insult the members of the
group, on the basis of attributes such as
race, religion, ethnic origin, sexual
orientation, disability, or gender is hate
speech” - Founta et al. 2018
13
Challenges in Online Harassment Detection
Ambiguity
Researchers have defined harassment using jargon that overlaps, causing ambiguity in
annotations
“Profanity, strongly impolite, rude or
vulgar language expressed with fighting
or hurtful words in order to insult a
targeted individual or group is
offensive language ” - Founta et al. 2018
Ex. 1: @user_name nah you just a dumb
hoe who doesn’t know her place 😂 😂
This tweet belongs to hate speech and
offensive language based on above
definitions
Examples from Highschool students’
tweet corpus
Challenges in Online Harassment
Ex. 2: IS THAT A MICROAGRESSION AGAINST
MEXICANS BY STEREOTYPING THEM AS
ILLEGALS?!? only if you were vegan you wouldn’t
be such a racist pig
This tweet falls into the category of
hate speech but not necessarily
offensive language

12. 14
What? How? Who?
What causes online
harassment? [REASON]
How can online
harassment happen?
[METHOD]
What are the effects of
online harassment?
[RESULT]
Victim
Appearance/
Religion/
Race etc.
Harasser
Xenophobia/
Homophobia/
Intolerance
etc.
Victim feels:
~ Offended
~ Discriminated
~ Afraid of losing life or
losing social capital
~ Depressed
~ Suicidal
Direct vs. Indirect
Where?
~ Flaming (the act of posting or
sending offensive messages over the
Internet)
~ Doxxing (broadcasting private
or identifying information of
individuals)
~ Dogpiling (several people in
twitter addressing someone, usually
negatively in a short period of time)
~ Impersonation
~ Public shaming
~ Threats
[ACTORS]
~ Harasser
~ Victim
~ Bystanders:
a. Aggravators
(People who try to
fuel a harassing
situation indirectly,
for example by
retweeting
harassing tweets)
a. Empathizers
(People who
empathize with the
victim, providing
support)
~ Social Media
(Twitter,
Facebook,
Instagram)
~ Discussion
Boards
(Reddit, 4chan)
~ Email
~ Private
Messaging
~ Online
Gaming
* Policies of
platforms
[PLATFORM]
Frequently vs. One-
time
Cyber Bullying vs. Cyber
Aggression
Online Harassment - Dimensions

13. 16
Resolving Data Scarcity Issue
Using Generative adversarial networks (GANs) to generate text,
increasing the positive(harassing) examples in a dataset
Changing the generator objective function in the GAN to incorporate
domain specific knowledge
Multiclass classification of harassing tweets
Harassment type prediction was done using a multiclass classifier
The process of tweet vectorization leveraged domain knowledge in the
form of an offensive words lexicon
Current Research Directions Pursued

14. 17
Key Takeaways
● In spite of recognition of importance, the problem is
still not well-understood, and not well-defined.
● Use cases and data show that the problem is far more
complex and nuanced. Understanding context is
critical.
● Current social media platforms appear to have little or
no automated processes for detection and prevention.
Oversimplification of problem definition largely
relying on machine learning without significant domain
specific knowledge has rendered solutions practically
useless.

15. 18
2.
Online Extremist
Communications
Islamist Extremism and White Supremacy

16. 19
Challenges in Online Harassment Detection
Efforts by High-Tech Companies
Capabilities of social media
companies (Twitter, Facebook
and Google) are inadequate and
ineffective.
Governments insisted that the
industry had a ‘social
responsibility’ to do more to
remove harmful content.
If unsolved, social media
platforms will continue to
negatively impact the society.
Unsolved: Detection of Extremism on Social Media

17. 20
Challenges in Online Harassment Detection
● One thousand Americans
between 1980 and 2011.
300 Americans since 2011
attempted or traveled.
● > 5 thousand individuals from
Europe traveled to Join
Extremist Terrorist Groups
(ISIS, Al-Qaeda) abroad
through 2015,
● Most inspired and persuaded
online.
“The Travelers”
*George Washington University, Program on Extremism

18. ● 24 year old college student from Alabama became
radicalized on Twitter. After a year, moved to Syria
to join ISIS.
● Self-taught, she read verses from the Qur’an, but
interpreted them with others in the extremist
network.
● Persuaded that when the true Islamic State is
declared, it is obligatory to do hijrah, which they
see as the pilgrimage to ’the State’.
21
Illustrative Case
*New York Times: “Alabama Woman Who Joined ISIS Can’t Return Home, U.S. Says”

19. 22
Challenges in Online Harassment DetectionRadicalization Scale (Achilov et al.)
0
None
Mainstream
religious
views and
orientations
Indicator:
Islam; Allah;
jihad (self
struggle);
halal;
democracy,
islam, salah,
fatwa, hajj.
1
Low
Attitudinal
support for
politically
moderate
Islamism
Indicator:
Hadith;
Caliphate
(Khilafah)
justified;
Sharia better
(than secular
law);
Hypocrisy
west.
2
Elevated
Emergent
support for
exclusive rule
of the Shari’a
law
Indicator:
Shariah best;
revenge
(justified);
jihad (against
West); justify
Daesh (ISIS)
3
High
Support for
extremist
networks and
travel to
“Darul Islam”
Indicator:
Kafir; infidel;
hijrah to Darul-
Islam;
(supporting)
fatwa Al-
Awlaki;
mushrikeen.
4
Severe
Call for action
to join the
fight and the
use of
violence.
Indicator:
apostate;
sahwat;
taghut; kill;
kafir; kuffar;
murtadd;
tawaghit;
al_baghdadi;
martyrdom
khilafah

20. 23
Challenges in Online Harassment DetectionRadicalization Process over time
Ultimately, analysis of content in context will provide better
finer-granular understanding the underlying factors in the
radicalization process.
Non-extremist
ordinary
individual
Radicalized
extremist
individual
0 1 2 4
SevereHighLowNon
e
Elevated
3

21. Islamist Extremism on Social Media
(e.g., recruiter, follower) with
respect to different stages of
radicalization.
Modeling users
psychological process
over a time period.
Persuasive
relevant to Islamist
extremism.
Domain Knowledge
of the context (“jihad” has
different meaning in
different context)
Multidimensionality
Radicalization

22. Security Implications
Specifically, unfair
classification of non-
extremist individuals as
extremist.
False alarm might potentially
impact millions of innocent
people.
25
Local and Global security implications,
while predicting online terrorist activities
and involved individuals.

23. 26
Multidimensionality of Extremist Content
● Dimensions to define the context:
○ Based on literature and our empirical study of the
data, three contextual dimensions are identified:
Religion, Ideology, Hate
● The distribution of prevalent terms (i.e., words, phrases,
concepts) in each dimension is different.
● These terms should be represented in different
dimensions, to disambiguate especially diagnostic
terms (e.g., jihad): .

24. Extremist Content
27
Prevalent Key Phrases Prevalent Topics
isis, syria, kill, iraq, muslim, allah, attack, break,
aleppo, assad, islamicstate, army, soldier,
cynthiastruth, islam, support, mosul, libya, rebel,
destroy, airstrike
Caliphate_news, islamic_state, iraq_army, soldier_kill,
iraqi_army, syria_isis, syria_iraq, assad_army,
terror_group, shia_militia, isis_attack, aleppo_syria,
martyrdom_operation, ahrar_sham, assad_regime,
follow_support, lead_coalition, turkey_army, isis_claim,
kill_isis
Imam_anwar_awlaki, video_message_islamicstate,
fight_islamic_state, isisclaim_responsibility_attack,
muwahideen_powerful_middleeast, isis_tikrit_tikritop,
amaqagency_islamicstate_fighter,
sinai_explosion_target, alone_state_fighter,
intelligence_reportedly_kill,
khilafahnew_islamic_state,
yemanqaida_commander_kill, isis_militant_hasakah,
breakingnew_assad_army, isis_explode_middle,
hater_trier_haleemah, trust_isis_tighten,
qamishlus_isis_fighting, defeat_enemy_allah,
kill_terrorist_baby, ahrar_sham_leader
islamic state, syria, isis, kill, allah, video, minute
propaganda video scenes, jaish islam release, restock
missile, kaffir, join isis, aftermath, mercy, martyrdom
operation syrian opposition, punish libya isis, syria
assad, islam sunni, swat, lose head, wilayatalfurat,
somali, child kill, takfir, jaish fateh, baghdad, iraq,
kashmir muslim, capture, damascus, report rebel,
british, qala moon, jannat, isis capture, border cross,
aleppo, iranian soldier, tikrit tikrittop, lead shia military
kill, saleh abdeslam refuse cooperate
Green: Religion
Blue: Ideology
Red: Hate
Corpus: 538 Twiter verified extremists, 48K tweets

25. 28
“Reportedly, a number of
apostates were killed in
the process. Just
because they like it I
guess.. #SpringJihad
#CountrysideCleanup”
“Kindness is a language
which the blind can see
and the deaf can hear
#MyJihad be kind
always”
“By the Lord of Muhammad (blessings and peace be upon
him) The nation of Jihad and martyrdom can never be
defeated”
“Jihad” can appear in tweets with different meanings in different dimensions
of the context.
H
I
R
Example Tweets with “Jihad”

26. 29
Challenges in Online Harassment Detection
● Same term can have
different meanings for each
dimensions.
● Example:
“Meaning of Jihad”
is different for extremists
and non-extremists.
○ For extremists, meaning closer
to “awlaki”, “islamic state”,
“aqeedah”
For non-extremists, closer to
“muslims”, “quran”, “imams”
Ambiguity of Diagnostic terms/phrases
ExtremistsNon-Extremists

27. ● Different Contextual Dimensions
incorporating:
○ Knowledge Graphs
○ Dimension Corpora
● Utilization of Deep Learning models,
generate knowledge-enhanced
representations
● KG creation:
Religion: Qur’an, Hadith
Ideology: Books, lectures of
ideologues
[Not KG: Hate: Hate Speech Corpuss
(Davidson et al. 2017)]
● Can be applied over many social
problems.
30
Modeling
Modeling
Modeling
Dimension 1
Dimension 2
Dimension 3
DimensionDimensionDimension
Dimension Modeling
Process
Dimension based
Knowledge
enhanced
Representation
Contextual Dimension Modeling

28. (Hate)
Capturing similarity:
● Learning word similarities from a substantial knowledge
graph
● A solution via distance between concepts in the knowledge
graph.
Modeling
31
Using a Knowledge Graph
“You shall know a word by the company it keeps” (J. R. Firth 1957: 11)

29. Capturing similarity (and resolving ambiguity):
● Learning word similarities from a large corpora.
● A solution via distributional similarity-based
representations.
Modeling
32
(Hate)
Using a Corpus
“You shall know a word by the company it keeps” (J. R. Firth 1957: 11)

30. 33
● Found two distinct groups
employing different contexts with
different density.
● Religion and Hate are usually
mixed, suggesting that extremists
might employ different hate
tactics.
● A small group of users employ
ideological context far more often
than others, suggesting these
users might be disseminators of
ideologically intense content.
Density of Dimensions in Extremist Content

31. 34
● Tri-dimension model
performs best.
● Precision used as
metric, to emphasize
reduction on
misclassification of
non-extremist
content.
● Implications in a large
scale application.
Results

32. ● False alarms: significantly reduced via incorporation of three
specific dimensions of context.
● Extremist users employ religion along with hate, suggesting they
employ different hate tactics for their targets.
● Inclusion of all three contextual dimensions significantly reduces
the likelihood of an unfair mistreatment towards non-extremist
individuals, in a real world application.
● Each dimension plays different roles in different levels of
radicalization, capturing nuances as well as linguistic and
semantic cues better throughout the radicalization process.
35
Key Insights

33. 36
Public/
Society
Social
Interactions
Cognitive
Neuro
Cognitive
Process
● Human brain processes information
from extremist narratives on social
media, that includes different
contexts, emotions, sentiment, etc.
● Individuals change behavior, make
choices in consuming/sharing
content with an intent.
● Coordination, information flow and
diffusion on social networks.
● Outcomes/impact on society through
events and collective actions (eg,
civil war or result of an election).
Our Highly Multidisciplinary Approach

34. 37
Weaponized Ambiguity
Sparsity Complexity

35. 38
Context-Aware Harassment Detection on Social Media(wiki link)
is an interdisciplinary project among the Ohio Center of
Excellence in Knowledge-enabled Computing (Kno.e.sis), the
Department of Psychology, and Center for Urban and Public
Affairs (CUPA) at Wright State University.
We are supported by the NSF Award#: CNS 1513721
Supporting Grants

36. 39
Thank You!
Special Thanks: Ugur Kursuncu and Thilini Wijesiriwardene

37. 1. Hinduja, S. and Patchin, J.W., 2010. Bullying, cyberbullying, and suicide. Archives of suicide research,
14(3), pp.206-221
2. Rezvan, M., Shekarpour, S., Balasuriya, L., Thirunarayan, K., Shalin, V.L. and Sheth, A., 2018, May. A
Quality Type-aware Annotated Corpus and Lexicon for Harassment Research. In Proceedings of the
10th ACM Conference on Web Science (pp. 33-36). ACM.
3. Zeerak Waseem. Are you a racist or am i seeing things? annotator influence on hate speech detection
on twitter. In Proc. of the Workshop on NLP and Computational Social Science, pages 138–142.
Association for Computational Linguistics, 2016.
4. Thoams Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech
detection and the problem of offensive language. In Proceedings of the 11th Conference on Web and
Social Media. AAAI, 2017.
5. Zhang, Ziqi & Luo, Lei. (2018). Hate Speech Detection: A Solved Problem? The Challenging Case of
Long Tail on Twitter. Semantic Web. Accepted. 10.3233/SW-180338.
40
References