Towards a More Holistic Approach on Online Abuse and Antisemitism

Towards A More Holistic Approach
On Online Abuse and Antisemitism
Mohit Chandra
mohit.chandra@research.iiit.ac.in
1

Disclaimer
This work deals with the topic online abuse and contains examples of hateful content
used only for illustrative purposes, reader/viewer discretion is advised.
2

Abstract Outline
Introduction & Motivation
Online abuse and Gab
Abuse Detection, Severity and Target Prediction for Gab Posts
Online Antisemitism Detection Using Multimodal Deep Learning
Conclusion & Future Work
3

Outline
Introduction
Motivation
Consequences and the Impact of Online Abuse
Why Gab ?
Thesis Work
4

Introduction
Social media has become an indispensable part of our lives and with the ever rising amount of
user generated content on these platforms, there has been a steady rise in the cases on online
abuse.
Purpose of attacking a person or a group on the basis of attributes such as race, religion, ethnic
origin, sexual orientation, disability, or gender (Johnson et al., 2019)
5

Outline
Introduction
Motivation
Why Gab ?
Proposed Work
6

Motivation
As of April 2020, there are 3.81 billion active social
media users spread across different social media
platforms.(Link)
8

Motivation
According to the Anti-DefamationLeague’s 2019 report, there has been a jump of 12% in the total cases of antisemitism,
and a disturbing rise of 56% in antisemitic assaults.
9

Outline
Introduction
Motivation
Why Gab ?
Proposed Work
11

Psychological Effects on People
Direct and indirect effects on individuals’ psychological well being, with the amount of damage signiﬁcantly
bigger in case of victimisation, compared to mere witnessing
12
Radicalization and Increased Hate Crimes
Social media platforms are being manipulated by far-right groups and nefarious states to increase political
polarisation to their advantage.
Inequality in the Society
Minority religion communities, LGBTQ+ and females are some of the common targets to online abuse which
creates a sense of inequality among the members of the affected community.

Outline
Introduction
Motivation
Why Gab ?
Proposed Work
13

Why Gab ?
Gab has seen a signiﬁcant rise in the number of registered users to 1,000,000 users along with a daily
web trafﬁc of 5.1 million visits per day by the end of July 2019
The platform is relatively unexplored and presents a wider spectrum of online abusive behaviour due
to its liberal moderation policy.
Gab played a pivotal role in Pittsburgh synagogue shooting and Brazil's Presidential elections.
14

Outline
Introduction
Motivation
Why Gab ?
Proposed Work
15

The Proposed Work
16
General Purpose Abuse Presence, Severity and
Target Detection
Take a more holistic approach towards
categorising online abuse and its classiﬁcation.
Multimodal Antisemitism Detection Using
Deep Learning
Present the ﬁrst multimodal quantitative
study for online antisemitism.

AbuseAnalyzer: Abuse Detection,
Severity and Target Prediction
for Gab Posts
17

Outline
Challenges in detection of Online Abuse
Related Work
Abuse Severity and Targets
Abuse Analysis Dataset
AbuseAnalyzer: A Transformer based approach
Experiments & Case Studies
18

Variety in the forms of online abuse: Posts on OSM network demonstrate various kind of abuse
varying in terms of severity and nature ( Vidgen et al., 2019)
Vocabulary richness: OSM networks are full of slang words which are geography speciﬁc and
evolving.
Natural language aspects: The grammatical structure followed on OSM platforms varies and the data
is noisy in nature. ( Yang et al., 2011 )
Variety of targets and impact set: Abuse could be targeted towards individuals or groups and it is
important to study the targets to understand the impact set. Eg: Capitol Hill Violence
19

Outline
Related Work
20

Related Work
21
Sub-areas of Abuse found on
Web Communities
Multiple subareas of abuse
have been considered
individually like Racism, Sexism,
Sarcasm (Tulkens et al., 2016,
Jha and Mamidi, 2017,
Chatzakou et al., 2017)
Combination of
aforementioned areas like
racism & sexism, sexism &
cyber-bullying (Chatzakou et al.,
2017, Founta et al., 2019)
Datasets and ML Approaches for
Abuse Detection
Traditionally studied platforms like
Twitter and on some newer web
communities like 4chan and Whisper
(Hine et al., 2016, Silva et al., 2016)
Use of statistical methods like SVM,
Logistic Regression. Present day
approaches use deep learning based
approaches (Transformers, LSTMs,
CNNs, Hybrid Networks) (Badjatiya
et al., 2017, Serrà et al., 2017, Park
and Fung, 2017)
Analysis of Gab Posts
Studies related to user
dynamics and nature of content
shared (Zannettou et al., 2018,
Lima et al., 2018)
Multiple dataset related
studies, but no prior work
focussed on ﬁne grained abuse
classiﬁcation. (Fair and
Wesslen, 2019, Zannettou et
al., 2020)

Outline
Related Work
22

Abuse Severity Classes
Based on the concept Anti-Defamation League' s pyramid of hate. As one
moves up the pyramid, the behaviors have more life threatening
consequences.
Biased Attitude: Contains posts related to trolling, accusations, sarcasm
and insensitive remarks.
Act of Bias and Discrimination: Consists of posts lying in the category of
sexism, racism, xenophobia and homophobia . This class also contains
instances on dehumanizing and devaluation speech.
Violence and Genocide: Contains abusive behaviours like violent threat,
intimidation and extremism.These have statements of intent to inﬂict harm.
24
Pyramid of Hate

Abuse Severity Classes Examples
25
Abuse Severity Class Example 1 Example 2
Biased Attitude At least my choice of president
waits till they are out of diapers.
Good news. No wonder they tried
to fuck her nomination over hard.
Act of Bias and
Discrimination
Jews are tragic for the world You’re are a real fucking brain
dead piece of trash bitchboy.
Violence and
Genocide
Antiwhite whites deserve public
hanging
I know what is going on! I fuck his
mammy and I hurt her. Sorry
mam!

Abuse Target Classes
Individual (Second-Person): The posts in this class target the person being mentioned in the
post.Generally, there is a usage of terms like ‘@username’, ‘you’ and ‘your’ to refer to the target.
Individual (Third-Person): Posts classiﬁed under this category target a third person. Usually, these
posts use terms like ‘he’, ‘she’, etc. or many a times the posts mention the name of the target.
Group: This category represents those posts which target a group/organization based on
ideologies,race, gender or some other basis.
26

Abuse Target Classes Examples
27
Abuse Target Class Example 1 Example 2
Individual
(Second-Person)
No but I do realize that you're full
of shit and I know it.
@username is serving a purpose
or just a load of hot air
Individual
(Third-Person)
His predatory sexual behavior is
still evident.
Another pedophile circles the
wagons.
Group We have some shit stirrers afoot
today, ignore.
Why not set dead muslims on the
curb in a trash bagthe?

Outline
Related Work
28

7,601 posts extracted from Gab are classiﬁed on three different aspects: abuse presence or not, abuse
severity and abuse target
Used a high precision lexicon gathered by aggregating from multiple sources. The annotations were
done in a 2-step fashion
29
Distribution of posts among different categories

Annotation Procedure
The annotations were done in a 2-step fashion:
Whether the post is abusive in nature or not
We observed Cohen’s Kappa Score as (1) 0.719 for presence/absence of abuse, (2) 0.720 for
presence+target, and (3) 0.683 for presence+severity classiﬁcation.
30
Annotate in one of the three ‘Abuse Severity’
categories.
Annotate in one of the three ‘Abuse Target’
categories.
If labelled 'Abusive'
If labelled 'Abusive'

Dataset Statistics and Analysis
31

Dataset Statistics and Analysis
32

Outline
Related Work
33

Text Classiﬁers for Abuse Detection
We experimented with different text classiﬁcation approaches on the three abuse prediction tasks
(Abuse presence, Abuse severity, Abuse targets) and propose a transformer based method.
We experimented with different Machine Learning approaches
Statistical methods based approaches (SVM, Logistic Regression, XG Boost) with TF-IDF
feature vectors.
Deep Learning based approaches (LSTMs and Transformers)
34

Based on the concept of Transfer Learning through
re-training.
Text pre-processing for input: We remove the
punctuation, hashtags, external URLs and convert the
‘@usermentions’ to a standard token name ‘usermention’.
We keep truncate/pad the inputs for the length of 100
tokens.
Network Architecture: Uses Batch Normalization with
momentum, Dropout layers.
Hyper-Parameters: We use Adam optimizer with lr=
1e−5. We train the network for maximum 20 epochs
35

Outline
Related Work
36

Experiments
Binary Abuse Presence
Detection
Three-Class Abuse Target
Detection
Three-Class Abuse Severity
Detection
37

Case Studies
38
Confusion matrix for Abuse Target detection task. (Sum of 5 fold
cross validation)
Confusion matrix for Abuse Severity detection task. (Sum of 5 fold
cross validation)

Case Studies
For the task of detection of
presence of abuse, terms like
‘black’, ‘muslims’ which are prone
to online abuse pose a challenge.
The complexity in the intent of
the post (e.g: sarcasm, trolling) in
addition to the difference in
addressing the subject creates a
problem.
The subtlety in the abuse poses a
major challenge for the classiﬁers.
39

Further Insights
Abuse severity prediction proved to be the hardest task among is the three abuse prediction tasks.
One of the possible reason being the presence of a large spectrum of abuse.
While important keywords helped in the task of 'abuse target prediction', the dependence on
keywords caused misclassiﬁcations in case of other tasks.
Statistical methods like SVM with simple features are still useful as a baseline, especially in tasks like
abuse detection where the availability of large amount of data is a costly affair.
40

Online Antisemitism Detection
Using Multimodal Deep Learning
41

Outline
Antimsemitism in OSM and Challenges
Related Work
Antisemitism Categorization
Online Antisemitism Dataset
Multimodal Antisemitism Categorization System
Conclusion & Future Works
42

Antisemitism on OSM
According to International Holocaust Remembrance Alliance (IHRA):
“Antisemitism is a certain perception of Jews, which may be expressed as hatred toward Jews. Rhetorical
and physical manifestations of antisemitism are directed toward Jewish or non-Jewish individuals and/or
their property, towards Jewish community institutions and religious facilities.”
43

Challenges
Clarity in the deﬁnition: The guidelines and the criteria for a content to
be classiﬁed as antisemitic tend to be minimalistic and vague.
44
Presence of data with multiple modalities: In many cases, the content
on social media involves multimodal data (images, videos, text, speech). A
post with benign text may as well be antisemitic due to a hateful image
Context associated with the content: Knowledge of the context is crucial
in cases involving online antisemitism.

Challenges
45
I see the blews are at it again. Even grandma can see what’s going on

Outline
Related Work
46

Related Work
Antisemitism as a social phenomenon has been studied extensively especially as part of social science literature
(Schwarz-Friesel et.al, 2017 , Salwen, 2009)
Other sets of studies have focused on the categorization of antisemitic behaviour (Dencik and Marosi, 2016, Bilewicz
et al., 2013) and effects of antisemitism (Ben-Moshe and Halafoff, 2014)
Deep Learning architectures like RNNs (Founta et al., 2019), LSTMs (Serrà et al., 2017) and CNNs (Gambäck and
Sikdar, 2017) have been used for hate speech detection.
Multimodal deep learning has been harnessed to improve the accuracy for various tasks like Visual Question
Answering (VQA) (Antol et al., 2015) , fake news/rumour detection (Jin et al., 2017, Khattar et al., 2019), and hate
speech detection (Yang et al., 2019, Sabat et al., 2019).
47

Outline
Related Work
48

We used the interpretation of IHRA’s detailed definition for antisemitism as the basis for our
annotation guidelines.
Besides annotating every post as antisemitic or not, we also annotate them for finer subcategories of
online antisemitism. (W. Brustein, 2003)
Political Antisemitism: Defined as the hatred toward Jews based on the stereotype that Jews seek national or
world power.
Economic Antisemitism: Based on the implicit belief that Jews perform and control the economic activities
which are harmful for others or the society.
Religious Antisemitism: Deals with bias and discrimination against Jewsdue to their religious belief in Judaism.
Racial Antisemitism: Unlike religious antisemitism, it is based on the prejudice against Jews as a race/ethnic
group.
49

50

Outline
Related Work
51

We obtained 3,509 English posts from Gab such that each post contains both text as well as image.
The posts were collected based on an extensive lexicon containing negative & neutral keywords (like
‘Jewish’, ‘Hasidic’, ‘Hebrew’, ‘Semitic’, ‘Judaistic’, ‘israeli’, ‘yahudi’, ‘yehudi’)
Each example was annotated on two levels after looking at the text as well as the image –
(1) binary label (whether the example in antisemitic or not)
(2) if the example is antisemitic then assign the respective category of antisemitism.
The Fleiss kappa score came out to be 0.707 which translates to a substantial agreement between all
the 4 annotators.
52

Data Statistics and Analysis
Out of the total 3,509 posts, 1,877 (53.5%)
posts have been annotated as antisemitic.
Among the antisemitic posts, the
distribution is as follows
‘Political Antisemitism’: 736 posts
‘Economic Antisemitism’: 118 posts
‘Religious Antisemitism’: 144 posts
‘Racial Antisemitism’: 879 posts
87.5% of the total images have some form of
text in them. On average, post text has ∼45
words, while the OCR output is ∼50 words
long.
53

Outline
Related Work
54

The system comprises of two broad
modules – (1) Text+OCR Module and (2)
Image Module.
The proposed system uses two pre-trained
networks: BERT and DenseNet-161,
ﬁne-tuned on our dataset.
We use Adam as the optimizer. We
experimented with a range of learning rates
and found lr=2e−6 as the best one.
We train our system for a max of 30 epochs
with early stopping, with a batch size of 4.
55
Text +
OCR
Module
Image
Module
Fusion
Module
OCR
Image
Text
(768,1)
Binary/ Multiclass
Labels
Proposed multimodal system architecture which uses the information from text, OCR-text and images
from the post for the classiﬁcation task.

Outline
Related Work
56

Experiments
We conduct multiple experiments, to show the efficacy of our proposed system. Through the series of
experiments we answer the following questions:
Whether adding signals from multiple modalities improves the performance over the single modality classifiers?
What is the best combination of pre-trained networks for the proposed system ?
Which fusion technique works the best for the the proposed task ?
We evaluate each model on two tasks – (1) binary classification of posts as antisemitic or not (2)
multiclass antisemitism categorization.
57

OCR Experiments
For getting the OCR output from the images we experimented
with three different services (Google’s Vision API, Microsoft’s
Computer Vision API and Open source tesseract).
Open source tesseract performed the worst, especially in cases
where the text was not horizontally aligned or the text present
was in different text styles.
In cases where the text written was not easily recognizable or
written poorly, wefound Google’s Vision API to perform better
than Microsoft’s Computer Vision API.
58

Experiments (Single Modal Classifiers)
We experiment with five popular pre-trained text embedding/network based classifiers and four pre-trained image
network classifiers.
For the text-only classifiers we use GloVe + LSTMs, FastText + LSTMs, BERT , XLNet and RoBERTa. We also
experiment with the method proposed by (Founta et al., 2019).
For the image only classifiers we experiment with ResNet-152, DenseNet-161 , VGG-19 and ResNeXt-101 with
MLP.
Compared to the text-only methods, the image-only models provide much lower accuracy. We believe this is
because of the text-heavy nature of the images in the dataset
59

Experiments (Single Modal Classiﬁers)
60
Comparison of performance of popular text-only and image-only classiﬁers.

Experiments (Multimodal Classifiers)
We experiment with different combinations of text-based and image-based methods. We look at three different
modality combinations – (1) Text + OCR (2) Text + Image (3) Text + Image + OCR.
For Text + OCR based models, we compared BERT and RoBERTa as these gave the best results in single modal
experiments.
For Text+Image classifiers. We combined the two best text classifiers (BERT and RoBERTa) with the two best image
classifiers (ResNeXt-101-32x8d and DenseNet-161).
61

62
Comparison of performance of Text+OCR classiﬁers.
Comparison of performance of Text+Image multimodal classiﬁers.

Text+OCR+Image classifiers with the concatenation fusion architecture. The variants having BERT in the ‘Text+OCR’
module outperform their RoBERTa-based counterparts.
In summary, we conclude the following in terms of efficacy:
Image-only/Text-only classifiers < Text+Image classifiers < Text+OCR classifier ∼ Text+OCR+Image classifiers
63
Comparison of performance of Text+OCR+Image multimodal classifiers.

64
Comparison of different fusion techniques for Text + OCR + Image models
We experimented with different tensor fusion approaches - (1) Concatenation, (2) Hadamard Product, (3) Gated MCB.
Hadamard Product based fusion technique performed the worst.

Attention Visualization
To gain better insights into the the proposed system, we
visualize attention weights for both the Text+OCR(using
bertviz) and the Image modules (using GradCAM).
We took an antisemitic example having the text content as
“some people have jew parasites embedded in their brains”
and the OCR text being “liberals”.
The word ‘liberals’ present in the OCR text output shares
higher attention weights with the word ‘jew’ from the post
text content showing the cross-attention learnt by the
system.
GradCAM visualization of the image in the post, we
observe higher attention on the region of the Happy
Merchant Meme face.
65
Image module attention visualization using GradCam
Text+OCR module attention visualization using BertViz

Error Analysis and Case Studies
We observe that the classifier is most confused between
the ‘Political’ and ‘Racial’ classes. This could be because
many politically oriented posts against Jews also used
racial prejudices.
Posts exhibiting multiple facets of antisemitism (like some
posts abusing Jews on the basis of political, racial and
religious basis) caused confusion for the system.
66
Confusion matrix for the multiclass classification task.
Confusion matrix for the binary classification task.

67

Phrases such as “jew hating” and “nazi” causes the system to commit mistake.
Presence of religious and racial terms also caused the confusion for the system.
Subtle expression of hate involving sarcasm/trolling along with absence of visual cues caused confusion in the
classifier.
News articles and tweet texts reporting news/information confused the classifier. The prominent reason being the
multi-column format of the text in the image.
Longer text in the OCR caused misclassification in many cases.
68

Outline
Related Work
69

Conclusion
We presented a holistic view towards analyzing and categorizing different types of abusive behaviour onthe basis of the
severity and targets.
We presented guidelines for the three abuse prediction tasks using which we created the proposed corpus with 7,601
Gab posts.
Proposed Transformer based text classifier which uses BERT for each of the abuse detection task. We performed a
comparative study of proposed system with the baselines which involved both statistical method and deep learning
methods.
Apart from general abuse detection, we focused on the problem of online antisemitism.
W proposed a detailed guideline defining cases of antisemitism along with presenting definitions for 4 finer categories.
Using the proposed guidelines we created a multimodal corpus with 3,509 posts which was used for our multimodal
system.
70

Conclusion
We presented our study on the multimodal deep learning, we proposed multimodal system which uses information
from the post text, post images and OCR text to predict the instances of antisemitism.
We performed an extensive set of experiments in which to answer three questions:
Whether adding signals from multiple modalities improves the performance over the single modality classiﬁers?
What is the best combination of pre-trained networks for the proposed system ?
Which fusion technique works the best for the the proposed task ?
We presented an attention visualization experiment for our proposed system along with the error analysis and case
studies.
71

Future Work
We are currently working on extending our antisemitism work to a mainstream social media platform
(Twitter). It will be interesting to compare the performance of the system on a platform with different
demographics.
As a natural extension to our ﬁrst work on general abuse detection, we would like to develop methods
in multimodal settings where we would consider information from images.
One of the most interesting direction is along the lines of contextual abuse. We would like to study
comment threads for posts where most of times the comments are related to the previous comments
and/or the post.
We noticed that some speciﬁc type of content gathered more comments, replies and likes, hence it will
be interesting to study the user behaviour on the fringe web communities from the angle of online
abuse.
72

Publications
Related Publications:
Chandra, M., Pathak, A., Dutta, E., Jain, P., Gupta, M., Shrivastava, M., Kumaraguru, P. AbuseAnalyzer: Abuse Detection,
Severity and Target Prediction for Gab Posts. In Proceedings of 28th International Conference on Computational
Linguistics (COLING) 2020.
Chandra, M., Pailla, D., Bhatia, H., Sanchawala, A., Gupta, M., Shrivastava, M., Kumaraguru, P. “Subverting the
Jewtocracy”: Online Antisemitism Detection Using Multimodal Deep Learning. (Under-Review)
Other Publications:
Chandra, M., Reddy, M., Sehgal, S., Gupta, S., Buduru, A., Kumaraguru, P. “CoronaJihad”: Analyzing Islamophobia During
the COVID-19 Outbreak. (Under-Review)
N. Manwani and M. Chandra, "Exact Passive-Aggressive Algorithms for Ordinal Regression Using Interval Labels," in
IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, pp. 3259-3268, Sept. 2020
73

Acknowledgement
One slide cannot do the justice to the fact that how grateful I am to everyone who have been the part
of this journey.
Prof. Ponnurangam Kumaraguru, Dr. Manish Shrivastava for constant support and guidance
throughout my research.
Dr. Manish Gupta for shaping up the works and providing valuable inputs.
My Co-authors who have been a constant source of inspiration and learning.
The entire Precog family.
My Mom and my friends.
74

Towards a More Holistic Approach on Online Abuse and Antisemitism

More Related Content

What's hot

Similar to Towards a More Holistic Approach on Online Abuse and Antisemitism

More from IIIT Hyderabad

Recently uploaded

Towards a More Holistic Approach on Online Abuse and Antisemitism