Towards a More Holistic Approach on Online Abuse and Antisemitism

Towards a More Holistic Approach on Online Abuse and Antisemitism
Towards A More Holistic Approach On Online Abuse and Antisemitism Mohit Chandra mohit.chandra@research.iiit.ac.in 1
Disclaimer This work deals with the topic online abuse and contains examples of hateful content used only for illustrative purposes, reader/viewer discretion is advised. 2
Abstract Outline Introduction & Motivation Online abuse and Gab Abuse Detection, Severity and Target Prediction for Gab Posts Online Antisemitism Detection Using Multimodal Deep Learning Conclusion & Future Work 3
Outline Introduction Motivation Consequences and the Impact of Online Abuse Why Gab ? Thesis Work 4
Introduction Social media has become an indispensable part of our lives and with the ever rising amount of user generated content on these platforms, there has been a steady rise in the cases on online abuse. Purpose of attacking a person or a group on the basis of attributes such as race, religion, ethnic origin, sexual orientation, disability, or gender (Johnson et al., 2019) 5
Outline Introduction Motivation Consequences and the Impact of Online Abuse Why Gab ? Proposed Work 6
Motivation 7
Motivation As of April 2020, there are 3.81 billion active social media users spread across different social media platforms.(Link) 8
Motivation According to the Anti-DefamationLeague’s 2019 report, there has been a jump of 12% in the total cases of antisemitism, and a disturbing rise of 56% in antisemitic assaults. 9
Motivation 10
Consequences and the Impact of Online Abuse Psychological Effects on People Direct and indirect effects on individuals’ psychological well being, with the amount of damage significantly bigger in case of victimisation, compared to mere witnessing 12 Radicalization and Increased Hate Crimes Social media platforms are being manipulated by far-right groups and nefarious states to increase political polarisation to their advantage. Inequality in the Society Minority religion communities, LGBTQ+ and females are some of the common targets to online abuse which creates a sense of inequality among the members of the affected community.
Why Gab ? Gab has seen a significant rise in the number of registered users to 1,000,000 users along with a daily web traffic of 5.1 million visits per day by the end of July 2019 The platform is relatively unexplored and presents a wider spectrum of online abusive behaviour due to its liberal moderation policy. Gab played a pivotal role in Pittsburgh synagogue shooting and Brazil's Presidential elections. 14
The Proposed Work 16 General Purpose Abuse Presence, Severity and Target Detection Take a more holistic approach towards categorising online abuse and its classification. Multimodal Antisemitism Detection Using Deep Learning Present the first multimodal quantitative study for online antisemitism.
AbuseAnalyzer: Abuse Detection, Severity and Target Prediction for Gab Posts 17
Outline Challenges in detection of Online Abuse Related Work Abuse Severity and Targets Abuse Analysis Dataset AbuseAnalyzer: A Transformer based approach Experiments & Case Studies 18
Challenges in detection of Online Abuse Variety in the forms of online abuse: Posts on OSM network demonstrate various kind of abuse varying in terms of severity and nature ( Vidgen et al., 2019) Vocabulary richness: OSM networks are full of slang words which are geography specific and evolving. Natural language aspects: The grammatical structure followed on OSM platforms varies and the data is noisy in nature. ( Yang et al., 2011 ) Variety of targets and impact set: Abuse could be targeted towards individuals or groups and it is important to study the targets to understand the impact set. Eg: Capitol Hill Violence 19
Related Work 21 Sub-areas of Abuse found on Web Communities Multiple subareas of abuse have been considered individually like Racism, Sexism, Sarcasm (Tulkens et al., 2016, Jha and Mamidi, 2017, Chatzakou et al., 2017) Combination of aforementioned areas like racism & sexism, sexism & cyber-bullying (Chatzakou et al., 2017, Founta et al., 2019) Datasets and ML Approaches for Abuse Detection Traditionally studied platforms like Twitter and on some newer web communities like 4chan and Whisper (Hine et al., 2016, Silva et al., 2016) Use of statistical methods like SVM, Logistic Regression. Present day approaches use deep learning based approaches (Transformers, LSTMs, CNNs, Hybrid Networks) (Badjatiya et al., 2017, Serrà et al., 2017, Park and Fung, 2017) Analysis of Gab Posts Studies related to user dynamics and nature of content shared (Zannettou et al., 2018, Lima et al., 2018) Multiple dataset related studies, but no prior work focussed on fine grained abuse classification. (Fair and Wesslen, 2019, Zannettou et al., 2020)
Abuse Severity and Targets 23
Abuse Severity Classes Based on the concept Anti-Defamation League' s pyramid of hate. As one moves up the pyramid, the behaviors have more life threatening consequences. Biased Attitude: Contains posts related to trolling, accusations, sarcasm and insensitive remarks. Act of Bias and Discrimination: Consists of posts lying in the category of sexism, racism, xenophobia and homophobia . This class also contains instances on dehumanizing and devaluation speech. Violence and Genocide: Contains abusive behaviours like violent threat, intimidation and extremism.These have statements of intent to inflict harm. 24 Pyramid of Hate
Abuse Severity Classes Examples 25 Abuse Severity Class Example 1 Example 2 Biased Attitude At least my choice of president waits till they are out of diapers. Good news. No wonder they tried to fuck her nomination over hard. Act of Bias and Discrimination Jews are tragic for the world You’re are a real fucking brain dead piece of trash bitchboy. Violence and Genocide Antiwhite whites deserve public hanging I know what is going on! I fuck his mammy and I hurt her. Sorry mam!
Abuse Target Classes Individual (Second-Person): The posts in this class target the person being mentioned in the post.Generally, there is a usage of terms like ‘@username’, ‘you’ and ‘your’ to refer to the target. Individual (Third-Person): Posts classified under this category target a third person. Usually, these posts use terms like ‘he’, ‘she’, etc. or many a times the posts mention the name of the target. Group: This category represents those posts which target a group/organization based on ideologies,race, gender or some other basis. 26
Abuse Target Classes Examples 27 Abuse Target Class Example 1 Example 2 Individual (Second-Person) No but I do realize that you're full of shit and I know it. @username is serving a purpose or just a load of hot air Individual (Third-Person) His predatory sexual behavior is still evident. Another pedophile circles the wagons. Group We have some shit stirrers afoot today, ignore. Why not set dead muslims on the curb in a trash bagthe?
Abuse Analysis Dataset 7,601 posts extracted from Gab are classified on three different aspects: abuse presence or not, abuse severity and abuse target Used a high precision lexicon gathered by aggregating from multiple sources. The annotations were done in a 2-step fashion 29 Distribution of posts among different categories
Annotation Procedure The annotations were done in a 2-step fashion: Whether the post is abusive in nature or not We observed Cohen’s Kappa Score as (1) 0.719 for presence/absence of abuse, (2) 0.720 for presence+target, and (3) 0.683 for presence+severity classification. 30 Annotate in one of the three ‘Abuse Severity’ categories. Annotate in one of the three ‘Abuse Target’ categories. If labelled 'Abusive' If labelled 'Abusive'
Dataset Statistics and Analysis 31
Dataset Statistics and Analysis 32
Text Classifiers for Abuse Detection We experimented with different text classification approaches on the three abuse prediction tasks (Abuse presence, Abuse severity, Abuse targets) and propose a transformer based method. We experimented with different Machine Learning approaches Statistical methods based approaches (SVM, Logistic Regression, XG Boost) with TF-IDF feature vectors. Deep Learning based approaches (LSTMs and Transformers) 34
AbuseAnalyzer: A Transformer based approach Based on the concept of Transfer Learning through re-training. Text pre-processing for input: We remove the punctuation, hashtags, external URLs and convert the ‘@usermentions’ to a standard token name ‘usermention’. We keep truncate/pad the inputs for the length of 100 tokens. Network Architecture: Uses Batch Normalization with momentum, Dropout layers. Hyper-Parameters: We use Adam optimizer with lr= 1e−5. We train the network for maximum 20 epochs 35
Experiments Binary Abuse Presence Detection Three-Class Abuse Target Detection Three-Class Abuse Severity Detection 37
Case Studies 38 Confusion matrix for Abuse Target detection task. (Sum of 5 fold cross validation) Confusion matrix for Abuse Severity detection task. (Sum of 5 fold cross validation)
Case Studies For the task of detection of presence of abuse, terms like ‘black’, ‘muslims’ which are prone to online abuse pose a challenge. The complexity in the intent of the post (e.g: sarcasm, trolling) in addition to the difference in addressing the subject creates a problem. The subtlety in the abuse poses a major challenge for the classifiers. 39
Further Insights Abuse severity prediction proved to be the hardest task among is the three abuse prediction tasks. One of the possible reason being the presence of a large spectrum of abuse. While important keywords helped in the task of 'abuse target prediction', the dependence on keywords caused misclassifications in case of other tasks. Statistical methods like SVM with simple features are still useful as a baseline, especially in tasks like abuse detection where the availability of large amount of data is a costly affair. 40
Online Antisemitism Detection Using Multimodal Deep Learning 41
Outline Antimsemitism in OSM and Challenges Related Work Antisemitism Categorization Online Antisemitism Dataset Multimodal Antisemitism Categorization System Experiments & Case Studies Conclusion & Future Works 42
Antisemitism on OSM According to International Holocaust Remembrance Alliance (IHRA): “Antisemitism is a certain perception of Jews, which may be expressed as hatred toward Jews. Rhetorical and physical manifestations of antisemitism are directed toward Jewish or non-Jewish individuals and/or their property, towards Jewish community institutions and religious facilities.” 43
Challenges Clarity in the definition: The guidelines and the criteria for a content to be classified as antisemitic tend to be minimalistic and vague. 44 Presence of data with multiple modalities: In many cases, the content on social media involves multimodal data (images, videos, text, speech). A post with benign text may as well be antisemitic due to a hateful image Context associated with the content: Knowledge of the context is crucial in cases involving online antisemitism.
Challenges 45 I see the blews are at it again. Even grandma can see what’s going on
Related Work Antisemitism as a social phenomenon has been studied extensively especially as part of social science literature (Schwarz-Friesel et.al, 2017 , Salwen, 2009) Other sets of studies have focused on the categorization of antisemitic behaviour (Dencik and Marosi, 2016, Bilewicz et al., 2013) and effects of antisemitism (Ben-Moshe and Halafoff, 2014) Deep Learning architectures like RNNs (Founta et al., 2019), LSTMs (Serrà et al., 2017) and CNNs (Gambäck and Sikdar, 2017) have been used for hate speech detection. Multimodal deep learning has been harnessed to improve the accuracy for various tasks like Visual Question Answering (VQA) (Antol et al., 2015) , fake news/rumour detection (Jin et al., 2017, Khattar et al., 2019), and hate speech detection (Yang et al., 2019, Sabat et al., 2019). 47
Antisemitism Categorization We used the interpretation of IHRA’s detailed definition for antisemitism as the basis for our annotation guidelines. Besides annotating every post as antisemitic or not, we also annotate them for finer subcategories of online antisemitism. (W. Brustein, 2003) Political Antisemitism: Defined as the hatred toward Jews based on the stereotype that Jews seek national or world power. Economic Antisemitism: Based on the implicit belief that Jews perform and control the economic activities which are harmful for others or the society. Religious Antisemitism: Deals with bias and discrimination against Jewsdue to their religious belief in Judaism. Racial Antisemitism: Unlike religious antisemitism, it is based on the prejudice against Jews as a race/ethnic group. 49
Antisemitism Categorization 50
Online Antisemitism Dataset We obtained 3,509 English posts from Gab such that each post contains both text as well as image. The posts were collected based on an extensive lexicon containing negative & neutral keywords (like ‘Jewish’, ‘Hasidic’, ‘Hebrew’, ‘Semitic’, ‘Judaistic’, ‘israeli’, ‘yahudi’, ‘yehudi’) Each example was annotated on two levels after looking at the text as well as the image – (1) binary label (whether the example in antisemitic or not) (2) if the example is antisemitic then assign the respective category of antisemitism. The Fleiss kappa score came out to be 0.707 which translates to a substantial agreement between all the 4 annotators. 52
Data Statistics and Analysis Out of the total 3,509 posts, 1,877 (53.5%) posts have been annotated as antisemitic. Among the antisemitic posts, the distribution is as follows ‘Political Antisemitism’: 736 posts ‘Economic Antisemitism’: 118 posts ‘Religious Antisemitism’: 144 posts ‘Racial Antisemitism’: 879 posts 87.5% of the total images have some form of text in them. On average, post text has ∼45 words, while the OCR output is ∼50 words long. 53
Multimodal Antisemitism Categorization System The system comprises of two broad modules – (1) Text+OCR Module and (2) Image Module. The proposed system uses two pre-trained networks: BERT and DenseNet-161, fine-tuned on our dataset. We use Adam as the optimizer. We experimented with a range of learning rates and found lr=2e−6 as the best one. We train our system for a max of 30 epochs with early stopping, with a batch size of 4. 55 Text + OCR Module Image Module Fusion Module OCR Image Text (768,1) Binary/ Multiclass Labels Proposed multimodal system architecture which uses the information from text, OCR-text and images from the post for the classification task.
Experiments We conduct multiple experiments, to show the efficacy of our proposed system. Through the series of experiments we answer the following questions: Whether adding signals from multiple modalities improves the performance over the single modality classifiers? What is the best combination of pre-trained networks for the proposed system ? Which fusion technique works the best for the the proposed task ? We evaluate each model on two tasks – (1) binary classification of posts as antisemitic or not (2) multiclass antisemitism categorization. 57
OCR Experiments For getting the OCR output from the images we experimented with three different services (Google’s Vision API, Microsoft’s Computer Vision API and Open source tesseract). Open source tesseract performed the worst, especially in cases where the text was not horizontally aligned or the text present was in different text styles. In cases where the text written was not easily recognizable or written poorly, wefound Google’s Vision API to perform better than Microsoft’s Computer Vision API. 58
Experiments (Single Modal Classifiers) We experiment with five popular pre-trained text embedding/network based classifiers and four pre-trained image network classifiers. For the text-only classifiers we use GloVe + LSTMs, FastText + LSTMs, BERT , XLNet and RoBERTa. We also experiment with the method proposed by (Founta et al., 2019). For the image only classifiers we experiment with ResNet-152, DenseNet-161 , VGG-19 and ResNeXt-101 with MLP. Compared to the text-only methods, the image-only models provide much lower accuracy. We believe this is because of the text-heavy nature of the images in the dataset 59
Experiments (Single Modal Classifiers) 60 Comparison of performance of popular text-only and image-only classifiers.
Experiments (Multimodal Classifiers) We experiment with different combinations of text-based and image-based methods. We look at three different modality combinations – (1) Text + OCR (2) Text + Image (3) Text + Image + OCR. For Text + OCR based models, we compared BERT and RoBERTa as these gave the best results in single modal experiments. For Text+Image classifiers. We combined the two best text classifiers (BERT and RoBERTa) with the two best image classifiers (ResNeXt-101-32x8d and DenseNet-161). 61
Experiments (Multimodal Classifiers) 62 Comparison of performance of Text+OCR classifiers. Comparison of performance of Text+Image multimodal classifiers.
Experiments (Multimodal Classifiers) Text+OCR+Image classifiers with the concatenation fusion architecture. The variants having BERT in the ‘Text+OCR’ module outperform their RoBERTa-based counterparts. In summary, we conclude the following in terms of efficacy: Image-only/Text-only classifiers < Text+Image classifiers < Text+OCR classifier ∼ Text+OCR+Image classifiers 63 Comparison of performance of Text+OCR+Image multimodal classifiers.
Experiments (Multimodal Classifiers) 64 Comparison of different fusion techniques for Text + OCR + Image models We experimented with different tensor fusion approaches - (1) Concatenation, (2) Hadamard Product, (3) Gated MCB. Hadamard Product based fusion technique performed the worst.
Attention Visualization To gain better insights into the the proposed system, we visualize attention weights for both the Text+OCR(using bertviz) and the Image modules (using GradCAM). We took an antisemitic example having the text content as “some people have jew parasites embedded in their brains” and the OCR text being “liberals”. The word ‘liberals’ present in the OCR text output shares higher attention weights with the word ‘jew’ from the post text content showing the cross-attention learnt by the system. GradCAM visualization of the image in the post, we observe higher attention on the region of the Happy Merchant Meme face. 65 Image module attention visualization using GradCam Text+OCR module attention visualization using BertViz
Error Analysis and Case Studies We observe that the classifier is most confused between the ‘Political’ and ‘Racial’ classes. This could be because many politically oriented posts against Jews also used racial prejudices. Posts exhibiting multiple facets of antisemitism (like some posts abusing Jews on the basis of political, racial and religious basis) caused confusion for the system. 66 Confusion matrix for the multiclass classification task. Confusion matrix for the binary classification task.
Error Analysis and Case Studies 67
Error Analysis and Case Studies Phrases such as “jew hating” and “nazi” causes the system to commit mistake. Presence of religious and racial terms also caused the confusion for the system. Subtle expression of hate involving sarcasm/trolling along with absence of visual cues caused confusion in the classifier. News articles and tweet texts reporting news/information confused the classifier. The prominent reason being the multi-column format of the text in the image. Longer text in the OCR caused misclassification in many cases. 68
Conclusion We presented a holistic view towards analyzing and categorizing different types of abusive behaviour onthe basis of the severity and targets. We presented guidelines for the three abuse prediction tasks using which we created the proposed corpus with 7,601 Gab posts. Proposed Transformer based text classifier which uses BERT for each of the abuse detection task. We performed a comparative study of proposed system with the baselines which involved both statistical method and deep learning methods. Apart from general abuse detection, we focused on the problem of online antisemitism. W proposed a detailed guideline defining cases of antisemitism along with presenting definitions for 4 finer categories. Using the proposed guidelines we created a multimodal corpus with 3,509 posts which was used for our multimodal system. 70
Conclusion We presented our study on the multimodal deep learning, we proposed multimodal system which uses information from the post text, post images and OCR text to predict the instances of antisemitism. We performed an extensive set of experiments in which to answer three questions: Whether adding signals from multiple modalities improves the performance over the single modality classifiers? What is the best combination of pre-trained networks for the proposed system ? Which fusion technique works the best for the the proposed task ? We presented an attention visualization experiment for our proposed system along with the error analysis and case studies. 71
Future Work We are currently working on extending our antisemitism work to a mainstream social media platform (Twitter). It will be interesting to compare the performance of the system on a platform with different demographics. As a natural extension to our first work on general abuse detection, we would like to develop methods in multimodal settings where we would consider information from images. One of the most interesting direction is along the lines of contextual abuse. We would like to study comment threads for posts where most of times the comments are related to the previous comments and/or the post. We noticed that some specific type of content gathered more comments, replies and likes, hence it will be interesting to study the user behaviour on the fringe web communities from the angle of online abuse. 72
Publications Related Publications: Chandra, M., Pathak, A., Dutta, E., Jain, P., Gupta, M., Shrivastava, M., Kumaraguru, P. AbuseAnalyzer: Abuse Detection, Severity and Target Prediction for Gab Posts. In Proceedings of 28th International Conference on Computational Linguistics (COLING) 2020. Chandra, M., Pailla, D., Bhatia, H., Sanchawala, A., Gupta, M., Shrivastava, M., Kumaraguru, P. “Subverting the Jewtocracy”: Online Antisemitism Detection Using Multimodal Deep Learning. (Under-Review) Other Publications: Chandra, M., Reddy, M., Sehgal, S., Gupta, S., Buduru, A., Kumaraguru, P. “CoronaJihad”: Analyzing Islamophobia During the COVID-19 Outbreak. (Under-Review) N. Manwani and M. Chandra, "Exact Passive-Aggressive Algorithms for Ordinal Regression Using Interval Labels," in IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, pp. 3259-3268, Sept. 2020 73
Acknowledgement One slide cannot do the justice to the fact that how grateful I am to everyone who have been the part of this journey. Prof. Ponnurangam Kumaraguru, Dr. Manish Shrivastava for constant support and guidance throughout my research. Dr. Manish Gupta for shaping up the works and providing valuable inputs. My Co-authors who have been a constant source of inspiration and learning. The entire Precog family. My Mom and my friends. 74
Fin! Thank You 75

Check these out next

Towards a More Holistic Approach on Online Abuse and Antisemitism

Recommended

More Related Content

Slideshows for you(19)

Similar to Towards a More Holistic Approach on Online Abuse and Antisemitism(20)

More from IIIT Hyderabad (20)

Recently uploaded(20)

Towards a More Holistic Approach on Online Abuse and Antisemitism