This document presents a framework for detecting hate intent in social media posts using hybrid semantic feature representations. It evaluates different feature types, including corpus-based features like TF-IDF, knowledge-based features from Hatebase and FrameNet, and distributional semantic features like word embeddings. Evaluating these features on two Twitter datasets, it finds that while TF-IDF performs well, knowledge base and embedding features improve generalization across datasets. The proposed approach achieves up to a 3.0% absolute gain in F1 score over baselines, demonstrating the value of diverse semantic representations for hate speech detection.

1. EVALUATING SEMANTIC FEATURE REPRESENTATIONS TO
EFFICIENTLY DETECT HATE INTENT ON SOCIAL MEDIA
@wayasas @hemant_pt
Yasas Senarath Hemant Purohit
2020 IEEE International Conference on Semantic Computing
(IEEE-ICSC ’20)
San Diego, California
Feb 04, 2020

2. Outline
2
¨ Introduction
¤ Social Media
¤ Malicious Intent
¤ Background
¨ Problem and Contribution
¤ Data and Task
¤ Contributions
¨ Methodology
¤ Hybrid Feature Representation Framework
¤ Features
¨ Results and Discussion
¨ Conclusion
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

3. Outline
3
¨ Introduction
¤ Social Media
¤ Malicious Intent
¤ Background
¨ Problem and Contribution
¤ Data and Task
¤ Contributions
¨ Methodology
¤ Hybrid Feature Representation Framework
¤ Features
¨ Results and Discussion
¨ Conclusion
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

4. Motivation: Diverse Intent behind Social Media
Sharing
4
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
Social Media is an integral part of many of our daily lives!

5. Motivation: Malicious Intent on Social Media
¨ Social Media
¤ Malicious intent highly
profound in recent years
¨ Challenges
¤ Distinguishing intent: hate
speech vs. sarcasm vs. angry
rant
¤ Inefficiency in formalizing &
representing the context
5
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
members of nontraditional
religions r all subhuman trash
•hateful
you sure u ain’t colored?
• hateful
such a sucker 4some Oreos.
• Normal
Image source:
https://www.deviantart.com/ryujin2490/art/ANGRY-TWITTER-BIRD-252230315

6. Background
6
¨ Levels of Hate Speech
¤ Presence of Hate Speech [2, 6]
¤ Type of Hate Speech: offensive, abusive, hateful speech, aggressive,
and cyberbullying [2, 4]
¨ Classifiers [1]
¤ Naive Bayes
¤ Logistic Regression [3, 8]
¤ Random Forest [3, 7]
¤ Support Vector Machine* [6-9]
¤ Deep Learning [5]
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

7. Background: Features [1]
7
¨ Surface-level features
¤ Bag-of-Words, TFIDF
¨ Lexical Resources
¤ Hate Speech Lexicons
¤ Sentiment Lexicons
¨ Linguistic Features
¤ POS tags
¨ Knowledge-based Features
¤ ConceptNet (with custom rules)
¨ Meta-information
¤ User relevant information
¨ Transfer Learning
¤ Sentiment Analysis
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

8. Outline
8
¨ Introduction
¤ Social Media
¤ Malicious Intent
¤ Background
¨ Problem and Contribution
¤ Data and Task
¤ Contributions
¨ Methodology
¤ Hybrid Feature Representation Framework
¤ Features
¨ Results and Discussion
¨ Conclusion
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

9. Task and Data
¨ Task:
¤ Given a social media post, detect
whether it has hateful intent
¨ Datasets:
¤ DWMW17 [3]
n ~25k tweets
n query for words in Hatebase
n labels: Hate, Offensive and
Neither
¤ FDCL18 [4]
n ~ 60k tweets
n randomly sampled from Twitter
stream
n labels: Normal, Spam, Abusive,
Hateful
9
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
0%
20%
40%
60%
80%
100%
DWMW17
(24783)
FDCL18 (60227)
Label Distribution
Nomal/Spam/Neither
Hate/Offense/Abusive
#BendersRule
English language tweets

10. Contribution
10
¨ Proposed a set of diverse features capturing a variety of data
semantics for learning a hate speech classification model
¨ Validated the significance of proposed features on each dataset
¨ Evaluated prediction performance on each dataset based on models
trained on the other (cross-prediction performance)
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

11. Outline
11
¨ Introduction
¤ Social Media
¤ Malicious Intent
¤ Background
¨ Problem and Contribution
¤ Data and Task
¤ Contributions
¨ Methodology
¤ Hybrid Feature Representation Framework
¤ Features
¨ Results and Discussion
¨ Conclusion
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

12. Methodology: Pipeline
¨ Classical Data Mining Pipeline
¨ Preprocess
¤ Normalize (Usernames and
URLs)
¤ Tokenization
¨ Features*
¨ Classifier
¤ Liner SVM
12
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
Tweet Text
Preprocess
Feature Extractor
Classifier
Label
(Hate Speech / Normal)

13. Methodology: Feature Extractor
¨ Corpus-based semantic features
¤ TFIDF
¤ N-gram for N=[1, 2, 3]
¨ Distributional semantics-based
features
¤ Average of word embeddings
¨ Declarative knowledge-based
semantic features
¤ Hatebase
¤ FrameNet
13
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
Tweet Text
Preprocess
Feature
Extractor
Classifier
Label
(Hate Speech / Normal)
Corpus Based
Features
Distributional
Semantic Features
Knowledge Based
Features

14. Methodology: Hatebase
¨ Let 𝑓! be function mapping a
word to feature vector based
on some parameter/s in our
KB
14
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
∑!"#
$
𝑓𝑗 𝑤𝑖
𝑛
Knowledge
Base
Tweet
FKB(Tweet)
Tweet Text
Preprocess
Feature
Extractor
Classifier
Label
(Hate Speech / Normal)
Corpus Based
Features
Distributional
Semantic Features
Knowledge Based
Features
Hatebase
FrameNet

15. Methodology: FKB(Tweet) | KB = Hatebase
¨ Offensiveness (𝑓")
¤ discretized Value
¤ Freedman Diaconis Estimator
¨ Unambiguous (𝑓%)
¤ 1D vector with Boolean-
value
¨ Hateful-Meaning (𝑓#)
¤ bag-of-words vector of the
hateful definition
¨ Non-hateful-Meaning (𝑓$)
¤ bag-of-words vector of the
non-hateful definition
15
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
Tweet Text
Preprocess
Feature
Extractor
Classifier
Label
(Hate Speech / Normal)
Corpus Based Features
Distributional Semantic Features
Knowledge
Based Features
Hatebase
FrameNet

16. Methodology: FrameNet Features
16
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
Tweet Text
Preprocess
Feature
Extractor
Classifier
Label
(Hate Speech / Normal)
Corpus Based Features
Distributional Semantic Features
Knowledge
Based Features
SLINGTweet
Frames
(PropBank)
Mapping
Frames
(FrameNet)
Bag of Frames
Features
Hatebase
FrameNet

17. Outline
17
¨ Introduction
¤ Social Media
¤ Malicious Intent
¤ Background
¨ Problem and Contribution
¤ Data and Task
¤ Contributions
¨ Methodology
¤ Hybrid Feature Representation Framework
¤ Features
¨ Results and Discussion
¨ Conclusion
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

18. Results
18
¨ Five-fold cross validation performance
*baseline
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
Features
M1* TFIDF
M2
Hatebase
Features
+ Offensiveness
M3 + Unambiguous
M4 + Hateful Meaning
M5 + Non-Hateful
Meaning
M6 + FrameNet Features
M7 + Mean Embedding

19. Cross-Predication Performance
19
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20
0
10
20
30
40
50
60
70
80
90
100
DWMW17/FDCL18 FDCL18/DWMW17
F1Score
Train/Test Dataset
M1 M7
Features
M1* TFIDF
M2
Hatebase
Features
+ Offensiveness
M3 + Unambiguous
M4 + Hateful Meaning
M5 + Non-Hateful
Meaning
M6 + FrameNet Features
M7 + Mean Embedding

20. Discussion
¨ TFIDF features – Highly
Predictive
¤ However, do not help in
generalizing the models
¨ Knowledge base features
enhance precision
¨ Larger vocabulary of Word
Embeddings improve
performance
20
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

21. Outline
21
¨ Introduction
¤ Social Media
¤ Malicious Intent
¤ Background
¨ Problem and Contribution
¤ Data and Task
¤ Contributions
¨ Methodology
¤ Hybrid Feature Representation Framework
¤ Features
¨ Results and Discussion
¨ Conclusion
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

22. Conclusion
22
¨ Limitations and Future Work
¤ Polysemy words with multiple meanings can hinder the actual text
interpretation
¤ Multilingual social media posts
¨ Conclusions
¤ Novel empirical study of diverse semantic feature representations
for hate speech detection on social media
¤ Absolute gain in F1 score up to 3.0% for the models with hybrid
feature representation
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

23. References
23
[1] A. Schmidt and M. Wiegand, “A survey on hate speech detection using natural language processing,” in Proc.
of the Fifth Int’l Workshop on Natural Language Processing for Social Media, 2017, pp. 1–10.
[2] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar, “Semeval-2019 task 6: Identifying
and categorizing offensive language in social media (offenseval),” in SemEval, 2019, pp. 75–86.
[3] T. Davidson, D. Warmsley, M. Macy, and I. Weber, “Automated hate speech detection and the problem of
offensive language,” in ICWSM, 2017.
[4] A. M. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, and
N. Kourtellis, “Large scale crowdsourcing and characterization of twitter abusive behavior,” in ICWSM, 2018.
[5] K.Dinakar,B.Jones,C.Havasi,H.Lieberman,andR.Picard,“Common sense reasoning for detection, prevention, and
mitigation of cyberbullying,” ACM Tran. on Interactive Intelligent Systems, vol. 2, no. 3, p. 18, 2012.
[6] P. Burnap and M. L. Williams, “Cyber hate speech on twitter: An application of machine classification and
statistical modeling for policy and decision making,” Policy & Internet, vol. 7, no. 2, pp. 223–242, 2015.
[7] Y. Chen, Y. Zhou, S. Zhu, and H. Xu, “Detecting offensive lan- guage in social media to protect adolescent
online safety,” in PASSAT- SOCIALCOM. IEEE, 2012, pp. 71–80.
[8] Y. Mehdad and J. Tetreault, “Do characters abuse more than words?” in Proc. of the 17th Annual Meeting of
the Special Interest Group on Discourse and Dialogue, 2016, pp. 299–303.
[9] G. Xiang, B. Fan, L. Wang, J. Hong, and C. Rose, “Detecting offensive tweets via topical feature discovery
over a large scale twitter corpus,” in CIKM. ACM, 2012, pp. 1980–1984.
Evaluating Semantic Feature Representations to Efficiently Detect Hate Intent on Social Media, IEEE-ICSC ‘20

24. Acknowledgement Resources
Questions?24
Thank you
IIS #657379
@wayasas
ywijesu@gmu.edu
More Questions?
https://git.gmu.edu/ysenarath
/public/hate-intent-detection

25. Results
25