The document discusses using knowledge-infused natural language understanding to analyze social media and dark web data to monitor trends in substance use and mental health related to addiction research. It presents several use cases applying this approach to study the relationship between opioid use and mental health, cannabis and synthetic cannabinoid use, and emerging trends in illicit synthetic opioids sold on cryptomarkets. The goal is to generate actionable insights and early warnings to inform public health surveillance and policy.

1. Video at: https://youtu.be/pRUXTuxm3as
Knowledge-infused NLU for Addiction and
Mental Health Research
AI Institute
public health, epidemiology, substance use
Invited Talk at MAISoN 2021 in conjunction with IJCAI2021
Amit Sheth, Founding Director, AI Institute at UofSC #AIISC
Technical Team: Usha Lokala, Manas Gaur, Kaushik Roy
Experts/Collaborators: Raminta Daniulaityte, Francois Lamy, Jyotishman Pathak
Collaborations: School of Medicine at UofSC, Wright State University, Boonshoft School of Medicine and Weill Cornell
Medicine. Funding: NIDA/NIH and NSF funded projects on Addiction and Mental Health.

2. Outline of the talk
❏ Why NLP is not enough, why we need NLU? Challenges in Deep
Learning (DL)
❏ Duality of Data and Knowledge &
Adding Knowledge to DL
❏ What is Knowledge Graph?
❏ Towards Knowledge Infused Learning (K-IL)
❏ Shades of K-IL
❏ Use-cases
❏ Addiction X Epidemiology
❏ Mental Health
While we will focus on social media data, the ideas and methods are adaptable to
clinical, conversational and other types of data.
2

3. NLP is not enough!
Data alone is not enough!
Why we need NLU?
Why knowledge is indispensable?
3
NLU: Natural Language Understanding

4. Deep Learning:
Deeper you go, Darker it gets.
4
Performance
Interpretability
Rule
Based
Learning
Linear/
Logistic
Regression
Decision
Trees
k-NN
Generalized
Additive
Models
Bayesian
Models
Support
Vector
Machines
Ensembles
Deep
Learning
Statistical
Symbolic
Solution: Hybrid systems.

5. Duality of Data and Knowledge
Layered hybrid system using
neural networks and deep
learning for perception with
knowledge-based reasoning
system for decision making (one
of several architectures for
neuro-symbolic AI)
Sheth, A., & Thirunarayan, K. (2021). The duality of data and knowledge across the three waves of AI. IT Professional, 23(3), 35–45.
Decisions/Actions
Symbolic
Reasoning
Linking Data to
Knowledge
Neural Network & Deep
Learning
Data Sensing & Collection
Feature Engineering,
Recommendation
Deduction, Abduction, Non-
monotonic and Probabilistic
Inferencing, Planning
Construction and/or
Instantiation
Upper Level
(Second Stage)
Lower Level
(First Stage)

6. Example: NLP vs NLU

7. Limitations of NLP
Presence of Long Tail
Context
Personalization
Knowledge infused
Learning (explicit
knowledge +
statistical AI)
NLU for
Decisions/Actions/Wisdom
supported by XAI
Abstraction
NLP Limitations
Approach
Real Impact
Reasoning
A. Sheth, M. Gaur, K. Roy, K. Faldu, "Knowledge-intensive Language Understanding for Explainable AI." arXiv e-prints (2021): arXiv-2108. [ In
print: IEEE Internet Computing, September/October, 2021.]

8. Context: Opioids are not legalized for recreation and medical
purposes. Pain relievers are used legally with prescription. Synthetic
Opioids like Fentanyl are being used for similar purposes illegally as
consumers are addicted to it and often use them without
prescription, contributing to Opioid overdoses.
Q: Does Fentanyl cause Opioid Overdose?
8
Opioids are a class of drugs that include the illegal drug
heroin, synthetic opioids such as fentanyl.
Pain relievers available legally by prescription, such as
oxycodone, morphine
Misuse of and addiction to opioids—including prescription pain
relievers causes Opioid Overdoses.
NLU - Learning the Long Tail
Domain
Specific
Low
Resource
Symbolic
Knowledge
Multi Hop Procedural
Process of learning longtail which would be
difficult to support with statistical AI alone.

9. Language Model Prediction Vs Symbolic Knowledge
9
“Who was the 44th President of the United States of
America?”
Barack Obama.
Explain Why? See Info box or equivalently see KG .
Who Succeeded? Who Preceded? Term?
With
Knowledge
and limited
data
Without
Knowledge
GPT-3, ~175 Billion
Parameters
Barack Obama (71%),
Donald Trump (18%), ..
Explain Why? …..
● Probabilistic
vs assertional
● Named
relationships

10. ConceptNet
Common sense knowledge helps us identify commonly related context for the concepts in the text
Here we use a single
knowledge graph such
as ConceptNet to
contextualize
Task: Respond to user appropriately if they are absent minded
Contextualization
Faldu, Keyur, Amit Sheth, Prashant Kikani, and Hemang Akabari. "KI-BERT: Infusing Knowledge Context for Better Language and Domain
Understanding." arXiv preprint arXiv:2104.08145 (2021)

11. Sentence 1: How does benzodiazepine withdrawal happen by increasing
stimulating chemicals?
Sentence 2: Does heightened stimulating chemical production such as
serotonin cause benzodiazepine withdrawals?
ConceptNet
Neurotransmitter
isa
chemical
isa
relatedTo
withdrawals
Are these sentences related to withdrawal from benzodiazepine?
Abstraction

12. I’ve had a long day at work and
I would like to go grab a drink
Sure, there is a bar at […..]
that’s got great reviews
Alright, thank
you.
I’ve had a long day at work and
I would like to go grab a drink
Your depression typically gets
worse with alcohol, are you sure?
You’re right.
Thanks
Without personalization With a personalization
Adam Alcoholism
Suffers from
PKG:
Personaling knowledge
graph (PKG) support the
wisdom to prevent Adam
from going out to a bar,
which may be a norm with
extenuating circumstances
Personalization

13. 15
Really struggling with my bisexuality which is causing chaos in my relationship with a girl.
Being a fan of LGBTQ community, I am equal to worthless for her. I’m now starting to get
drunk because I can’t cope with the obsessive, intrusive thoughts, and need to get it out of
my head.
Is mental health related ? Yes: 0.71 , No: 0.29
Which Mental Health condition?
Predicted: Depression (False)
True: Obsessive Compulsive Disorder
Reasoning over Model:
Why model predicted
Depression?
Unknown
Manas Gaur, Ugur Kursuncu, Amanuel Alambo, Amit Sheth,
Raminta Daniulaityte, Krishnaprasad Thirunarayan, and
Jyotishman Pathak. "" Let Me Tell You About Your Mental
Health!" Contextualized Classification of Reddit Posts to DSM-5
for Web-based Intervention." CIKM 2018. [CIKM 2018]
Reasoning

14. 16
Really struggling with my [health-related behavior] which is causing [health-related
behavior] with a girl. Being a fan of [XXX] community, I am equal to [level of mood] for
her. I’m now starting to [drinking] because I can’t cope with the [obsessive compulsive
personality disorder] [disturbance in thinking], and [disturbance in thinking].
Is mental health related ? Yes: 0.96 , No: 0.04
Which Mental Health condition?
Predicted: Obsessive Compulsive Disorder(True)
True: Obsessive Compulsive Disorder
DSM-5 Knowledge
Graph
DSM-5 and Post
Correlation Matrix
Reasoning over Model:
Why model predicted
Obsessive Compulsive
Disorder ? known
Interpretable and
Explainable Learning
Reasoning

15. “
Adding knowledge to data-
intensive learning for
interpretability and explainability
17

16. What is Knowledge Graph
Knowledge Graph (KG) is a
structural representation of
entities (entity and entity type)
and relationships in various
forms (e.g. labeled property
graphs, RDFs) create to
facilitate reasoning over the
outcome.
Commonsense Reasoning Graph
Event Ontology
Crisis Ontology
18

17. 19
Shades of Knowledge-infused Learning (K-
IL)
of knowledge graphs
to improve the
semantic and
conceptual
processing of data.
Semi-Deep Infusion
Deeper and congruent
incorporation or
integration of the
knowledge graphs in the
learning techniques. Deep Infusion
(Part of Future KG Strategy)
combines a stratified
representation of knowledge
representing abstractions levels to
be transferred in different layers of
a deep learning model.
Shallow Infusion
Sheth, Gaur, Kursuncu, Wickramarachchi: Shades of Knowledge-Infused Learning for Enhancing Deep Learning

18. Dataset
enrich
Deep Learning
Model
Tacit
Knowledge
Hypothesis
testing or
similarity-based
verification
Shallow Infusion
Tacit
Knowledge
Self-aware or
External Knowledge
Self-aware or
External Knowledge
Similarity
based
verification
Semi-Deep Infusion
Dataset
Deep Learning
Model
Knowledge-infused (Deep) Learning
Kursuncu, Ugur, Manas Gaur, and Amit Sheth. "Knowledge Infused Learning (K-IL): Towards
Deep Incorporation of Knowledge in Deep Learning." AAAI Fall Symposium (2020).

19. Tacit
Knowledge
Similarity based
verification
Deep Infusion
Deep Learning Model
Stratified layer 1 Stratified layer 2
Stratified Knowledge 1 Stratified Knowledge 2
Knowledge-infused (Deep) Learning
Kursuncu, Ugur, Manas Gaur, and Amit Sheth. "Knowledge Infused Learning (K-IL): Towards Deep Incorporation
of Knowledge in Deep Learning." AAAI Fall Symposium (2020).

20. Knowledge Infused Language Understanding (KILU)
A.Sheth, M. Gaur, K. Roy, K.Faldu. "Knowledge-intensive Language Understanding for Explainable AI," IEEE Internet Computing,
September/October 2021.

21. Addiction (Opioid, Cannabis, Synthetic
Cannabinoid, Prescription Drug Abuse) X
Epidemiology
Visit Project page: http://wiki.aiisc.ai/index.php/Public_Health_Addictions_Research_at_AIISC

22. Motivation
The opioid epidemic is entrenched in USA
● Study the prevalence of opioid and
its impact on the well-being of
individuals and the society.
○ Mental Health & Suicide Risk
Questions
1. How can we use social media to measure
mental health impact of opioid
prevalence?
1. Are there association between opioid and
mental health/suicide risk?
Approach
Monitoring the prevalence of opioid and its impact on mental health and suicide utilizing a
scalable knowledge and data driven BIGDATA (BD) approach via social media.
Opioid and Substance Use

23. Score
Calculation
Opioid
Mental Health
Depression
Addiction
Suicide Risk
Ideation, Behavior
Attempt
Correlations
● Sheth, Amit, and Pavan Kapanipathi. "Semantic filtering for social data." IEEE Internet Computing 20, no. 4 (2016): 74-78.
● Hussein S. Al-Olimat, Krishnaprasad Thirunarayan, Valerie Shalin, and Amit Sheth. 2018. Location Name Extraction from Targeted Text Streams using Gazetteer-based Statistical
Language Models. In Proceedings of the 27th International Conference on Computational Linguistics (COLING2018), pages 1986-1997. Association for Computational Linguistics
● Gaur, M., Kursuncu, U., Alambo, A., Sheth, A., Daniulaityte, R., Thirunarayan, K., & Pathak, J. (2018, October). " Let Me Tell You About Your Mental Health!" Contextualized
Classification of Reddit Posts to DSM-5 for Web-based Intervention. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (pp. 753-
762).
● Gaur, M., Alambo, A., Sain, J. P., Kursuncu, U., Thirunarayan, K., Kavuluru, R., ... & Pathak, J. (2019, May). Knowledge-aware assessment of severity of suicide risk for early
intervention. In The World Wide Web Conference (pp. 514-525).
● Yazdavar, A. H., Al-Olimat, H. S., Ebrahimi, M., Bajaj, G., Banerjee, T., Thirunarayan, K., ... & Sheth, A. (2017, July). Semi-supervised approach to monitoring clinical depressive
symptoms in social media. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017 (pp. 1191-1198).
● Daniulaityte, R., Nahhas, R. W., Wijeratne, S., Carlson, R. G., Lamy, F. R., Martins, S. S., ... & Sheth, A. (2015). “Time for dabs”: Analyzing Twitter data on marijuana concentrates
News
Articles
Twitter
Data
Domain
Knowledg
e
Content
Enrichment
DAO
DSM-5
Location Extraction
Keyphrase Extraction
Age-based
Clustering
Semantic Filtering
Entity
Extraction
NLM Training
f(.)
Knowledge Infused
Natural Language
Understanding (Ki-
NLU)
Semantic
Mapping
Semantic
Proximity
Topic Model
Language Model
DAO
DSM-5
Dashboard
Visualizations
(Online)
Offline
Analysis
&
Visualizations
Opioid and Substance Use

24. Social Media Analysis to understand
Cannabis and Synthetic Cannabinoid use
Visit Project page: http://wiki.aiisc.ai/index.php/EDrugTrends

25. Study of Cannabis use on social media
Semi-automated platform to identify emerging trends in cannabis and synthetic
cannabinoid use in the U.S.
◎ To analyze characteristics of marijuana concentrate users, describe patterns and
reasons of use.
◎ To identify factors associated with daily use of concentrates among U.S.-based
cannabis users recruited via a Twitter-based online survey
◎ Identify and compare trends in knowledge, attitudes, and behaviors related to
cannabis and synthetic cannabinoid use across U.S. regions with different
cannabis legalization policies using Twitter and Web forum data.
◎ Analyze social network characteristics and identify key influencers (opinions
leaders) in cannabis and synthetic cannabinoid-related discussions on Twitter
28

26. DAO
Cannabis Study on social Media
The DAO was expanded further to include more
comprehensive representation of emerging
cannabis products, synthetic cannabinoid
products, health-related consequences and
mental health conditions.
NLU
http://wiki.aiisc.ai/index.php/EDrugTrends

27. Drug Abuse Ontology as knowledge
source for analyzing web-based data and
use the extracted wisdom to inform public
health surveillance as insights and
actionable items.
Ontology as a Knowledge Source to understand
Addiction and Mental Health
Roy, Kaushik, et al. "" Is depression related to cannabis?": A knowledge-infused model
for Entity and Relation Extraction with Limited Supervision." arXiv preprint
arXiv:2102.01222 (2021).

28. Knowledge-infused via DAO Ontology for solving relation
between Cannabis and Depression
Yadav, Shweta, et al. "“When they say weed causes depression, but it’s your fav antidepressant”: Knowledge-aware
Attention Framework for Relationship Extraction." PloS one 16.3 (2021): e0248299.

29. Monitoring Cryptomarkets to Identify
Emerging Trends of Illicit Synthetic Opioids
Use
Visit Project page: http://wiki.aiisc.ai/index.php/Public_Health_Addictions_Research_at_AIISC

30. Why study cryptomarkets
To monitor Cryptomarkets to Identify Emerging Trends of Illicit Synthetic Opioids Use
Semi automated platform to monitor illicit online transactions of several illicit synthetic
opioids in dark web.
To design effective and responsive prevention and policies for public health professionals
Epidemiological surveillance by providing timely data regarding emerging substances and
product form
To monitor Darknet supply and marketing trends over time.
Enhancing the capacities of early warning systems like NDEWS
33

32. Wisdom
Knowledge
Wisdom as emerging
trend alert to NDEWS
Task: to harness cryptomarket data on illicit fentanyl and other novel synthetic opioid
over time and identify new substances
NLU

33. Actionable insight to National Drug Early Warning
System (ndews.org)
Reports about Novel Synthetic
Opioids being sold on
cryptomarkets. Early
identification of emerging trends.
Fentanyl Analogs (Ex:
Carfentanil, acetyl fentanyl,
furanyl fentanyl) , Novel
Synthetic Opioids, variation in
potency and other
pharmacological features.

34. Shallow knowledge infusion to identify unique vendors
in cryptomarkets - Sybil Account Detection
Kumar, Ramnath, et al. "edarkfind: Unsupervised multi-view learning for sybil account detection." Proceedings of The
Web Conference 2020. 2020.

35. Knowledge Infusion with DAO in DL model to detect trends in
cryptomarkets - Reflection on social Media
Table: Sample properties derived from cryptomarket with DAO
Lokala, Usha, et al. "eDarkTrends: Harnessing Social Media Trends
in Substance Use Disorders for Opioid Listings on Cryptomarket."
ICLR AI for Public Health Workshop 2021.

36. “
Mental Health Disorders
Wisdom as insights,
emerging alerts and
actionable items
Drug Discovery
Emerging Trends
Enabling explainability
Public Health surveillance
User Roles and Behaviors
Knowledge Sources: DAO
and ODKG
Takeaway - Epidemiology
Real impact
Cannabis study- Social
Media
Problems addressed
Fentanyl- Cryptomarkets
Web forums, BlueLight,
Reddit, Twitter,
Cryptomarkets
How does Knowledge
help in Epidemiology
domain?
Public Health Addictions Wiki Page:
http://wiki.aiisc.ai/index.php/Public_Health_Addictions_Research_at_AIISC

37. Online Mental Health Support
Social Media: Reddit
Visit Project page: Mental Health

38. Interest
We are interested in
Matching Support Seekers
-SSs (left) with Support
Providers - SPs (right)
Current State
Currently, moderators
(center) do this matching
41
Proposal
SS-SP matcher will replace/assist the moderators that use
medical knowledge, information about the user, extracted
from the posts to perform this matching

39. 42
Example of matching on Reddit - SS-SP on Reddit
Opiate addiction and recovery SS-SP matching Anxiety and depression SS-SP matching

40. Using knowledge and lexicons with deep learning for
matching users on Reddit
Event-Specific Filtering
Business Closure
School Closure
Lockdown
Shelter-in-place
Hospitalization
Domain-Specific
Filtering
Anxiety
Depression
First Person Pronouns
Subordinate Conjunction
Max Height
Max Verb Phrase Length
Syntactic Features
Sentiment and Emotions
Focus Future
Cognitive Features
Biological Features
Psycholinguistic
Gaur 2018, Gkotsis 2017
Linguistic Inquiry and Word Count http://liwc.wpengine.com/

41. Mental Health Dialogue
44

42. Construction of Personalized Knowledge Graph

43. Knowledge-infused Reinforcement Learning
● The input to the agent is sequential through many steps, it gets an input and a reward at every step and
learns the right output gradually through reinforcement.

44. APH Self Management: Reinforcement Learning

45. Conclusion
48

46. Knowledge Integration:
● Variety of knowledge is
available ,
● therefore depending on use-
case/application it can be
combined/integrated
This will enable support for the
right kind of wisdom depending on
the task
Integrate
Integrated KG
Stratified knowledge types with one example
knowledge that can be directly used or
extracted from, relevant to NLU Sheth, A., & Thirunarayan, K. (2021). The duality of data and knowledge across the three waves
of AI. IT Professional, 23(3), 35–45.
Language Syntactic
Structure and Grammar

47. Data Sources
◎ Social Media (Reddit,
Twitter,Web Forums)
◎ Conversations: Patient-
Clinician, Virtual Health
Assistant-Patient
AI Techniques and
Technologies
◎ Knowledge Graphs/Ontologies
(contextualization, personalization,
abstraction)
◎ NLP/NLU
◎ Machine Learning/Deep Learning
(RL, GAN, CNN, LSTM,....)
◎ Conversational AI, Q/A
◎ mApp, Virtual Health Assistants
(Chatbots)
◎ Health sensors/IoTs/mobile
devices

48. “
51
AIISC in core AI areas, and
interdisciplinary AI/AI applications
>> 25 researchers
including 4 faculty
(6 in Fall 2021), 2-3
postdocs, ~20 PhD
students, >10
MS/BS and several
interns/associates

49. Addiction and Mental Health Projects at AIISC
➢ Improving mental health of COVID-19 patients with an Artificial Intelligence-based chatbot
➢ Personalized Virtual Health Assistant Enabled by Knowledge-infused Reinforcement Learning for
Adaptive Mental Health Self-management
➢ Characterizing and supporting help seekers on social media using expert-in-the-loop learning
➢ Modeling Social Behavior for Healthcare Utilization in Depression (NIMH)
➢ eDarkTrends : Monitoring Cryptomarkets to Identify Emerging Trends of Illicit Synthetic Opioids
Use (NIDA)
➢ Trending: Social media analysis to monitor cannabis and synthetic cannabinoid use (NIDA)
➢ Innovative NIDA National Early Warning System Network (iN3) (NIDA)
➢ PREDOSE: PREscription Drug abuse Online Surveillance and Epidemiology (NIDA)
➢ Community-Driven Data Engineering for Substance Abuse Prevention in the Rural Midwest (NSF)
➢ BD Spoke: Opioid and Substance Use in Ohio
➢ more ...

50. http://aiisc.ai, http://wiki.aiisc.ai
https://scholarcommons.sc.edu/aii_fac_pub/
Many thanks to our sponsors, esp. ~10 NIH
grants (four R01s, three R21s, R56, etc) from
NIMH, NIDA, NICHD, and other institutes.
Acknowledgement
Usha Lokala, Raminta Daniulaityte, Francois Lamy, Manas Gaur, Jyotishman
Pathak, and collaborators on NIDA/NIH and NSF funded projects on Addiction
and Mental Health.