Social media platforms facilitate the emergence of citizen communities that discuss real-world events, and generate/share content with a variety of intent ranging from social good (e.g., volunteering to help) to commercial interest (e.g., criticizing product features). Hence, mining intent from social data can aid in filtering social media to support organizations, such as an emergency management unit for resource planning. However, effective intent mining is inherently challenging due to ambiguity in interpretation, and sparsity of relevant behaviors in social data. In this research, we address the problem of multiclass classification of intent with a use-case of social data generated during crisis events. Our novel method exploits a hybrid feature representation created by combining top-down processing using knowledge-guided patterns with bottom-up processing using a bag-of-tokens model. We employ pattern-set creation from a variety of knowledge sources including psycholinguistics to tackle the ambiguity challenge, social behavior about conversations to enrich context, and contrast patterns to tackle the sparsity challenge.

1. Intent Classification of Short-text
Social Media
Dec 19 2015
The 8th
IEEE SocialCom-2015
Hemant Purohit
Information Sciences and Technology, George Mason U
Guozhu Dong, Valerie Shalin,
Krishnaprasad Thirunarayan, Amit Sheth
Kno.e.sis, Wright State U

2. @hemant_pt IEEE SocialCom-2015
Outline
● Intention
● Social Media Short-text
● Intent Classification Problem
● Feature Representation
● Bottom-Up
● Bag of Tokens model
● Top-Down
● Set of Patterns:
● Declarative Knowledge & Social Behavior Knowledge
● Contrast Mining based Patterns
● Experiments & Results
● Limitations & Future Work
22

3. @hemant_pt IEEE SocialCom-2015
Intention
● Intent: Purpose or aim for an action
● ‘we are tempted to speak of “different senses” of a
word which is clearly not equivocal, we may infer that
we are pretty much in the dark about the character of
the concept which it represents’ (Anscombe 1963, p. 1) [Stanford
Encyclopedia of Philosophy]
● Latent in the utterance
3

4. @hemant_pt IEEE SocialCom-2015
Social Media Short-text & Intent
Social media text: unstructured, informal language, short
4
DOCUMENT INTENT
Text REDCROSS to 90999 to donate 10$ to help the
victims of hurricane sandy
SEEKING HELP
Anyone know where the nearest #RedCross is? I wanna
give blood today to help the victims of hurricane Sandy
OFFERING HELP
Would like to urge all citizens to make the proper
preparations for Hurricane #Sandy - prep is key - http://t.
co/LyCSprbk has valuable info!
ADVISING
4

5. @hemant_pt IEEE SocialCom-2015
Short-text Document Intent
● Intent: Aim of action
DOCUMENT INTENT
Text REDCROSS to 90999 to donate 10$ to help the
victims of hurricane sandy
SEEKING HELP
Anyone know where the nearest #RedCross is? I wanna
give blood today to help the victims of hurricane Sandy
OFFERING HELP
Would like to urge all citizens to make the proper
preparations for Hurricane #Sandy - prep is key - http://t.
co/LyCSprbk has valuable info!
ADVISING
5
How to identify relevant intent from ambiguous, unconstrained
natural language text?
Relevant intent ➔ Articulation of organizational tasks
(e.g., Seeking vs. Offering resources)
5

6. @hemant_pt IEEE SocialCom-2015
Intent Classification: Problem
Formulation
● Given a set of user-generated text documents, identify
existing intents
● Variety of interpretations
● Problem statement: a multi-class classification task
approximate f: S → C , where
C = {C1
, C2
, …, CK
}
is a set of predefined K intent classes, and
S = {m1
, m2
… mN
}
is a set of N short text documents
Focus - Cooperation-assistive intent classes, C= {Seeking, Offering, None}
66

7. @hemant_pt IEEE SocialCom-2015
Intent Classification: Related Work
TEXT
CLASSIFICATION
TYPE
FOCUS EXAMPLE
Topic predominant
subject matter
sports or entertainment
Sentiment/Emotion/
Opinion
focus on present state
of emotional affairs
negative or positive;
happy emotion
Intent Focus on action, hence,
future state of affairs
offer to help after floods
e.g., I am going to watch the awesome Fast and Furious movie!! #Excited
77

8. @hemant_pt IEEE SocialCom-2015
Intent Classification: Related Work
DATA TYPE APPROACH FOCUS LIMITED APPLICABILITY
8
Formal text on
Webpages/blogs
(Kröll and Strohmaier 2009, -15;
Raslan et al. 2013, -14)
Knowledge
Acquisition:
via Rules, Clustering
• Lack of large corpora with
proper grammatical structure
• Poor quality text hard to parse
for dependencies
Commercial Reviews,
marketplace
(Hollerit et al. 2013, Chen et al.
2013, Wang et al. 2015, Wu et al.
2011, Ramanand et al. 2010, Carlos
& Yalamanchi 2012)
Classification:
via Rules, Lexical
template based,
Pattern
• More generalized intents (e.
g., ‘help’ broader than ‘sell’)
• Patterns implicit to capture than
for buying/selling
Search Queries
(Broder 2002, Downey et al. 2008,,
Case 2012, Wu et al. 2010,
Strohmaier & Kröll 2012)
User Profiling:
Query Classification
• Lack of large query logs, click
graphs
• Existence of social conversation
8

9. @hemant_pt IEEE SocialCom-2015
Intent Classification: Challenges
● Unconstrained Natural Language in small space
● Ambiguity in interpretation
● Sparsity of low ‘signal-to-noise’: Imbalanced classes
● 1% signals (Seeking/Offering) in 4.9 million tweets #Sandy
● Hard-to-predict problem
● e.g., commercial intent, F-1 score 65% on Twitter [Hollerit et al. 2013]
@Zuora wants to help @Network4Good with Hurricane Relief. Text SANDY to
80888 & donate $10 to @redcross @AmeriCares & @SalvationArmyUS #help
*Blue: offering intent, *Red: seeking intent
99

10. @hemant_pt IEEE SocialCom-2015
Intent Classification: Domain & Features
10
Intent
Binary
Crisis Domain:
- [Varga et al. 2013] Problem & Aid (Japanese)
- Purohit et al. 2013, 2014: Seeking & Offering
- Features: N-grams, Rules, Noun-Verb templates, etc.
Commercial Domain:
- [Hollerit et al. 2013] Buy vs. Sell intent
- Features: N-grams, Part-of-Speech
Multiclass
Commercial Domain:
- [Wang et al. 2015] Semi-supervised
- Features: N-grams, Part-of-speech
10

11. @hemant_pt IEEE SocialCom-2015
TOP-DOWN
Pattern Rules:
Declarative (DK) & Social Behavior (SK)
Knowledge, Contrast Mining (CTK,CPK)
(patterns defined for intent association)
BOTTOM-UP
Bag of N-grams Tokens:
Independent Tokens
(patterns derived from the data)
Our
Hybrid
Approach
Learning
Improves
Expressivity
Increases
11

12. @hemant_pt IEEE SocialCom-2015
Intent Classification Hybrid:
Multiclass Classifier – Feature Creation
1. (T) Bag of Tokens
Abstraction: due to importance in info sharing [Nagarajan et al. 2010]
- Numeric (e.g., $10) → _NUM_
- Interactions (e.g., RT & @user) → _RT_ , _MENTION_
- Links (e.g., http://bit.ly) → _URL_
N-grams: after stemming and abstraction [Hollerit et al. 2013]
TOKENIZER ( mi
) → { bi-, tri-gram }
12
TOKENIZER(mi ,
min, max)
12

13. @hemant_pt IEEE SocialCom-2015
Leveraging Declarative Knowledge
● Conceptual Dependency Theory [Schank, 1972]
● Make meaning independent from the actual words in input
● e.g., Class in an Ontology abstracts similar instances
● Verb Lexicon [Hollerit et al. 2013]
● Verb reflects action
● Relevant Levin’s Verb categories [Levin, 1993] , e.g., give, send, etc.
● Syntactic Pattern
● Auxiliary & modals: e.g., ‘be’, ‘do’, ‘could’, etc. [Ramanand et al. 2010]
● Word order: Verb-Subject positions, etc.
1313

14. @hemant_pt IEEE SocialCom-2015
Leveraging Social Behavior Knowledge
● Conversation indicators often thrown away in Text Mining
14
CATEGORY Hj
Hj
SET
H1 - Determiners (the)
H3 - Subject pronouns (she, he, we, they)
H9 - Dialogue management indicators (thanks, yes, ok, sorry, hi, hello, bye,
anyway, how about, so, what do you
mean, please, {could, would, should, can,
will} followed by pronoun)
H11 - Hedge words (kinda, sorta)
• Feature_Hj
(mi
) = term-frequency ( Hj
-set, mi
)
• Normalized
• Total 14 feature categories

15. @hemant_pt IEEE SocialCom-2015
Intent Classification Hybrid:
Multiclass Classifier - Feature Creation
2. (DK) Declarative Knowledge Patterns
● Domain expert guidance
● Psycholinguistics syntactic & semantic rules
● Expand by WordNet and Levin Verbs
e.g.,
3. (SK) Social Knowledge Indicators
● Offline conversation indicators
e.g., Hj
= Dialogue Management, Hj-set = {Thanks, anyway,..}
15
Feature_Pj
(mi
) = 1 if Pj
exists in mi
, else 0
Feature_Hj
(mi
) = term-frequency ( Hj
-set, mi
)

16. @hemant_pt IEEE SocialCom-2015
Intent Classification Hybrid:
Multiclass Classifier - Feature Creation
4. (CTK) Contrast Knowledge Patterns
INPUT: corpus {mi
} cleaned and abstracted, min. support, X
For each class Cj
● Find contrasting pattern using sequential pattern mining
OUTPUT: contrast patterns set {P} for each class Cj
5. (CPK) Contrast Patterns: on Part-of-Speech tags of {mi
}
16
e.g., unique sequential patterns:
SEEKING: help .* victim .* _url_ .*
OFFERING: anyon .* know .* cloth .*

17. @hemant_pt IEEE SocialCom-2015
Contrast Mining based Patterns
Finding CTK (CPK): Contrast Knowledge Patterns
For each class Cj
1. Tokenize the cleaned, abstracted text of {mi
}
2. Mine Sequential Patterns, {P}: using SPADE Algorithm
3. Reduce to minimal sequences {P}
4. Compute growth rate & contrast strength for P with all other Ck
5. Top-K ranked {P} by contrast strength
OUTPUT: contrast patterns set {P} for each class Cj
17
gr(P,Cj,Ck) = support (P,Cj) / support (P,Ck) .. (1)
Contrast-Growth (P,Cj) = 1/(|Cj| -1) ΣCk, k=/=j
gr(P,Cj,Ck)/ (1 + gr(P,Cj,Ck)) ..(2)
Sparse-Contrast-Strength(P,Cj) = support(P,Cj)*Contrast-Growth(P,Cj) .. (3)

18. @hemant_pt IEEE SocialCom-2015
CORPUS
Set of
short text
documents,
S
FEATURES
Knowledge-driven
features
XT
,
y
M_1
M_2
M_K
.
.
.
Subset Xj
T
⊂ S such that, Xj
T
includes
all the labeled instances of class Cj
for
model M_j
Binarization Frameworks for Multiclass
Classifier: 1 vs. All (OVA)
P(c2
)
P(c1
)
X1
T
, y1
X2
T
, y2
XK
T
, yK
P(cK
)
18
(In 1 vs. 1 (OVO) framework: K*(K-1)/2 classifiers, for each Cj,Ck pair)

19. @hemant_pt IEEE SocialCom-2015
Intent Classification Hybrid:
Multiclass Classifier - Experiments
● Datasets
● Dataset-1: Hurricane Sandy, Oct 27 – Nov 7, 2012
● Dataset-2: Philippines Typhoon, Nov 7 – Nov 17, 2013
● Parameters
● Base Learner M_j: Random Forest, 10 trees with 100 features
● bi-, tri-gram for (T)
● K=100% & min. support 10% for CTK, 50% for CPK
19

20. @hemant_pt
Intent Classification:
Multiclass Classifier – Results
20
Avg. F-1 Score
(10-fold CV)
Frameworks:
Gain 7%, p < 0.05
Dataset-1 (Hurricane Sandy, 2012)
(Declarative)
(Social)
(Contrast)
T,DK,SK,CTK,CPK
T,CTK,CPK

21. @hemant_pt
Intent Classification:
Multiclass Classifier - Results
21
Frameworks:
Gain 6%, p < 0.05
Dataset-2 (Philippines Typhoon, 2013)
(Declarative)
(Social)
(Contrast)
Avg. F-1 Score
(10-fold CV)
T,DK,SK,CTK,CPK
T,CTK,CPK

22. @hemant_pt IEEE SocialCom-2015
Lessons
1. Top-down & Bottom-up hybrid approach improves data
representation for learning (complementary) intent classes
- Top 1% discriminative features contained 50% knowledge driven
2. Offline theoretic social conversation (SK) features (the,
thanks, etc.), often removed for text mining are valuable for
intent mining.
3. There is a varying effect of knowledge types (SK vs. DK vs.
CTK/CPK) in different types of real world event datasets
➢ Culturally-sensitive psycholinguistics knowledge in future
22

23. @hemant_pt IEEE SocialCom-2015
Limitations & Future Work Directions
-Non-cooperation assistive intent classes not considered
-Temporal drift of intent not considered
-Possibility for Multilabel intent classes with instances
-Mining actor-level intent beyond document level
23

24. @hemant_pt IEEE SocialCom-2015
Conclusion
A hybrid approach of interplaying features from
top-down representation via patterns using prior knowledge
of psycholinguistics, social behavior, & contrast mining
&
bottom-up representation via bag-of-tokens model
improves Intent Classification of short-text on social media.
24

25. @hemant_pt IEEE SocialCom-2015
25
TWITTER: @hemant_pt
MAIL: hpurohit@gmu.edu
Acknowledgement: Respective image sources, and
Questions?
Grant IIS-1111182