This document discusses character-based hybrid sentiment analysis. It begins by outlining reasons for performing sentiment analysis, such as determining if movie reviews are positive or negative. It then discusses challenges like informal language with misspellings and new words. Different neural network approaches are reviewed, including recurrent neural networks, convolutional neural networks, and hybrid CNN-RNN models. A novel approach is proposed that uses a character-based CNN to generate word embeddings, followed by a CNN to extract local features and an RNN to model long-distance dependencies, for the purpose of sentiment analysis.

1. Character-based Hybrid
Sentiment Analysis
JITIN DUA, PRANAV DIXIT

2. Why sentiment analysis?
◦ Movie: Is this review positive or negative?
◦ Products: What do people think about the new iPhone?
◦ Public sentiment: How is consumer confidence?
◦ Politics: What do people think about Trump?
◦ Prediction: Predict market trends

3. Challenges
Ø Short in length - less contextual data to predict the sentiment
Ø Informal with misspellings (often intentional, like different spellings of ”cool”: coool, coolll,
coooolll!!), new words, slangs, and URLs
Ø New words, new trends in using abbreviations - leads to the problem of out-of-vocabulary
words
Ø Special symbols like emoticons and hashtags

4. Related Work:
1. Recurrent Neural Network
a. Word based
b. Character based
2. Convolutional Neural Network
a. Word based
b. Character based
3. Hybrid (CNN-RNN)
a. Word based
b. Character based (our approach)

5. RNN based models
RNN takes words in a sentence in a sequential order and is able to learn the long-term
dependencies of texts rather than local features.

6. CNN based models
Similar to the translation, rotation and scale invariance of images in CNN, CNN is able to learn
the local features from words or phrases in different places of texts.
Uses different size filters to get n-grams, ex. unigram, bigram, etc.
◦ contiguous sequence of n items from a given sequence
Example: “the country of my birth” computes n-grams as follows:
the country, country of, of my, my birth, the country of, country of my, of my birth, the country of
my, country of my birth
Kim, Y. Convolutional Neural Networks for Sentence Classification. In Proceedings of EMNLP. 2014.

7. CNN based models
Kim, Y. Convolutional Neural Networks for Sentence Classification. In Proceedings of EMNLP. 2014

8. Word Based
Word Embedding
Ø Expresses a word as a real number vector of low dimension (about 200 dimensions or higher)
Ø Words with similar meaning corresponds to close vector and obtain meaningful results (e.g.
king – man + women = queen) by adding (subtracting) vectors.
Ø Types:
Ø Train from data
Ø Pre-trained models (static, non-static)
Ø Popular pre-trained models:
Ø Google word2vec
Ø Glove

9. http://suriyadeepan.github.io/2016-06-28-easy-seq2seq/

10. Word based models: Issues
• Word based models are blind to sub-word information (e.g. morphemes – eventful, eventfully,
uneventful)
• Most of the training parameters are from the word embedding
• Word2Vec is usually very big in size
• Google word2vec: 3.5 GB
• Word based models can not handle
• New words
• Typos

11. CharCNN
Ø Detects sub-word information (e.g. morphemes)
Ø Does not require morphological tagging as a pre-processing step
Ø Handles typos, slangs and new words
Ø As most of the parameters are from the word embeddings, the proposed model has
significantly fewer parameters
Ø Makes it attractive for applications where model size may be an issue (e.g. cell phones).

12. CharCNN
Character based word embedding
Ø Creates a word-level embedding using one-hot representation of characters
Ø Uses multiple filters of varying widths to obtain the feature vector for a word
Ø Each filter of the CharCNN is essentially learning to detect particular character n-grams
Ø If we have a total of h filters H1,...,Hh, then yk = [y1k , . . . , yhk ] is the input representation of
word k.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. (AAAI 2016)

13. z = t ⊙ g(WH y + bH ) + (1 − t) ⊙ y
t = σ(WT y + bT )
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. (AAAI 2016)

14. Hybrid approach
CNN + RNN architecture
Takes advantage of the coarse-grained local features generated by CNN and long-distance
dependencies learned via RNN for sentiment analysis.
Word2Vec
CNN
(local features)
RNN
(long-distance
dependencies)
Words Sentiment

15. Wang, Jiang, Luo. Combination of Convolutional and Recurrent Neural Network for Sentiment Analysis of Short Texts . Dec 2016.

16. Novel approach
CharCNN + Hybrid-CNN-RNN
Ø Advantages of character based models
Ø Coarse-grained local features generated by CNN
Ø Long-distance dependencies learned via RNN for sentiment analysis
CharCNN
(word-embedding)
CNN
(local features)
RNN
(long-distance
dependencies)
Characters Sentiment

17. Future of sentiment analysis:
Better T-800!..

18. References:
1. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural
language models. (AAAI 2016)
2. X Wang, W Jiang, Z Luo. 2016. Combination of Convolutional and Recurrent Neural Network
for Sentiment Analysis of Short Texts.
3. Y Xiao, K Cho. 2016. Efficient Character-level Document Classification by Combining
Convolution and Recurrent Layers. arXiv preprint arXiv:1602.00367.
4. Kim, Y. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of
EMNLP.

19. Q&A

20. Thanks..