This document discusses Twitter sentiment analysis using a lexicon-based approach. It covers tokenization, stop word removal, stemming, lemmatization, and using a lexicon to calculate sentiment polarity. Naive Bayes algorithms are described for classification, with applications including real-time prediction, multi-class prediction, text classification, and recommendation systems. Tools mentioned include Anaconda and Spider.

1. Twitter Sentimental Analysis
Under the Guidance of:
Dr. Ashish khare
ANIL KUMAR MAURYA
M.TECH (COMPUTER TECHNOLOGY.)
EN NO-(15AU/995)
DEPARTMENT OF ELECTRONICS &
COMMUNICATION,
UNIVERSITY OF ALLAHABAD

2. Out line
 Type of data
 Tokenization
 Stop word removal
 Stemming
 Lemmatization
 Lexicon base approach
 Lexicon calculation

5. Tokenization
 Given a character sequence and a defined document unit, tokenization is
the task of chopping it up into pieces, called tokens , perhaps at the same
time throwing away certain characters, such as punctuation. Here is an
example of tokenization.
 Input:
 Output:

6. Stop word removal
 Stop word , by definition ,are meaningless word that have low
discrimination power.

8. Stemming
 For grammatical reasons, documents are going to use different forms of a word, such
as organize, organizes, and organizing. Additionally, there are families of derivationally related words
with similar meanings, such as democracy, democratic, and democratization. In many situations, it
seems as if it would be useful for a search for one of these words to return documents that contain
another word in the set.
 The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes
derivationally related forms of a word to a common base form. For instance:
 am, are, is= be
car, cars, car's, cars' = car
 The result of this mapping of text will be something like:
 the boy's cars are different colors
the boy car be differ color

9. Lemmatization
 . Lemmatization usually refers to
doing things properly with the use
of a vocabulary and morphological
analysis of words, normally aiming
to remove inflectional endings
only and to return the base or
dictionary form of a word, which
is known as the lemma

10. What is?
SENTIMENT DETECTION ORIENTATION USING LEXICON-BASED
APPROACH

18. Lexicon method:
 Machine Learning Methods: Such techniques require creating a model by
training the classifier with labeled examples. This means that you must first
gather a dataset with examples for positive, negative and neutral classes, extract
the features from the examples and then train the algorithm based on the
examples. These methods are used mainly for computing the polarity of the
document.
 Choice of the method heavily depends on the application, domain and language.
Using lexicon based techniques with large dictionaries enables us to achieve
very good results. Nevertheless they require using a lexicon, something which is
not always available in all languages.
On the other hand Machine Learning based techniques deliver good results but
they require obtaining training on labeled data.

19. Naïve Bayer algorithms
 Bayes theorem provides a way of calculating posterior probability P(c|x)
from P(c), P(x) and P(x|c). Look at the equation below:


20.  Above,
 P(c|x) is the posterior probability of class (c, target)
given predictor (x, attributes).
 P(c) is the prior probability of class.
 P(x|c) is the likelihood which is the probability of predictor given class.
 P(x) is the prior probability of predictor.

21. How Naive Bayes algorithm works?
 Let’s understand it using an example. Below I have a training data set of
weather and corresponding target variable ‘Play’ (suggesting possibilities
of playing). Now, we need to classify whether players will play or not based
on weather condition. Let’s follow the below steps to perform it.
 Step 1: Convert the data set into a frequency table
 Step 2: Create Likelihood table by finding the probabilities like Overcast
probability = 0.29 and probability of playing is 0.64.

22. WEATHER CONDITION AND PLAYING
SITUATION

23. NBC OF SUNNY DAY
P(YES/SUNNY)= (P(SUNNY/YES) * P(YES))/P(SUNNY)
=((5/14)*(9/14))/5/14=.22
P(NO/SUNNY)=(P(SUNNY/NO)*P(NO))/P(SUNNY)
=((5/14*5/14))/5/14=.12
Normalization of yes=(.22)/(.22+.12)=.68
Normalization of no=(.12)/(.12+.22)=.35
P(yes/sunny)>p(no/sunny) so probability of day is playing game .

24. Applications of Naive Bayes Algorithms
 Real time Prediction: Naive Bayes is an eager learning classifier and it is sure fast.
Thus, it could be used for making predictions in real time.
 Multi class Prediction: This algorithm is also well known for multi class prediction
feature. Here we can predict the probability of multiple classes of target variable.
 Text classification/ Spam Filtering/ Sentiment Analysis: Naive Bayes classifiers
mostly used in text classification (due to better result in multi class problems and
independence rule) have higher success rate as compared to other algorithms. As a
result, it is widely used in Spam filtering (identify spam e-mail) and Sentiment Analysis
(in social media analysis, to identify positive and negative customer sentiments)
 Recommendation System: Naive Bayes Classifier and Collaborative Filtering together
builds a Recommendation System that uses machine learning and data mining
techniques to filter unseen information and predict whether a user would like a given
resource or no

25. Tools for data analyses
 Anaconda-3.6
 Spider application

27. REFERENCES…………..
 [1] Paul C. Zikopoulos, Chris Eaton, Dirk deRoos “Understanding Big Data”, ISBN 978-07179053-
 6.
 [2] Penchalaiah.C, Murali.GSuresh Babu.A, Effective Sentiment Analysis on Twitter Data using: Apache Flume and Hive, Computer Science and
EngineeringDept, JNTUACEP, Pulivendula, Vol. 1 Issue 8, October 2014.
 [3] Mr. Swapnil A. Kale, Prof. Sangram S.Dandge, Understanding the Big Data problems and their solutions using Hadoop MapReduce, ISSN 2319 – 4847,Volume
3.
 [4] K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The Hadoop Distributed File System,” in the 26th IEEE Symposium on Mass Storage Systems and
Technologies, pp. 1-10, May 2010.
 [5] J. Dean and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,”
 Communications of the ACM, Vol. 51, Iss. 1, pp. 107-113, January 2008.
 [6] T. White, "The Hadoop Distributed Filesystem," Hadoop: The Definitive Guide, pp. 41-73,
 GravensteinHighwaNorth, Sebastopol: O’Reilly Media, Inc., 2010.
 [7] Chansup Byun, William Arcand, David Bestor, Bill Bergeron, Matthew Hubbell, Jeremy
 Kepner, Andrew McCabe, Peter Michaleas, Julie Mullen, David O’Gwynn, Andrew Prout, Albert Reuther, Antonio Rosa, Charles Yee, “ Driving Big Data With Big
Compute”, MIT Lincoln Laboratory, Lexington, MA, U.S.A.
 [8](OnlineResource) http://www.ibmbigdatahub.com/infographic/four-vs-big-data
 [9](OnlineResource)http://hadoop.apache.org/docs/r2.5.0