SlideShare a Scribd company logo
1 of 30
Download to read offline
R by example: mining Twitter for consumer
attitudes towards airlines

presented at the

Boston Predictive Analytics
MeetUp
by


Jeffrey Breen
President
Cambridge Aviation Research

jbreen@cambridge.aero

June 2011




  Cambridge Aviation Research   • 245 First Street • Suite 1800 • Cambridge, MA 02142 • cambridge.aero




                                                                                                         © Copyright 2010 by Cambridge Aviation Research. All rights reserved.
Airlines top customer satisfaction... alphabetically




http://www.theacsi.org/                                     3
Actually, they rank below the Post
    Office and health insurers




                                     4
which gives us plenty to listen to
                               Completely unimpressed with @continental or @united.
RT @dave_mcgregor:       Poor communication, goofy reservations systems and
Publicly pledging to                       all to turn my trip into a mess.
never fly @delta again.
The worst airline ever.
U have lost my patronage     @united #fail on wifi in red carpet clubs (too
forever due to ur            slow), delayed flight, customer service in red
incompetence                 carpet club (too slow), hmmm do u see a trend?



@United Weather delays may not be your fault,
but you are in the customer service business.
It's atrocious how people are getting treated!
We were just told we are delayed 1.5        @SouthwestAir I know you don't make the
hrs & next announcement on @JetBlue -      weather. But at least pretend I am not a
“We're selling headsets.” Way to           bother when I ask if the delay will make
capitalize on our misfortune.                                    miss my connection
         @SouthwestAir
     I hate you with every            Hey @delta - you suck! Your prices
    single bone in my body          are over the moon & to move a flight
   for delaying my flight by         a cpl of days is $150.00. Insane. I
   3 hours, 30mins before I              hate you! U ruined my vacation!
    was supposed to board.
             #hate
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                14
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                15
Searching Twitter in one line
R’s XML and RCurl packages make it easy to grab web data, but Jeff
Gentry’s twitteR package makes searching Twitter almost too easy:

> # load the package
> library(twitteR)
> # get the 1,500 most recent tweets mentioning ‘@delta’:
> delta.tweets = searchTwitter('@delta', n=1500)




See what we got in return:              A “list” in R is a collection of
                                        objects and its elements may be
> length(delta.tweets)                  named or just numbered.
[1] 1500
> class(delta.tweets)
[1] "list"
                                        “[[ ]]” is used to access elements.
Examine the output
Let’s take a look at the first tweet in the output list:

    > tweet = delta.tweets[[1]]
                                       tweet is an object of type “status”
                                       from the “twitteR” package.
    > class(tweet)
    [1] "status"
    attr(,"package")                   It holds all the information about
    [1] "twitteR"                      the tweet returned from Twitter.



The help page (“?status”) describes some accessor methods like
getScreenName() and getText() which do what you would expect:

    > tweet$getScreenName()
    [1] "Alaqawari"
    > tweet$getText()
    [1] "I am ready to head home. Inshallah will try to get on the earlier
    flight to Fresno. @Delta @DeltaAssist"
Extract the tweet text
R has several (read: too many) ways to apply functions iteratively.
•The plyr package unifies them all with a consistent naming convention.
•The function name is determined by the input and output data types. We
have a list and would like a simple array output, so we use “laply”:

> delta.text = laply(delta.tweets, function(t) t$getText() )


> length(delta.text)[1] 1500
> head(delta.text, 5)
[1] "I am ready to head home. Inshallah will try to get on the earlier
flight to Fresno. @Delta @DeltaAssist"
[2] "@Delta Releases 2010 Corporate Responsibility Report - @PRNewswire
(press release) : http://tinyurl.com/64mz3oh"
[3] "Another week, another upgrade! Thanks @Delta!"
[4] "I'm not able to check in or select a seat for flight DL223/KL6023 to
Seattle tomorrow. Help? @KLM @delta"
[5] "In my boredom of waiting realized @deltaairlines is now @delta
seriously..... Stil waiting and your not even unloading status yet"
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                19
Estimating Sentiment

There are many good papers and resources describing methods to
estimate sentiment. These are very complex algorithms.



For this tutorial, we use a very simple algorithm which assigns a score by
simply counting the number of occurrences of “positive” and “negative”
words in a tweet. The code for our score.sentiment() function can be
found at the end of this deck.


Hu & Liu have published an “opinion lexicon” which categorizes
approximately 6,800 words as positive or negative and which can be
downloaded.


            Positive: love, best, cool, great, good, amazing
            Negative: hate, worst, sucks, awful, nightmare
                                                                        20
Load sentiment word lists
1. Download Hu & Liu’s opinion lexicon:


   http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html


2. Loading data is one of R’s strengths. These are simple text files,
though they use “;” as a comment character at the beginning:

   > hu.liu.pos = scan('../data/opinion-lexicon-English/positive-
   words.txt', what='character', comment.char=';')

   > hu.liu.neg = scan('../data/opinion-lexicon-English/negative-
   words.txt', what='character', comment.char=';')



3. Add a few industry-specific and/or especially emphatic terms:

   > pos.words = c(hu.liu.pos, 'upgrade')         The c() function
   > neg.words = c(hu.liu.neg, 'wtf', 'wait',     combines objects
     'waiting', 'epicfail', 'mechanical')         into vectors or lists
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                22
Algorithm sanity check
    > sample = c("You're awesome and I love you",
          "I hate and hate and hate. So angry. Die!",
          "Impressed and amazed: you are peerless in your achievement of
          unparalleled mediocrity.")
    > result = score.sentiment(sample, pos.words, neg.words)
    > class(result)
                                   data.frames hold tabular data so they
    [1] "data.frame"
                                   consist of columns & rows which can
    > result$score
                                   be accessed by name or number.
    [1]   2 -5   4
                                   Here, “score” is the name of a column.


So, not so good with sarcasm. Here are a couple of real tweets:

    > score.sentiment(c("@Delta I'm going to need you to get it together.
    Delay on tarmac, delayed connection, crazy gate changes... #annoyed",
    "Surprised and happy that @Delta helped me avoid the 3.5 hr layover I
    was scheduled for. Patient and helpful agents. #remarkable"),
    pos.words, neg.words)$score
    [1] -4   5
Accessing data.frames
Here’s the data.frame just returned from score.sentiment():
   > result
       score                                                                                     text

   1           2                                                         You're awesome and I love you

   2      -5                                                   I hate and hate and hate. So angry. Die!

   3           4 Impressed and amazed: you are peerless in your achievement of unparalleled mediocrity.



Elements can be accessed by name or position, and positions can be
ranges:
   > result[1,1]
   [1] 2
   > result[1,'score']
   [1] 2
   > result[1:2, 'score']
   [1]         2 -5
   > result[c(1,3), 'score']
   [1] 2 4
   > result[,'score']
   [1]         2 -5     4
Score the tweets
To score all of the Delta tweets, just feed their text into
score.sentiment():

    > delta.scores = score.sentiment(delta.text, pos.words,     Progress bar
    neg.words, .progress='text')                                provided by
    |==================================================| 100%   plyr

Let’s add two new columns to identify the airline for when we
combine all the scores later:
    > delta.scores$airline = 'Delta'
    > delta.scores$code = 'DL’
Plot Delta’s score distribution
R’s built-in hist() function will create and plot histograms of your data:
    > hist(delta.scores$score)
The ggplot2 alternative
ggplot2 is an alternative graphics package which generates more refined
graphics:
   > qplot(delta.scores$score)
Lather. Rinse. Repeat
To see how the other airlines fare, collect & score tweets for other
airlines.


Then combine all the results into a single “all.scores” data.frame:

    > all.scores = rbind( american.scores, continental.scores, delta.scores,
    jetblue.scores, southwest.scores, united.scores, us.scores )



                                                rbind() combines
                                                rows from
                                                data.frames, arrays,
                                                and matrices
Compare score distributions
   ggplot2 implements “grammar of graphics”, building plots in layers:
       > ggplot(data=all.scores) + # ggplot works on data.frames, always
            geom_bar(mapping=aes(x=score, fill=airline), binwidth=1) +
            facet_grid(airline~.) + # make a separate plot for each airline
            theme_bw() + scale_fill_brewer() # plain display, nicer colors




ggplot2’s faceting
capability makes it
easy to generate the
same graph for
different values of a
variable, in this case
“airline”.
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                30
Ignore the middle
Let’s focus on very negative (<-2) and positive (>2) tweets:
    > all.scores$very.pos = as.numeric( all.scores$score >= 2 )
    > all.scores$very.neg = as.numeric( all.scores$score <= -2 )


For each airline ( airline + code ), let’s use the ratio of very positive to
very negative tweets as the overall sentiment score for each airline:
    > twitter.df = ddply(all.scores, c('airline', 'code'), summarise,
    pos.count = sum( very.pos ), neg.count = sum( very.neg ) )
    > twitter.df$all.count = twitter.df$pos.count + twitter.df$neg.count
    > twitter.df$score = round( 100 * twitter.df$pos.count /
                 twitter.df$all.count )

Sort with orderBy() from the doBy package:
        > orderBy(~-score, twitter.df)
Any relation to ACSI’s airline scores?




http://www.theacsi.org/index.php?option=com_content&view=article&id=147&catid=&Itemid=212&i=Airlines

                                                                                                       18
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                33
Scrape, don’t type
XML package provides amazing readHTMLtable() function:
    > library(XML)
    > acsi.url = 'http://www.theacsi.org/index.php?
    option=com_content&view=article&id=147&catid=&Itemid=212&i=Airlines'
    > acsi.df = readHTMLTable(acsi.url, header=T, which=1,
    stringsAsFactors=F)
    > # only keep column #1 (name) and #18 (2010 score)
    > acsi.df = acsi.df[,c(1,18)]
    > head(acsi.df,1)
                         10
    1 Southwest Airlines 79



Well, typing metadata is OK, I guess... clean up column names, etc:

    > colnames(acsi.df) = c('airline', 'score')              NA (as in “n/a”) is
    > acsi.df$code = c('WN', NA, 'CO', NA, 'AA', 'DL',       supported as a
                       'US', 'NW', 'UA')                     valid value
    > acsi.df$score = as.numeric(acsi.df$score)              everywhere in R.
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                35
Join and compare
merge() joins two data.frames by the specified “by=” fields. You can
specify ‘suffixes’ to rename conflicting column names:

    > compare.df = merge(twitter.df, acsi.df, by='code',
        suffixes=c('.twitter', '.acsi'))




Unless you specify “all=T”, non-matching rows are dropped (like a SQL
INNER JOIN), and that’s what happened to top scoring JetBlue.


With a very low score, and low traffic to boot, soon-to-disappear
Continental looks like an outlier. Let’s exclude:
    > compare.df = subset(compare.df, all.count > 100)
an actual result!
ggplot will even run lm() linear
(and other) regressions for you
 with its geom_smooth() layer:

> ggplot( compare.df ) +
geom_point(aes(x=score.twitter,
y=score.acsi,
color=airline.twitter), size=5) +
geom_smooth(aes(x=score.twitter,
y=score.acsi, group=1), se=F,
method="lm") +
theme_bw() +
opts(legend.position=c(0.2,
0.85))




                                         37
                                         21
http://www.despair.com/cudi.html
R code for example scoring function
    score.sentiment = function(sentences, pos.words, neg.words, .progress='none')
{
	   require(plyr)
	   require(stringr)
	
	   # we got a vector of sentences. plyr will handle a list or a vector as an "l" for us
	   # we want a simple array of scores back, so we use "l" + "a" + "ply" = laply:
	   scores = laply(sentences, function(sentence, pos.words, neg.words) {
	   	
	   	      # clean up sentences with R's regex-driven global substitute, gsub():
	   	      sentence = gsub('[[:punct:]]', '', sentence)
	   	      sentence = gsub('[[:cntrl:]]', '', sentence)
	   	      sentence = gsub('d+', '', sentence)
	   	      # and convert to lower case:
	   	      sentence = tolower(sentence)

	   	     # split into words. str_split is in the stringr package
	   	     word.list = str_split(sentence, 's+')
	   	     # sometimes a list() is one level of hierarchy too much
	   	     words = unlist(word.list)

	   	     # compare our words to the dictionaries of positive & negative terms
	   	     pos.matches = match(words, pos.words)
	   	     neg.matches = match(words, neg.words)
	
	   	     # match() returns the position of the matched term or NA
	   	     # we just want a TRUE/FALSE:
	   	     pos.matches = !is.na(pos.matches)
	   	     neg.matches = !is.na(neg.matches)

	   	     # and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum():
	   	     score = sum(pos.matches) - sum(neg.matches)

	   	      return(score)
	   }, pos.words, neg.words, .progress=.progress )

	   scores.df = data.frame(score=scores, text=sentences)
	   return(scores.df)
}                                                                                          39

More Related Content

Viewers also liked

Sentiment Analysis via R Programming
Sentiment Analysis via R ProgrammingSentiment Analysis via R Programming
Sentiment Analysis via R ProgrammingSkillspeed
 
Sentiment Analysis in R
Sentiment Analysis in RSentiment Analysis in R
Sentiment Analysis in REdureka!
 
How to Win Machine Learning Competitions ?
How to Win Machine Learning Competitions ? How to Win Machine Learning Competitions ?
How to Win Machine Learning Competitions ? HackerEarth
 
Tweets Classification using Naive Bayes and SVM
Tweets Classification using Naive Bayes and SVMTweets Classification using Naive Bayes and SVM
Tweets Classification using Naive Bayes and SVMTrilok Sharma
 
Twitter sentiment-analysis Jiit2013-14
Twitter sentiment-analysis Jiit2013-14Twitter sentiment-analysis Jiit2013-14
Twitter sentiment-analysis Jiit2013-14Rachit Goel
 
Sentiment analysis of tweets
Sentiment analysis of tweetsSentiment analysis of tweets
Sentiment analysis of tweetsVasu Jain
 
Introduction to Sentiment Analysis
Introduction to Sentiment AnalysisIntroduction to Sentiment Analysis
Introduction to Sentiment AnalysisJaganadh Gopinadhan
 
How Sentiment Analysis works
How Sentiment Analysis worksHow Sentiment Analysis works
How Sentiment Analysis worksCJ Jenkins
 
Sentiment Analysis in Twitter
Sentiment Analysis in TwitterSentiment Analysis in Twitter
Sentiment Analysis in TwitterAyushi Dalmia
 
Tutorial of Sentiment Analysis
Tutorial of Sentiment AnalysisTutorial of Sentiment Analysis
Tutorial of Sentiment AnalysisFabio Benedetti
 
Sentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSumit Raj
 
The Who What Where When And Why Of Social Media Lead Generation
The Who What Where When And Why Of Social Media Lead GenerationThe Who What Where When And Why Of Social Media Lead Generation
The Who What Where When And Why Of Social Media Lead GenerationAbhishek Shah
 
Social Media Secrets
Social Media SecretsSocial Media Secrets
Social Media SecretsGuy Kawasaki
 
Zenpayroll Pitch Deck Template
Zenpayroll Pitch Deck TemplateZenpayroll Pitch Deck Template
Zenpayroll Pitch Deck TemplateJoseph Hsieh
 
AdPushup Fundraising Deck - First Pitch
AdPushup Fundraising Deck - First PitchAdPushup Fundraising Deck - First Pitch
AdPushup Fundraising Deck - First Pitchadpushup
 
AppVirality.com - Investor Pitch Deck
AppVirality.com - Investor Pitch DeckAppVirality.com - Investor Pitch Deck
AppVirality.com - Investor Pitch DeckLaxman Papineni
 
How Wealthsimple raised $2M in 2 weeks
How Wealthsimple raised $2M in 2 weeksHow Wealthsimple raised $2M in 2 weeks
How Wealthsimple raised $2M in 2 weeksWealthsimple
 
500’s Demo Day Batch 16 >> Podozi
500’s Demo Day Batch 16 >>  Podozi500’s Demo Day Batch 16 >>  Podozi
500’s Demo Day Batch 16 >> Podozi500 Startups
 
Swipes pitch deck for Beta Pitch 2013 Finals in Berlin
Swipes pitch deck for Beta Pitch 2013 Finals in BerlinSwipes pitch deck for Beta Pitch 2013 Finals in Berlin
Swipes pitch deck for Beta Pitch 2013 Finals in BerlinSwipes App
 

Viewers also liked (20)

Sentiment Analysis via R Programming
Sentiment Analysis via R ProgrammingSentiment Analysis via R Programming
Sentiment Analysis via R Programming
 
Sentiment Analysis in R
Sentiment Analysis in RSentiment Analysis in R
Sentiment Analysis in R
 
How to Win Machine Learning Competitions ?
How to Win Machine Learning Competitions ? How to Win Machine Learning Competitions ?
How to Win Machine Learning Competitions ?
 
Tweets Classification using Naive Bayes and SVM
Tweets Classification using Naive Bayes and SVMTweets Classification using Naive Bayes and SVM
Tweets Classification using Naive Bayes and SVM
 
Twitter sentiment-analysis Jiit2013-14
Twitter sentiment-analysis Jiit2013-14Twitter sentiment-analysis Jiit2013-14
Twitter sentiment-analysis Jiit2013-14
 
Sentiment analysis of tweets
Sentiment analysis of tweetsSentiment analysis of tweets
Sentiment analysis of tweets
 
Introduction to Sentiment Analysis
Introduction to Sentiment AnalysisIntroduction to Sentiment Analysis
Introduction to Sentiment Analysis
 
How Sentiment Analysis works
How Sentiment Analysis worksHow Sentiment Analysis works
How Sentiment Analysis works
 
Sentiment Analysis in Twitter
Sentiment Analysis in TwitterSentiment Analysis in Twitter
Sentiment Analysis in Twitter
 
Tutorial of Sentiment Analysis
Tutorial of Sentiment AnalysisTutorial of Sentiment Analysis
Tutorial of Sentiment Analysis
 
Sentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSentiment Analysis of Twitter Data
Sentiment Analysis of Twitter Data
 
The Who What Where When And Why Of Social Media Lead Generation
The Who What Where When And Why Of Social Media Lead GenerationThe Who What Where When And Why Of Social Media Lead Generation
The Who What Where When And Why Of Social Media Lead Generation
 
Social Media Secrets
Social Media SecretsSocial Media Secrets
Social Media Secrets
 
SteadyBudget's Seed Funding Pitch Deck
SteadyBudget's Seed Funding Pitch DeckSteadyBudget's Seed Funding Pitch Deck
SteadyBudget's Seed Funding Pitch Deck
 
Zenpayroll Pitch Deck Template
Zenpayroll Pitch Deck TemplateZenpayroll Pitch Deck Template
Zenpayroll Pitch Deck Template
 
AdPushup Fundraising Deck - First Pitch
AdPushup Fundraising Deck - First PitchAdPushup Fundraising Deck - First Pitch
AdPushup Fundraising Deck - First Pitch
 
AppVirality.com - Investor Pitch Deck
AppVirality.com - Investor Pitch DeckAppVirality.com - Investor Pitch Deck
AppVirality.com - Investor Pitch Deck
 
How Wealthsimple raised $2M in 2 weeks
How Wealthsimple raised $2M in 2 weeksHow Wealthsimple raised $2M in 2 weeks
How Wealthsimple raised $2M in 2 weeks
 
500’s Demo Day Batch 16 >> Podozi
500’s Demo Day Batch 16 >>  Podozi500’s Demo Day Batch 16 >>  Podozi
500’s Demo Day Batch 16 >> Podozi
 
Swipes pitch deck for Beta Pitch 2013 Finals in Berlin
Swipes pitch deck for Beta Pitch 2013 Finals in BerlinSwipes pitch deck for Beta Pitch 2013 Finals in Berlin
Swipes pitch deck for Beta Pitch 2013 Finals in Berlin
 

Similar to Here are a few ways to access elements of the data.frame returned by score.sentiment():1. Select a column by name:result$score2. Select a row by index: result1, 3. Select a specific element:result1,14. Select multiple rows/columns:resultc(1,3), c("score","text")5. Use logical conditions to select rows:resultresult$score < 0,So in summary, data.frames allow flexible access to elements by name, index, or condition for both rows and columns. The $ operator selects columns, while

Sentiment analysis on airlines
Sentiment analysis on airlinesSentiment analysis on airlines
Sentiment analysis on airlinesPiyush Srivastava
 
Final Presentation
Final PresentationFinal Presentation
Final PresentationLove Tyagi
 
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...Amazon Web Services
 
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...PAPIs.io
 
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docxDashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docxsimonithomas47935
 
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...AWS Germany
 
Exploring Equilibrium Login to the computer and open a web browser.docx
Exploring Equilibrium Login to the computer and open a web browser.docxExploring Equilibrium Login to the computer and open a web browser.docx
Exploring Equilibrium Login to the computer and open a web browser.docxssuser454af01
 
(BDT302) Real-World Smart Applications With Amazon Machine Learning
(BDT302) Real-World Smart Applications With Amazon Machine Learning(BDT302) Real-World Smart Applications With Amazon Machine Learning
(BDT302) Real-World Smart Applications With Amazon Machine LearningAmazon Web Services
 
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter AnalysisIBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter AnalysisTorsten Steinbach
 
Building an Autonomous Data Layer
Building an Autonomous Data LayerBuilding an Autonomous Data Layer
Building an Autonomous Data LayerMartyPitt1
 
Flutter Festivals IIT Goa Session 2
Flutter Festivals IIT Goa Session 2Flutter Festivals IIT Goa Session 2
Flutter Festivals IIT Goa Session 2SEJALGUPTA44
 
IST365 - Project Deliverable #3Create the corresponding relation.docx
IST365 - Project Deliverable #3Create the corresponding relation.docxIST365 - Project Deliverable #3Create the corresponding relation.docx
IST365 - Project Deliverable #3Create the corresponding relation.docxpriestmanmable
 
"R & Text Analytics" (15 January 2013)
"R & Text Analytics" (15 January 2013)"R & Text Analytics" (15 January 2013)
"R & Text Analytics" (15 January 2013)Portland R User Group
 
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI servicesAWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI servicesAmazon Web Services
 
Command Query Responsibility Segregation at Enterprise Scale
Command Query Responsibility Segregation at Enterprise ScaleCommand Query Responsibility Segregation at Enterprise Scale
Command Query Responsibility Segregation at Enterprise ScaleTechWell
 
Databaseconcepts
DatabaseconceptsDatabaseconcepts
Databaseconceptsdilipkkr
 

Similar to Here are a few ways to access elements of the data.frame returned by score.sentiment():1. Select a column by name:result$score2. Select a row by index: result1, 3. Select a specific element:result1,14. Select multiple rows/columns:resultc(1,3), c("score","text")5. Use logical conditions to select rows:resultresult$score < 0,So in summary, data.frames allow flexible access to elements by name, index, or condition for both rows and columns. The $ operator selects columns, while (20)

Sentiment analysis on airlines
Sentiment analysis on airlinesSentiment analysis on airlines
Sentiment analysis on airlines
 
Final Presentation
Final PresentationFinal Presentation
Final Presentation
 
The Value of Twitter
The Value of TwitterThe Value of Twitter
The Value of Twitter
 
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
 
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
 
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docxDashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
 
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
 
Exploring Equilibrium Login to the computer and open a web browser.docx
Exploring Equilibrium Login to the computer and open a web browser.docxExploring Equilibrium Login to the computer and open a web browser.docx
Exploring Equilibrium Login to the computer and open a web browser.docx
 
(BDT302) Real-World Smart Applications With Amazon Machine Learning
(BDT302) Real-World Smart Applications With Amazon Machine Learning(BDT302) Real-World Smart Applications With Amazon Machine Learning
(BDT302) Real-World Smart Applications With Amazon Machine Learning
 
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter AnalysisIBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
 
Building an Autonomous Data Layer
Building an Autonomous Data LayerBuilding an Autonomous Data Layer
Building an Autonomous Data Layer
 
Flutter Festivals IIT Goa Session 2
Flutter Festivals IIT Goa Session 2Flutter Festivals IIT Goa Session 2
Flutter Festivals IIT Goa Session 2
 
REST APIs, Girls Who Code
REST APIs, Girls Who CodeREST APIs, Girls Who Code
REST APIs, Girls Who Code
 
IST365 - Project Deliverable #3Create the corresponding relation.docx
IST365 - Project Deliverable #3Create the corresponding relation.docxIST365 - Project Deliverable #3Create the corresponding relation.docx
IST365 - Project Deliverable #3Create the corresponding relation.docx
 
"R & Text Analytics" (15 January 2013)
"R & Text Analytics" (15 January 2013)"R & Text Analytics" (15 January 2013)
"R & Text Analytics" (15 January 2013)
 
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI servicesAWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
 
Command Query Responsibility Segregation at Enterprise Scale
Command Query Responsibility Segregation at Enterprise ScaleCommand Query Responsibility Segregation at Enterprise Scale
Command Query Responsibility Segregation at Enterprise Scale
 
Databaseconcepts
DatabaseconceptsDatabaseconcepts
Databaseconcepts
 
Blind sql injection
Blind sql injectionBlind sql injection
Blind sql injection
 
Blind sql injection
Blind sql injectionBlind sql injection
Blind sql injection
 

More from Jeffrey Breen

Tapping the Data Deluge with R
Tapping the Data Deluge with RTapping the Data Deluge with R
Tapping the Data Deluge with RJeffrey Breen
 
Getting started with R & Hadoop
Getting started with R & HadoopGetting started with R & Hadoop
Getting started with R & HadoopJeffrey Breen
 
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)Jeffrey Breen
 
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....Jeffrey Breen
 
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2Jeffrey Breen
 
Big Data Step-by-Step: Infrastructure 1/3: Local VM
Big Data Step-by-Step: Infrastructure 1/3: Local VMBig Data Step-by-Step: Infrastructure 1/3: Local VM
Big Data Step-by-Step: Infrastructure 1/3: Local VMJeffrey Breen
 
Move your data (Hans Rosling style) with googleVis + 1 line of R code
Move your data (Hans Rosling style) with googleVis + 1 line of R codeMove your data (Hans Rosling style) with googleVis + 1 line of R code
Move your data (Hans Rosling style) with googleVis + 1 line of R codeJeffrey Breen
 
Accessing Databases from R
Accessing Databases from RAccessing Databases from R
Accessing Databases from RJeffrey Breen
 
Grouping & Summarizing Data in R
Grouping & Summarizing Data in RGrouping & Summarizing Data in R
Grouping & Summarizing Data in RJeffrey Breen
 
R + 15 minutes = Hadoop cluster
R + 15 minutes = Hadoop clusterR + 15 minutes = Hadoop cluster
R + 15 minutes = Hadoop clusterJeffrey Breen
 
FAA Aviation Forecasts 2011-2031 overview
FAA Aviation Forecasts 2011-2031 overviewFAA Aviation Forecasts 2011-2031 overview
FAA Aviation Forecasts 2011-2031 overviewJeffrey Breen
 

More from Jeffrey Breen (12)

Tapping the Data Deluge with R
Tapping the Data Deluge with RTapping the Data Deluge with R
Tapping the Data Deluge with R
 
Getting started with R & Hadoop
Getting started with R & HadoopGetting started with R & Hadoop
Getting started with R & Hadoop
 
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
 
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
 
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
 
Big Data Step-by-Step: Infrastructure 1/3: Local VM
Big Data Step-by-Step: Infrastructure 1/3: Local VMBig Data Step-by-Step: Infrastructure 1/3: Local VM
Big Data Step-by-Step: Infrastructure 1/3: Local VM
 
Move your data (Hans Rosling style) with googleVis + 1 line of R code
Move your data (Hans Rosling style) with googleVis + 1 line of R codeMove your data (Hans Rosling style) with googleVis + 1 line of R code
Move your data (Hans Rosling style) with googleVis + 1 line of R code
 
Accessing Databases from R
Accessing Databases from RAccessing Databases from R
Accessing Databases from R
 
Reshaping Data in R
Reshaping Data in RReshaping Data in R
Reshaping Data in R
 
Grouping & Summarizing Data in R
Grouping & Summarizing Data in RGrouping & Summarizing Data in R
Grouping & Summarizing Data in R
 
R + 15 minutes = Hadoop cluster
R + 15 minutes = Hadoop clusterR + 15 minutes = Hadoop cluster
R + 15 minutes = Hadoop cluster
 
FAA Aviation Forecasts 2011-2031 overview
FAA Aviation Forecasts 2011-2031 overviewFAA Aviation Forecasts 2011-2031 overview
FAA Aviation Forecasts 2011-2031 overview
 

Recently uploaded

Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialJoão Esperancinha
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
Français Patch Tuesday - Avril
Français Patch Tuesday - AvrilFrançais Patch Tuesday - Avril
Français Patch Tuesday - AvrilIvanti
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 

Recently uploaded (20)

Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorial
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
How Tech Giants Cut Corners to Harvest Data for A.I.
How Tech Giants Cut Corners to Harvest Data for A.I.How Tech Giants Cut Corners to Harvest Data for A.I.
How Tech Giants Cut Corners to Harvest Data for A.I.
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
Français Patch Tuesday - Avril
Français Patch Tuesday - AvrilFrançais Patch Tuesday - Avril
Français Patch Tuesday - Avril
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 

Here are a few ways to access elements of the data.frame returned by score.sentiment():1. Select a column by name:result$score2. Select a row by index: result1, 3. Select a specific element:result1,14. Select multiple rows/columns:resultc(1,3), c("score","text")5. Use logical conditions to select rows:resultresult$score < 0,So in summary, data.frames allow flexible access to elements by name, index, or condition for both rows and columns. The $ operator selects columns, while

  • 1. R by example: mining Twitter for consumer attitudes towards airlines presented at the Boston Predictive Analytics MeetUp by Jeffrey Breen President Cambridge Aviation Research jbreen@cambridge.aero June 2011 Cambridge Aviation Research • 245 First Street • Suite 1800 • Cambridge, MA 02142 • cambridge.aero © Copyright 2010 by Cambridge Aviation Research. All rights reserved.
  • 2. Airlines top customer satisfaction... alphabetically http://www.theacsi.org/ 3
  • 3. Actually, they rank below the Post Office and health insurers 4
  • 4. which gives us plenty to listen to Completely unimpressed with @continental or @united. RT @dave_mcgregor: Poor communication, goofy reservations systems and Publicly pledging to all to turn my trip into a mess. never fly @delta again. The worst airline ever. U have lost my patronage @united #fail on wifi in red carpet clubs (too forever due to ur slow), delayed flight, customer service in red incompetence carpet club (too slow), hmmm do u see a trend? @United Weather delays may not be your fault, but you are in the customer service business. It's atrocious how people are getting treated! We were just told we are delayed 1.5 @SouthwestAir I know you don't make the hrs & next announcement on @JetBlue - weather. But at least pretend I am not a “We're selling headsets.” Way to bother when I ask if the delay will make capitalize on our misfortune. miss my connection @SouthwestAir I hate you with every Hey @delta - you suck! Your prices single bone in my body are over the moon & to move a flight for delaying my flight by a cpl of days is $150.00. Insane. I 3 hours, 30mins before I hate you! U ruined my vacation! was supposed to board. #hate
  • 5. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 14
  • 6. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 15
  • 7. Searching Twitter in one line R’s XML and RCurl packages make it easy to grab web data, but Jeff Gentry’s twitteR package makes searching Twitter almost too easy: > # load the package > library(twitteR) > # get the 1,500 most recent tweets mentioning ‘@delta’: > delta.tweets = searchTwitter('@delta', n=1500) See what we got in return: A “list” in R is a collection of objects and its elements may be > length(delta.tweets) named or just numbered. [1] 1500 > class(delta.tweets) [1] "list" “[[ ]]” is used to access elements.
  • 8. Examine the output Let’s take a look at the first tweet in the output list: > tweet = delta.tweets[[1]] tweet is an object of type “status” from the “twitteR” package. > class(tweet) [1] "status" attr(,"package") It holds all the information about [1] "twitteR" the tweet returned from Twitter. The help page (“?status”) describes some accessor methods like getScreenName() and getText() which do what you would expect: > tweet$getScreenName() [1] "Alaqawari" > tweet$getText() [1] "I am ready to head home. Inshallah will try to get on the earlier flight to Fresno. @Delta @DeltaAssist"
  • 9. Extract the tweet text R has several (read: too many) ways to apply functions iteratively. •The plyr package unifies them all with a consistent naming convention. •The function name is determined by the input and output data types. We have a list and would like a simple array output, so we use “laply”: > delta.text = laply(delta.tweets, function(t) t$getText() ) > length(delta.text)[1] 1500 > head(delta.text, 5) [1] "I am ready to head home. Inshallah will try to get on the earlier flight to Fresno. @Delta @DeltaAssist" [2] "@Delta Releases 2010 Corporate Responsibility Report - @PRNewswire (press release) : http://tinyurl.com/64mz3oh" [3] "Another week, another upgrade! Thanks @Delta!" [4] "I'm not able to check in or select a seat for flight DL223/KL6023 to Seattle tomorrow. Help? @KLM @delta" [5] "In my boredom of waiting realized @deltaairlines is now @delta seriously..... Stil waiting and your not even unloading status yet"
  • 10. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 19
  • 11. Estimating Sentiment There are many good papers and resources describing methods to estimate sentiment. These are very complex algorithms. For this tutorial, we use a very simple algorithm which assigns a score by simply counting the number of occurrences of “positive” and “negative” words in a tweet. The code for our score.sentiment() function can be found at the end of this deck. Hu & Liu have published an “opinion lexicon” which categorizes approximately 6,800 words as positive or negative and which can be downloaded. Positive: love, best, cool, great, good, amazing Negative: hate, worst, sucks, awful, nightmare 20
  • 12. Load sentiment word lists 1. Download Hu & Liu’s opinion lexicon: http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html 2. Loading data is one of R’s strengths. These are simple text files, though they use “;” as a comment character at the beginning: > hu.liu.pos = scan('../data/opinion-lexicon-English/positive- words.txt', what='character', comment.char=';') > hu.liu.neg = scan('../data/opinion-lexicon-English/negative- words.txt', what='character', comment.char=';') 3. Add a few industry-specific and/or especially emphatic terms: > pos.words = c(hu.liu.pos, 'upgrade') The c() function > neg.words = c(hu.liu.neg, 'wtf', 'wait', combines objects 'waiting', 'epicfail', 'mechanical') into vectors or lists
  • 13. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 22
  • 14. Algorithm sanity check > sample = c("You're awesome and I love you", "I hate and hate and hate. So angry. Die!", "Impressed and amazed: you are peerless in your achievement of unparalleled mediocrity.") > result = score.sentiment(sample, pos.words, neg.words) > class(result) data.frames hold tabular data so they [1] "data.frame" consist of columns & rows which can > result$score be accessed by name or number. [1] 2 -5 4 Here, “score” is the name of a column. So, not so good with sarcasm. Here are a couple of real tweets: > score.sentiment(c("@Delta I'm going to need you to get it together. Delay on tarmac, delayed connection, crazy gate changes... #annoyed", "Surprised and happy that @Delta helped me avoid the 3.5 hr layover I was scheduled for. Patient and helpful agents. #remarkable"), pos.words, neg.words)$score [1] -4 5
  • 15. Accessing data.frames Here’s the data.frame just returned from score.sentiment(): > result score text 1 2 You're awesome and I love you 2 -5 I hate and hate and hate. So angry. Die! 3 4 Impressed and amazed: you are peerless in your achievement of unparalleled mediocrity. Elements can be accessed by name or position, and positions can be ranges: > result[1,1] [1] 2 > result[1,'score'] [1] 2 > result[1:2, 'score'] [1] 2 -5 > result[c(1,3), 'score'] [1] 2 4 > result[,'score'] [1] 2 -5 4
  • 16. Score the tweets To score all of the Delta tweets, just feed their text into score.sentiment(): > delta.scores = score.sentiment(delta.text, pos.words, Progress bar neg.words, .progress='text') provided by |==================================================| 100% plyr Let’s add two new columns to identify the airline for when we combine all the scores later: > delta.scores$airline = 'Delta' > delta.scores$code = 'DL’
  • 17. Plot Delta’s score distribution R’s built-in hist() function will create and plot histograms of your data: > hist(delta.scores$score)
  • 18. The ggplot2 alternative ggplot2 is an alternative graphics package which generates more refined graphics: > qplot(delta.scores$score)
  • 19. Lather. Rinse. Repeat To see how the other airlines fare, collect & score tweets for other airlines. Then combine all the results into a single “all.scores” data.frame: > all.scores = rbind( american.scores, continental.scores, delta.scores, jetblue.scores, southwest.scores, united.scores, us.scores ) rbind() combines rows from data.frames, arrays, and matrices
  • 20. Compare score distributions ggplot2 implements “grammar of graphics”, building plots in layers: > ggplot(data=all.scores) + # ggplot works on data.frames, always geom_bar(mapping=aes(x=score, fill=airline), binwidth=1) + facet_grid(airline~.) + # make a separate plot for each airline theme_bw() + scale_fill_brewer() # plain display, nicer colors ggplot2’s faceting capability makes it easy to generate the same graph for different values of a variable, in this case “airline”.
  • 21. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 30
  • 22. Ignore the middle Let’s focus on very negative (<-2) and positive (>2) tweets: > all.scores$very.pos = as.numeric( all.scores$score >= 2 ) > all.scores$very.neg = as.numeric( all.scores$score <= -2 ) For each airline ( airline + code ), let’s use the ratio of very positive to very negative tweets as the overall sentiment score for each airline: > twitter.df = ddply(all.scores, c('airline', 'code'), summarise, pos.count = sum( very.pos ), neg.count = sum( very.neg ) ) > twitter.df$all.count = twitter.df$pos.count + twitter.df$neg.count > twitter.df$score = round( 100 * twitter.df$pos.count / twitter.df$all.count ) Sort with orderBy() from the doBy package: > orderBy(~-score, twitter.df)
  • 23. Any relation to ACSI’s airline scores? http://www.theacsi.org/index.php?option=com_content&view=article&id=147&catid=&Itemid=212&i=Airlines 18
  • 24. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 33
  • 25. Scrape, don’t type XML package provides amazing readHTMLtable() function: > library(XML) > acsi.url = 'http://www.theacsi.org/index.php? option=com_content&view=article&id=147&catid=&Itemid=212&i=Airlines' > acsi.df = readHTMLTable(acsi.url, header=T, which=1, stringsAsFactors=F) > # only keep column #1 (name) and #18 (2010 score) > acsi.df = acsi.df[,c(1,18)] > head(acsi.df,1) 10 1 Southwest Airlines 79 Well, typing metadata is OK, I guess... clean up column names, etc: > colnames(acsi.df) = c('airline', 'score') NA (as in “n/a”) is > acsi.df$code = c('WN', NA, 'CO', NA, 'AA', 'DL', supported as a 'US', 'NW', 'UA') valid value > acsi.df$score = as.numeric(acsi.df$score) everywhere in R.
  • 26. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 35
  • 27. Join and compare merge() joins two data.frames by the specified “by=” fields. You can specify ‘suffixes’ to rename conflicting column names: > compare.df = merge(twitter.df, acsi.df, by='code', suffixes=c('.twitter', '.acsi')) Unless you specify “all=T”, non-matching rows are dropped (like a SQL INNER JOIN), and that’s what happened to top scoring JetBlue. With a very low score, and low traffic to boot, soon-to-disappear Continental looks like an outlier. Let’s exclude: > compare.df = subset(compare.df, all.count > 100)
  • 28. an actual result! ggplot will even run lm() linear (and other) regressions for you with its geom_smooth() layer: > ggplot( compare.df ) + geom_point(aes(x=score.twitter, y=score.acsi, color=airline.twitter), size=5) + geom_smooth(aes(x=score.twitter, y=score.acsi, group=1), se=F, method="lm") + theme_bw() + opts(legend.position=c(0.2, 0.85)) 37 21
  • 30. R code for example scoring function score.sentiment = function(sentences, pos.words, neg.words, .progress='none') { require(plyr) require(stringr) # we got a vector of sentences. plyr will handle a list or a vector as an "l" for us # we want a simple array of scores back, so we use "l" + "a" + "ply" = laply: scores = laply(sentences, function(sentence, pos.words, neg.words) { # clean up sentences with R's regex-driven global substitute, gsub(): sentence = gsub('[[:punct:]]', '', sentence) sentence = gsub('[[:cntrl:]]', '', sentence) sentence = gsub('d+', '', sentence) # and convert to lower case: sentence = tolower(sentence) # split into words. str_split is in the stringr package word.list = str_split(sentence, 's+') # sometimes a list() is one level of hierarchy too much words = unlist(word.list) # compare our words to the dictionaries of positive & negative terms pos.matches = match(words, pos.words) neg.matches = match(words, neg.words) # match() returns the position of the matched term or NA # we just want a TRUE/FALSE: pos.matches = !is.na(pos.matches) neg.matches = !is.na(neg.matches) # and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum(): score = sum(pos.matches) - sum(neg.matches) return(score) }, pos.words, neg.words, .progress=.progress ) scores.df = data.frame(score=scores, text=sentences) return(scores.df) } 39

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. delta.tweets[1] \n\n&quot;0533370677: RT @g: To the @Delta attendant at gate D39 in ATL: your attitude isn&apos;t the best, but you did get me on the earlier flight. Thank you. #ncslatl&quot;\n\n&amp;#x2018;original tweet has two parts: screenname and the text; we need to separate out the text&amp;#x201D;\n
  8. &amp;#x2018;This is one tweet, next will batch process the set&amp;#x2019;\n
  9. &amp;#x2018;can see here the text has been extracted&amp;#x2019;\n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n