Trend Detection and Analysis on Twitter Leveraging Twitter's Streaming API to collect over 80 million tweets and extracting trends. Project for first project paper during master studies. Together with Lukas Masuch and Benjamin Raethlein

1. Trend Detection and Analysis
on Twitter
Benjamin Räthlein
Henning Muszynski
Lukas Masuch

2. 2
Agenda
Motivation
Architecture
Data Preparation
Trend Analysis
Analyzed Trends
Conclusion

3. 3
Motivation
Predict the stock market in real time
source
source
Detecting influenza epidemics
Automatic crime prediction
source
“Successful results of mainly research-based projects
helped to open up new business opportunities”

4. 4
Twitter

5. 5
Early Trend Detector
Bag-of-words (Hashtags, Mentions)
Twitter Streaming API (Twython)
Architecture
Bag of Words
Bags Count
#newyear 7
#christmas 6
@bigdata 2
@sap 3

6. 6
Statistical MeasurementEarly Trend Detector
Bag-of-words (Hashtags, Mentions)
Twitter Streaming API (Twython)
Architecture
Statistical Measurement
(growth, average usage, retweets, participating users…)
Report statistics (every 20 minutes):
• Total hashtags & user mentions
• Hashtag/mentions count
• Usage growth per hashtag/mention
• Participating users per hashtag/mention
• Retweet count per hashtag/mention

7. 7
Early Trend Detector
Bag-of-words (Hashtags, Mentions)
Twitter Streaming API (Twython)
Architecture
Statistical Measurement
(growth, average usage, retweets, participating users…)
Anomaly Detection
Time Series Analysis
Calculated for every hashtag / user mention
Every 2 / 4 hours based on reports
Anomaly detection using:
• Relative & absolute fluctuation
• Total occurrences (sum)
• Minimum occurrences
• Maximum occurrences
• Average occurrences
Time Series Analysis

8. 8
Twitter Streaming API (Twython)
Architecture
Trend Analyzer
Text Preprocessing (Python NLTK)
Lowercasing & tokenizing
URL & stopword removal
Stop Word Removal
This sample text shows which words will
be removed when applying stop word
removal. Mostly words like the, a or and.
This sample text shows which words will
be removed when applying stop word
removal. Mostly words like the, a or and.

9. 9
Twitter Streaming API (Twython)
Architecture
Trend Analyzer
Text Preprocessing (Python NLTK)
URL & stopword removal
Lowercasing & tokenizing
Word stemming
Stemming
Amazing
Amazement
Amazed
amaze

10. 10
Twitter Streaming API (Twython)
Architecture
Trend Analyzer
Text Preprocessing (Python NLTK)
URL & stopword removal
Lowercasing & tokenizing
Word stemming
Sentiment Analysis
Sentiment Analysis
I love cookiesI hate cookies

11. 11
Twitter Streaming API (Twython)
Architecture
Trend Analyzer
Text Preprocessing (Python NLTK)
URL & stopword removal
Lowercasing & tokenizing
Word stemming
Sentiment Analysis
Topic Modeling (LDA)
Topic Modeling
Topics
• …
• …
• …
Trend Classification

12. 14
Trend Analyzer
Text Preprocessing (Python NLTK)
URL & stopword removal
Lowercasing & tokenizing
Word stemming
Sentiment Analysis
Topic Modeling (LDA)
Wordcloud Visualization
Wordfreq.js
Wordcloud2.js
GeoSpatial Visualization
CartoDB
Early Trend Detector
Bag-of-words (Hashtags, Mentions)
Anomaly Detection
Statistical Measurement
(growth, average usage, retweets, participating users…)
Time Series Analysis
Trend Classification
Twitter Streaming API (Twython)
Architecture

13. 15
Analyzed Trends

14. 16
Limitations
Tweets collected: 38 million (70GB)
Only English tweets from the USA
Twitter Streaming API

15. 17
New Year
Time Series

16. 18
New Year
Word Cloud

17. 19
New Year
Geospatial Analysis
Midnight Los Angeles Midnight New York

18. 20
New Year
Sentiment Analysis
Positive Neutral Negative
Home sick on #nye. Horrible timing
stupid cold. Ugh. My date is my
couch & pillow watching.
#HappyNewYear everyone.
#HappyNewYear from the Youth for
Astronomy and Engineering Program
at Space Telescope Science Institute!
Happy New Year! Last year was
amazing, and here’s to another great
year of love & happiness! #NYE2015

19. 21
Air Asia Tragedy

20. 22
Air Asia Tragedy
Time Series

21. 23
Air Asia Tragedy
Word Cloud

22. 24
Air Asia Tragedy
Topic Modeling
News
airasia, missing, flight, air,
Indonesia, singapore, asia
Search for the Plane
airasia, missing, plane, find,
plane, world, technology
Sympathy
Prayers, families, thoughts,
airasia, crash, thought, airfrance
Cause
airasia, weather, flight,
pilots, fly, bad, path
International Help
raaf, butterworth, china, australia,
Russia, trndnl, trending

23. 25
Air Asia Tragedy
Sentiment Analysis
Neutral Negative Positive
Prayers are USELESS! Stop repeating
meaningless crap, pretending that
you care … #PrayForAirAsia #QZ8501
#GrowABrain #ReligousNonsense
#BREAKING #AirAsia Flight #8501
likely “at the bottom of the sea”
rescue officials says.
May God’s great love shine on the
families and loved ones of all
passengers and crew #AirAsia #8501

24. 26
Air Asia Tragedy
Google Trends Comparison
Google Trends Twitter Sample

25. 27
Air Asia Tragedy
Google Trends Comparison
Google Trends Twitter Sample

26. 28
Sony Hack

27. 29
Sony Hack
Time Series

28. 30
Sony Hack
Word Cloud

29. 31
Sony Hack
Topic Modeling
Christmas Release
theinterview, christmas, day,
theaters, freedom, theater, showing
Reviews
theinterview, jamesfrancotv, sethrogen,
movie, interview, funny, hilarious
Suspicions
northkorea, sonyhack, korea,
north, internet, sony, amp
News
theinterview, sonypictures, sony,
movie, korea, north, interview
Insider Joke
theinterview, aint, hate, cuz,
jealous, anus, peanutbutter

30. 32
Sony Hack
Geospatial Analysis

31. 33
Sony Hack
Sentiment Analysis
Neutral Negative Positive
#TheInterview SUCKS!!! @sethrogen
Like I knew it would #Stupid
#NotFunny
#Sony says #TheInterview made
more than $1 million at the box office
on in 1 single day on Dec. 25.
Happy I joined my fellow Americans
in the great #TheInterview Christmas
Day Viewing. Plus it was pretty funny,
truth be told.

32. 34
Network Outage

33. 35
Network Outage
Time Series

34. 36
Network Outage
Word Cloud

35. 37
Network Outage
Topic Modeling
Network Error
xbox, psn, sign, connect,
live, error, account, issues
Connection between Hacks
xbox, playstation, watch, movie,
fuckcrucifix, north, korea, interview
Xbox Down
xbox, christmas, play, xboxlivedown,
live, xboxlive, xboxsupport, day
Caused Damage
playstation, dollar, psn, company,
lizardsquad, sony, billion, multi
Hacker Group
fuckcrucifix, lizardmafia, lizardsquad,
fuck,lizard, squad, finestsquad, stop
Restored
psn, back, playstation, online,
askplaystation, network, psndown, working

36. 38
Network Outage
Sentiment Analysis
Neutral Negative Positive
@XboxSupport f*** your servers, a
big ass company like you should
handle these teenage kids, terrible
@AskPlayStation when will the
service be back online because it says
there’s maintenance?
@PlayStation thanks for the great
year. I am sure this new year will be
amazing. Don’t allow yourselves to
be hacked ever again.

37. 39
Conclusion
High quality insights into world’s interest
Twitter is very good for detecting and predicting trends
Maintaining a high data quality is important

38. 40
#Questions
Benjamin Räthlein
@B3nRa
Henning Muszynski
@henningmus
Lukas Masuch
@LukasMasuch