Download as PDF, PPTX
Uploaded byBenjamin Raethlein
Trend Detection and Analysis on Twitter
The document describes a trend detection and analysis system built using Twitter data. It outlines the architecture which includes collecting tweets using the Twitter API, performing natural language processing, and analyzing trends over time using metrics like sentiment, topics, locations and growth rates. It then provides examples of trends analyzed including New Year's, an airliner crash, a movie hack and a gaming network outage. It finds Twitter is good for trend detection and predicting interests but maintaining high data quality is important.
More Related Content
Similar to Trend Detection and Analysis on Twitter
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Trend Detection and Analysis on Twitter
- 1. Trend Detection andAnalysis on Twitter Benjamin Räthlein Henning Muszynski Lukas Masuch
- 2.
- 3. 3 Motivation Predict the stockmarket in real time source source Detecting influenza epidemics Automatic crime prediction source “Successful results of mainly research-based projects helped to open up new business opportunities”
- 4.
- 5. 5 Early Trend Detector Bag-of-words(Hashtags, Mentions) Twitter Streaming API (Twython) Architecture Bag of Words Bags Count #newyear 7 #christmas 6 @bigdata 2 @sap 3
- 6. 6 Statistical MeasurementEarly TrendDetector Bag-of-words (Hashtags, Mentions) Twitter Streaming API (Twython) Architecture Statistical Measurement (growth, average usage, retweets, participating users…) Report statistics (every 20 minutes): • Total hashtags & user mentions • Hashtag/mentions count • Usage growth per hashtag/mention • Participating users per hashtag/mention • Retweet count per hashtag/mention
- 7. 7 Early Trend Detector Bag-of-words(Hashtags, Mentions) Twitter Streaming API (Twython) Architecture Statistical Measurement (growth, average usage, retweets, participating users…) Anomaly Detection Time Series Analysis Calculated for every hashtag / user mention Every 2 / 4 hours based on reports Anomaly detection using: • Relative & absolute fluctuation • Total occurrences (sum) • Minimum occurrences • Maximum occurrences • Average occurrences Time Series Analysis
- 8. 8 Twitter Streaming API(Twython) Architecture Trend Analyzer Text Preprocessing (Python NLTK) Lowercasing & tokenizing URL & stopword removal Stop Word Removal This sample text shows which words will be removed when applying stop word removal. Mostly words like the, a or and. This sample text shows which words will be removed when applying stop word removal. Mostly words like the, a or and.
- 9. 9 Twitter Streaming API(Twython) Architecture Trend Analyzer Text Preprocessing (Python NLTK) URL & stopword removal Lowercasing & tokenizing Word stemming Stemming Amazing Amazement Amazed amaze
- 10. 10 Twitter Streaming API(Twython) Architecture Trend Analyzer Text Preprocessing (Python NLTK) URL & stopword removal Lowercasing & tokenizing Word stemming Sentiment Analysis Sentiment Analysis I love cookiesI hate cookies
- 11. 11 Twitter Streaming API(Twython) Architecture Trend Analyzer Text Preprocessing (Python NLTK) URL & stopword removal Lowercasing & tokenizing Word stemming Sentiment Analysis Topic Modeling (LDA) Topic Modeling Topics • … • … • … Trend Classification
- 12. 14 Trend Analyzer Text Preprocessing(Python NLTK) URL & stopword removal Lowercasing & tokenizing Word stemming Sentiment Analysis Topic Modeling (LDA) Wordcloud Visualization Wordfreq.js Wordcloud2.js GeoSpatial Visualization CartoDB Early Trend Detector Bag-of-words (Hashtags, Mentions) Anomaly Detection Statistical Measurement (growth, average usage, retweets, participating users…) Time Series Analysis Trend Classification Twitter Streaming API (Twython) Architecture
- 13.
- 14. 16 Limitations Tweets collected: 38million (70GB) Only English tweets from the USA Twitter Streaming API
- 15.
- 16.
- 17. 19 New Year Geospatial Analysis MidnightLos Angeles Midnight New York
- 18. 20 New Year Sentiment Analysis PositiveNeutral Negative Home sick on #nye. Horrible timing stupid cold. Ugh. My date is my couch & pillow watching. #HappyNewYear everyone. #HappyNewYear from the Youth for Astronomy and Engineering Program at Space Telescope Science Institute! Happy New Year! Last year was amazing, and here’s to another great year of love & happiness! #NYE2015
- 19.
- 20. 22 Air Asia Tragedy TimeSeries
- 21.
- 22. 24 Air Asia Tragedy TopicModeling News airasia, missing, flight, air, Indonesia, singapore, asia Search for the Plane airasia, missing, plane, find, plane, world, technology Sympathy Prayers, families, thoughts, airasia, crash, thought, airfrance Cause airasia, weather, flight, pilots, fly, bad, path International Help raaf, butterworth, china, australia, Russia, trndnl, trending
- 23. 25 Air Asia Tragedy SentimentAnalysis Neutral Negative Positive Prayers are USELESS! Stop repeating meaningless crap, pretending that you care … #PrayForAirAsia #QZ8501 #GrowABrain #ReligousNonsense #BREAKING #AirAsia Flight #8501 likely “at the bottom of the sea” rescue officials says. May God’s great love shine on the families and loved ones of all passengers and crew #AirAsia #8501
- 24. 26 Air Asia Tragedy GoogleTrends Comparison Google Trends Twitter Sample
- 25. 27 Air Asia Tragedy GoogleTrends Comparison Google Trends Twitter Sample
- 26.
- 27.
- 28.
- 29. 31 Sony Hack Topic Modeling ChristmasRelease theinterview, christmas, day, theaters, freedom, theater, showing Reviews theinterview, jamesfrancotv, sethrogen, movie, interview, funny, hilarious Suspicions northkorea, sonyhack, korea, north, internet, sony, amp News theinterview, sonypictures, sony, movie, korea, north, interview Insider Joke theinterview, aint, hate, cuz, jealous, anus, peanutbutter
- 30.
- 31. 33 Sony Hack Sentiment Analysis NeutralNegative Positive #TheInterview SUCKS!!! @sethrogen Like I knew it would #Stupid #NotFunny #Sony says #TheInterview made more than $1 million at the box office on in 1 single day on Dec. 25. Happy I joined my fellow Americans in the great #TheInterview Christmas Day Viewing. Plus it was pretty funny, truth be told.
- 32.
- 33.
- 34.
- 35. 37 Network Outage Topic Modeling NetworkError xbox, psn, sign, connect, live, error, account, issues Connection between Hacks xbox, playstation, watch, movie, fuckcrucifix, north, korea, interview Xbox Down xbox, christmas, play, xboxlivedown, live, xboxlive, xboxsupport, day Caused Damage playstation, dollar, psn, company, lizardsquad, sony, billion, multi Hacker Group fuckcrucifix, lizardmafia, lizardsquad, fuck,lizard, squad, finestsquad, stop Restored psn, back, playstation, online, askplaystation, network, psndown, working
- 36. 38 Network Outage Sentiment Analysis NeutralNegative Positive @XboxSupport f*** your servers, a big ass company like you should handle these teenage kids, terrible @AskPlayStation when will the service be back online because it says there’s maintenance? @PlayStation thanks for the great year. I am sure this new year will be amazing. Don’t allow yourselves to be hacked ever again.
- 37. 39 Conclusion High quality insightsinto world’s interest Twitter is very good for detecting and predicting trends Maintaining a high data quality is important
- 38. 40 #Questions Benjamin Räthlein @B3nRa Henning Muszynski @henningmus LukasMasuch @LukasMasuch