Present an overview of Twitter APIs that can be used for Data Mining and Data Analytics purposes.

1. Gathering Data from Twitter
Marcello Tomasini – mtomasini@my.fit.edu
Marcos Oliveira – moliveirajun2013@my.fit.edu
Adapted from 2013 Hugo Serrano’s and 2014 Diogo Pacheco’s Lectures

2. Introduction
▪ What is Twitter
▪ Twitter is an online social network that allows users
to send and read messages of up to 140
characters, known as tweets.
▪ https://twitter.com

3. Introduction
▪ Twitter
▪ Online Social Network
▪ Users can follow and be followed by other users.
▪ A Real Time Information Network
▪ Tweets can be replied, retweeted, and favorited.

4. Twitter APIs
▪ REST APIs
▪ Access core primitives of Twitter
▪ provides programmatic access to read and write Twitter data
▪ Author a new Tweet, retweet a Tweet, read author profile and follower data,
and more
▪ Search API
▪ Finding a set of Tweets, Tweets from a particular user, or Tweets with
specific keywords/hashtags
▪ focused on relevance and not completeness
▪ Streaming APIs
▪ Continuously deliver new responses to REST API queries
▪ Useful for Data Mining & Analytics research
▪ When application is rate-limited for over-polling the REST APIs
▪ Large quantity of keywords
▪ Ads APIs
▪ https://dev.twitter.com/overview/documentation

5. REST APIs
▪ Communicate over HTTP with the same HTTP
verbs (GET, POST)
▪ Twitter limits the number of requests per 15
minutes intervals
▪ https://dev.twitter.com/rest/public/rate-limiting
▪ https://dev.twitter.com/rest/public/rate-limits
▪ Detailed reference of Search API endpoint
▪ https://dev.twitter.com/rest/public/search
▪ https://dev.twitter.com/rest/reference/get/search/
tweets

6. Streaming APIs
▪ Low latency access to Twitter’s global stream of Tweet
data.
▪ Twitter offers several streaming endpoints
▪ Public Streams
▪ https://dev.twitter.com/streaming/public
▪ Streams of the public data flowing through Twitter.
▪ Suitable for following specific users or topics, and data mining
▪ User Streams
▪ https://dev.twitter.com/streaming/userstreams
▪ Contain roughly all of the data corresponding with a single user’s view of Twitter.
▪ Site Streams
▪ https://dev.twitter.com/streaming/sitestreams
▪ Multi-user version of user streams. Site streams are intended for servers which must
connect to Twitter on behalf of many users.
▪ Require a long-lived HTTP connection

7. API Objects
▪ The most frequently observed objects in Twitter
ecosystem:
▪ Tweets
▪ Users
▪ Entities
▪ Places

8. API Object: Tweets
▪ Basic building blocks
▪ Also known as “status updates”
▪ Fields such as
▪ created_at: UTC time when this Tweet was created.
▪ coordinates: the geographic location of this Tweet.
▪ in_reply_to_screen_name: the name of the original Tweet's
author.
▪ in_reply_to_status_id: the id of the original Tweet's author.
▪ retweeted_status: a representation of the original Tweet that
was retweeted.
▪ https://dev.twitter.com/overview/api/tweets

9. API Object: Users
▪ They tweet, follow, retweet, are mentioned etc.
▪ Fields such as:
▪ id: the unique identification of a user
▪ screen_name: the screen name, handle, or alias that this
user identifies themselves with
▪ followers_count: the number of followers this account
currently has
▪ https://dev.twitter.com/overview/api/users

10. API Object: Entities
▪ Provide metadata and additional information about
content posted on Twitter:
▪ media
▪ urls
▪ hashtags
▪ user_mentions
▪ https://dev.twitter.com/overview/api/entities
▪ https://dev.twitter.com/overview/api/entities-in-
twitter-objects

11. API Objects: Places
▪ Specific, named locations with corresponding geo
coordinates
▪ Tweets associated with places are not necessarily
issued from that location but could also potentially
be about that location
▪ Fields such as
▪ country
▪ bounding_box
▪ full_name
▪ place_type
▪ https://dev.twitter.com/overview/api/places

12. Twitter APIs: OAuth Authentication
▪ Create a Twitter profile
▪ Create an application (you need a phone number)
▪ Create Access Tokens
▪ Use the OAuth tool to get your keys, tokens and
secrets
▪ https://apps.twitter.com

13. Twitter Libraries
▪ For some libraries (python), the Twitter API website is
the documentation for the developers
▪ https://dev.twitter.com/overview/api/twitter-libraries
-> api.search.tweets(q=“#HowToComplexNetworks”)

14. Useful Hints
▪ Get accustomed with Twitter App

15. Useful Hints
▪ Try first Twitter Console before coding
https://dev.twitter.com/rest/tools/console

16. Useful Hints
▪ Check rate limits in your code – try/catch loops
▪ Do you need database? Relational one?
▪ Remember 2-step process:
▪ Collect data
▪ Construct the network
▪ Intensive traffic/volume, try MongoDB

17. Useful Hints?
▪ 2014 FIFA World Cup Dataset
▪ +50 millions tweets
▪ +1 million geo-tagged tweets
▪ 1 month window period
▪ Database Optimization
▪ Memory Consuming
▪ Programs Crashes

18. Twitter api resources

19. Twitter API - Resources
▪ Timelines
▪ Search
▪ Tweets
▪ Streaming
▪ Direct Messages
▪ Friends & Followers
▪ Users
▪ Suggested Users
▪ Favorites
▪ Lists
▪ Saved Searches
▪ Places & Geo
▪ Trends
▪ Spam Reporting
▪ OAuth
▪ Help

20. Twitter API - Resources
▪ Search
▪ Let’s search for 5 most recent tweets with “Florida
Tech” on it
▪ Let’s search for 5 most recent tweets within 1mi

21. Twitter API – Resources
▪ Friends & Followers
▪ Let’s query for the friends of a particular user
▪ Let’s query for the first 100 followers of a particular
user

22. Twitter API - Resources
▪ Users
▪ Let’s query for the usernames of the first 100
followers of a particular user

23. © Ronaldo Menezes, Florida Tech
Twitter API - Resources
▪ Streaming
▪ Let’s track tweets with the hashtag #cse5656 and
#netsci
▪ Let’s track the tweets in Brevard County

24. Questions?
Marcello Tomasini – mtomasini@my.fit.edu
Marcos Oliveira – moliveirajun2013@my.fit.edu
http://my.fit.edu/~mtomasini/classes/ComplexNetworks.html