Consuming the Twitter Streaming API                           with Ruby and MongoDB                                      J...
Goals                  • Watch for any                          tweet that contains                          certain keywo...
Demo (@aplusk)Friday, August 12, 2011
Twitter Streaming API+                         Mongo DB                     • Twitter Streaming API                     • ...
The many APIs of                              Twitter                     • Twitter Streaming API                     • Us...
Twitter Streaming API                     • Keywords                     • Users                     • Locations          ...
TweetStream gem                          by Michael Bleigh                          https://github.com/intridea/tweetstrea...
Connecting to the                          Twitter Streaming API                     • JSON responses                     ...
Limitations of Twitter                             Streaming API                     • 400 Keywords                     • ...
A Tweet, in JSONFriday, August 12, 2011
Intro to MongoDB                     • NoSQL - what does that mean?                     • Great fit for JSON-oriented appli...
Storing data in MongoDB                     • Native format of MongoDB is BSON,                          similar to JSON  ...
MongoDB + Ruby                     • mongo gem                     • bson_ext gem                     • http://www.mongodb...
Considerations for                                MongoDB                     • Tweets - very verbose JSON                ...
Here’s the code                     • Okay, the whole thing is really done in                          about three lines.F...
Where do you run this?                     • Command line                     • Your own server                     • Hero...
MongoLabFriday, August 12, 2011
Now what?                     • Step 1: Collect Tweets                     • Step 2: ????                     • Step 3: Pr...
To Learn More                     • https://dev.twitter.com/docs/streaming-api/                          concepts         ...
Upcoming SlideShare
Loading in …5
×

Consuming the Twitter Streaming API with Ruby and MongoDB

6,378 views

Published on

A talk on connecting to the Twitter Streaming API and storing the tweets in MongoDB with Ruby.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,378
On SlideShare
0
From Embeds
0
Number of Embeds
1,162
Actions
Shares
0
Downloads
36
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Consuming the Twitter Streaming API with Ruby and MongoDB

  1. 1. Consuming the Twitter Streaming API with Ruby and MongoDB Jeff Linwood Lone Star Ruby Conference V August 12, 2011 http://www.jefflinwood.com @jefflinwoodFriday, August 12, 2011
  2. 2. Goals • Watch for any tweet that contains certain keywords • Store those tweets into the MongoDB databaseFriday, August 12, 2011
  3. 3. Demo (@aplusk)Friday, August 12, 2011
  4. 4. Twitter Streaming API+ Mongo DB • Twitter Streaming API • MongoDB • Not a web applicationFriday, August 12, 2011
  5. 5. The many APIs of Twitter • Twitter Streaming API • User Streams • Site Streams (for the big boys) • REST API • Search APIFriday, August 12, 2011
  6. 6. Twitter Streaming API • Keywords • Users • Locations Great photo is by rachel_thecat http://www.flickr.com/photos/23209605@N00/2786126623/Friday, August 12, 2011
  7. 7. TweetStream gem by Michael Bleigh https://github.com/intridea/tweetstreamFriday, August 12, 2011
  8. 8. Connecting to the Twitter Streaming API • JSON responses • HTTP Basic Authentication • One stream per account (dev/prod) • Leave it open! • Don’t constantly reconnect, and if you do, back offFriday, August 12, 2011
  9. 9. Limitations of Twitter Streaming API • 400 Keywords • 5,000 User Ids • 25 Location Boxes • Can ask Twitter for increased accessFriday, August 12, 2011
  10. 10. A Tweet, in JSONFriday, August 12, 2011
  11. 11. Intro to MongoDB • NoSQL - what does that mean? • Great fit for JSON-oriented applications • If you don’t know your schema in advance • Query language • Map ReduceFriday, August 12, 2011
  12. 12. Storing data in MongoDB • Native format of MongoDB is BSON, similar to JSON • Connect to a database (similar to MySQL) • Connect to a collection (created if it doesn’t exist) • Insert JSON (in our case, a tweet)Friday, August 12, 2011
  13. 13. MongoDB + Ruby • mongo gem • bson_ext gem • http://www.mongodb.org/display/DOCS/ Ruby+Language+CenterFriday, August 12, 2011
  14. 14. Considerations for MongoDB • Tweets - very verbose JSON • Date format in Tweets not same as MongoDB • May want to pre-process Tweets • Can use both MongoDB and MySQL in same app if you wantFriday, August 12, 2011
  15. 15. Here’s the code • Okay, the whole thing is really done in about three lines.Friday, August 12, 2011
  16. 16. Where do you run this? • Command line • Your own server • Heroku + MongoLab • Other cloud servicesFriday, August 12, 2011
  17. 17. MongoLabFriday, August 12, 2011
  18. 18. Now what? • Step 1: Collect Tweets • Step 2: ???? • Step 3: Profit!Friday, August 12, 2011
  19. 19. To Learn More • https://dev.twitter.com/docs/streaming-api/ concepts • http://www.mongodb.org/ • https://github.com/jefflinwood/Tweeter- Keeper • http://www.jefflinwood.com/ • @jefflinwood on TwitterFriday, August 12, 2011

×