Consuming the Twitter Streaming API with Ruby and MongoDB

Consuming the Twitter Streaming API with Ruby and MongoDB Jeff Linwood Lone Star Ruby Conference V August 12, 2011 http://www.jefﬂinwood.com @jefﬂinwoodFriday, August 12, 2011

Goals • Watch for any tweet that contains certain keywords • Store those tweets into the MongoDB databaseFriday, August 12, 2011

Demo (@aplusk)Friday, August 12, 2011

Twitter Streaming API+ Mongo DB • Twitter Streaming API • MongoDB • Not a web applicationFriday, August 12, 2011

The many APIs of Twitter • Twitter Streaming API • User Streams • Site Streams (for the big boys) • REST API • Search APIFriday, August 12, 2011

Twitter Streaming API • Keywords • Users • Locations Great photo is by rachel_thecat http://www.ﬂickr.com/photos/23209605@N00/2786126623/Friday, August 12, 2011

TweetStream gem by Michael Bleigh https://github.com/intridea/tweetstreamFriday, August 12, 2011

Connecting to the Twitter Streaming API • JSON responses • HTTP Basic Authentication • One stream per account (dev/prod) • Leave it open! • Don’t constantly reconnect, and if you do, back offFriday, August 12, 2011

Limitations of Twitter Streaming API • 400 Keywords • 5,000 User Ids • 25 Location Boxes • Can ask Twitter for increased accessFriday, August 12, 2011

A Tweet, in JSONFriday, August 12, 2011

Intro to MongoDB • NoSQL - what does that mean? • Great ﬁt for JSON-oriented applications • If you don’t know your schema in advance • Query language • Map ReduceFriday, August 12, 2011

Storing data in MongoDB • Native format of MongoDB is BSON, similar to JSON • Connect to a database (similar to MySQL) • Connect to a collection (created if it doesn’t exist) • Insert JSON (in our case, a tweet)Friday, August 12, 2011

MongoDB + Ruby • mongo gem • bson_ext gem • http://www.mongodb.org/display/DOCS/ Ruby+Language+CenterFriday, August 12, 2011

Considerations for MongoDB • Tweets - very verbose JSON • Date format in Tweets not same as MongoDB • May want to pre-process Tweets • Can use both MongoDB and MySQL in same app if you wantFriday, August 12, 2011

Here’s the code • Okay, the whole thing is really done in about three lines.Friday, August 12, 2011

Where do you run this? • Command line • Your own server • Heroku + MongoLab • Other cloud servicesFriday, August 12, 2011

MongoLabFriday, August 12, 2011

Now what? • Step 1: Collect Tweets • Step 2: ???? • Step 3: Proﬁt!Friday, August 12, 2011

To Learn More • https://dev.twitter.com/docs/streaming-api/ concepts • http://www.mongodb.org/ • https://github.com/jefflinwood/Tweeter- Keeper • http://www.jefflinwood.com/ • @jefflinwood on TwitterFriday, August 12, 2011

http://www.jefflinwood.com	631
http://lanyrd.com	148
http://feeds.feedburner.com	10
http://www.hanrss.com	3
http://reader.youdao.com	1
http://10.170.30.107	1

Consuming the Twitter Streaming API with Ruby and MongoDB

by Jeff Linwood on Aug 12, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

6 Embeds 794

Statistics

Consuming the Twitter Streaming API with Ruby and MongoDB — Presentation Transcript