SlideShare a Scribd company logo
1 of 19
Download to read offline
Consuming the Twitter Streaming API
                           with Ruby and MongoDB

                                      Jeff Linwood
                             Lone Star Ruby Conference V
                                     August 12, 2011
                              http://www.jefflinwood.com
                                      @jefflinwood




Friday, August 12, 2011
Goals

                  • Watch for any
                          tweet that contains
                          certain keywords
                  • Store those tweets
                          into the MongoDB
                          database



Friday, August 12, 2011
Demo (@aplusk)




Friday, August 12, 2011
Twitter Streaming API+
                         Mongo DB

                     • Twitter Streaming API
                     • MongoDB
                     • Not a web application


Friday, August 12, 2011
The many APIs of
                              Twitter
                     • Twitter Streaming API
                     • User Streams
                     • Site Streams (for the big boys)
                     • REST API
                     • Search API

Friday, August 12, 2011
Twitter Streaming API

                     • Keywords
                     • Users
                     • Locations
                                                                Great photo is by rachel_thecat
                                      http://www.flickr.com/photos/23209605@N00/2786126623/




Friday, August 12, 2011
TweetStream gem
                          by Michael Bleigh
                          https://github.com/intridea/tweetstream




Friday, August 12, 2011
Connecting to the
                          Twitter Streaming API
                     • JSON responses
                     • HTTP Basic Authentication
                     • One stream per account (dev/prod)
                     • Leave it open!
                     • Don’t constantly reconnect, and if you do,
                          back off


Friday, August 12, 2011
Limitations of Twitter
                             Streaming API
                     • 400 Keywords
                     • 5,000 User Ids
                     • 25 Location Boxes
                     • Can ask Twitter for increased access

Friday, August 12, 2011
A Tweet, in JSON




Friday, August 12, 2011
Intro to MongoDB

                     • NoSQL - what does that mean?
                     • Great fit for JSON-oriented applications
                     • If you don’t know your schema in advance
                     • Query language
                     • Map Reduce

Friday, August 12, 2011
Storing data in MongoDB

                     • Native format of MongoDB is BSON,
                          similar to JSON
                     • Connect to a database (similar to MySQL)
                     • Connect to a collection (created if it
                          doesn’t exist)
                     • Insert JSON (in our case, a tweet)

Friday, August 12, 2011
MongoDB + Ruby

                     • mongo gem
                     • bson_ext gem
                     • http://www.mongodb.org/display/DOCS/
                          Ruby+Language+Center




Friday, August 12, 2011
Considerations for
                                MongoDB
                     • Tweets - very verbose JSON
                     • Date format in Tweets not same as
                          MongoDB
                     • May want to pre-process Tweets
                     • Can use both MongoDB and MySQL in
                          same app if you want


Friday, August 12, 2011
Here’s the code


                     • Okay, the whole thing is really done in
                          about three lines.




Friday, August 12, 2011
Where do you run this?

                     • Command line
                     • Your own server
                     • Heroku + MongoLab
                     • Other cloud services

Friday, August 12, 2011
MongoLab




Friday, August 12, 2011
Now what?

                     • Step 1: Collect Tweets
                     • Step 2: ????
                     • Step 3: Profit!

Friday, August 12, 2011
To Learn More
                     • https://dev.twitter.com/docs/streaming-api/
                          concepts
                     • http://www.mongodb.org/
                     • https://github.com/jefflinwood/Tweeter-
                          Keeper
                     • http://www.jefflinwood.com/
                     • @jefflinwood on Twitter
Friday, August 12, 2011

More Related Content

What's hot

Fostering Community
Fostering CommunityFostering Community
Fostering Community
kmtracey
 

What's hot (14)

Build automation for XPages - AUSLUG 2015
Build automation for XPages - AUSLUG 2015Build automation for XPages - AUSLUG 2015
Build automation for XPages - AUSLUG 2015
 
Developers! Y U No Open Source Ur Code?
Developers! Y U No Open Source Ur Code?Developers! Y U No Open Source Ur Code?
Developers! Y U No Open Source Ur Code?
 
Business of APIs
Business of APIsBusiness of APIs
Business of APIs
 
WordPress: Getting Under the Hood
WordPress: Getting Under the HoodWordPress: Getting Under the Hood
WordPress: Getting Under the Hood
 
eMusic: WordPress in the Enterprise
eMusic: WordPress in the EnterpriseeMusic: WordPress in the Enterprise
eMusic: WordPress in the Enterprise
 
Bringing WordPress to the front-end. o2 is the new P2
Bringing WordPress to the front-end. o2 is the new P2Bringing WordPress to the front-end. o2 is the new P2
Bringing WordPress to the front-end. o2 is the new P2
 
WordPress Under Control
WordPress Under ControlWordPress Under Control
WordPress Under Control
 
WordPress Security 101 - Meetup Nairobi March 2020
WordPress Security 101 - Meetup Nairobi March 2020 WordPress Security 101 - Meetup Nairobi March 2020
WordPress Security 101 - Meetup Nairobi March 2020
 
Interoperability with Standardless IoT (Global IoT Day Wien)
Interoperability with Standardless IoT (Global IoT Day Wien)Interoperability with Standardless IoT (Global IoT Day Wien)
Interoperability with Standardless IoT (Global IoT Day Wien)
 
Novmeber 2015 WordPress News 'n' Stuff
Novmeber 2015 WordPress News 'n' StuffNovmeber 2015 WordPress News 'n' Stuff
Novmeber 2015 WordPress News 'n' Stuff
 
What a Thing API Should Look Like (Global IoT Day Wien)
What a Thing API Should Look Like (Global IoT Day Wien)What a Thing API Should Look Like (Global IoT Day Wien)
What a Thing API Should Look Like (Global IoT Day Wien)
 
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...
 
Fostering Community
Fostering CommunityFostering Community
Fostering Community
 
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
 

Similar to Consuming the Twitter Streaming API with Ruby and MongoDB

Sean coates fifty things and tricks, confoo 2011
Sean coates fifty things and tricks, confoo 2011Sean coates fifty things and tricks, confoo 2011
Sean coates fifty things and tricks, confoo 2011
Bachkoutou Toutou
 
Clouds against the Floods (RubyConfBR2011)
Clouds against the Floods (RubyConfBR2011) Clouds against the Floods (RubyConfBR2011)
Clouds against the Floods (RubyConfBR2011)
Leonardo Borges
 
Conquistando el Servidor con Node.JS
Conquistando el Servidor con Node.JSConquistando el Servidor con Node.JS
Conquistando el Servidor con Node.JS
Caridy Patino
 
HootSuite Dev 2
HootSuite Dev 2HootSuite Dev 2
HootSuite Dev 2
ujihisa
 

Similar to Consuming the Twitter Streaming API with Ruby and MongoDB (20)

Iwmn architecture
Iwmn architectureIwmn architecture
Iwmn architecture
 
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
 
BRAINREPUBLIC - Powered by no-SQL
BRAINREPUBLIC - Powered by no-SQLBRAINREPUBLIC - Powered by no-SQL
BRAINREPUBLIC - Powered by no-SQL
 
Flowdock's full-text search with MongoDB
Flowdock's full-text search with MongoDBFlowdock's full-text search with MongoDB
Flowdock's full-text search with MongoDB
 
AppScale Talk at SBonRails
AppScale Talk at SBonRailsAppScale Talk at SBonRails
AppScale Talk at SBonRails
 
Sean coates fifty things and tricks, confoo 2011
Sean coates fifty things and tricks, confoo 2011Sean coates fifty things and tricks, confoo 2011
Sean coates fifty things and tricks, confoo 2011
 
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...
 
Web micro-framework BATTLE!
Web micro-framework BATTLE!Web micro-framework BATTLE!
Web micro-framework BATTLE!
 
Clouds against the Floods (RubyConfBR2011)
Clouds against the Floods (RubyConfBR2011) Clouds against the Floods (RubyConfBR2011)
Clouds against the Floods (RubyConfBR2011)
 
Java EE and Google App Engine
Java EE and Google App EngineJava EE and Google App Engine
Java EE and Google App Engine
 
HTML5 & CSS3 in Drupal (on the Bayou)
HTML5 & CSS3 in Drupal (on the Bayou)HTML5 & CSS3 in Drupal (on the Bayou)
HTML5 & CSS3 in Drupal (on the Bayou)
 
A Look at the Future of HTML5
A Look at the Future of HTML5A Look at the Future of HTML5
A Look at the Future of HTML5
 
Apcug 2011 07-17-intro_to_drupal_jeff_schuler
Apcug 2011 07-17-intro_to_drupal_jeff_schulerApcug 2011 07-17-intro_to_drupal_jeff_schuler
Apcug 2011 07-17-intro_to_drupal_jeff_schuler
 
Caridy patino - node-js
Caridy patino - node-jsCaridy patino - node-js
Caridy patino - node-js
 
Conquistando el Servidor con Node.JS
Conquistando el Servidor con Node.JSConquistando el Servidor con Node.JS
Conquistando el Servidor con Node.JS
 
App Engine Meetup
App Engine MeetupApp Engine Meetup
App Engine Meetup
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disqus
 
HootSuite Dev 2
HootSuite Dev 2HootSuite Dev 2
HootSuite Dev 2
 
Damien Tanner, Pusher
Damien Tanner, PusherDamien Tanner, Pusher
Damien Tanner, Pusher
 
MongoDB at Sailthru: Scaling and Schema Design
MongoDB at Sailthru: Scaling and Schema DesignMongoDB at Sailthru: Scaling and Schema Design
MongoDB at Sailthru: Scaling and Schema Design
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Peter Udo Diehl
 

Recently uploaded (20)

Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, Ocado
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Buy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxBuy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptx
 

Consuming the Twitter Streaming API with Ruby and MongoDB

  • 1. Consuming the Twitter Streaming API with Ruby and MongoDB Jeff Linwood Lone Star Ruby Conference V August 12, 2011 http://www.jefflinwood.com @jefflinwood Friday, August 12, 2011
  • 2. Goals • Watch for any tweet that contains certain keywords • Store those tweets into the MongoDB database Friday, August 12, 2011
  • 4. Twitter Streaming API+ Mongo DB • Twitter Streaming API • MongoDB • Not a web application Friday, August 12, 2011
  • 5. The many APIs of Twitter • Twitter Streaming API • User Streams • Site Streams (for the big boys) • REST API • Search API Friday, August 12, 2011
  • 6. Twitter Streaming API • Keywords • Users • Locations Great photo is by rachel_thecat http://www.flickr.com/photos/23209605@N00/2786126623/ Friday, August 12, 2011
  • 7. TweetStream gem by Michael Bleigh https://github.com/intridea/tweetstream Friday, August 12, 2011
  • 8. Connecting to the Twitter Streaming API • JSON responses • HTTP Basic Authentication • One stream per account (dev/prod) • Leave it open! • Don’t constantly reconnect, and if you do, back off Friday, August 12, 2011
  • 9. Limitations of Twitter Streaming API • 400 Keywords • 5,000 User Ids • 25 Location Boxes • Can ask Twitter for increased access Friday, August 12, 2011
  • 10. A Tweet, in JSON Friday, August 12, 2011
  • 11. Intro to MongoDB • NoSQL - what does that mean? • Great fit for JSON-oriented applications • If you don’t know your schema in advance • Query language • Map Reduce Friday, August 12, 2011
  • 12. Storing data in MongoDB • Native format of MongoDB is BSON, similar to JSON • Connect to a database (similar to MySQL) • Connect to a collection (created if it doesn’t exist) • Insert JSON (in our case, a tweet) Friday, August 12, 2011
  • 13. MongoDB + Ruby • mongo gem • bson_ext gem • http://www.mongodb.org/display/DOCS/ Ruby+Language+Center Friday, August 12, 2011
  • 14. Considerations for MongoDB • Tweets - very verbose JSON • Date format in Tweets not same as MongoDB • May want to pre-process Tweets • Can use both MongoDB and MySQL in same app if you want Friday, August 12, 2011
  • 15. Here’s the code • Okay, the whole thing is really done in about three lines. Friday, August 12, 2011
  • 16. Where do you run this? • Command line • Your own server • Heroku + MongoLab • Other cloud services Friday, August 12, 2011
  • 18. Now what? • Step 1: Collect Tweets • Step 2: ???? • Step 3: Profit! Friday, August 12, 2011
  • 19. To Learn More • https://dev.twitter.com/docs/streaming-api/ concepts • http://www.mongodb.org/ • https://github.com/jefflinwood/Tweeter- Keeper • http://www.jefflinwood.com/ • @jefflinwood on Twitter Friday, August 12, 2011