SlideShare a Scribd company logo
1 of 19
Download to read offline
Consuming the Twitter Streaming API
                           with Ruby and MongoDB

                                      Jeff Linwood
                             Lone Star Ruby Conference V
                                     August 12, 2011
                              http://www.jefflinwood.com
                                      @jefflinwood




Friday, August 12, 2011
Goals

                  • Watch for any
                          tweet that contains
                          certain keywords
                  • Store those tweets
                          into the MongoDB
                          database



Friday, August 12, 2011
Demo (@aplusk)




Friday, August 12, 2011
Twitter Streaming API+
                         Mongo DB

                     • Twitter Streaming API
                     • MongoDB
                     • Not a web application


Friday, August 12, 2011
The many APIs of
                              Twitter
                     • Twitter Streaming API
                     • User Streams
                     • Site Streams (for the big boys)
                     • REST API
                     • Search API

Friday, August 12, 2011
Twitter Streaming API

                     • Keywords
                     • Users
                     • Locations
                                                                Great photo is by rachel_thecat
                                      http://www.flickr.com/photos/23209605@N00/2786126623/




Friday, August 12, 2011
TweetStream gem
                          by Michael Bleigh
                          https://github.com/intridea/tweetstream




Friday, August 12, 2011
Connecting to the
                          Twitter Streaming API
                     • JSON responses
                     • HTTP Basic Authentication
                     • One stream per account (dev/prod)
                     • Leave it open!
                     • Don’t constantly reconnect, and if you do,
                          back off


Friday, August 12, 2011
Limitations of Twitter
                             Streaming API
                     • 400 Keywords
                     • 5,000 User Ids
                     • 25 Location Boxes
                     • Can ask Twitter for increased access

Friday, August 12, 2011
A Tweet, in JSON




Friday, August 12, 2011
Intro to MongoDB

                     • NoSQL - what does that mean?
                     • Great fit for JSON-oriented applications
                     • If you don’t know your schema in advance
                     • Query language
                     • Map Reduce

Friday, August 12, 2011
Storing data in MongoDB

                     • Native format of MongoDB is BSON,
                          similar to JSON
                     • Connect to a database (similar to MySQL)
                     • Connect to a collection (created if it
                          doesn’t exist)
                     • Insert JSON (in our case, a tweet)

Friday, August 12, 2011
MongoDB + Ruby

                     • mongo gem
                     • bson_ext gem
                     • http://www.mongodb.org/display/DOCS/
                          Ruby+Language+Center




Friday, August 12, 2011
Considerations for
                                MongoDB
                     • Tweets - very verbose JSON
                     • Date format in Tweets not same as
                          MongoDB
                     • May want to pre-process Tweets
                     • Can use both MongoDB and MySQL in
                          same app if you want


Friday, August 12, 2011
Here’s the code


                     • Okay, the whole thing is really done in
                          about three lines.




Friday, August 12, 2011
Where do you run this?

                     • Command line
                     • Your own server
                     • Heroku + MongoLab
                     • Other cloud services

Friday, August 12, 2011
MongoLab




Friday, August 12, 2011
Now what?

                     • Step 1: Collect Tweets
                     • Step 2: ????
                     • Step 3: Profit!

Friday, August 12, 2011
To Learn More
                     • https://dev.twitter.com/docs/streaming-api/
                          concepts
                     • http://www.mongodb.org/
                     • https://github.com/jefflinwood/Tweeter-
                          Keeper
                     • http://www.jefflinwood.com/
                     • @jefflinwood on Twitter
Friday, August 12, 2011

More Related Content

What's hot

Build automation for XPages - AUSLUG 2015
Build automation for XPages - AUSLUG 2015Build automation for XPages - AUSLUG 2015
Build automation for XPages - AUSLUG 2015gregorbyte
 
Developers! Y U No Open Source Ur Code?
Developers! Y U No Open Source Ur Code?Developers! Y U No Open Source Ur Code?
Developers! Y U No Open Source Ur Code?Craig Marvelley
 
WordPress: Getting Under the Hood
WordPress: Getting Under the HoodWordPress: Getting Under the Hood
WordPress: Getting Under the HoodScott Taylor
 
eMusic: WordPress in the Enterprise
eMusic: WordPress in the EnterpriseeMusic: WordPress in the Enterprise
eMusic: WordPress in the EnterpriseScott Taylor
 
Bringing WordPress to the front-end. o2 is the new P2
Bringing WordPress to the front-end. o2 is the new P2Bringing WordPress to the front-end. o2 is the new P2
Bringing WordPress to the front-end. o2 is the new P2Beau Lebens
 
WordPress Under Control
WordPress Under ControlWordPress Under Control
WordPress Under ControlMatt Bernhardt
 
Interoperability with Standardless IoT (Global IoT Day Wien)
Interoperability with Standardless IoT (Global IoT Day Wien)Interoperability with Standardless IoT (Global IoT Day Wien)
Interoperability with Standardless IoT (Global IoT Day Wien)David Janes
 
WordPress Security 101 - Meetup Nairobi March 2020
WordPress Security 101 - Meetup Nairobi March 2020 WordPress Security 101 - Meetup Nairobi March 2020
WordPress Security 101 - Meetup Nairobi March 2020 stk_jj
 
Novmeber 2015 WordPress News 'n' Stuff
Novmeber 2015 WordPress News 'n' StuffNovmeber 2015 WordPress News 'n' Stuff
Novmeber 2015 WordPress News 'n' StuffWordPress Sydney
 
What a Thing API Should Look Like (Global IoT Day Wien)
What a Thing API Should Look Like (Global IoT Day Wien)What a Thing API Should Look Like (Global IoT Day Wien)
What a Thing API Should Look Like (Global IoT Day Wien)David Janes
 
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...Nuxeo
 
Fostering Community
Fostering CommunityFostering Community
Fostering Communitykmtracey
 
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)Robert Metzger
 

What's hot (14)

Build automation for XPages - AUSLUG 2015
Build automation for XPages - AUSLUG 2015Build automation for XPages - AUSLUG 2015
Build automation for XPages - AUSLUG 2015
 
Developers! Y U No Open Source Ur Code?
Developers! Y U No Open Source Ur Code?Developers! Y U No Open Source Ur Code?
Developers! Y U No Open Source Ur Code?
 
Business of APIs
Business of APIsBusiness of APIs
Business of APIs
 
WordPress: Getting Under the Hood
WordPress: Getting Under the HoodWordPress: Getting Under the Hood
WordPress: Getting Under the Hood
 
eMusic: WordPress in the Enterprise
eMusic: WordPress in the EnterpriseeMusic: WordPress in the Enterprise
eMusic: WordPress in the Enterprise
 
Bringing WordPress to the front-end. o2 is the new P2
Bringing WordPress to the front-end. o2 is the new P2Bringing WordPress to the front-end. o2 is the new P2
Bringing WordPress to the front-end. o2 is the new P2
 
WordPress Under Control
WordPress Under ControlWordPress Under Control
WordPress Under Control
 
Interoperability with Standardless IoT (Global IoT Day Wien)
Interoperability with Standardless IoT (Global IoT Day Wien)Interoperability with Standardless IoT (Global IoT Day Wien)
Interoperability with Standardless IoT (Global IoT Day Wien)
 
WordPress Security 101 - Meetup Nairobi March 2020
WordPress Security 101 - Meetup Nairobi March 2020 WordPress Security 101 - Meetup Nairobi March 2020
WordPress Security 101 - Meetup Nairobi March 2020
 
Novmeber 2015 WordPress News 'n' Stuff
Novmeber 2015 WordPress News 'n' StuffNovmeber 2015 WordPress News 'n' Stuff
Novmeber 2015 WordPress News 'n' Stuff
 
What a Thing API Should Look Like (Global IoT Day Wien)
What a Thing API Should Look Like (Global IoT Day Wien)What a Thing API Should Look Like (Global IoT Day Wien)
What a Thing API Should Look Like (Global IoT Day Wien)
 
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...
[Nuxeo World 2013] DID YOU SAY DAM? DIGITAL ASSET MANAGEMENT WITH THE NUXEO P...
 
Fostering Community
Fostering CommunityFostering Community
Fostering Community
 
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
 

Similar to Twitter streamingapi rubymongodbv2

2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Goikailan
 
BRAINREPUBLIC - Powered by no-SQL
BRAINREPUBLIC - Powered by no-SQLBRAINREPUBLIC - Powered by no-SQL
BRAINREPUBLIC - Powered by no-SQLAndreas Jung
 
Flowdock's full-text search with MongoDB
Flowdock's full-text search with MongoDBFlowdock's full-text search with MongoDB
Flowdock's full-text search with MongoDBFlowdock
 
AppScale Talk at SBonRails
AppScale Talk at SBonRailsAppScale Talk at SBonRails
AppScale Talk at SBonRailsChris Bunch
 
Sean coates fifty things and tricks, confoo 2011
Sean coates fifty things and tricks, confoo 2011Sean coates fifty things and tricks, confoo 2011
Sean coates fifty things and tricks, confoo 2011Bachkoutou Toutou
 
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...Brian Huff
 
Web micro-framework BATTLE!
Web micro-framework BATTLE!Web micro-framework BATTLE!
Web micro-framework BATTLE!Richard Jones
 
Clouds against the Floods (RubyConfBR2011)
Clouds against the Floods (RubyConfBR2011) Clouds against the Floods (RubyConfBR2011)
Clouds against the Floods (RubyConfBR2011) Leonardo Borges
 
Java EE and Google App Engine
Java EE and Google App EngineJava EE and Google App Engine
Java EE and Google App EngineArun Gupta
 
HTML5 & CSS3 in Drupal (on the Bayou)
HTML5 & CSS3 in Drupal (on the Bayou)HTML5 & CSS3 in Drupal (on the Bayou)
HTML5 & CSS3 in Drupal (on the Bayou)Mediacurrent
 
A Look at the Future of HTML5
A Look at the Future of HTML5A Look at the Future of HTML5
A Look at the Future of HTML5Tim Wright
 
Apcug 2011 07-17-intro_to_drupal_jeff_schuler
Apcug 2011 07-17-intro_to_drupal_jeff_schulerApcug 2011 07-17-intro_to_drupal_jeff_schuler
Apcug 2011 07-17-intro_to_drupal_jeff_schulerhewie
 
Conquistando el Servidor con Node.JS
Conquistando el Servidor con Node.JSConquistando el Servidor con Node.JS
Conquistando el Servidor con Node.JSCaridy Patino
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disquszeeg
 
HootSuite Dev 2
HootSuite Dev 2HootSuite Dev 2
HootSuite Dev 2ujihisa
 
Damien Tanner, Pusher
Damien Tanner, PusherDamien Tanner, Pusher
Damien Tanner, PusherMashery
 
MongoDB at Sailthru: Scaling and Schema Design
MongoDB at Sailthru: Scaling and Schema DesignMongoDB at Sailthru: Scaling and Schema Design
MongoDB at Sailthru: Scaling and Schema DesignDATAVERSITY
 

Similar to Twitter streamingapi rubymongodbv2 (20)

Iwmn architecture
Iwmn architectureIwmn architecture
Iwmn architecture
 
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
 
BRAINREPUBLIC - Powered by no-SQL
BRAINREPUBLIC - Powered by no-SQLBRAINREPUBLIC - Powered by no-SQL
BRAINREPUBLIC - Powered by no-SQL
 
Flowdock's full-text search with MongoDB
Flowdock's full-text search with MongoDBFlowdock's full-text search with MongoDB
Flowdock's full-text search with MongoDB
 
AppScale Talk at SBonRails
AppScale Talk at SBonRailsAppScale Talk at SBonRails
AppScale Talk at SBonRails
 
Sean coates fifty things and tricks, confoo 2011
Sean coates fifty things and tricks, confoo 2011Sean coates fifty things and tricks, confoo 2011
Sean coates fifty things and tricks, confoo 2011
 
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...
Integrating ECM (WebCenter Content) with your Enterprise! 5 Tips to Try, 5 Tr...
 
Web micro-framework BATTLE!
Web micro-framework BATTLE!Web micro-framework BATTLE!
Web micro-framework BATTLE!
 
Clouds against the Floods (RubyConfBR2011)
Clouds against the Floods (RubyConfBR2011) Clouds against the Floods (RubyConfBR2011)
Clouds against the Floods (RubyConfBR2011)
 
Java EE and Google App Engine
Java EE and Google App EngineJava EE and Google App Engine
Java EE and Google App Engine
 
HTML5 & CSS3 in Drupal (on the Bayou)
HTML5 & CSS3 in Drupal (on the Bayou)HTML5 & CSS3 in Drupal (on the Bayou)
HTML5 & CSS3 in Drupal (on the Bayou)
 
A Look at the Future of HTML5
A Look at the Future of HTML5A Look at the Future of HTML5
A Look at the Future of HTML5
 
Apcug 2011 07-17-intro_to_drupal_jeff_schuler
Apcug 2011 07-17-intro_to_drupal_jeff_schulerApcug 2011 07-17-intro_to_drupal_jeff_schuler
Apcug 2011 07-17-intro_to_drupal_jeff_schuler
 
Caridy patino - node-js
Caridy patino - node-jsCaridy patino - node-js
Caridy patino - node-js
 
Conquistando el Servidor con Node.JS
Conquistando el Servidor con Node.JSConquistando el Servidor con Node.JS
Conquistando el Servidor con Node.JS
 
App Engine Meetup
App Engine MeetupApp Engine Meetup
App Engine Meetup
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disqus
 
HootSuite Dev 2
HootSuite Dev 2HootSuite Dev 2
HootSuite Dev 2
 
Damien Tanner, Pusher
Damien Tanner, PusherDamien Tanner, Pusher
Damien Tanner, Pusher
 
MongoDB at Sailthru: Scaling and Schema Design
MongoDB at Sailthru: Scaling and Schema DesignMongoDB at Sailthru: Scaling and Schema Design
MongoDB at Sailthru: Scaling and Schema Design
 

Recently uploaded

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentationyogeshlabana357357
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessUXDXConf
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxFIDO Alliance
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsLeah Henrickson
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireExakis Nelite
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimaginedpanagenda
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...marcuskenyatta275
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024Lorenzo Miniero
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxFIDO Alliance
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfUK Journal
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...ScyllaDB
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?Paolo Missier
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 

Recently uploaded (20)

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 

Twitter streamingapi rubymongodbv2

  • 1. Consuming the Twitter Streaming API with Ruby and MongoDB Jeff Linwood Lone Star Ruby Conference V August 12, 2011 http://www.jefflinwood.com @jefflinwood Friday, August 12, 2011
  • 2. Goals • Watch for any tweet that contains certain keywords • Store those tweets into the MongoDB database Friday, August 12, 2011
  • 4. Twitter Streaming API+ Mongo DB • Twitter Streaming API • MongoDB • Not a web application Friday, August 12, 2011
  • 5. The many APIs of Twitter • Twitter Streaming API • User Streams • Site Streams (for the big boys) • REST API • Search API Friday, August 12, 2011
  • 6. Twitter Streaming API • Keywords • Users • Locations Great photo is by rachel_thecat http://www.flickr.com/photos/23209605@N00/2786126623/ Friday, August 12, 2011
  • 7. TweetStream gem by Michael Bleigh https://github.com/intridea/tweetstream Friday, August 12, 2011
  • 8. Connecting to the Twitter Streaming API • JSON responses • HTTP Basic Authentication • One stream per account (dev/prod) • Leave it open! • Don’t constantly reconnect, and if you do, back off Friday, August 12, 2011
  • 9. Limitations of Twitter Streaming API • 400 Keywords • 5,000 User Ids • 25 Location Boxes • Can ask Twitter for increased access Friday, August 12, 2011
  • 10. A Tweet, in JSON Friday, August 12, 2011
  • 11. Intro to MongoDB • NoSQL - what does that mean? • Great fit for JSON-oriented applications • If you don’t know your schema in advance • Query language • Map Reduce Friday, August 12, 2011
  • 12. Storing data in MongoDB • Native format of MongoDB is BSON, similar to JSON • Connect to a database (similar to MySQL) • Connect to a collection (created if it doesn’t exist) • Insert JSON (in our case, a tweet) Friday, August 12, 2011
  • 13. MongoDB + Ruby • mongo gem • bson_ext gem • http://www.mongodb.org/display/DOCS/ Ruby+Language+Center Friday, August 12, 2011
  • 14. Considerations for MongoDB • Tweets - very verbose JSON • Date format in Tweets not same as MongoDB • May want to pre-process Tweets • Can use both MongoDB and MySQL in same app if you want Friday, August 12, 2011
  • 15. Here’s the code • Okay, the whole thing is really done in about three lines. Friday, August 12, 2011
  • 16. Where do you run this? • Command line • Your own server • Heroku + MongoLab • Other cloud services Friday, August 12, 2011
  • 18. Now what? • Step 1: Collect Tweets • Step 2: ???? • Step 3: Profit! Friday, August 12, 2011
  • 19. To Learn More • https://dev.twitter.com/docs/streaming-api/ concepts • http://www.mongodb.org/ • https://github.com/jefflinwood/Tweeter- Keeper • http://www.jefflinwood.com/ • @jefflinwood on Twitter Friday, August 12, 2011