SlideShare a Scribd company logo
Do More, Faster
The Twitter Streaming API
1. Create a HTTP Request

GET /1/statuses/sample.json HTTP/1.1
Authorization: OAuth realm="",oauth_no...
Host: stream.twitter.com
Accept: */*


             Sign it with OAuth
2. Handle the response
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.25)

{"in_reply_to_status_id_str":null,"geo":n
ull,"text":"Hey fat boyyyyy!!...


   What’s missing? No Content-Length header!
HTTP Libraries won’t know when to stop consuming
3. Keep handling it

• Twitter will keep sending you JSON chunks
  until you kill the connection
• Normal HTTP libraries make this awkward
• Try non-blocking / event-based I/O
The Streaming API lets
  you do cool stuff
• Sample the firehose, free ~1%
• Get interesting subsets of all tweets
 • By filtering, can get all tweets on a topic
• Interact in near-realtime
• Process more efficiently
Also...


• User Streams
• Site Streams
At Conversocial, we built a way to search twitter
            on behalf of our clients.




There’s a stream client that receives tweets from the
                    Streaming API
We monitor the stream client from a separate thread
We still have to poll the REST API to cover gaps between
                        connections
Need to disconnect stream client from processing tasks.
               We do this via a queue.
Searches have to be matched back to the terms that
    produced them - the stream is multiplexed.
Thanks for listening!

   gareth@conversocial.com
http://www.github.com/gareth-lloyd/twistedstream

More Related Content

Similar to Twitter Streaming API

OAuth for your API - The Big Picture
OAuth for your API - The Big PictureOAuth for your API - The Big Picture
OAuth for your API - The Big Picture
Apigee | Google Cloud
 
Networked APIs with swift
Networked APIs with swiftNetworked APIs with swift
Networked APIs with swift
Tim Burks
 
Http_Protocol.pptx
Http_Protocol.pptxHttp_Protocol.pptx
Http_Protocol.pptx
Abshar Fatima
 
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
Cory Forsyth
 
Testing REST Web Services
Testing REST Web ServicesTesting REST Web Services
Testing REST Web Services
Jan Algermissen
 
ASP.NET Mvc 4 web api
ASP.NET Mvc 4 web apiASP.NET Mvc 4 web api
ASP.NET Mvc 4 web api
Tiago Knoch
 
The Open Source... Behind the Tweets
The Open Source... Behind the TweetsThe Open Source... Behind the Tweets
The Open Source... Behind the Tweets
Chris Aniszczyk
 
Twitter Update for Social Developers London - March 2013
Twitter Update for Social Developers London - March 2013Twitter Update for Social Developers London - March 2013
Twitter Update for Social Developers London - March 2013
Angus Fox
 
Web technology
Web technologyWeb technology
Web technology
Anuj Singh Rajput
 
HTTP Services & REST API Security
HTTP Services & REST API SecurityHTTP Services & REST API Security
HTTP Services & REST API Security
Taiseer Joudeh
 
Http
HttpHttp
Http
Eri Alam
 
Interactive web. O rly?
Interactive web. O rly?Interactive web. O rly?
Interactive web. O rly?
timbc
 
Understanding the Web through HTTP
Understanding the Web through HTTPUnderstanding the Web through HTTP
Understanding the Web through HTTP
Olivia Brundage
 
Secure Webservices
Secure WebservicesSecure Webservices
Secure Webservices
Matthias Käppler
 
Java Servlet
Java Servlet Java Servlet
Java Servlet
Rajiv Gupta
 
Http-protocol
Http-protocolHttp-protocol
Http-protocol
Toushik Paul
 
Ftp servlet
Ftp servletFtp servlet
Ftp servlet
Alexander Decker
 
Real-time Communications with SignalR
Real-time Communications with SignalRReal-time Communications with SignalR
Real-time Communications with SignalR
Shravan Kumar Kasagoni
 
Rest WebAPI with OData
Rest WebAPI with ODataRest WebAPI with OData
Rest WebAPI with OData
Mahek Merchant
 
Python tools for testing web services over HTTP
Python tools for testing web services over HTTPPython tools for testing web services over HTTP
Python tools for testing web services over HTTP
Mykhailo Kolesnyk
 

Similar to Twitter Streaming API (20)

OAuth for your API - The Big Picture
OAuth for your API - The Big PictureOAuth for your API - The Big Picture
OAuth for your API - The Big Picture
 
Networked APIs with swift
Networked APIs with swiftNetworked APIs with swift
Networked APIs with swift
 
Http_Protocol.pptx
Http_Protocol.pptxHttp_Protocol.pptx
Http_Protocol.pptx
 
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
HTTP by Hand: Exploring HTTP/1.0, 1.1 and 2.0
 
Testing REST Web Services
Testing REST Web ServicesTesting REST Web Services
Testing REST Web Services
 
ASP.NET Mvc 4 web api
ASP.NET Mvc 4 web apiASP.NET Mvc 4 web api
ASP.NET Mvc 4 web api
 
The Open Source... Behind the Tweets
The Open Source... Behind the TweetsThe Open Source... Behind the Tweets
The Open Source... Behind the Tweets
 
Twitter Update for Social Developers London - March 2013
Twitter Update for Social Developers London - March 2013Twitter Update for Social Developers London - March 2013
Twitter Update for Social Developers London - March 2013
 
Web technology
Web technologyWeb technology
Web technology
 
HTTP Services & REST API Security
HTTP Services & REST API SecurityHTTP Services & REST API Security
HTTP Services & REST API Security
 
Http
HttpHttp
Http
 
Interactive web. O rly?
Interactive web. O rly?Interactive web. O rly?
Interactive web. O rly?
 
Understanding the Web through HTTP
Understanding the Web through HTTPUnderstanding the Web through HTTP
Understanding the Web through HTTP
 
Secure Webservices
Secure WebservicesSecure Webservices
Secure Webservices
 
Java Servlet
Java Servlet Java Servlet
Java Servlet
 
Http-protocol
Http-protocolHttp-protocol
Http-protocol
 
Ftp servlet
Ftp servletFtp servlet
Ftp servlet
 
Real-time Communications with SignalR
Real-time Communications with SignalRReal-time Communications with SignalR
Real-time Communications with SignalR
 
Rest WebAPI with OData
Rest WebAPI with ODataRest WebAPI with OData
Rest WebAPI with OData
 
Python tools for testing web services over HTTP
Python tools for testing web services over HTTPPython tools for testing web services over HTTP
Python tools for testing web services over HTTP
 

Recently uploaded

20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
SitimaJohn
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
jpupo2018
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 

Recently uploaded (20)

20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 

Twitter Streaming API

  • 1. Do More, Faster The Twitter Streaming API
  • 2. 1. Create a HTTP Request GET /1/statuses/sample.json HTTP/1.1 Authorization: OAuth realm="",oauth_no... Host: stream.twitter.com Accept: */* Sign it with OAuth
  • 3. 2. Handle the response HTTP/1.1 200 OK Content-Type: application/json Transfer-Encoding: chunked Server: Jetty(6.1.25) {"in_reply_to_status_id_str":null,"geo":n ull,"text":"Hey fat boyyyyy!!... What’s missing? No Content-Length header! HTTP Libraries won’t know when to stop consuming
  • 4. 3. Keep handling it • Twitter will keep sending you JSON chunks until you kill the connection • Normal HTTP libraries make this awkward • Try non-blocking / event-based I/O
  • 5. The Streaming API lets you do cool stuff • Sample the firehose, free ~1% • Get interesting subsets of all tweets • By filtering, can get all tweets on a topic • Interact in near-realtime • Process more efficiently
  • 7. At Conversocial, we built a way to search twitter on behalf of our clients. There’s a stream client that receives tweets from the Streaming API
  • 8. We monitor the stream client from a separate thread
  • 9. We still have to poll the REST API to cover gaps between connections
  • 10. Need to disconnect stream client from processing tasks. We do this via a queue.
  • 11. Searches have to be matched back to the terms that produced them - the stream is multiplexed.
  • 12. Thanks for listening! gareth@conversocial.com http://www.github.com/gareth-lloyd/twistedstream

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n