This document discusses collecting photos from Twitter streams in real-time using Node.js on Heroku. It describes tracking hashtags and photo URLs to filter tweets containing images, and extracting the photo URLs. The architecture involves using the Twitter Streaming API and socket.io for real-time streaming to browsers, with fallbacks for Heroku which does not yet support WebSockets. The goal is to present the photo stream to users in real-time.

1. Collecting Photo Stream
in Node.js on Heroku
@stomita (ja)
@shinichitomita (en)

2. Who am I ?
• CEO at Mashmatrix, Inc.
• JavaScript Developer
• Formerly (almost forgot)
• Identity (OpenID/SAML/SSO) @stomita
• JavaEE
• Oracle
• Ex-Salesforce.com (2006/12-2007/12)

3. Objective
• Collect and store photos tweeted in the
world
• Add metadata to photos manually by crowd
• Category ? (food/people/landscape)
• Venue ? (foursquare)

4. Twitter Streaming API
https://dev.twitter.com/docs/streaming-api

5. Photo Stream
• Set tracking keywords to filter tweets with
photo uploading services URL
• e.g.
tracks=lockerz,plixi,tweetphoto,twitpic,yfrog,
photozou,twitgoo,instagr
• Extract photo URLs by pre-given rule
• e.g. http://twitpic.com/7wiwlw =>
http://twitpic.com/show/thumb/7wiwlw

6. Architecture
Streaming JSON

7. Twitter-Node
https://github.com/technoweenie/twitter-node

8. Visualization
• Present photo stream to users in real-time
• Not our primary goal, but just interesting
• Don’t think, FEEL.

9. tweet-photo-stream
http://tweet-photo-stream.herokuapp.com/
https://github.com/stomita/tweet-photo-stream

10. socket.io
• A Node.js library, enabling push messaging
to browsers
• Stream Transport Supported :
• WebSocket/FlashSocket/XHR-Multipart/
XHR-polling/JSONP-polling
• Used as fallback of SocketStream

11. Architecture
Streaming socket.io

12. Similar Work - Twitcaps
http://twitcaps.com/

13. Twitcaps - Not Real-time
Delayed Streaming Ajax Polling

14. Issue
• Heroku Node.js doesn’t support
WebSocket yet (as of Dec 2011)
• Fallbacks to XHR-polling
• c.f. Joyent No.de SmartMachine

15. End