TWITTER
AWESOMENESS
Damon P. Cortesi

@DACORT
By Day By Night
Founder/Director of Twitter Addict, including
Alchemy Security
TweetStats
Web app security Twitter Timeline Viz
Security Product Dev TweepSearch
Twitter Profile Search
Firewall Visualization

TWEETSTATS?
GRAPHING TWITTER SINCE 2007
Visualize your Twitter timeline - tweetstats.com
Tweets per month/hour
Top replies and apps
TweetCloud
Trend monitoring since September ’08
Global Twitter usage since December ’08
Over 250M tweets (no content) logged
Thanks, Gnip (http://gnip.com)

CHALLENGES
1. Data, data, data
Twitter allows retrieval of up to 3,200 tweets/user
Whitelisting allows 20,000 API queries/hour
Not uncommon now for 20k + tweets
Disparate - timeline vs. friends/followers
2. User experience
It takes time to gather data, graph

TWEETSTATS V1
OMG, SCREENSCRAPING

TWEETSTATS V2
BEAUTIFUL GRAPHS

GRAPH LIBRARIES
JAVASCRIPT
JavaScript
Honestly...
Kind of ugly
Limited functionality
Limited support

GRAPH LIBRARIES
FLASH
Flash - Interactive, beautiful
TweetStats currently uses FusionCharts Free edition
Gantt charts in disguise!!

BUT WAIT...
Raphaël JS - Hot new JavaScript graphing library
Vector Library - SVG/VML
Impressive capabilities
http://raphaeljs.com

TWEETSTATS V2
TWEETCLOUD

TWEETSTATS V2
TWEETCLOUD VIA WORDLE

USER EXPERIENCE
Average Twitter user attention span = seconds
Keep them informed
Queues and Quotes and Status Bars, oh my!

USER EXPERIENCE
BACKGROUND QUEUES
Generate XML data
asynchronously
Activate charts via JavaScript

USER EXPERIENCE
K.I.S.S.
Tons of data, easy to get lost
Lots of features TweetStats _could_ have...
Important to keep the experience simple
And this doesn’t pay the bills ... yet. ;)
TweetStats v3 - More features? More analytics?

SCALING TWEETSTATS
The story of the Blockheads ...

SCALING TWEETSTATS
WHY IT BROKE?
One server (affectionately named after @darkgracie)
Web server + Rails stack
Database w/150,000 tables
= #FAIL
Various other applications
Synchronous DB queries
Thousands of NKOTB Fans

SCALING TWEETSTATS
TYPICAL TWITTER TRAFFIC
Peaks and valleys
Influential tweet = immediate need for resources

SCALING TWEETSTATS
“FIXING” IT
Nods to:
Amazon EC2
Scalr.net
Architecture: Scaled!
Wallet: Empty.

POST-NKOTB
New avatar for @TweetStats ->

DATA MINING
Other services - TweepSearch, TweetSum
Background Queues are King
TweetStats - BackgroundRB
TweepSearch - Workling
TweetSum - BackgroundJob
Why so many? Can’t judge without experience.

DATA MINING
VOLUME
TweepSearch - ~4.5M Twitter Profiles Indexed
Just a small fraction of actual users
TweetStats - Global twitter usage via Gnip
Previously - 2M updates/day
Now - ~10M updates/day
~9,000 updates/minute
aka ... my servers hate me.

DATA MINING
OPTIONS
Brute Force (historical)
Crawl profiles, timelines
Twitter Search - 1,500 results or 20-30(??) days
Official Twitter Streaming API
“spritzer”, “gardenhose”, “firehose” streams
Gnip - Not just Twitter!

Other Glances
Friends and Followers

Other Glances
Friends and Followers

Other Glances
Friends and Followers

Other Glances
Friends and Followers

Other Glances
Friends and favorites

Other Glances
Friends and favorites

Other Glances
Friends and favorites

THANKS
TweetStats - Graph your Twitter Timeline
TweepSearch - Profile Search Awesomeness
Damon P. Cortesi
@dacort
dacort@tweetstats.com