By Day By Night
Founder/Director of Twitter Addict, including
Web app security Twitter Timeline Viz
Security Product Dev TweepSearch
Twitter Proﬁle Search
GRAPHING TWITTER SINCE 2007
Visualize your Twitter timeline - tweetstats.com
Tweets per month/hour
Top replies and apps
Trend monitoring since September ’08
Global Twitter usage since December ’08
Over 250M tweets (no content) logged
Thanks, Gnip (http://gnip.com)
1. Data, data, data
Twitter allows retrieval of up to 3,200 tweets/user
Whitelisting allows 20,000 API queries/hour
Not uncommon now for 20k + tweets
Disparate - timeline vs. friends/followers
2. User experience
It takes time to gather data, graph
Average Twitter user attention span = seconds
Keep them informed
Queues and Quotes and Status Bars, oh my!
Generate XML data
Tons of data, easy to get lost
Lots of features TweetStats _could_ have...
Important to keep the experience simple
And this doesn’t pay the bills ... yet. ;)
TweetStats v3 - More features? More analytics?
The story of the Blockheads ...
WHY IT BROKE?
One server (affectionately named after @darkgracie)
Web server + Rails stack
Database w/150,000 tables
Various other applications
Synchronous DB queries
Thousands of NKOTB Fans
TYPICAL TWITTER TRAFFIC
Peaks and valleys
Inﬂuential tweet = immediate need for resources
Other services - TweepSearch, TweetSum
Background Queues are King
TweetStats - BackgroundRB
TweepSearch - Workling
TweetSum - BackgroundJob
Why so many? Can’t judge without experience.
TweepSearch - ~4.5M Twitter Proﬁles Indexed
Just a small fraction of actual users
TweetStats - Global twitter usage via Gnip
Previously - 2M updates/day
Now - ~10M updates/day
aka ... my servers hate me.
Brute Force (historical)
Crawl proﬁles, timelines
Twitter Search - 1,500 results or 20-30(??) days
Ofﬁcial Twitter Streaming API
“spritzer”, “gardenhose”, “ﬁrehose” streams
Gnip - Not just Twitter!