Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Twitter Awesomeness


Published on

The story and process behind visualizing Twitter data as told by TweetStats creator, Damon Cortesi (@dacort).

Published in: Technology
  • Be the first to comment

Twitter Awesomeness

  1. 1. TWITTER AWESOMENESS Damon P. Cortesi
  2. 2. @DACORT By Day By Night Founder/Director of Twitter Addict, including Alchemy Security TweetStats Web app security Twitter Timeline Viz Security Product Dev TweepSearch Twitter Profile Search Firewall Visualization
  3. 3. TWEETSTATS? GRAPHING TWITTER SINCE 2007 Visualize your Twitter timeline - Tweets per month/hour Top replies and apps TweetCloud Trend monitoring since September ’08 Global Twitter usage since December ’08 Over 250M tweets (no content) logged Thanks, Gnip (
  4. 4. CHALLENGES 1. Data, data, data Twitter allows retrieval of up to 3,200 tweets/user Whitelisting allows 20,000 API queries/hour Not uncommon now for 20k + tweets Disparate - timeline vs. friends/followers 2. User experience It takes time to gather data, graph
  7. 7. GRAPH LIBRARIES JAVASCRIPT JavaScript Honestly... Kind of ugly Limited functionality Limited support
  8. 8. GRAPH LIBRARIES FLASH Flash - Interactive, beautiful TweetStats currently uses FusionCharts Free edition Gantt charts in disguise!!
  9. 9. BUT WAIT... Raphaël JS - Hot new JavaScript graphing library Vector Library - SVG/VML Impressive capabilities
  12. 12. USER EXPERIENCE Average Twitter user attention span = seconds Keep them informed Queues and Quotes and Status Bars, oh my!
  13. 13. USER EXPERIENCE BACKGROUND QUEUES Generate XML data asynchronously Activate charts via JavaScript
  14. 14. USER EXPERIENCE K.I.S.S. Tons of data, easy to get lost Lots of features TweetStats _could_ have... Important to keep the experience simple And this doesn’t pay the bills ... yet. ;) TweetStats v3 - More features? More analytics?
  15. 15. SCALING TWEETSTATS The story of the Blockheads ...
  16. 16. SCALING TWEETSTATS WHY IT BROKE? One server (affectionately named after @darkgracie) Web server + Rails stack Database w/150,000 tables = #FAIL Various other applications Synchronous DB queries Thousands of NKOTB Fans
  17. 17. SCALING TWEETSTATS TYPICAL TWITTER TRAFFIC Peaks and valleys Influential tweet = immediate need for resources
  18. 18. SCALING TWEETSTATS “FIXING” IT Nods to: Amazon EC2 Architecture: Scaled! Wallet: Empty.
  19. 19. POST-NKOTB New avatar for @TweetStats ->
  20. 20. DATA MINING Other services - TweepSearch, TweetSum Background Queues are King TweetStats - BackgroundRB TweepSearch - Workling TweetSum - BackgroundJob Why so many? Can’t judge without experience.
  21. 21. DATA MINING VOLUME TweepSearch - ~4.5M Twitter Profiles Indexed Just a small fraction of actual users TweetStats - Global twitter usage via Gnip Previously - 2M updates/day Now - ~10M updates/day ~9,000 updates/minute aka ... my servers hate me.
  22. 22. DATA MINING OPTIONS Brute Force (historical) Crawl profiles, timelines Twitter Search - 1,500 results or 20-30(??) days Official Twitter Streaming API “spritzer”, “gardenhose”, “firehose” streams Gnip - Not just Twitter!
  23. 23. Other Glances Friends and Followers
  24. 24. Other Glances Friends and Followers
  25. 25. Other Glances Friends and Followers
  26. 26. Other Glances Friends and Followers
  27. 27. Other Glances Friends and favorites
  28. 28. Other Glances Friends and favorites
  29. 29. Other Glances Friends and favorites
  30. 30. THANKS TweetStats - Graph your Twitter Timeline TweepSearch - Profile Search Awesomeness Damon P. Cortesi @dacort