TWITTER
AWESOMENESS

  Damon P. Cortesi
@DACORT

By Day                   By Night



Founder/Director of      Twitter Addict, including
Alchemy Security
        ...
TWEETSTATS?
 GRAPHING TWITTER SINCE 2007

Visualize your Twitter timeline - tweetstats.com
  Tweets per month/hour
  Top r...
CHALLENGES

1. Data, data, data
  Twitter allows retrieval of up to 3,200 tweets/user
  Whitelisting allows 20,000 API que...
TWEETSTATS V1
OMG, SCREENSCRAPING
TWEETSTATS V2
BEAUTIFUL GRAPHS
GRAPH LIBRARIES
                JAVASCRIPT

JavaScript



Honestly...
Kind of ugly


Limited functionality
Limited support
GRAPH LIBRARIES
                      FLASH

Flash - Interactive, beautiful

TweetStats currently uses FusionCharts Free e...
BUT WAIT...

Raphaël JS - Hot new JavaScript graphing library

  Vector Library - SVG/VML

  Impressive capabilities

  ht...
TWEETSTATS V2
  TWEETCLOUD
TWEETSTATS V2
TWEETCLOUD VIA WORDLE
USER EXPERIENCE

Average Twitter user attention span = seconds

Keep them informed
Queues and Quotes and Status Bars, oh m...
USER EXPERIENCE
        BACKGROUND QUEUES

Generate XML data
asynchronously

Activate charts via JavaScript
USER EXPERIENCE
                      K.I.S.S.


Tons of data, easy to get lost

Lots of features TweetStats _could_ have....
SCALING TWEETSTATS

The story of the Blockheads ...
SCALING TWEETSTATS
             WHY IT BROKE?

One server (affectionately named after @darkgracie)

  Web server + Rails s...
SCALING TWEETSTATS
     TYPICAL TWITTER TRAFFIC

Peaks and valleys

Influential tweet = immediate need for resources
SCALING TWEETSTATS
                 “FIXING” IT

Nods to:

  Amazon EC2

  Scalr.net



Architecture: Scaled!

Wallet: Emp...
POST-NKOTB




New avatar for @TweetStats ->
DATA MINING

Other services - TweepSearch, TweetSum

Background Queues are King

  TweetStats - BackgroundRB

  TweepSearc...
DATA MINING
                   VOLUME

TweepSearch - ~4.5M Twitter Profiles Indexed

  Just a small fraction of actual user...
DATA MINING
                   OPTIONS

Brute Force (historical)

  Crawl profiles, timelines

  Twitter Search - 1,500 res...
Other Glances
 Friends and Followers
Other Glances
 Friends and Followers
Other Glances
 Friends and Followers
Other Glances
 Friends and Followers
Other Glances
  Friends and favorites
Other Glances
  Friends and favorites
Other Glances
  Friends and favorites
THANKS

 TweetStats - Graph your Twitter Timeline
TweepSearch - Profile Search Awesomeness

            Damon P. Cortesi

...
Upcoming SlideShare
Loading in …5
×

Twitter Awesomeness

1,893 views

Published on

The story and process behind visualizing Twitter data as told by TweetStats creator, Damon Cortesi (@dacort).

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,893
On SlideShare
0
From Embeds
0
Number of Embeds
37
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Twitter Awesomeness

  1. 1. TWITTER AWESOMENESS Damon P. Cortesi
  2. 2. @DACORT By Day By Night Founder/Director of Twitter Addict, including Alchemy Security TweetStats Web app security Twitter Timeline Viz Security Product Dev TweepSearch Twitter Profile Search Firewall Visualization
  3. 3. TWEETSTATS? GRAPHING TWITTER SINCE 2007 Visualize your Twitter timeline - tweetstats.com Tweets per month/hour Top replies and apps TweetCloud Trend monitoring since September ’08 Global Twitter usage since December ’08 Over 250M tweets (no content) logged Thanks, Gnip (http://gnip.com)
  4. 4. CHALLENGES 1. Data, data, data Twitter allows retrieval of up to 3,200 tweets/user Whitelisting allows 20,000 API queries/hour Not uncommon now for 20k + tweets Disparate - timeline vs. friends/followers 2. User experience It takes time to gather data, graph
  5. 5. TWEETSTATS V1 OMG, SCREENSCRAPING
  6. 6. TWEETSTATS V2 BEAUTIFUL GRAPHS
  7. 7. GRAPH LIBRARIES JAVASCRIPT JavaScript Honestly... Kind of ugly Limited functionality Limited support
  8. 8. GRAPH LIBRARIES FLASH Flash - Interactive, beautiful TweetStats currently uses FusionCharts Free edition Gantt charts in disguise!!
  9. 9. BUT WAIT... Raphaël JS - Hot new JavaScript graphing library Vector Library - SVG/VML Impressive capabilities http://raphaeljs.com
  10. 10. TWEETSTATS V2 TWEETCLOUD
  11. 11. TWEETSTATS V2 TWEETCLOUD VIA WORDLE
  12. 12. USER EXPERIENCE Average Twitter user attention span = seconds Keep them informed Queues and Quotes and Status Bars, oh my!
  13. 13. USER EXPERIENCE BACKGROUND QUEUES Generate XML data asynchronously Activate charts via JavaScript
  14. 14. USER EXPERIENCE K.I.S.S. Tons of data, easy to get lost Lots of features TweetStats _could_ have... Important to keep the experience simple And this doesn’t pay the bills ... yet. ;) TweetStats v3 - More features? More analytics?
  15. 15. SCALING TWEETSTATS The story of the Blockheads ...
  16. 16. SCALING TWEETSTATS WHY IT BROKE? One server (affectionately named after @darkgracie) Web server + Rails stack Database w/150,000 tables = #FAIL Various other applications Synchronous DB queries Thousands of NKOTB Fans
  17. 17. SCALING TWEETSTATS TYPICAL TWITTER TRAFFIC Peaks and valleys Influential tweet = immediate need for resources
  18. 18. SCALING TWEETSTATS “FIXING” IT Nods to: Amazon EC2 Scalr.net Architecture: Scaled! Wallet: Empty.
  19. 19. POST-NKOTB New avatar for @TweetStats ->
  20. 20. DATA MINING Other services - TweepSearch, TweetSum Background Queues are King TweetStats - BackgroundRB TweepSearch - Workling TweetSum - BackgroundJob Why so many? Can’t judge without experience.
  21. 21. DATA MINING VOLUME TweepSearch - ~4.5M Twitter Profiles Indexed Just a small fraction of actual users TweetStats - Global twitter usage via Gnip Previously - 2M updates/day Now - ~10M updates/day ~9,000 updates/minute aka ... my servers hate me.
  22. 22. DATA MINING OPTIONS Brute Force (historical) Crawl profiles, timelines Twitter Search - 1,500 results or 20-30(??) days Official Twitter Streaming API “spritzer”, “gardenhose”, “firehose” streams Gnip - Not just Twitter!
  23. 23. Other Glances Friends and Followers
  24. 24. Other Glances Friends and Followers
  25. 25. Other Glances Friends and Followers
  26. 26. Other Glances Friends and Followers
  27. 27. Other Glances Friends and favorites
  28. 28. Other Glances Friends and favorites
  29. 29. Other Glances Friends and favorites
  30. 30. THANKS TweetStats - Graph your Twitter Timeline TweepSearch - Profile Search Awesomeness Damon P. Cortesi @dacort dacort@tweetstats.com

×