We are losing our tweets!An analysis, a prototype, lessons learned, and proposed third party solution to the problemJohn O’Brien III@jobrieniiihttp://www.linkedin.com/in/jobrieniii

Twitter “Primer”Social network / micro blogging siteSend / read 140 character messagesYou can follow anyone, and they can follow youSent messages are delivered to all your followersSent messages are also publically indexed and searchablePermissions can be established to restrict delivery, but this is not the norm

ProblemAs the usage of Twitter has exploded, Twitter’s ability to provide long term access to tweets that mention key events (typically #hashtag’ed) has eroded

First, who cares?IndividualsBloggersConference Attendees / LeadersAcademia / “Web” EcologistsMedia OutletsCompaniesGovernment

So lets dive into the problem...FollowersSearch

Search UI / API ConstraintsLimited to keywords, #hashtags, or @mentions within 140 char body of tweet 100 tweets x 15 pages = 1500 per search termFor a given keyword, exists in search for “around 1.5 weeks but is dynamic and subject to shrink as the number of tweets per day continues to grow.” – Twitter website

Hmmmm….No other ‘in the cloud’ sites were found back in June, only client side applications and ‘hacked’ custom scriptsRSS feeds were considered but initially dismissed because they typically require an end user clientDecision was to “build our own” and see if we can solve the problem

A little bit about my thoughts on the SDLC process…**FOCUS**ONLEARNING“Minimally Viable”PROTOTYPE

“Minimally Viable” Micro AppWhat if we could get ahead of the problem and store the data before Twitter “loses” it?Functional RequirementsAbility for user to define #hashtags of importanceCreate a background script that leverages the Twitter /search REST API to keep an eye on each hash tag and store data in local database**Sweep, grab, and record…**Must be running at all times and publically availableTechnical SpecsBuild on LAMP stack, put into the cloud, running 24/7/365

“Minimally Viable” Micro Appinternetphp script to query each#hashtagTwitter/searchAPIOur Database

TwapperKeeper.com “BETA”was born on Saturday and released to public on Sunday…

And we started to grow and get customer feedback…

And we lived through a key world event…http://mashable.com/2009/09/16/white-house-records/

So what did we learn?We need to be whitelistedPeople often don’t start the archiving until after they start using #hashtagsThus, point forward solution not enough, need to reach back as wellWhile hashtags are the norm, some people would just like to track keywordsVelocity of tweets can be a major issueWhat if a hashtag results are greater than 1500 tweets per minute? Hashtags of archive interest typically spike in velocity and die off in traffic.However some archives get VERY, VERY big!

And more learning…URL shortening services are of long time concern to users and archiving communityTwitter /search REST API periodically is unresponsive Twitter /search REST API sometimes glitches and returns duplicate dataPeople want not only output in html, but raw exports for publication, analysis and real time consumption (txt, csv, xml, json, etc)Twitter engineers contact us and recommend also incorporating newly releasedreal time streams /track, /sample , /firehose

Recommended “out-of-beta” V2.0Anticipate #hashtags to archive based upon Twitter trending stats and autocreate archivesHybrid approach of using /search and /track (real time stream) APIs to handle velocity issuesCheck for duplicates “before” insertsImplement monitoring and “self healing” servicesShortened URLs should be resolved into fully qualified URLs and stored separately for reference (at time of capture)Create TwapperKeeper API by modularizing the archiving engine into a SOA architecture (/create, /info, /get) for internal and external consumptionInclude additional output formats to be provided for download“Extracts” of large archives should be automatically generated on a daily basis and made available for downloadVERSION 2.0

Recommended “out-of-beta” V2.0Twitter/trackAPIhybrid php / curl scriptto archive per #hashtagMonitor Health and Self HealTwitter/searchAPIautocreate trendsTwitter/trendsAPIOur DatabaseFile extractorapi/create/info/getexternal sitesshort url lookup

Questions?