The document summarizes the current server configuration and archiving processes for the Twapper Keeper application as of April 25, 2010. It notes issues around high server load, lack of optimized backups, and lack of visibility into system issues. The plan is to upgrade server resources, improve backups, add monitoring and a health status page, allow archive resets to requery tweets, and implement OAuth for API calls by May 14th to address these issues.

1. Twapper Keeper Operations / Infrastructure Enhancements04/25/2010

2. High Level Server Configuration – as of 4/25/2010 Twitter Primary Twapper Keeper Server /track Streaming API /search REST API mysql apache jobs export Backup Twapper Keeper Server Backup extract

3. Primary Archiving Process for Hashtags and Keywords – as of 4/25/2010 Twitter Primary Twapper Keeper Server Jobs mysql apache phire_track.php create persistent connection to Twitter Streaming API for each archive predicate listed in twappers, and store tweets in rawstream table [1 instance] twappers rawstream /track Streaming API sweeper_stream.php pull chunks of tweets, compare to archives on file, insert Into proper archive table, delete from processing when complete with chunk [11 instances] archive tables z_{notebook_id}

4. New / Missed Archiving Process for Hashtags and Keywords – as of 4/25/2010 Twitter Primary Twapper Keeper Server Jobs mysql apache sweeper_new.php When archive is first created, it is flagged as new. This process picks up new archives and uses Twitter /search REST API to find all tweets available and stores them in appropriate archive. Once more than 1 tweet is found it unflags archive as no longer new. (NOTE: Sometimes calls are rate limited and thus need to ensure some data is rec’d before unflagging.) twappers sweeper_missed.php For each archive in the system, this is constantly crawling the Twitter /search REST API for all tweets in Twitter cache, checks to make sure they have been archived, and if not archives them. This is important in during Streaming API reconnects or prolonged stream disconnects. archive tables z_{notebook_id}

5. New / Primary Archiving Process for Persons – as of 4/25/2010 Twitter Primary Twapper Keeper Server Jobs mysql apache sweeper_personal_new.php When archive is first created, it set flagged as new. This process picks up new archives and uses Twitter /timeline REST API to find all tweets available and stores them in appropriate archive. Once more than 1 tweet is found it unflags archive as no longer new. (NOTE: Sometimes calls are rate limited and thus need to ensure some data is rec’d before unflagging.) twappers sweeper_personal.php for each keyword and hashtag archive in the system, this is constantly crawling the Twitter /search REST API for all tweets in Twitter cache, checks to make sure they have been archived, and if not archives them. This is important in during Streaming API reconnects or prolonged disconnects. archive tables z_{notebook_id}

7. Users do not have automatic visibility to system issue and have to ask questions too often about archives, especially when things are queued in the RAWSTREAM table awaiting processing

8. There is contention for access to RAWSTREAM table causing a latency on the inbound /STREAM HTTP connection resulting in 30-40 min latency, which can cause lost tweets during reconnects (which happen when new archives are created)

10. Upgrade the Primary Twapper Keeper server VPS configuration to include larger CPU and RAM configuration

11. Procure additional disk space for Primary and Backup server

12. Incorporate automated export / import routine for backup server and rsynch apache and jobs files to ensure secondary server can be brought online more rapidly

13. Implement a layer of monitoring for administrator and provide a “system health” page for users for transparency

14. Provide a user feature to “reset” archives to kick off the requery process if they are in a hurry to see tweets (will force re-call of /search instead of waiting for streaming API)

15. Implement OAuth for all REST API calls

16. Implement an approach to reduce contention on RAWSTREAM table to reduce latency on inbound Streaming API

17. Timeline

18. Target completion by May 14, 2010