Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Yusuke Wada, profile picture
Uploaded byYusuke Wada
PDF, PPTX1,851 views

YAPC::Asia 2010 Twitter解析サービス

This document describes the architecture of a Twitter scraping application. It uses Gearman for asynchronous job processing to fetch Twitter data from the API or stream. Workers register fetch jobs and process the requests. The application includes modules for the CLI, reader, and streaming components and uses JSON and LWP libraries. It retrieves tweets and stores them in a database for analysis.

•  •  •  • 
•  • 
• 
•  –  •  –  •  –  •  – 
•  • 
•  •  •  • 
•  •  • 
•  •  •  • 
•  •  •  • 
•  • 
•  • 
yusuke@dev:~/work/twib/Twib/trunk$ ls  bot  cli  etc  migration  schema  web 
•  •  •  •  • 
•  Package Twib::CLI::Reader::Feed;  use JSON::XS;  use LWP::UserAgent;  Package Twib::CLI::Reader::Stream;  use AnyEvent::Twitter::Stream; 
•  • 
•  –  •  –  •  – 
#API Reader  my $client = Gearman::Client‐>new;  $client‐>job_servers(qw|localhost:7003|);  $client‐>dispatch_background( "fetch", $json, {} );  #Worker  my $worker = Gearman::Worker‐>new;  $worker‐>job_servers(qw|localhost:7003|);  $worker‐>register_function( "fetch" => &fetch );  $worker‐>work while 1;  sub fetch {      my $job      = shift;      my $args = $job‐>arg;      $fetcher‐>fetch( decode_json($args) );  } 
•  –  –  •  –  •  –  •  – 
–  –  –  –  –  –  –  –  – 
•  •  •  • 
•  –  •  – 
•  •  •  • 
•  •  •  –  • 
•  •  •  –  – 
•  –  •  –  • 
YAPC::Asia 2010 Twitter解析サービス