Asynchronous Programming FTW! 2 (with AnyEvent)
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Asynchronous Programming FTW! 2 (with AnyEvent)

  • 2,915 views
Uploaded on

This is a revamped Asynchronous Programming FTW talk, given at YAPC::NA 2013, Cluj.pm, and AmsterdamX.pm.

This is a revamped Asynchronous Programming FTW talk, given at YAPC::NA 2013, Cluj.pm, and AmsterdamX.pm.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,915
On Slideshare
2,907
From Embeds
8
Number of Embeds
2

Actions

Shares
Downloads
23
Comments
0
Likes
4

Embeds 8

https://twitter.com 7
https://duckduckgo.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. AsynchronousAsynchronous ProgrammingProgramming FTW!FTW!(Sawyer X) @PerlSawyer Theory Practice
  • 2. TheoryTheory We think in non-blocking terms We act in non-blocking terms We program in blocking terms An example? Sure!
  • 3. The exampleThe example use LWP::UserAgent; my @urls = ( 'https://duckduckgo.com', 'http://cluj.pm', 'http://telaviv.pm.org', ); my $ua = LWP::UserAgent->new; foreach my $url (@urls) { my $res = $ua->get($url); say $res->decoded_content; }
  • 4. Problem?Problem? Wait on each URL Ineffecient waste of time Pointless Senseless CRIME AGAINST HUMANITY (maybe just ineffecient)
  • 5. How we thinkHow we think Nothing to work on? Defer to later! Like cooking!
  • 6. Making dinnerMaking dinner Boil water. Wait? No! Salad time! (water gets deferred to later) Cook potatoes. Wait? No! Sauce time! (potatoes get deferred to later) We think and act non-blocking
  • 7. TIMTOWTDITIMTOWTDI Forks Threads Event-loops <- this talk
  • 8. Event loop?Event loop? Single thread, single process Like a big while(1) {} Heavy usage of code references No real race conditions No locking, semaphores, etc. Can yield better speed But... nothing can be blocking! (I/O)
  • 9. Event loops in PerlEvent loops in Perl POE Reflex IO::Async IO::Lambda Qt, GTK, Tk, Glib, Cocoa::EventLoop, EV, FLTK, Irssi AnyEvent <- this talk
  • 10. AnyEventAnyEvent Light-weight Fast Very fast Slim interface Good support for other event loops Lots of modules
  • 11. PracticePractice Cooking with AnyEventCooking with AnyEvent (timers)(timers) use AnyEvent; say '>> Water boiling...'; my $water = AnyEvent->timer( after => 5*60, cb => sub { say '<< Water boiled!'; say '>> Potatos being prepared...'; my $potatoes; $potatoes = AnyEvent->timer( after => 10*60, cb => sub { undef $potatoes; say '<< Potatos ready!'; } ); } ); say '>> Salad being prepared...'; my $salad = AnyEvent->timer( after => 6*60, cb => sub { say '<< Salad ready!'; say '>> Sauce being prepared...'; my $sauce; $sauce = AnyEvent->timer( after => 12*60, cb => sub { undef $sauce; say '<< Sauce ready!'; } ); } );
  • 12. Water - 5 minutes Potatos - 10 minutes after water Salad - 6 minutes Sauce - 12 minutes after salad Water, salad, potatoes, sauce >> Water boiling... >> Salad being prepared... << Water boiled! >> Potatos being prepared... << Salad ready! >> Sauce being prepared... << Potatos ready! << Sauce ready!
  • 13. Condition variablesCondition variables A condition waiting to be fulfilled Starting at false, hoping to get to true Helps waiting for other events to finish Call ->recv() to wait for stuff to finish Call ->send() when finished doing your stuff Without condition variables, we reach the end of programs We reach the end == program closes Program closes == our code doesn't get executed Our code doesn't get executed == we get executed!
  • 14. Simple exampleSimple example use AnyEvent; my $count = 1; my $cv = AnyEvent->condvar; my $timer = AnyEvent->timer( interval => 1, cb => sub { say "Count: $count"; $count++ == 5 and $cv->send; }, ); $cv->recv; say 'We counted to 5!';
  • 15. Bigger exampleBigger example my $cv = AnyEvent->condvar; my $water = AnyEvent->timer( after => 5*60, cb => sub { my $potatoes; $potatoes = AnyEvent->timer( after => 10*60, cb => sub { undef $potatoes; } ); } ); my $salad = AnyEvent->timer( after => 6*60, cb => sub { my $sauce; $sauce = AnyEvent->timer( after => 12*60, cb => sub { undef $sauce; $cv->send; } ); } ); $cv->recv;
  • 16. Better example :: LWPBetter example :: LWP use LWP::UserAgent; my @urls = ( 'https://whatismyip.com', 'http://checkip.dyndns.org', 'http://wtfismyipaddress.org', ); my $ua = LWP::UserAgent->new; foreach my $url (@urls) { my $res = $ua->get($url); say $res->decoded_content; }
  • 17. Better example :: AnyEventBetter example :: AnyEvent use AnyEvent; use AnyEvent::HTTP; my $cv = AnyEvent->condvar; foreach my $url (@urls) { $cv->begin; http_get $url, sub { my $body = shift; say $body; $cv->end; } } $cv->recv;
  • 18. POP QUIZ TIME!POP QUIZ TIME! A few examples Spot the error(s) and get a prize Available prizes: handshake, high five, hug, kiss (not necessarily from me)
  • 19. Round 1Round 1 use AnyEvent; use AnyEvent::HTTP; http_post $url, $body, sub { say 'Successfully made an API call to do something cool!'; }; # wait for all events to finish AnyEvent->condvar->recv;
  • 20. Round 2Round 2 use AnyEvent; my $timer = AnyEvent->timer( interval => 5, cb => sub { sleep 1; say 'Bwahaha!'; } );
  • 21. Round 3Round 3 use AnyEvent; my $cv = AnyEvent->condvar; my $player1 = AnyEvent->timer( after => 3.7, cb => sub { say 'Player 1 joins the game!'; $cv->send; }, ); my $player2 = AnyEvent->timer( after => 7.2, cb => sub { say 'Player 2 joins the game!'; $cv->send; }, ); $cv->recv;
  • 22. Round 4Round 4 use AnyEvent; use HTTP::Tiny; my $cv = AnyEvent->condvar; $cv->begin for 1 .. 2; my $player1 = AnyEvent->timer( after => 3.7, cb => sub { say 'Player 1 joins the game! Alerting server!'; my $res = HTTP::Tiny->new->get('http://server:3030/add/1'); $res->{'success'} or die 'Failed to insert player 1 to game!'; $cv->end; }, ); my $player2 = AnyEvent->timer( after => 7.2, cb => sub { say 'Player 2 joins the game!'; $cv->end; }, ); $cv->recv;
  • 23. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface
  • 24. WWW::xkcdWWW::xkcd use WWW::xkcd; my $xkcd = WWW::xkcd->new; $xkcd->fetch( sub { my ( $img, $comic ) = @_; say "Today's comic is titled: ", $comic->{'title'}; ... } );
  • 25. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server
  • 26. TwiggyTwiggy use Twiggy::Server; my $app = sub { # PSGI app }; my $server = Twiggy::Server->new(...); $server->register_service($app); AnyEvent->condvar->recv;
  • 27. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface
  • 28. AnyEvent::EC2::TinyAnyEvent::EC2::Tiny use AnyEvent::EC2::Tiny; my $ec2 = AnyEvent::EC2::Tiny->new(...); my $xml = $ec2->send( 'RegionName.1' => 'us-east-1', Action => 'DescribeRegions', success_cb => sub { my $xml = shift; # prints ec2.us-east-1.amazonaws.com say $xml->{'regionInfo'}{'item'}[0]{'regionEndpoint'}; }, fail_cb => sub { my $error = shift; ... }, );
  • 29. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework
  • 30. JunoJuno use Juno; my $juno = Juno->new( hosts => [ 'server1', 'server2' ], interval => 10, checks => { HTTP => { headers => { { 'Host', 'example.com' }, }, on_result => sub { my $result = shift; ... }, }, }, );
  • 31. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework Real-time network path resolution
  • 32. SIP RouteSIP Route use AnyEvent; use Anyevent::Util 'fork_call'; use AnyEvent::Socket; use AnyEvent::Handle; use Data::Dump 'dump'; my $tcp_server_guard = tcp_server '127.0.0.1', 2020, sub { my ($fh) = @_; my $handle = AnyEvent::Handle->new( fh => $fh ); $handle->push_read( line => sub {...} ); };
  • 33. my ( $hdl, $target ) = @_; if ( ! defined $target ) { $hdl->push_write($default_route); $hdl->destroy; return; } chomp $target; if ( ! exists $data{$target} ) { $hdl->push_write($default_route); $hdl->destroy; return; }
  • 34. my @sorted = sort { $data{$target}{$a} <=> $data{$target}{$b} } keys %{ $data{$target} }; $hdl->push_write( $sorted[0] ); $hdl->destroy;
  • 35. foreach my $target (@targets) { foreach my $iface (@interfaces) { my $cmd = "sipsak --sip-uri=sip:$target -D $timeout -X $iface"; push @timers, AE::timer $start_after, $check_every, sub { my $time = AE::now; fork_call { system "$cmd >/dev/null 2>&1"; return $? >> 8; } sub { my $exit = shift; ( $exit == 0 || $exit == 1 ) or return delete $data{$target}{$iface}; $data{$target}{$iface} = AE::now - $time; }; }; } } AE::cv->recv;
  • 36. if ($verbose) { push @timers, AE::timer 5, 5, sub { dump %data }; }
  • 37. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework Real-time network path resolution API interfaces to services
  • 38. AnyEvent::RiakAnyEvent::Riak use AnyEvent::Riak; my $riak = AnyEvent::Riak->new( host => 'http://127.0.0.1:8098', path => 'riak', ); $riak->list_bucket( 'mybucket', {props => 'true', keys => 'false'}, sub { my $res = shift; ... } );
  • 39. AnyEvent::SNMPAnyEvent::SNMP use AnyEvent::SNMP; use Net::SNMP; my $cv = AnyEvent->condvar; Net::SNMP->session( -hostname => "127.0.0.1", -community => "public", -nonblocking => 1 )->get_request( -callback => sub { $cv->send (@_) } ); my @result = $cv->recv;
  • 40. AnyEvent::XMLRPCAnyEvent::XMLRPC use AnyEvent::XMLRPC; my $serv = AnyEvent::XMLRPC->new( methods => { 'echo' => &echo, }, );
  • 41. AnyEvent::DNSAnyEvent::DNS use AnyEvent::DNS; my $cv = AnyEvent->condvar; AnyEvent::DNS::a "www.google.ro", $cv; # later... my @addrs = $cv->recv;
  • 42. AnyEvent::TwitterAnyEvent::Twitter my $ua = AnyEvent::Twitter->new( consumer_key => 'consumer_key', consumer_secret => 'consumer_secret', access_token => 'access_token', access_token_secret => 'access_token_secret', ); my $cv = AnyEvent->condvar; # GET request $cv->begin; $ua->get( 'account/verify_credentials', sub { my ($header, $response, $reason) = @_; say $response->{screen_name}; $cv->end; } );
  • 43. AnyEvent::GraphiteAnyEvent::Graphite my $graphite = AnyEvent::Graphite->new( host => '127.0.0.1', port => '2003', ); $graphite->send("a.b.c.d", $value, $timestamp);
  • 44. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework Real-time network path resolution API interfaces to services Find matching special actors in TV series
  • 45. Actor matchesActor matches use AnyEvent; use AnyEvent::HTTP; my $base = 'http://www.imdb.com/title'; my %titles = ( tt1196946 => { name => 'Mentalist, the' }, tt1219024 => { name => 'Castle' }, tt0773262 => { name => 'Dexter' }, tt0491738 => { name => 'Psych' }, tt0247082 => { name => 'CSI Las Vegas' }, tt0458253 => { name => 'Closer, the' }, ); # DON'T PARSE HTML WITH REGEX!!!!1127 my $episode_regex = qr{...}; my $cast_regex = qr{...};
  • 46. # Fetching number of seasons for each title my $cv = AE::cv; foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'}; print "[$title] Fetching number of seasons.n"; my $url = "$base/$title_id/"; my $regex = qr{/title/$title_id/episodes?season=([0-9]+)}; $cv->begin; http_get $url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$regex/mg; $titles{$title_id}{'episodes'}{$_} = [] for @found; $cv->end; }; } $cv->recv;
  • 47. # fetching the list of episodes for each season foreach my $title_id ( keys %titles ) { my $cv = AE::cv; foreach my $season ( keys %{ $titles{$title_id}{'episodes'} } ) { my $title = $titles{$title_id}{'name'}; print "[$title] [Season $season] fetching episodesn"; my $season_url = "$base/$title_id/episodes?season=$season"; $cv->begin; http_get $season_url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$episode_regex/mg; while (@found) { my $id = shift @found; my $name = shift @found;
  • 48. foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'}; foreach my $season ( keys %{ $titles{$title_id}{'episodes'} } ) { my @episodes = @{ $titles{$title_id}{'episodes'}{$season} }; print "[$title] [$season] Fetching cast for season $seasonn"; my $cv = AE::cv; foreach my $episode_id (@episodes) { print " -> [$title] [$season] $episode_idn"; my $url = "$base/$episode_id/fullcredits"; $cv->begin; http_get $url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$cast_regex/mg;
  • 49. print "Populating actorsn"; my %actors = (); foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'}; foreach my $cast_sets ( @{ $titles{$title_id}{'cast'} } ) { my ( $name, $role ) = @{$cast_sets}; $actors{$name}{$title} = $role; } }
  • 50. print "Cross referencingn"; foreach my $name ( keys %actors ) { scalar keys %{ $actors{$name} } > 1 or next; my @where = (); foreach my $title ( keys %{ $actors{$name} } ) { my $role = $actors{$name}{$title}; push @where, "$title ($role)"; } printf "Actor $name appeared in %sn", join ', ', @where; }
  • 51. Some dataSome data 6 TV series 42 total seasons 1,510 total episodes 6,484 total actors 7,493 total roles 380,520 total requests (6 * 42 * 1,510) ~ 2s/req = 761,040s = 12,684m = 211h = 8.8 days Time to run (incl. analysis) with AnyEvent: 6 minutes Matches found... 1,095!
  • 52. Actor Michael Patrick McGill appeared in: CSI Las Vegas (Ruben Caster) Dexter (Troy) Actor Robert Esser appeared in: CSI Las Vegas (Waiter) Castle (Officer #2 (uncredited)) Actor Andre Alexsen appeared in: Closer, the (Adam) Mentalist, the (CA State Govenor (uncredited)) Actor Becky O'Donohue appeared in: CSI Las Vegas (Ava Rendell) Mentalist, the (Sasha) Psych (Molly Gogolack)
  • 53. Mark PellegrinoMark Pellegrino Tom Dempsey / Tom Dempsey III in Castle Gavin Q. Baker III in The Closer Bruno Curtis in CSI Last Vegas Paul Bennett in Dexter Von McBride in The Mentalist Most accomplishedMost accomplished cross-actorcross-actor
  • 54. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework Real-time network path resolution API interfaces to services Find matching special actors in TV series Whole lot of other awesome stuff (which I'm not gonna show you now)
  • 55. Thank youThank you