Asynchronous Programming FTW! 2 (with AnyEvent)

4,137 views
3,885 views

Published on

This is a revamped Asynchronous Programming FTW talk, given at YAPC::NA 2013, Cluj.pm, and AmsterdamX.pm.

Published in: Lifestyle, Technology, Business
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,137
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
35
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Asynchronous Programming FTW! 2 (with AnyEvent)

  1. 1. AsynchronousAsynchronous ProgrammingProgramming FTW!FTW!(Sawyer X) @PerlSawyer Theory Practice
  2. 2. TheoryTheory We think in non-blocking terms We act in non-blocking terms We program in blocking terms An example? Sure!
  3. 3. The exampleThe example use LWP::UserAgent; my @urls = ( 'https://duckduckgo.com', 'http://cluj.pm', 'http://telaviv.pm.org', ); my $ua = LWP::UserAgent->new; foreach my $url (@urls) { my $res = $ua->get($url); say $res->decoded_content; }
  4. 4. Problem?Problem? Wait on each URL Ineffecient waste of time Pointless Senseless CRIME AGAINST HUMANITY (maybe just ineffecient)
  5. 5. How we thinkHow we think Nothing to work on? Defer to later! Like cooking!
  6. 6. Making dinnerMaking dinner Boil water. Wait? No! Salad time! (water gets deferred to later) Cook potatoes. Wait? No! Sauce time! (potatoes get deferred to later) We think and act non-blocking
  7. 7. TIMTOWTDITIMTOWTDI Forks Threads Event-loops <- this talk
  8. 8. Event loop?Event loop? Single thread, single process Like a big while(1) {} Heavy usage of code references No real race conditions No locking, semaphores, etc. Can yield better speed But... nothing can be blocking! (I/O)
  9. 9. Event loops in PerlEvent loops in Perl POE Reflex IO::Async IO::Lambda Qt, GTK, Tk, Glib, Cocoa::EventLoop, EV, FLTK, Irssi AnyEvent <- this talk
  10. 10. AnyEventAnyEvent Light-weight Fast Very fast Slim interface Good support for other event loops Lots of modules
  11. 11. PracticePractice Cooking with AnyEventCooking with AnyEvent (timers)(timers) use AnyEvent; say '>> Water boiling...'; my $water = AnyEvent->timer( after => 5*60, cb => sub { say '<< Water boiled!'; say '>> Potatos being prepared...'; my $potatoes; $potatoes = AnyEvent->timer( after => 10*60, cb => sub { undef $potatoes; say '<< Potatos ready!'; } ); } ); say '>> Salad being prepared...'; my $salad = AnyEvent->timer( after => 6*60, cb => sub { say '<< Salad ready!'; say '>> Sauce being prepared...'; my $sauce; $sauce = AnyEvent->timer( after => 12*60, cb => sub { undef $sauce; say '<< Sauce ready!'; } ); } );
  12. 12. Water - 5 minutes Potatos - 10 minutes after water Salad - 6 minutes Sauce - 12 minutes after salad Water, salad, potatoes, sauce >> Water boiling... >> Salad being prepared... << Water boiled! >> Potatos being prepared... << Salad ready! >> Sauce being prepared... << Potatos ready! << Sauce ready!
  13. 13. Condition variablesCondition variables A condition waiting to be fulfilled Starting at false, hoping to get to true Helps waiting for other events to finish Call ->recv() to wait for stuff to finish Call ->send() when finished doing your stuff Without condition variables, we reach the end of programs We reach the end == program closes Program closes == our code doesn't get executed Our code doesn't get executed == we get executed!
  14. 14. Simple exampleSimple example use AnyEvent; my $count = 1; my $cv = AnyEvent->condvar; my $timer = AnyEvent->timer( interval => 1, cb => sub { say "Count: $count"; $count++ == 5 and $cv->send; }, ); $cv->recv; say 'We counted to 5!';
  15. 15. Bigger exampleBigger example my $cv = AnyEvent->condvar; my $water = AnyEvent->timer( after => 5*60, cb => sub { my $potatoes; $potatoes = AnyEvent->timer( after => 10*60, cb => sub { undef $potatoes; } ); } ); my $salad = AnyEvent->timer( after => 6*60, cb => sub { my $sauce; $sauce = AnyEvent->timer( after => 12*60, cb => sub { undef $sauce; $cv->send; } ); } ); $cv->recv;
  16. 16. Better example :: LWPBetter example :: LWP use LWP::UserAgent; my @urls = ( 'https://whatismyip.com', 'http://checkip.dyndns.org', 'http://wtfismyipaddress.org', ); my $ua = LWP::UserAgent->new; foreach my $url (@urls) { my $res = $ua->get($url); say $res->decoded_content; }
  17. 17. Better example :: AnyEventBetter example :: AnyEvent use AnyEvent; use AnyEvent::HTTP; my $cv = AnyEvent->condvar; foreach my $url (@urls) { $cv->begin; http_get $url, sub { my $body = shift; say $body; $cv->end; } } $cv->recv;
  18. 18. POP QUIZ TIME!POP QUIZ TIME! A few examples Spot the error(s) and get a prize Available prizes: handshake, high five, hug, kiss (not necessarily from me)
  19. 19. Round 1Round 1 use AnyEvent; use AnyEvent::HTTP; http_post $url, $body, sub { say 'Successfully made an API call to do something cool!'; }; # wait for all events to finish AnyEvent->condvar->recv;
  20. 20. Round 2Round 2 use AnyEvent; my $timer = AnyEvent->timer( interval => 5, cb => sub { sleep 1; say 'Bwahaha!'; } );
  21. 21. Round 3Round 3 use AnyEvent; my $cv = AnyEvent->condvar; my $player1 = AnyEvent->timer( after => 3.7, cb => sub { say 'Player 1 joins the game!'; $cv->send; }, ); my $player2 = AnyEvent->timer( after => 7.2, cb => sub { say 'Player 2 joins the game!'; $cv->send; }, ); $cv->recv;
  22. 22. Round 4Round 4 use AnyEvent; use HTTP::Tiny; my $cv = AnyEvent->condvar; $cv->begin for 1 .. 2; my $player1 = AnyEvent->timer( after => 3.7, cb => sub { say 'Player 1 joins the game! Alerting server!'; my $res = HTTP::Tiny->new->get('http://server:3030/add/1'); $res->{'success'} or die 'Failed to insert player 1 to game!'; $cv->end; }, ); my $player2 = AnyEvent->timer( after => 7.2, cb => sub { say 'Player 2 joins the game!'; $cv->end; }, ); $cv->recv;
  23. 23. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface
  24. 24. WWW::xkcdWWW::xkcd use WWW::xkcd; my $xkcd = WWW::xkcd->new; $xkcd->fetch( sub { my ( $img, $comic ) = @_; say "Today's comic is titled: ", $comic->{'title'}; ... } );
  25. 25. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server
  26. 26. TwiggyTwiggy use Twiggy::Server; my $app = sub { # PSGI app }; my $server = Twiggy::Server->new(...); $server->register_service($app); AnyEvent->condvar->recv;
  27. 27. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface
  28. 28. AnyEvent::EC2::TinyAnyEvent::EC2::Tiny use AnyEvent::EC2::Tiny; my $ec2 = AnyEvent::EC2::Tiny->new(...); my $xml = $ec2->send( 'RegionName.1' => 'us-east-1', Action => 'DescribeRegions', success_cb => sub { my $xml = shift; # prints ec2.us-east-1.amazonaws.com say $xml->{'regionInfo'}{'item'}[0]{'regionEndpoint'}; }, fail_cb => sub { my $error = shift; ... }, );
  29. 29. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework
  30. 30. JunoJuno use Juno; my $juno = Juno->new( hosts => [ 'server1', 'server2' ], interval => 10, checks => { HTTP => { headers => { { 'Host', 'example.com' }, }, on_result => sub { my $result = shift; ... }, }, }, );
  31. 31. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework Real-time network path resolution
  32. 32. SIP RouteSIP Route use AnyEvent; use Anyevent::Util 'fork_call'; use AnyEvent::Socket; use AnyEvent::Handle; use Data::Dump 'dump'; my $tcp_server_guard = tcp_server '127.0.0.1', 2020, sub { my ($fh) = @_; my $handle = AnyEvent::Handle->new( fh => $fh ); $handle->push_read( line => sub {...} ); };
  33. 33. my ( $hdl, $target ) = @_; if ( ! defined $target ) { $hdl->push_write($default_route); $hdl->destroy; return; } chomp $target; if ( ! exists $data{$target} ) { $hdl->push_write($default_route); $hdl->destroy; return; }
  34. 34. my @sorted = sort { $data{$target}{$a} <=> $data{$target}{$b} } keys %{ $data{$target} }; $hdl->push_write( $sorted[0] ); $hdl->destroy;
  35. 35. foreach my $target (@targets) { foreach my $iface (@interfaces) { my $cmd = "sipsak --sip-uri=sip:$target -D $timeout -X $iface"; push @timers, AE::timer $start_after, $check_every, sub { my $time = AE::now; fork_call { system "$cmd >/dev/null 2>&1"; return $? >> 8; } sub { my $exit = shift; ( $exit == 0 || $exit == 1 ) or return delete $data{$target}{$iface}; $data{$target}{$iface} = AE::now - $time; }; }; } } AE::cv->recv;
  36. 36. if ($verbose) { push @timers, AE::timer 5, 5, sub { dump %data }; }
  37. 37. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework Real-time network path resolution API interfaces to services
  38. 38. AnyEvent::RiakAnyEvent::Riak use AnyEvent::Riak; my $riak = AnyEvent::Riak->new( host => 'http://127.0.0.1:8098', path => 'riak', ); $riak->list_bucket( 'mybucket', {props => 'true', keys => 'false'}, sub { my $res = shift; ... } );
  39. 39. AnyEvent::SNMPAnyEvent::SNMP use AnyEvent::SNMP; use Net::SNMP; my $cv = AnyEvent->condvar; Net::SNMP->session( -hostname => "127.0.0.1", -community => "public", -nonblocking => 1 )->get_request( -callback => sub { $cv->send (@_) } ); my @result = $cv->recv;
  40. 40. AnyEvent::XMLRPCAnyEvent::XMLRPC use AnyEvent::XMLRPC; my $serv = AnyEvent::XMLRPC->new( methods => { 'echo' => &echo, }, );
  41. 41. AnyEvent::DNSAnyEvent::DNS use AnyEvent::DNS; my $cv = AnyEvent->condvar; AnyEvent::DNS::a "www.google.ro", $cv; # later... my @addrs = $cv->recv;
  42. 42. AnyEvent::TwitterAnyEvent::Twitter my $ua = AnyEvent::Twitter->new( consumer_key => 'consumer_key', consumer_secret => 'consumer_secret', access_token => 'access_token', access_token_secret => 'access_token_secret', ); my $cv = AnyEvent->condvar; # GET request $cv->begin; $ua->get( 'account/verify_credentials', sub { my ($header, $response, $reason) = @_; say $response->{screen_name}; $cv->end; } );
  43. 43. AnyEvent::GraphiteAnyEvent::Graphite my $graphite = AnyEvent::Graphite->new( host => '127.0.0.1', port => '2003', ); $graphite->send("a.b.c.d", $value, $timestamp);
  44. 44. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework Real-time network path resolution API interfaces to services Find matching special actors in TV series
  45. 45. Actor matchesActor matches use AnyEvent; use AnyEvent::HTTP; my $base = 'http://www.imdb.com/title'; my %titles = ( tt1196946 => { name => 'Mentalist, the' }, tt1219024 => { name => 'Castle' }, tt0773262 => { name => 'Dexter' }, tt0491738 => { name => 'Psych' }, tt0247082 => { name => 'CSI Las Vegas' }, tt0458253 => { name => 'Closer, the' }, ); # DON'T PARSE HTML WITH REGEX!!!!1127 my $episode_regex = qr{...}; my $cast_regex = qr{...};
  46. 46. # Fetching number of seasons for each title my $cv = AE::cv; foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'}; print "[$title] Fetching number of seasons.n"; my $url = "$base/$title_id/"; my $regex = qr{/title/$title_id/episodes?season=([0-9]+)}; $cv->begin; http_get $url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$regex/mg; $titles{$title_id}{'episodes'}{$_} = [] for @found; $cv->end; }; } $cv->recv;
  47. 47. # fetching the list of episodes for each season foreach my $title_id ( keys %titles ) { my $cv = AE::cv; foreach my $season ( keys %{ $titles{$title_id}{'episodes'} } ) { my $title = $titles{$title_id}{'name'}; print "[$title] [Season $season] fetching episodesn"; my $season_url = "$base/$title_id/episodes?season=$season"; $cv->begin; http_get $season_url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$episode_regex/mg; while (@found) { my $id = shift @found; my $name = shift @found;
  48. 48. foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'}; foreach my $season ( keys %{ $titles{$title_id}{'episodes'} } ) { my @episodes = @{ $titles{$title_id}{'episodes'}{$season} }; print "[$title] [$season] Fetching cast for season $seasonn"; my $cv = AE::cv; foreach my $episode_id (@episodes) { print " -> [$title] [$season] $episode_idn"; my $url = "$base/$episode_id/fullcredits"; $cv->begin; http_get $url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$cast_regex/mg;
  49. 49. print "Populating actorsn"; my %actors = (); foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'}; foreach my $cast_sets ( @{ $titles{$title_id}{'cast'} } ) { my ( $name, $role ) = @{$cast_sets}; $actors{$name}{$title} = $role; } }
  50. 50. print "Cross referencingn"; foreach my $name ( keys %actors ) { scalar keys %{ $actors{$name} } > 1 or next; my @where = (); foreach my $title ( keys %{ $actors{$name} } ) { my $role = $actors{$name}{$title}; push @where, "$title ($role)"; } printf "Actor $name appeared in %sn", join ', ', @where; }
  51. 51. Some dataSome data 6 TV series 42 total seasons 1,510 total episodes 6,484 total actors 7,493 total roles 380,520 total requests (6 * 42 * 1,510) ~ 2s/req = 761,040s = 12,684m = 211h = 8.8 days Time to run (incl. analysis) with AnyEvent: 6 minutes Matches found... 1,095!
  52. 52. Actor Michael Patrick McGill appeared in: CSI Las Vegas (Ruben Caster) Dexter (Troy) Actor Robert Esser appeared in: CSI Las Vegas (Waiter) Castle (Officer #2 (uncredited)) Actor Andre Alexsen appeared in: Closer, the (Adam) Mentalist, the (CA State Govenor (uncredited)) Actor Becky O'Donohue appeared in: CSI Las Vegas (Ava Rendell) Mentalist, the (Sasha) Psych (Molly Gogolack)
  53. 53. Mark PellegrinoMark Pellegrino Tom Dempsey / Tom Dempsey III in Castle Gavin Q. Baker III in The Closer Bruno Curtis in CSI Last Vegas Paul Bennett in Dexter Von McBride in The Mentalist Most accomplishedMost accomplished cross-actorcross-actor
  54. 54. Stuff to do with AnyEventStuff to do with AnyEvent xkcd interface PSGI/Plack server Async AWS EC2 interface Monitoring framework Real-time network path resolution API interfaces to services Find matching special actors in TV series Whole lot of other awesome stuff (which I'm not gonna show you now)
  55. 55. Thank youThank you

×