YAPC::Asia 2009 Tokyo 2009/9/10 Gosuke Miyashita
Speaker <ul><li>Gosuke Miyashita ( mizzy ) </li></ul><ul><li>Technical manager at paperboy&co. </li></ul><ul><li>Have most...
 
 
Agenda <ul><li>Asynchronous processing with event driven programming </li></ul><ul><li>Asynchronous processing with Danga:...
 
Event driven programming <ul><li>A programming paradigm in which the flow of the program is determined by events </li></ul...
Events <ul><li>I/O </li></ul><ul><ul><li>On I/O read ready or write ready </li></ul></ul><ul><li>Timers </li></ul><ul><ul>...
Main loop <ul><li>Also called “event loop” </li></ul><ul><li>On each loop, check that events are occurred or not, and proc...
Eaxmple of main loop Process timers Wait I/O events Process I/O events Post main loop process
 
Events supported by Danga::Socket <ul><li>I/O </li></ul><ul><li>Timers </li></ul>
Process I/O events <ul><li>Danga::Socket supports 3 I/O event notification facility </li></ul><ul><ul><li>kqueue </li></ul...
Add I/O watcher with Danga::Socket <ul><li>use   base  ‘Danga::Socket’; </li></ul><ul><li>sub   new  { </li></ul><ul><li>#...
Add I/O wathcer (pattern 2) <ul><li>Danga::Socket->AddOtherFds( </li></ul><ul><li>$fd  =>  sub  { </li></ul><ul><li># proc...
Add I/O watcher (pattern 3) <ul><li>Danga::Socket::Callback->new( </li></ul><ul><li>handle   =>  $fd , </li></ul><ul><li>o...
Add timer watcher with Danga::Socket <ul><li>Danga::Socket->AddTimer( </li></ul><ul><li>10, </li></ul><ul><li>sub  { </li>...
Start main loop with Danga::Socket <ul><li>Danga::Socket->EventLoop(); </li></ul>
Main loop(again) Process Timers Wait I/O events Process I/O events Post main loop process
 
Perlbal mechanism(as reverse proxy) BackendHTTP ClientProxy TCPListener Client Server ClientProxy Client BackendHTTP Danga...
Perlbal with a plugin (on start_proxy_request hook) BackendHTTP ClientProxy TCPListener Client Server Plugin::Hoge Target ...
 
Hooks <ul><li>Perlbal has many plugin hooks </li></ul><ul><li>Explain with start_proxy_request hook in this session </li><...
In case of a synchronous plugin ClientProxy TCPListener Client Server Plugin::Sync Client
In case of an asynchronous plugin ClientProxy TCPListener Client Server Plugin::Async Client ClientProxy
How to write asynchronous plugins? <ul><li>Write the main process based on Danga::Socket </li></ul><ul><ul><li>process wit...
How to write asynchronous plugins? <ul><li>When finish asynchronous process, go to next phase </li></ul><ul><ul><li>Call b...
Main process of a plugin <ul><li>package   My::Drizzle ; </li></ul><ul><li>use   base   ‘Danga::Socket’ ; </li></ul><ul><l...
Attention <ul><li>You can also use Danga::Socket::Callback </li></ul><ul><li>You cannot use AddOtherFds() </li></ul><ul><u...
plugin register <ul><li>package   Perlbal::Plugin::AsyncDb ; </li></ul><ul><li>sub   register  { </li></ul><ul><li>my  ( $...
Call asynchronous process <ul><li>sub   request_db  { </li></ul><ul><li>my  Perlbal::ClientProxy $client =  shift ; </li><...
In case of “return 0” BackendHTTP ClientProxy TCPListener Client Server Plugin::Async Go to next step even if Plugin::Asyn...
“ return 1” & “call back” BackendHTTP ClientProxy TCPListener Client Plugin::Async Stop the process and go to main loop Se...
Call back function <ul><li>If plugin process finished, ClientProxy must go to the next process(call BackendHTTP) </li></ul...
Call back function <ul><li>my  Perlbal::ClientProxy  </li></ul><ul><li>$client =  shift ; </li></ul><ul><li>My::Drizzle->n...
Patch of ClientProxy::handle_request <ul><li>-  return   if  $svc->run_hook( </li></ul><ul><li>-   'start_proxy_request', ...
 
Points of asynchronous Perlbal plugins <ul><li>Process within the Danga::Socket main loop </li></ul><ul><li>return 1 when ...
 
Upcoming SlideShare
Loading in...5
×

How Danga::Socket handles asynchronous processing and how to write asynchronous Perlbal plugins

2,719

Published on

Published in: Technology, Business
2 Comments
3 Likes
Statistics
Notes
No Downloads
Views
Total Views
2,719
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
2
Likes
3
Embeds 0
No embeds

No notes for slide

How Danga::Socket handles asynchronous processing and how to write asynchronous Perlbal plugins

  1. 1. YAPC::Asia 2009 Tokyo 2009/9/10 Gosuke Miyashita
  2. 2. Speaker <ul><li>Gosuke Miyashita ( mizzy ) </li></ul><ul><li>Technical manager at paperboy&co. </li></ul><ul><li>Have most kids in Japan Perl world? </li></ul><ul><ul><li>3 kids.Even with Yappo. </li></ul></ul><ul><ul><li>4 th kid will be born in November. </li></ul></ul>
  3. 5. Agenda <ul><li>Asynchronous processing with event driven programming </li></ul><ul><li>Asynchronous processing with Danga::Socket </li></ul><ul><li>Overview of Perlbal asynchronous processing </li></ul><ul><li>How to write asynchronous Perlbal plugins </li></ul><ul><li>Summary </li></ul>
  4. 7. Event driven programming <ul><li>A programming paradigm in which the flow of the program is determined by events </li></ul><ul><li>Counter paradigm of flow-based programming </li></ul>
  5. 8. Events <ul><li>I/O </li></ul><ul><ul><li>On I/O read ready or write ready </li></ul></ul><ul><li>Timers </li></ul><ul><ul><li>On time passed </li></ul></ul><ul><li>Signal </li></ul><ul><ul><li>On getting a signal </li></ul></ul><ul><li>Child process </li></ul><ul><ul><li>On child process exit </li></ul></ul>
  6. 9. Main loop <ul><li>Also called “event loop” </li></ul><ul><li>On each loop, check that events are occurred or not, and process the events. </li></ul><ul><li>After process the events, return to the loop and repeat the loop till next events occur </li></ul><ul><li>This is the flow of asynchronous processing </li></ul>
  7. 10. Eaxmple of main loop Process timers Wait I/O events Process I/O events Post main loop process
  8. 12. Events supported by Danga::Socket <ul><li>I/O </li></ul><ul><li>Timers </li></ul>
  9. 13. Process I/O events <ul><li>Danga::Socket supports 3 I/O event notification facility </li></ul><ul><ul><li>kqueue </li></ul></ul><ul><ul><li>epoll </li></ul></ul><ul><ul><li>poll </li></ul></ul><ul><li>Select adequate facility automatically </li></ul>
  10. 14. Add I/O watcher with Danga::Socket <ul><li>use base ‘Danga::Socket’; </li></ul><ul><li>sub new { </li></ul><ul><li># pass file descripter </li></ul><ul><li>$self->SUPER::new( $fd ); </li></ul><ul><li>} </li></ul><ul><li>sub event_read { </li></ul><ul><li># process when $fd is read ready </li></ul><ul><li>} </li></ul><ul><li>sub event_write { </li></ul><ul><li># procdess when $fd is write ready </li></ul><ul><li>} </li></ul>
  11. 15. Add I/O wathcer (pattern 2) <ul><li>Danga::Socket->AddOtherFds( </li></ul><ul><li>$fd => sub { </li></ul><ul><li># process when $fd is read ready </li></ul><ul><li>}, </li></ul><ul><li>); </li></ul>
  12. 16. Add I/O watcher (pattern 3) <ul><li>Danga::Socket::Callback->new( </li></ul><ul><li>handle => $fd , </li></ul><ul><li>on_read_ready => sub { </li></ul><ul><li># process on read ready </li></ul><ul><li>}, </li></ul><ul><li>on_write_ready => sub { </li></ul><ul><li># process on write ready </li></ul><ul><li>}, </li></ul><ul><li>); </li></ul>
  13. 17. Add timer watcher with Danga::Socket <ul><li>Danga::Socket->AddTimer( </li></ul><ul><li>10, </li></ul><ul><li>sub { </li></ul><ul><li># process after 10 seconds </li></ul><ul><li>}, </li></ul><ul><li>} </li></ul>
  14. 18. Start main loop with Danga::Socket <ul><li>Danga::Socket->EventLoop(); </li></ul>
  15. 19. Main loop(again) Process Timers Wait I/O events Process I/O events Post main loop process
  16. 21. Perlbal mechanism(as reverse proxy) BackendHTTP ClientProxy TCPListener Client Server ClientProxy Client BackendHTTP Danga::Socket based objects
  17. 22. Perlbal with a plugin (on start_proxy_request hook) BackendHTTP ClientProxy TCPListener Client Server Plugin::Hoge Target of this session
  18. 24. Hooks <ul><li>Perlbal has many plugin hooks </li></ul><ul><li>Explain with start_proxy_request hook in this session </li></ul>
  19. 25. In case of a synchronous plugin ClientProxy TCPListener Client Server Plugin::Sync Client
  20. 26. In case of an asynchronous plugin ClientProxy TCPListener Client Server Plugin::Async Client ClientProxy
  21. 27. How to write asynchronous plugins? <ul><li>Write the main process based on Danga::Socket </li></ul><ul><ul><li>process within the main loop of Danga::Socket </li></ul></ul><ul><ul><li>Create Danga::Socket based class </li></ul></ul><ul><ul><li>Or use Danga::Socket::Callback </li></ul></ul><ul><ul><li>Or use Danga::Socket::AddTimer </li></ul></ul><ul><ul><li>Also other libraries need to be non-blocking </li></ul></ul><ul><ul><ul><li>You can use Gearman::Client::Async with blocking processes </li></ul></ul></ul>
  22. 28. How to write asynchronous plugins? <ul><li>When finish asynchronous process, go to next phase </li></ul><ul><ul><li>Call back function </li></ul></ul><ul><ul><li>Also need to fix Perlbal itself </li></ul></ul>
  23. 29. Main process of a plugin <ul><li>package My::Drizzle ; </li></ul><ul><li>use base ‘Danga::Socket’ ; </li></ul><ul><li>use Net::Drizzle ':constants' ; </li></ul><ul><li>sub new { </li></ul><ul><li>$self->SUPER::new($fh); </li></ul><ul><li>$self->watch_read(1); </li></ul><ul><li>} </li></ul><ul><li>sub event_read { </li></ul><ul><li># Check db request status and call back when finished </li></ul><ul><li>} </li></ul>
  24. 30. Attention <ul><li>You can also use Danga::Socket::Callback </li></ul><ul><li>You cannot use AddOtherFds() </li></ul><ul><ul><li>File descriptors added by Add AddOtherFds() are only evaluated at the beginning of the main loop </li></ul></ul><ul><ul><li>So if already in the main loop, AddOtherFds() are meaningless. </li></ul></ul><ul><li>You can call epoll_ctl, EV_SET and so on directly, but no portability </li></ul>
  25. 31. plugin register <ul><li>package Perlbal::Plugin::AsyncDb ; </li></ul><ul><li>sub register { </li></ul><ul><li>my ( $class, $svc ) = @_; </li></ul><ul><li>$svc->register_hook( </li></ul><ul><li>'Async' => 'start_proxy_request', </li></ul><ul><li> &request_db , </li></ul><ul><li>); </li></ul><ul><li>return 1; </li></ul><ul><li>} </li></ul>
  26. 32. Call asynchronous process <ul><li>sub request_db { </li></ul><ul><li>my Perlbal::ClientProxy $client = shift ; </li></ul><ul><li>My::Drizzle->new( </li></ul><ul><li>callback => sub { </li></ul><ul><li># call back </li></ul><ul><li>}, </li></ul><ul><li>); </li></ul><ul><li>return 1; # very important! </li></ul><ul><li>} </li></ul>
  27. 33. In case of “return 0” BackendHTTP ClientProxy TCPListener Client Server Plugin::Async Go to next step even if Plugin::Async is not finished
  28. 34. “ return 1” & “call back” BackendHTTP ClientProxy TCPListener Client Plugin::Async Stop the process and go to main loop Server return 1 call back
  29. 35. Call back function <ul><li>If plugin process finished, ClientProxy must go to the next process(call BackendHTTP) </li></ul><ul><li>Stopped at the following code of handle_request() : </li></ul><ul><li>return if $svc->run_hook( </li></ul><ul><li>'start_proxy_request', $self </li></ul><ul><li>); </li></ul><ul><li>ClientProxy must start processing from the next of this code </li></ul>
  30. 36. Call back function <ul><li>my Perlbal::ClientProxy </li></ul><ul><li>$client = shift ; </li></ul><ul><li>My::Drizzle->new( </li></ul><ul><li>callback => sub { </li></ul><ul><li>$client->{async_complete} = 1; </li></ul><ul><li>$client->handle_request; </li></ul><ul><li>}, </li></ul><ul><li>); </li></ul>
  31. 37. Patch of ClientProxy::handle_request <ul><li>- return if $svc->run_hook( </li></ul><ul><li>- 'start_proxy_request', $self </li></ul><ul><li>- ); </li></ul><ul><li>+ unless ( $self->{async_complete} ) { </li></ul><ul><li>+ return if $svc->run_hook( </li></ul><ul><li>+ 'start_proxy_request', $self </li></ul><ul><li>+ ); </li></ul><ul><li>+ } </li></ul><ul><li>+ $self->{async_complete} = 0; </li></ul>
  32. 39. Points of asynchronous Perlbal plugins <ul><li>Process within the Danga::Socket main loop </li></ul><ul><li>return 1 when plugin called </li></ul><ul><li>Restart ClientProxy process by a call back function when plugin finished </li></ul><ul><li>Also need to fix Perlbal itself </li></ul><ul><li>Be careful with order of plugins </li></ul><ul><ul><li>Following plugins on a same hook are ignored when after a plugin returns 1 </li></ul></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×