YAPC::EU 2010 - Going Postal

1,327 views

Published on

Learn more about the evolution of inter-system communication from direct database access to message-queue based solutions.

It's not a Holy Grail or Silver Bullet but might prevent people making mistakes already made by others on their behalf.

The journey starts with a quick tour of things we used to do and progresses through time ... ending with an overview of an almost-in-production Net::ActiveMQ.

No Aardvark were harmed in the making of this presentation.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,327
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

YAPC::EU 2010 - Going Postal

  1. 1. Motivation Our Solution Issues Summary Going Postal Chisel Wright NET-A-PORTER YAPC::EU 2010 Chisel Wright Going Postal
  2. 2. Motivation Our Solution Talking To Strangers Issues Summary Motivation Why Bother? Chisel Wright Going Postal
  3. 3. Motivation Our Solution Talking To Strangers Issues Summary A Long Time Ago When There Weren’t Any Better Solutions Direct, far away database writes Need to know their dialect They can’t easily change their schema Tightly coupled Chisel Wright Going Postal
  4. 4. Motivation Our Solution Talking To Strangers Issues Summary A Long Time Ago When There Weren’t Any Better Solutions Direct, far away database writes Need to know their dialect They can’t easily change their schema Tightly coupled Chisel Wright Going Postal
  5. 5. Motivation Our Solution Talking To Strangers Issues Summary A Long Time Ago When There Weren’t Any Better Solutions Direct, far away database writes Need to know their dialect They can’t easily change their schema Tightly coupled Chisel Wright Going Postal
  6. 6. Motivation Our Solution Talking To Strangers Issues Summary A Long Time Ago When There Weren’t Any Better Solutions Direct, far away database writes Need to know their dialect They can’t easily change their schema Tightly coupled Chisel Wright Going Postal
  7. 7. Motivation Our Solution Talking To Strangers Issues Summary Not quite so long ago When There Were Options TheSchwartz resolved a slightly different issue first step in the right direction didn’t help with far away issue perl only Chisel Wright Going Postal
  8. 8. Motivation Our Solution Talking To Strangers Issues Summary Not quite so long ago When There Were Options TheSchwartz resolved a slightly different issue first step in the right direction didn’t help with far away issue perl only Chisel Wright Going Postal
  9. 9. Motivation Our Solution Talking To Strangers Issues Summary Not quite so long ago When There Were Options TheSchwartz resolved a slightly different issue first step in the right direction didn’t help with far away issue perl only Chisel Wright Going Postal
  10. 10. Motivation Our Solution Talking To Strangers Issues Summary Not quite so long ago When There Were Options TheSchwartz resolved a slightly different issue first step in the right direction didn’t help with far away issue perl only Chisel Wright Going Postal
  11. 11. Motivation Our Solution Talking To Strangers Issues Summary Not quite so long ago When There Were Options TheSchwartz resolved a slightly different issue first step in the right direction didn’t help with far away issue perl only Chisel Wright Going Postal
  12. 12. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Our Solution Chisel Wright Going Postal
  13. 13. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary ActiveMQ Chisel Wright Going Postal
  14. 14. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary ActiveMQ Java guys like it Perl guys can interact with it Reliable, well-used solution Chisel Wright Going Postal
  15. 15. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary ActiveMQ Java guys like it Perl guys can interact with it Reliable, well-used solution Chisel Wright Going Postal
  16. 16. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary ActiveMQ Java guys like it Perl guys can interact with it Reliable, well-used solution Chisel Wright Going Postal
  17. 17. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::Stomp STOMP is great for talking to ActiveMQ Net::Stomp excellent for quickly interacting with ActiveMQ Chisel Wright Going Postal
  18. 18. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::Stomp STOMP is great for talking to ActiveMQ Net::Stomp excellent for quickly interacting with ActiveMQ Chisel Wright Going Postal
  19. 19. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::Stomp Talking to a Queue # send a message to the queue ’foo’ use Net::Stomp; my $stomp = Net::Stomp->new({ hostname => ’localhost’, port => ’61613’, }); $stomp->connect({ login => ’hello’, passcode => ’there’, }); $stomp->send({ destination => ’/queue/foo’, body => ’test message’, }); $stomp->disconnect; Chisel Wright Going Postal
  20. 20. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Receiving with Net::Stomp Subscribing to a Queue # subscribe to messages from the queue ’foo’ use Net::Stomp; my $stomp = Net::Stomp->new({ hostname => ’localhost’, port => ’61613’, }); $stomp->connect({ login => ’hello’, passcode => ’there’, }); $stomp->subscribe({ destination => ’/queue/foo’, ’ack’ => ’client’, ’activemq.prefetchSize’ => 1, }); Chisel Wright Going Postal
  21. 21. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Receiving with Net::Stomp Plucking from the Queue while (1) { my $frame = $stomp->receive_frame; warn $frame->body; # do something here $stomp->ack( { frame => $frame } ); } $stomp->disconnect; Chisel Wright Going Postal
  22. 22. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::Stomp Thoughts Quickly and easily talk to ActiveMQ Does what it promises Low-level Copy-and-paste coding Error-handling, . . . Chisel Wright Going Postal
  23. 23. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::Stomp Thoughts Quickly and easily talk to ActiveMQ Does what it promises Low-level Copy-and-paste coding Error-handling, . . . Chisel Wright Going Postal
  24. 24. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::Stomp Thoughts Quickly and easily talk to ActiveMQ Does what it promises Low-level Copy-and-paste coding Error-handling, . . . Chisel Wright Going Postal
  25. 25. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::Stomp Thoughts Quickly and easily talk to ActiveMQ Does what it promises Low-level Copy-and-paste coding Error-handling, . . . Chisel Wright Going Postal
  26. 26. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::Stomp Thoughts Quickly and easily talk to ActiveMQ Does what it promises Low-level Copy-and-paste coding Error-handling, . . . Chisel Wright Going Postal
  27. 27. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::ActiveMQ Chisel Wright Going Postal
  28. 28. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::ActiveMQ Standardise our AMQ solution across apps Lower barrier to entry Chisel Wright Going Postal
  29. 29. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Net::ActiveMQ Standardise our AMQ solution across apps Lower barrier to entry Chisel Wright Going Postal
  30. 30. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary What Is It? Essentially: Net::Stomp Catalyst::Engine::Stomp with a lovely ribbon and bow around it. Message producers Message consumers all in one place. Chisel Wright Going Postal
  31. 31. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Talking to a Queue use Net::ActiveMQ::Producer; my $producer = Net::ActiveMQ::Producer->new({ hostname => ’localhost’, port => 61613 }); $producer->send( ’Some::Message’, { message => ’data’, goes => ’here’ } ); Net::ActiveMQ::Producer - message type does not exist - Some::Message at /path/to/.../Class/MOP/Method/Wrapped.pm line 159 Chisel Wright Going Postal
  32. 32. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ A Producer package Net::ActiveMQ::Producer::Some::Message; use Moose; with ’Net::ActiveMQ::Role::Producer’; sub transform { my ($self, $header, $data) = @_; # make sure it goes somewhere $header->{destination} ||= ’/queue/some-message’; # the desired action from the consumer $data->{’@type’} ||= ’action_method’; # "transform" the data $data->{process_time} = scalar localtime; return ($header, $data); } 1; Chisel Wright Going Postal
  33. 33. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Validate What You Send # This is completely optional! sub message_spec { # Data::Rx format return { type => ’//rec’, required => { message => ’//str’, }, optional => { ’@type’ => ’//str’, goes => ’//str’, process_time => ’//str’, }, }; } Chisel Wright Going Postal
  34. 34. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Catalyst Model # create your model class ./script/myapp_create.pl model MyMQ Net::ActiveMQ localhost 61613 YES # in your controller $c->model(’MyMQ’)->send( ’Some::Message’, { message => { a => ’shiny’, hash => ’reference’ } } ); Chisel Wright Going Postal
  35. 35. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Why did you do that? sub message_spec Used by proof-of-concept Can catch your own mistakes No compelling reason to remove it Data::Rx Nicer error messages Slightly more agnostic Chisel Wright Going Postal
  36. 36. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Why did you do that? sub message_spec Used by proof-of-concept Can catch your own mistakes No compelling reason to remove it Data::Rx Nicer error messages Slightly more agnostic Chisel Wright Going Postal
  37. 37. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Why did you do that? sub message_spec Used by proof-of-concept Can catch your own mistakes No compelling reason to remove it Data::Rx Nicer error messages Slightly more agnostic Chisel Wright Going Postal
  38. 38. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Why did you do that? sub message_spec Used by proof-of-concept Can catch your own mistakes No compelling reason to remove it Data::Rx Nicer error messages Slightly more agnostic Chisel Wright Going Postal
  39. 39. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Why did you do that? sub message_spec Used by proof-of-concept Can catch your own mistakes No compelling reason to remove it Data::Rx Nicer error messages Slightly more agnostic Chisel Wright Going Postal
  40. 40. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Why did you do that? sub message_spec Used by proof-of-concept Can catch your own mistakes No compelling reason to remove it Data::Rx Nicer error messages Slightly more agnostic Chisel Wright Going Postal
  41. 41. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Sending with Net::ActiveMQ Why did you do that? sub message_spec Used by proof-of-concept Can catch your own mistakes No compelling reason to remove it Data::Rx Nicer error messages Slightly more agnostic Chisel Wright Going Postal
  42. 42. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Receiving with Net::ActiveMQ Quick But Boring $ CATALYST_DEBUG=1 > net_activemq_consumer_server.pl [debug] Loaded engine "Catalyst::Engine::Stomp" ... [debug] Loaded components: .--------------------------------+----------. | Class | Type | +--------------------------------+----------+ | ...::Controller::Root | instance | ’--------------------------------+----------’ ... [info] Application powered by Catalyst 5.80016 Not very useful in this state Fairly easy to add something useful Chisel Wright Going Postal
  43. 43. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Receiving with Net::ActiveMQ Quick But Boring $ CATALYST_DEBUG=1 > net_activemq_consumer_server.pl [debug] Loaded engine "Catalyst::Engine::Stomp" ... [debug] Loaded components: .--------------------------------+----------. | Class | Type | +--------------------------------+----------+ | ...::Controller::Root | instance | ’--------------------------------+----------’ ... [info] Application powered by Catalyst 5.80016 Not very useful in this state Fairly easy to add something useful Chisel Wright Going Postal
  44. 44. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Receiving with Net::ActiveMQ Consuming Some::Message $ net_activemq_consumer_create.pl controller Some::Message ActiveMQ created "lib/Net/ActiveMQ/Consumer/Controller/Some" created "t" created "lib/Net/ActiveMQ/Consumer/Controller/Some/Message.pm" created "t/controller_Some-Message.t" Catalyst::Helper to do the hard work Chisel Wright Going Postal
  45. 45. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Receiving with Net::ActiveMQ Consuming Some::Message $ CATALYST_DEBUG=1 > PERL5LIB=$PWD/lib > net_activemq_consumer_server.pl [debug] Loaded engine "Catalyst::Engine::Stomp" ... [debug] Loaded components: .--------------------------------+----------. | Class | Type | +--------------------------------+----------+ | ...::Controller::Root | instance | | ...::Controller::Some::Message | instance | ’--------------------------------+----------’ ... [info] Application powered by Catalyst 5.80016 Chisel Wright Going Postal
  46. 46. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Receiving with Net::ActiveMQ Controller Actions package Net::ActiveMQ::Consumer::Controller::Some::Message; use Moose; BEGIN { extends ’Net::ActiveMQ::Consumer::ControllerBase::MessageDriven’ } __PACKAGE__->config( action_namespace => ’some-message’ ); # this will handle ’@type’ of ’action_method’ in # the ’some-message’ queue sub action_method :Local { my ($self, $c, $message) = @_; $c->log->warn(’Some::Message / some-message / action_method’); } __PACKAGE__->meta->make_immutable; 1; Chisel Wright Going Postal
  47. 47. Motivation ActiveMQ Our Solution Net::Stomp Issues Net::ActiveMQ Summary Receiving with Net::ActiveMQ Queue::Spec package Net::ActiveMQ::Consumer::Queue::Spec::Some::Message; use Moose; sub action_method { return { type => ’//any’ }; } 1; Chisel Wright Going Postal
  48. 48. Motivation Our Solution Peer Review Issues Implementation Summary Issues Chisel Wright Going Postal
  49. 49. Motivation Our Solution Peer Review Issues Implementation Summary Peer Review Only recently been used by the rest of the team Bugs Handling errors Lack of clear documentation POD How-To / “How do I. . . ?” Chisel Wright Going Postal
  50. 50. Motivation Our Solution Peer Review Issues Implementation Summary Peer Review Only recently been used by the rest of the team Bugs Handling errors Lack of clear documentation POD How-To / “How do I. . . ?” Chisel Wright Going Postal
  51. 51. Motivation Our Solution Peer Review Issues Implementation Summary Peer Review Only recently been used by the rest of the team Bugs Handling errors Lack of clear documentation POD How-To / “How do I. . . ?” Chisel Wright Going Postal
  52. 52. Motivation Our Solution Peer Review Issues Implementation Summary Peer Review Only recently been used by the rest of the team Bugs Handling errors Lack of clear documentation POD How-To / “How do I. . . ?” Chisel Wright Going Postal
  53. 53. Motivation Our Solution Peer Review Issues Implementation Summary Peer Review Only recently been used by the rest of the team Bugs Handling errors Lack of clear documentation POD How-To / “How do I. . . ?” Chisel Wright Going Postal
  54. 54. Motivation Our Solution Peer Review Issues Implementation Summary Peer Review Only recently been used by the rest of the team Bugs Handling errors Lack of clear documentation POD How-To / “How do I. . . ?” Chisel Wright Going Postal
  55. 55. Motivation Our Solution Peer Review Issues Implementation Summary Implementation General Some early features not fully phased out @type => ’...’ JMSType Chisel Wright Going Postal
  56. 56. Motivation Our Solution Peer Review Issues Implementation Summary Implementation General Some early features not fully phased out @type => ’...’ JMSType Chisel Wright Going Postal
  57. 57. Motivation Our Solution Peer Review Issues Implementation Summary Implementation General Some early features not fully phased out @type => ’...’ JMSType Chisel Wright Going Postal
  58. 58. Motivation Our Solution Peer Review Issues Implementation Summary Implementation Producers Shared spec files implementation not complete Still living in the ::Consumer:: namespace Not easily used by ::Producer:: classes message_spec vs build_message_spec Chisel Wright Going Postal
  59. 59. Motivation Our Solution Peer Review Issues Implementation Summary Implementation Producers Shared spec files implementation not complete Still living in the ::Consumer:: namespace Not easily used by ::Producer:: classes message_spec vs build_message_spec Chisel Wright Going Postal
  60. 60. Motivation Our Solution Peer Review Issues Implementation Summary Implementation Producers Shared spec files implementation not complete Still living in the ::Consumer:: namespace Not easily used by ::Producer:: classes message_spec vs build_message_spec Chisel Wright Going Postal
  61. 61. Motivation Our Solution Peer Review Issues Implementation Summary Implementation Producers Shared spec files implementation not complete Still living in the ::Consumer:: namespace Not easily used by ::Producer:: classes message_spec vs build_message_spec Chisel Wright Going Postal
  62. 62. Motivation Our Solution Peer Review Issues Implementation Summary Implementation Consumers Consumers limited to Net::ActiveMQ namespace Error-Handling still evolving Chisel Wright Going Postal
  63. 63. Motivation Our Solution Peer Review Issues Implementation Summary Implementation Consumers Consumers limited to Net::ActiveMQ namespace Error-Handling still evolving Chisel Wright Going Postal
  64. 64. Motivation Our Solution Issues Summary Summary Avoid direct database interaction Use message queues DRY The Future Evolution - final stages We’re starting to use it in production We hope to release to the CPAN soon Chisel Wright Going Postal
  65. 65. Motivation Our Solution Issues Summary Summary Avoid direct database interaction Use message queues DRY The Future Evolution - final stages We’re starting to use it in production We hope to release to the CPAN soon Chisel Wright Going Postal
  66. 66. Motivation Our Solution Issues Summary Summary Avoid direct database interaction Use message queues DRY The Future Evolution - final stages We’re starting to use it in production We hope to release to the CPAN soon Chisel Wright Going Postal
  67. 67. Motivation Our Solution Issues Summary Summary Avoid direct database interaction Use message queues DRY The Future Evolution - final stages We’re starting to use it in production We hope to release to the CPAN soon Chisel Wright Going Postal
  68. 68. Motivation Our Solution Issues Summary Summary Avoid direct database interaction Use message queues DRY The Future Evolution - final stages We’re starting to use it in production We hope to release to the CPAN soon Chisel Wright Going Postal
  69. 69. Motivation Our Solution Issues Summary Summary Avoid direct database interaction Use message queues DRY The Future Evolution - final stages We’re starting to use it in production We hope to release to the CPAN soon Chisel Wright Going Postal
  70. 70. Motivation Our Solution Issues Summary Summary Avoid direct database interaction Use message queues DRY The Future Evolution - final stages We’re starting to use it in production We hope to release to the CPAN soon Chisel Wright Going Postal
  71. 71. Motivation Our Solution Issues Summary QUESTIONS? Chisel Wright Going Postal
  72. 72. Motivation Our Solution Issues Summary Obligatory LOLCAT LOL-Rilla? chisel.wright@net-a-porter.com Chisel Wright Going Postal

×