Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Message passing

858 views

Published on

Presentation given at YAPC::EU 2012

Published in: Technology
  • Be the first to comment

Message passing

  1. 1. Easy Loggingwith Log::Message::Structured and Message::Passing
  2. 2. MeDamien dams KrotkineParis.pm memberFrench Perl Monger vice-presidentPerl Dancer developer11 modules on CPAN (pretty weak)Author of Moderne Perl (french book, go buy it)
  3. 3. • Let’s say I have a big application
  4. 4. • Let’s say I have a big application• It processes a lot of data
  5. 5. • Let’s say I have a big application• It processes a lot of data• It computes lots of lines
  6. 6. • Let’s say I have a big application• It processes a lot of data• It computes lots of lines• Per user, and per account
  7. 7. account 1user 1 linesuser 2 processuser 3 account n result running on multiple hosts
  8. 8. • It runs fine
  9. 9. • It runs fine• Well, more or less
  10. 10. • It runs fine• Well, more or less• I’d like to know what’s going on with it
  11. 11. When the app crashes
  12. 12. When the app crashes• Why did it crash ? ( reason, stack trace )
  13. 13. When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time
  14. 14. When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time • What was the situation ?
  15. 15. When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time • What was the situation ? • What was the data it was processing ?
  16. 16. When there are issues
  17. 17. When there are issues• How bad is this failure ?
  18. 18. When there are issues• How bad is this failure ?• Why did it fail to process this line ?
  19. 19. When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?
  20. 20. When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?• Is it reproducible ?
  21. 21. When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?• Is it reproducible ?• Does the rest fail ? (user, account)
  22. 22. When the app runs fine
  23. 23. When the app runs fine• Information harvesting loop
  24. 24. When the app runs fine• Information harvesting loop • 1: get more info about the process run
  25. 25. When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/...
  26. 26. When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/... • 3: (s)he asks something I don’t know
  27. 27. When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/... • 3: (s)he asks something I don’t know • 4: go back to step 1
  28. 28. So yeah, I need some logs
  29. 29. So yeah, I need some logsBut we knew that, didn’t we ?
  30. 30. Logs
  31. 31. Logs
  32. 32. Logs
  33. 33. Exceptions
  34. 34. Exceptions
  35. 35. Exceptions• An exception is an issue
  36. 36. Exceptions• An exception is an issue• All issues are not fatal (important)
  37. 37. Exceptions• An exception is an issue• All issues are not fatal (important)• So, try / catch !
  38. 38. Exception Module
  39. 39. Exception Module• Pick your own from CPAN
  40. 40. Exception Module• Pick your own from CPAN• I wrote mine anyway :)
  41. 41. Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception
  42. 42. Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module
  43. 43. Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module• Exceptions are objects with message pattern, stack trace... raise(), throw(), register(), introspection
  44. 44. Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module• Exceptions are objects with message pattern, stack trace... raise(), throw(), register(), introspection• Should end up on CPAN one day
  45. 45. foreach account { try { foreach user { try { foreach data line { try { process_line() } catch { log warning }; } } catch { log important }; } } catch { log important };}log fatal
  46. 46. Logs
  47. 47. Logging a line
  48. 48. Logging a line• say "warning, data is not good"
  49. 49. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"
  50. 50. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"
  51. 51. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")
  52. 52. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")• my $logger = MyOwnLogger->new(write_in => file, display_level => WARNING);
  53. 53. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")• my $logger = MyOwnLogger->new(write_in => file, display_level => WARNING);• $logger->log(warning =>"$Level, $data is not good");
  54. 54. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")• my $logger = MyOwnLogger->new(write_in => file, display_level => WARNING);• $logger->log(warning =>"$Level, $data is not good");• Put that in a function, then a module, etc
  55. 55. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")• my $logger = MyOwnLogger->new(write_in => file, display_level => WARNING);• $logger->log(warning =>"$Level, $data is not good");• Put that in a function, then a module, etc
  56. 56. The wheel...
  57. 57. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.
  58. 58. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.• Sometimes multiple home-made logging system in a single company
  59. 59. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.• Sometimes multiple home-made logging system in a single company• Sometimes multiple home-made logging system in a single project
  60. 60. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.• Sometimes multiple home-made logging system in a single company• Sometimes multiple home-made logging system in a single project• Its so easy to start with a print STDERR...
  61. 61. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.• Sometimes multiple home-made logging system in a single company• Sometimes multiple home-made logging system in a single project• Its so easy to start with a print STDERR...• Stop reinventing the wheel
  62. 62. Pick up a logger
  63. 63. Pick up a logger• Many Log modules on CPAN
  64. 64. Pick up a logger• Many Log modules on CPAN• Pick one you like
  65. 65. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch
  66. 66. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch• $log->log(level => info, message => Blah );
  67. 67. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch• $log->log(level => info, message => Blah );• Good ?
  68. 68. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch• $log->log(level => info, message => Blah );• Good ?• No, we are still nowhere !
  69. 69. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch• $log->log(level => info, message => Blah );• Good ?• No, we are still nowhere !• We want contextual information
  70. 70. Contextual information
  71. 71. Contextual information• What do we need to log ?
  72. 72. Contextual information• What do we need to log ?• A log message - ok, but
  73. 73. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)
  74. 74. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software info (time, user / account id...)
  75. 75. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software info (time, user / account id...)• Message (filename, line nb, stacktrace...)
  76. 76. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software info (time, user / account id...)• Message (filename, line nb, stacktrace...)• Log line = mix contextual and specific info
  77. 77. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software info (time, user / account id...)• Message (filename, line nb, stacktrace...)• Log line = mix contextual and specific info• Even with a great logger, need to build that
  78. 78. Richer logging
  79. 79. Richer logging• How to pack these information together ?
  80. 80. Richer logging• How to pack these information together ?• Use a key-value structure
  81. 81. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes
  82. 82. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes• So we can benefit of defaults, lazyness, etc
  83. 83. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes• So we can benefit of defaults, lazyness, etc• One log line = one class instance
  84. 84. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes• So we can benefit of defaults, lazyness, etc• One log line = one class instance• We need it as a string at the end
  85. 85. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes• So we can benefit of defaults, lazyness, etc• One log line = one class instance• We need it as a string at the end• So stringify to JSON,YAML...
  86. 86. Log::Message::Structured
  87. 87. Log::Message::Structured • A set of roles, consume in MyEvent class
  88. 88. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles :
  89. 89. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles : • L::M::S : basic setup + "" overloading
  90. 90. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles : • L::M::S : basic setup + "" overloading • L::M::S::Components : provides ready to use additional fields
  91. 91. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles : • L::M::S : basic setup + "" overloading • L::M::S::Components : provides ready to use additional fields • L::M::S::Stringify : provides stringifiers
  92. 92. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles : • L::M::S : basic setup + "" overloading • L::M::S::Components : provides ready to use additional fields • L::M::S::Stringify : provides stringifiers • Your custom fields : additional attributes
  93. 93. package MyLogEvent; use Moose;with qw/ Log::Message::Structured Log::Message::Structured::Component::Date Log::Message::Structured::Component::Hostname Log::Message::Structured::Stringify::AsJSON/;has message => ( is => ro, required => 1);has account_id => ( is => ro, required => 1);has user_id => ( is => ro );
  94. 94. package MyLogEvent; use Moose;with qw/ Log::Message::Structured Log::Message::Structured::Component::Date Log::Message::Structured::Component::Hostname Log::Message::Structured::Stringify::AsJSON/;has message => ( is => ro, required => 1);has account_id => ( is => ro, required => 1);has user_id => ( is => ro ); ...elsewhere...
  95. 95. package MyLogEvent; use Moose;with qw/ Log::Message::Structured Log::Message::Structured::Component::Date Log::Message::Structured::Component::Hostname Log::Message::Structured::Stringify::AsJSON/;has message => ( is => ro, required => 1);has account_id => ( is => ro, required => 1);has user_id => ( is => ro ); ...elsewhere...use My::Log::Event;$logger->log( warning => MyLogEvent->new( account_id => 42, user_id => 12, message => "watch out! behind you!" ));
  96. 96. package MyLogEvent; use Moose;with qw/ Log::Message::Structured Log::Message::Structured::Component::Date Log::Message::Structured::Component::Hostname Log::Message::Structured::Stringify::AsJSON/;has message => ( is => ro, required => 1);has account_id => ( is => ro, required => 1);has user_id => ( is => ro ); ...elsewhere...use My::Log::Event;$logger->log( warning => MyLogEvent->new( account_id => 42, user_id => 12, message => "watch out! behind you!" )); Thats sending the stringified JSON to Log::Dispatch
  97. 97. Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine.domain",message=>"watch out! behind you!"}
  98. 98. Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine.domain",message=>"watch out! behind you!"} L::M::S also supports options passed on the command line
  99. 99. Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine.domain",message=>"watch out! behind you!"} L::M::S also supports options passed on the command line$ my_program.pl --account_id=42
  100. 100. Lets shorten the codeIn my application I have:
  101. 101. Lets shorten the code In my application I have:while (my $account_id = AccountIterator->next) { while (my $user_id = UserIterator->next) { while (my $data_line = DataIterator->next) { ... do stuff ...
  102. 102. Lets shorten the code In my application I have:while (my $account_id = AccountIterator->next) { while (my $user_id = UserIterator->next) { while (my $data_line = DataIterator->next) { ... do stuff ...package MyLogEvent; use Moose;with qw/Log::Message::Structured .... /;use AccountIterator; use UserIterator;has account_id => ( is => ro, default => sub { AccountIterator->current } );has user_id => ( is => ro, default => sub { UserIterator->current } );
  103. 103. Lets shorten the code
  104. 104. Lets shorten the codeWrap Log::Dispatch call + L::M::S
  105. 105. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!");
  106. 106. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!"); actually calls
  107. 107. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!"); actually calls$logger->log( warning => MyLogEvent->new( @_ );
  108. 108. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!"); actually calls$logger->log( warning => MyLogEvent->new( @_ ); outputs
  109. 109. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!"); actually calls$logger->log( warning => MyLogEvent->new( @_ ); outputs{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine.domain",message=>"Achtung!"}
  110. 110. Exceptions...Remember ?
  111. 111. Exceptions... Remember ?try { ... } catch { log_warning(exception => $_ )};
  112. 112. Exceptions... Remember ? try { ... } catch { log_warning(exception => $_ ) };• To be able to write that, you need in your MyLogEvent class:
  113. 113. Exceptions... Remember ? try { ... } catch { log_warning(exception => $_ ) };• To be able to write that, you need in your MyLogEvent class:• these attributes : exception, stack_trace,
  114. 114. Exceptions... Remember ? try { ... } catch { log_warning(exception => $_ ) };• To be able to write that, you need in your MyLogEvent class:• these attributes : exception, stack_trace,• with a BUILDARGS
  115. 115. Exceptions... Remember ? try { ... } catch { log_warning(exception => $_ ) };• To be able to write that, you need in your MyLogEvent class:• these attributes : exception, stack_trace,• with a BUILDARGS• that looks at the exception, and set the message and the stack_trace arguments
  116. 116. around BUILDARGS => sub { my $orig = shift; my $class = shift; my $h = $class->$orig(@_); my $e = $h->{exception} or return $h; if ( blessed($e) && $e->isa(My::Exception::Base) ) { $h->{message} = $e->message; $h->{shortmess} = $e->{_shortmess}; } else { $h->{message} = $e } return $h;};
  117. 117. around BUILDARGS => sub { my $orig = shift; my $class = shift; my $h = $class->$orig(@_); my $e = $h->{exception} or return $h; if ( blessed($e) && $e->isa(My::Exception::Base) ) { $h->{message} = $e->message; $h->{shortmess} = $e->{_shortmess}; } else { $h->{message} = $e } return $h;};try { ... } catch { log_warning(exception => $_ ) That would now}; work !
  118. 118. Log::Message::Structured
  119. 119. Log::Message::Structured • Does the job
  120. 120. Log::Message::Structured • Does the job • Please contribute
  121. 121. Log::Message::Structured • Does the job • Please contribute • we need more components
  122. 122. Log::Message::Structured • Does the job • Please contribute • we need more components • we need more stringifiers
  123. 123. Log::Message::Structured • Does the job • Please contribute • we need more components • we need more stringifiers • compatibility with Moo
  124. 124. Log::Message::Structured • Does the job • Please contribute • we need more components • we need more stringifiers • compatibility with Moo • On CPAN / github
  125. 125. • We have all we need as JSON
  126. 126. • We have all we need as JSON• But thats not enough :(
  127. 127. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log
  128. 128. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface
  129. 129. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research
  130. 130. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research • thousands of files, grep not good enough
  131. 131. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research • thousands of files, grep not good enough • => need a powerful search engine
  132. 132. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research • thousands of files, grep not good enough • => need a powerful search engine• Also, what if you have different log sources
  133. 133. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research • thousands of files, grep not good enough • => need a powerful search engine• Also, what if you have different log sources • From DB events, Redis MQ, system logs ?
  134. 134. Web interfaces
  135. 135. Web interfaces• Two interfaces
  136. 136. Web interfaces• Two interfaces• First one is easy to use
  137. 137. Web interfaces• Two interfaces• First one is easy to use • Based on Dancer + MongoDB
  138. 138. Web interfaces• Two interfaces• First one is easy to use • Based on Dancer + MongoDB • Shows logs in natural manner, grouped by job, hostname, and presented by time
  139. 139. Web interfaces• Two interfaces• First one is easy to use • Based on Dancer + MongoDB • Shows logs in natural manner, grouped by job, hostname, and presented by time• Second one is more for data mining
  140. 140. Web interfaces• Two interfaces• First one is easy to use • Based on Dancer + MongoDB • Shows logs in natural manner, grouped by job, hostname, and presented by time• Second one is more for data mining • Elasticsearch: deep searching, filtering, etc
  141. 141. We need some kind of big stuff syslog Standard log filesRedis MQ logs $stuff Web interface Other logs Search Engine
  142. 142. We need some kind of big stuff syslog Standard log filesRedis MQ logs $stuff Web interface Other logs Search Engine $stuff = Message::Passing
  143. 143. Message::Passing
  144. 144. Message::Passing• Written by Tomas Doran
  145. 145. Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style
  146. 146. Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)
  147. 147. Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)• basic concepts: inputs, filters, outputs
  148. 148. Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)• basic concepts: inputs, filters, outputs• run it as : command line, using the DSL, using classes directly.
  149. 149. Real Life
  150. 150. Real Life• My software runs on many machines
  151. 151. Real Life• My software runs on many machines• Connected to one syslog-ng server
  152. 152. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files
  153. 153. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng to output to a pipe
  154. 154. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng to output to a pipe• Branch M::P on this named pipe
  155. 155. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng to output to a pipe• Branch M::P on this named pipe• Web interface: Dancer + MongoDB
  156. 156. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng to output to a pipe• Branch M::P on this named pipe• Web interface: Dancer + MongoDB• Search interface : Elasticsearch + plugin
  157. 157. syslog-ng many logsmachines standard log files
  158. 158. syslog-ng duplicated flow many logsmachines standard log files Message::PassingElasticSearch MongoDB Web GUI Web GUI
  159. 159. logs MQ Message::Passing manymachines In a better worldElasticSearch MongoDB Web GUI Web GUI
  160. 160. Lets use Message::Passing
  161. 161. Lets use Message::Passing • step 1 : configure & run M::P daemon
  162. 162. Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ...
  163. 163. Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ... • step 3 : PROFIT
  164. 164. Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ... • step 3 : PROFIT Remember the basic concepts? inputs, filters, outputs
  165. 165. use Message::Passing::DSL; use Moose;with Message::Passing::Role::Script;sub build_chain { message_chain { input filetail => ( class => FileTail, filename => ‘/path/to/logs’, output_to => cleanup_logs ); filter cleanup_logs => ( class => +My::Filter::DeJSON, output_to => [ elasticsearch, mongodb ] ); output elasticsearch_servers => ( class => ElasticSearch, elasticsearch_servers => localhost:9200 ); output mongodb => ( class => MongoDB, hostname => localhost, database => reporting, collection => logs, indexes => [ # specify fields to index ] );}}__PACKAGE__->start;
  166. 166. My::Filter::DeJSONpackage My::Filter::DeJSON;use Moose;sub consume { my ( $self, $message ) = @_; $message =~ m/(w+):JSON:(.*)/ or return; my $structure_log = from_json($2); $structure_log->{log_level} = $1; foreach my $output_to ( @{ $self->output_to } ) { $output_to->consume($structure_log); }}
  167. 167. Message::Passing::Output::MongoDBuse Moose; use MongoDB; use AnyEvent;with qw/    Message::Passing::Role::Output    Message::Passing::Role::HasUsernameAndPassword    Message::Passing::Role::HasHostnameAndPort/;# ... hostname, port, user/pass attributes ...has _db => (    is => ro, isa => MongoDB::Database, lazy => 1,    default => sub {        my $self = shift;        my $connection = MongoDB::Connection->new(            host => $self->hostname,            port => $self->port,        );        return $connection->get_database($self->database);    },);
  168. 168. Message::Passing::Output::MongoDB# ... _collection attributesub _build_logs_collection {    my ($self) = @_;    my $collection_name = $self->collection;    my $collection = $self->_db->$collection_name;     if ($self->_has_indexes) {        foreach my $index (@{$self->indexes}){            $collection->ensure_index(@$index);        }    }    return $collection;}
  169. 169. Message::Passing::Output::MongoDBsub consume {    my ($self, $data) = @_;    return unless $data;    my $date;    my $collection = $self->_collection;    $collection->insert($data)      or warn "Insertion failure: " . Dumper($data) . "n";    if ($self->verbose) {        $self->_inc_log_counter;        warn("Total " . $self->_log_counter . " records inserted in MongoDBn");    }} On CPAN, by Bin Shu
  170. 170. Dancer web GUI
  171. 171. Dancer web GUI• Very simple Dancer application
  172. 172. Dancer web GUI• Very simple Dancer application• Could have used Dancer::Plugin::Mongo
  173. 173. Dancer web GUI• Very simple Dancer application• Could have used Dancer::Plugin::Mongo• A screen to display log
  174. 174. Dancer web GUI• Very simple Dancer application• Could have used Dancer::Plugin::Mongo• A screen to display log• A screen to search (filter) logs
  175. 175. Dancer web GUI• Very simple Dancer application• Could have used Dancer::Plugin::Mongo• A screen to display log• A screen to search (filter) logs• Result page always show logs ordered by time
  176. 176. Dancer GUI
  177. 177. Dancer GUI
  178. 178. Dancer GUI routes
  179. 179. Dancer GUI routes• get / :
  180. 180. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result
  181. 181. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search :
  182. 182. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result
  183. 183. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result• get /result :
  184. 184. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result• get /result : • get search params from url, issue a find request
  185. 185. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result• get /result : • get search params from url, issue a find request • display in a table
  186. 186. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result• get /result : • get search params from url, issue a find request • display in a table• clicking on a result brings up a JS detailed result popup
  187. 187. Show me the video
  188. 188. Elasticsearch
  189. 189. Elasticsearch• Installation : trivial
  190. 190. Elasticsearch• Installation : trivial• Configuration : none
  191. 191. Elasticsearch• Installation : trivial• Configuration : none• Web GUI : elasticsearch-head
  192. 192. Elasticsearch• Installation : trivial• Configuration : none• Web GUI : elasticsearch-head• Code ? there is no code to show :)
  193. 193. Elasticsearch• Installation : trivial• Configuration : none• Web GUI : elasticsearch-head• Code ? there is no code to show :)• Usage : perform advanced search on massive backlog
  194. 194. Show me the video
  195. 195. Additional stuff• If there is an error in Message::Passing • You should use the error channel • To avoid loop of death
  196. 196. Conclusion• Thats all there is about it, really• Message::Passing is easy and work great• Log::Message::Structured : an awesome idea
  197. 197. Questions ? Thanks !Damien dams Krotkine

×