Easy Loggingwith Log::Message::Structured and Message::Passing
MeDamien dams KrotkineParis.pm memberFrench Perl Monger vice-presidentPerl Dancer developer11 modules on CPAN    (pretty w...
• Let’s say I have a big application
• Let’s say I have a big application• It processes a lot of data
• Let’s say I have a big application• It processes a lot of data• It computes lots of lines
• Let’s say I have a big application• It processes a lot of data• It computes lots of lines• Per user, and per account
account 1user 1         linesuser 2                                  processuser 3                      account n         ...
• It runs fine
• It runs fine• Well, more or less
• It runs fine• Well, more or less• I’d like to know what’s going on with it
When the app crashes
When the app crashes• Why did it crash ? ( reason, stack trace )
When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time
When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time • What was the situation ?
When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time • What was the situation ? • What was the ...
When there are issues
When there are issues• How bad is this failure ?
When there are issues• How bad is this failure ?• Why did it fail to process this line ?
When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?
When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?• Is it re...
When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?• Is it re...
When the app runs fine
When the app runs fine• Information harvesting loop
When the app runs fine• Information harvesting loop • 1: get more info about the process run
When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/...
When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/...
When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/...
So yeah, I need some logs
So yeah, I need some logsBut we knew that, didn’t we ?
Logs
Logs
Logs
Exceptions
Exceptions
Exceptions• An exception is an issue
Exceptions• An exception is an issue• All issues are not fatal (important)
Exceptions• An exception is an issue• All issues are not fatal (important)• So, try / catch !
Exception Module
Exception Module• Pick your own from CPAN
Exception Module• Pick your own from CPAN• I wrote mine anyway :)
Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception
Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module
Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module•...
Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module•...
foreach account {    try {        foreach user {            try {                foreach data line {                    tr...
Logs
Logging a line
Logging a line• say "warning, data is not good"
Logging a line• say "warning, data is not good"• say "warning, data $data is not good"
Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"
Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not goo...
Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not goo...
Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not goo...
Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not goo...
Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not goo...
The wheel...
The wheel...• Home-made logging system : Ive seen that  in every single company Ive worked.
The wheel...• Home-made logging system : Ive seen that  in every single company Ive worked.• Sometimes multiple home-made ...
The wheel...• Home-made logging system : Ive seen that  in every single company Ive worked.• Sometimes multiple home-made ...
The wheel...• Home-made logging system : Ive seen that  in every single company Ive worked.• Sometimes multiple home-made ...
The wheel...• Home-made logging system : Ive seen that  in every single company Ive worked.• Sometimes multiple home-made ...
Pick up a logger
Pick up a logger• Many Log modules on CPAN
Pick up a logger• Many Log modules on CPAN• Pick one you like
Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch
Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch•   $log->log(level => info, message ...
Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch•   $log->log(level => info, message ...
Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch•   $log->log(level => info, message ...
Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch•   $log->log(level => info, message ...
Contextual information
Contextual information• What do we need to log ?
Contextual information• What do we need to log ?• A log message - ok, but
Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)
Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software ...
Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software ...
Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software ...
Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software ...
Richer logging
Richer logging• How to pack these information together ?
Richer logging• How to pack these information together ?• Use a key-value structure
Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes
Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes...
Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes...
Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes...
Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes...
Log::Message::Structured
Log::Message::Structured • A set of roles, consume in MyEvent class
Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles :
Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles :  • L::M::S : basic setup + "" ove...
Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles :  • L::M::S : basic setup + "" ove...
Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles :  • L::M::S : basic setup + "" ove...
Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles :  • L::M::S : basic setup + "" ove...
package MyLogEvent; use Moose;with qw/    Log::Message::Structured    Log::Message::Structured::Component::Date    Log::Me...
package MyLogEvent; use Moose;with qw/    Log::Message::Structured    Log::Message::Structured::Component::Date    Log::Me...
package MyLogEvent; use Moose;with qw/    Log::Message::Structured    Log::Message::Structured::Component::Date    Log::Me...
package MyLogEvent; use Moose;with qw/    Log::Message::Structured    Log::Message::Structured::Component::Date    Log::Me...
Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine....
Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine....
Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine....
Lets shorten the codeIn my application I have:
Lets shorten the code      In my application I have:while (my $account_id = AccountIterator->next) {  while (my $user_id =...
Lets shorten the code      In my application I have:while (my $account_id = AccountIterator->next) {  while (my $user_id =...
Lets shorten the code
Lets shorten the codeWrap Log::Dispatch call + L::M::S
Lets shorten the code      Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!");
Lets shorten the code      Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!");       actually calls
Lets shorten the code      Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!");       actually calls$logg...
Lets shorten the code      Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!");       actually calls$logg...
Lets shorten the code      Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!");       actually calls$logg...
Exceptions...Remember ?
Exceptions...        Remember ?try { ... } catch {    log_warning(exception => $_ )};
Exceptions...           Remember ?   try { ... } catch {       log_warning(exception => $_ )   };• To be able to write tha...
Exceptions...            Remember ?   try { ... } catch {       log_warning(exception => $_ )   };• To be able to write th...
Exceptions...            Remember ?   try { ... } catch {       log_warning(exception => $_ )   };• To be able to write th...
Exceptions...            Remember ?   try { ... } catch {       log_warning(exception => $_ )   };• To be able to write th...
around BUILDARGS => sub {    my $orig = shift;    my $class = shift;     my $h = $class->$orig(@_);     my $e = $h->{excep...
around BUILDARGS => sub {    my $orig = shift;    my $class = shift;     my $h = $class->$orig(@_);     my $e = $h->{excep...
Log::Message::Structured
Log::Message::Structured • Does the job
Log::Message::Structured • Does the job • Please contribute
Log::Message::Structured • Does the job • Please contribute  • we need more components
Log::Message::Structured • Does the job • Please contribute  • we need more components  • we need more stringifiers
Log::Message::Structured • Does the job • Please contribute  • we need more components  • we need more stringifiers  • comp...
Log::Message::Structured • Does the job • Please contribute  • we need more components  • we need more stringifiers  • comp...
• We have all we need as JSON
• We have all we need as JSON• But thats not enough :(
• We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log
• We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface
• We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface...
• We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface...
• We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface...
• We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface...
• We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface...
Web interfaces
Web interfaces• Two interfaces
Web interfaces• Two interfaces• First one is easy to use
Web interfaces• Two interfaces• First one is easy to use  • Based on Dancer + MongoDB
Web interfaces• Two interfaces• First one is easy to use  • Based on Dancer + MongoDB  • Shows logs in natural manner, gro...
Web interfaces• Two interfaces• First one is easy to use  • Based on Dancer + MongoDB  • Shows logs in natural manner, gro...
Web interfaces• Two interfaces• First one is easy to use  • Based on Dancer + MongoDB  • Shows logs in natural manner, gro...
We need some kind of big stuff   syslog                                      Standard log filesRedis MQ logs          $stuf...
We need some kind of big stuff   syslog                                      Standard log filesRedis MQ logs          $stuf...
Message::Passing
Message::Passing• Written by Tomas Doran
Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style
Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)
Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)• basic c...
Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)• basic c...
Real Life
Real Life• My software runs on many machines
Real Life• My software runs on many machines• Connected to one syslog-ng server
Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files
Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng t...
Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng t...
Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng t...
Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng t...
syslog-ng many      logsmachines                  standard                   log files
syslog-ng    duplicated                                        flow many           logsmachines                         sta...
logs MQ                                    Message::Passing manymachines                In a better worldElasticSearch    ...
Lets use Message::Passing
Lets use Message::Passing • step 1 : configure & run M::P daemon
Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ...
Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ... • step 3 : PROFIT
Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ... • step 3 : PROFIT Remember the basic concep...
use Message::Passing::DSL; use Moose;with Message::Passing::Role::Script;sub build_chain { message_chain {  input filetail...
My::Filter::DeJSONpackage My::Filter::DeJSON;use Moose;sub consume {    my ( $self, $message ) = @_;    $message =~ m/(w+)...
Message::Passing::Output::MongoDBuse Moose; use MongoDB; use AnyEvent;with qw/    Message::Passing::Role::Output    Messag...
Message::Passing::Output::MongoDB# ... _collection attributesub _build_logs_collection {    my ($self) = @_;    my $collec...
Message::Passing::Output::MongoDBsub consume {    my ($self, $data) = @_;    return unless $data;    my $date;    my $coll...
Dancer web GUI
Dancer web GUI•   Very simple Dancer application
Dancer web GUI•   Very simple Dancer application•   Could have used Dancer::Plugin::Mongo
Dancer web GUI•   Very simple Dancer application•   Could have used Dancer::Plugin::Mongo•   A screen to display log
Dancer web GUI•   Very simple Dancer application•   Could have used Dancer::Plugin::Mongo•   A screen to display log•   A ...
Dancer web GUI•   Very simple Dancer application•   Could have used Dancer::Plugin::Mongo•   A screen to display log•   A ...
Dancer GUI
Dancer GUI
Dancer GUI routes
Dancer GUI routes•   get /   :
Dancer GUI routes•   get /   :    • a mongoDB group request per host, command, => /result
Dancer GUI routes•   get /   :    • a mongoDB group request per host, command, => /result•   get /search   :
Dancer GUI routes•   get /   :    • a mongoDB group request per host, command, => /result•   get /search   :    • a mapred...
Dancer GUI routes•   get /   :    • a mongoDB group request per host, command, => /result•   get /search   :    • a mapred...
Dancer GUI routes•   get /   :    • a mongoDB group request per host, command, => /result•   get /search   :    • a mapred...
Dancer GUI routes•   get /   :    • a mongoDB group request per host, command, => /result•   get /search   :    • a mapred...
Dancer GUI routes•   get /   :    • a mongoDB group request per host, command, => /result•   get /search   :    • a mapred...
Show me the video
Elasticsearch
Elasticsearch•   Installation : trivial
Elasticsearch•   Installation : trivial•   Configuration : none
Elasticsearch•   Installation : trivial•   Configuration : none•   Web GUI : elasticsearch-head
Elasticsearch•   Installation : trivial•   Configuration : none•   Web GUI : elasticsearch-head•   Code ? there is no code ...
Elasticsearch•   Installation : trivial•   Configuration : none•   Web GUI : elasticsearch-head•   Code ? there is no code ...
Show me the video
Additional stuff• If there is an error in Message::Passing • You should use the error channel • To avoid loop of death
Conclusion• Thats all there is about it, really• Message::Passing is easy and work great• Log::Message::Structured : an aw...
Questions ?      Thanks !Damien dams Krotkine
Message passing
Message passing
Message passing
Message passing
Message passing
Message passing
Message passing
Message passing
Upcoming SlideShare
Loading in …5
×

Message passing

758 views

Published on

Presentation given at YAPC::EU 2012

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
758
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Message passing

    1. 1. Easy Loggingwith Log::Message::Structured and Message::Passing
    2. 2. MeDamien dams KrotkineParis.pm memberFrench Perl Monger vice-presidentPerl Dancer developer11 modules on CPAN (pretty weak)Author of Moderne Perl (french book, go buy it)
    3. 3. • Let’s say I have a big application
    4. 4. • Let’s say I have a big application• It processes a lot of data
    5. 5. • Let’s say I have a big application• It processes a lot of data• It computes lots of lines
    6. 6. • Let’s say I have a big application• It processes a lot of data• It computes lots of lines• Per user, and per account
    7. 7. account 1user 1 linesuser 2 processuser 3 account n result running on multiple hosts
    8. 8. • It runs fine
    9. 9. • It runs fine• Well, more or less
    10. 10. • It runs fine• Well, more or less• I’d like to know what’s going on with it
    11. 11. When the app crashes
    12. 12. When the app crashes• Why did it crash ? ( reason, stack trace )
    13. 13. When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time
    14. 14. When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time • What was the situation ?
    15. 15. When the app crashes• Why did it crash ? ( reason, stack trace )• At crash time • What was the situation ? • What was the data it was processing ?
    16. 16. When there are issues
    17. 17. When there are issues• How bad is this failure ?
    18. 18. When there are issues• How bad is this failure ?• Why did it fail to process this line ?
    19. 19. When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?
    20. 20. When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?• Is it reproducible ?
    21. 21. When there are issues• How bad is this failure ?• Why did it fail to process this line ?• Is it only this line ?• Is it reproducible ?• Does the rest fail ? (user, account)
    22. 22. When the app runs fine
    23. 23. When the app runs fine• Information harvesting loop
    24. 24. When the app runs fine• Information harvesting loop • 1: get more info about the process run
    25. 25. When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/...
    26. 26. When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/... • 3: (s)he asks something I don’t know
    27. 27. When the app runs fine• Information harvesting loop • 1: get more info about the process run • 2: go see my boss/colleague/... • 3: (s)he asks something I don’t know • 4: go back to step 1
    28. 28. So yeah, I need some logs
    29. 29. So yeah, I need some logsBut we knew that, didn’t we ?
    30. 30. Logs
    31. 31. Logs
    32. 32. Logs
    33. 33. Exceptions
    34. 34. Exceptions
    35. 35. Exceptions• An exception is an issue
    36. 36. Exceptions• An exception is an issue• All issues are not fatal (important)
    37. 37. Exceptions• An exception is an issue• All issues are not fatal (important)• So, try / catch !
    38. 38. Exception Module
    39. 39. Exception Module• Pick your own from CPAN
    40. 40. Exception Module• Pick your own from CPAN• I wrote mine anyway :)
    41. 41. Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception
    42. 42. Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module
    43. 43. Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module• Exceptions are objects with message pattern, stack trace... raise(), throw(), register(), introspection
    44. 44. Exception Module• Pick your own from CPAN• I wrote mine anyway :)• Dancer::Exception• Extracted it as a standalone module• Exceptions are objects with message pattern, stack trace... raise(), throw(), register(), introspection• Should end up on CPAN one day
    45. 45. foreach account { try { foreach user { try { foreach data line { try { process_line() } catch { log warning }; } } catch { log important }; } } catch { log important };}log fatal
    46. 46. Logs
    47. 47. Logging a line
    48. 48. Logging a line• say "warning, data is not good"
    49. 49. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"
    50. 50. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"
    51. 51. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")
    52. 52. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")• my $logger = MyOwnLogger->new(write_in => file, display_level => WARNING);
    53. 53. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")• my $logger = MyOwnLogger->new(write_in => file, display_level => WARNING);• $logger->log(warning =>"$Level, $data is not good");
    54. 54. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")• my $logger = MyOwnLogger->new(write_in => file, display_level => WARNING);• $logger->log(warning =>"$Level, $data is not good");• Put that in a function, then a module, etc
    55. 55. Logging a line• say "warning, data is not good"• say "warning, data $data is not good"• say "$level, data $data is not good"• log( in_file => "$level, $data is not good")• my $logger = MyOwnLogger->new(write_in => file, display_level => WARNING);• $logger->log(warning =>"$Level, $data is not good");• Put that in a function, then a module, etc
    56. 56. The wheel...
    57. 57. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.
    58. 58. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.• Sometimes multiple home-made logging system in a single company
    59. 59. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.• Sometimes multiple home-made logging system in a single company• Sometimes multiple home-made logging system in a single project
    60. 60. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.• Sometimes multiple home-made logging system in a single company• Sometimes multiple home-made logging system in a single project• Its so easy to start with a print STDERR...
    61. 61. The wheel...• Home-made logging system : Ive seen that in every single company Ive worked.• Sometimes multiple home-made logging system in a single company• Sometimes multiple home-made logging system in a single project• Its so easy to start with a print STDERR...• Stop reinventing the wheel
    62. 62. Pick up a logger
    63. 63. Pick up a logger• Many Log modules on CPAN
    64. 64. Pick up a logger• Many Log modules on CPAN• Pick one you like
    65. 65. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch
    66. 66. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch• $log->log(level => info, message => Blah );
    67. 67. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch• $log->log(level => info, message => Blah );• Good ?
    68. 68. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch• $log->log(level => info, message => Blah );• Good ?• No, we are still nowhere !
    69. 69. Pick up a logger• Many Log modules on CPAN• Pick one you like• I choose Log::Dispatch• $log->log(level => info, message => Blah );• Good ?• No, we are still nowhere !• We want contextual information
    70. 70. Contextual information
    71. 71. Contextual information• What do we need to log ?
    72. 72. Contextual information• What do we need to log ?• A log message - ok, but
    73. 73. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)
    74. 74. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software info (time, user / account id...)
    75. 75. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software info (time, user / account id...)• Message (filename, line nb, stacktrace...)
    76. 76. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software info (time, user / account id...)• Message (filename, line nb, stacktrace...)• Log line = mix contextual and specific info
    77. 77. Contextual information• What do we need to log ?• A log message - ok, but• Hardware info (Mem, CPU, disk, host)• Software info (time, user / account id...)• Message (filename, line nb, stacktrace...)• Log line = mix contextual and specific info• Even with a great logger, need to build that
    78. 78. Richer logging
    79. 79. Richer logging• How to pack these information together ?
    80. 80. Richer logging• How to pack these information together ?• Use a key-value structure
    81. 81. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes
    82. 82. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes• So we can benefit of defaults, lazyness, etc
    83. 83. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes• So we can benefit of defaults, lazyness, etc• One log line = one class instance
    84. 84. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes• So we can benefit of defaults, lazyness, etc• One log line = one class instance• We need it as a string at the end
    85. 85. Richer logging• How to pack these information together ?• Use a key-value structure• Better: a class, fields are attributes• So we can benefit of defaults, lazyness, etc• One log line = one class instance• We need it as a string at the end• So stringify to JSON,YAML...
    86. 86. Log::Message::Structured
    87. 87. Log::Message::Structured • A set of roles, consume in MyEvent class
    88. 88. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles :
    89. 89. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles : • L::M::S : basic setup + "" overloading
    90. 90. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles : • L::M::S : basic setup + "" overloading • L::M::S::Components : provides ready to use additional fields
    91. 91. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles : • L::M::S : basic setup + "" overloading • L::M::S::Components : provides ready to use additional fields • L::M::S::Stringify : provides stringifiers
    92. 92. Log::Message::Structured • A set of roles, consume in MyEvent class • 4 types of roles : • L::M::S : basic setup + "" overloading • L::M::S::Components : provides ready to use additional fields • L::M::S::Stringify : provides stringifiers • Your custom fields : additional attributes
    93. 93. package MyLogEvent; use Moose;with qw/ Log::Message::Structured Log::Message::Structured::Component::Date Log::Message::Structured::Component::Hostname Log::Message::Structured::Stringify::AsJSON/;has message => ( is => ro, required => 1);has account_id => ( is => ro, required => 1);has user_id => ( is => ro );
    94. 94. package MyLogEvent; use Moose;with qw/ Log::Message::Structured Log::Message::Structured::Component::Date Log::Message::Structured::Component::Hostname Log::Message::Structured::Stringify::AsJSON/;has message => ( is => ro, required => 1);has account_id => ( is => ro, required => 1);has user_id => ( is => ro ); ...elsewhere...
    95. 95. package MyLogEvent; use Moose;with qw/ Log::Message::Structured Log::Message::Structured::Component::Date Log::Message::Structured::Component::Hostname Log::Message::Structured::Stringify::AsJSON/;has message => ( is => ro, required => 1);has account_id => ( is => ro, required => 1);has user_id => ( is => ro ); ...elsewhere...use My::Log::Event;$logger->log( warning => MyLogEvent->new( account_id => 42, user_id => 12, message => "watch out! behind you!" ));
    96. 96. package MyLogEvent; use Moose;with qw/ Log::Message::Structured Log::Message::Structured::Component::Date Log::Message::Structured::Component::Hostname Log::Message::Structured::Stringify::AsJSON/;has message => ( is => ro, required => 1);has account_id => ( is => ro, required => 1);has user_id => ( is => ro ); ...elsewhere...use My::Log::Event;$logger->log( warning => MyLogEvent->new( account_id => 42, user_id => 12, message => "watch out! behind you!" )); Thats sending the stringified JSON to Log::Dispatch
    97. 97. Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine.domain",message=>"watch out! behind you!"}
    98. 98. Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine.domain",message=>"watch out! behind you!"} L::M::S also supports options passed on the command line
    99. 99. Outputs as log:{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine.domain",message=>"watch out! behind you!"} L::M::S also supports options passed on the command line$ my_program.pl --account_id=42
    100. 100. Lets shorten the codeIn my application I have:
    101. 101. Lets shorten the code In my application I have:while (my $account_id = AccountIterator->next) { while (my $user_id = UserIterator->next) { while (my $data_line = DataIterator->next) { ... do stuff ...
    102. 102. Lets shorten the code In my application I have:while (my $account_id = AccountIterator->next) { while (my $user_id = UserIterator->next) { while (my $data_line = DataIterator->next) { ... do stuff ...package MyLogEvent; use Moose;with qw/Log::Message::Structured .... /;use AccountIterator; use UserIterator;has account_id => ( is => ro, default => sub { AccountIterator->current } );has user_id => ( is => ro, default => sub { UserIterator->current } );
    103. 103. Lets shorten the code
    104. 104. Lets shorten the codeWrap Log::Dispatch call + L::M::S
    105. 105. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!");
    106. 106. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!"); actually calls
    107. 107. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!"); actually calls$logger->log( warning => MyLogEvent->new( @_ );
    108. 108. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!"); actually calls$logger->log( warning => MyLogEvent->new( @_ ); outputs
    109. 109. Lets shorten the code Wrap Log::Dispatch call + L::M::Slog_warning( message => "Achtung!"); actually calls$logger->log( warning => MyLogEvent->new( @_ ); outputs{"__CLASS__":"MyLogEvent","account_id":42,"user_id":42,"date":"2010-03-28T23:15:52Z","hostname":"mymachine.domain",message=>"Achtung!"}
    110. 110. Exceptions...Remember ?
    111. 111. Exceptions... Remember ?try { ... } catch { log_warning(exception => $_ )};
    112. 112. Exceptions... Remember ? try { ... } catch { log_warning(exception => $_ ) };• To be able to write that, you need in your MyLogEvent class:
    113. 113. Exceptions... Remember ? try { ... } catch { log_warning(exception => $_ ) };• To be able to write that, you need in your MyLogEvent class:• these attributes : exception, stack_trace,
    114. 114. Exceptions... Remember ? try { ... } catch { log_warning(exception => $_ ) };• To be able to write that, you need in your MyLogEvent class:• these attributes : exception, stack_trace,• with a BUILDARGS
    115. 115. Exceptions... Remember ? try { ... } catch { log_warning(exception => $_ ) };• To be able to write that, you need in your MyLogEvent class:• these attributes : exception, stack_trace,• with a BUILDARGS• that looks at the exception, and set the message and the stack_trace arguments
    116. 116. around BUILDARGS => sub { my $orig = shift; my $class = shift; my $h = $class->$orig(@_); my $e = $h->{exception} or return $h; if ( blessed($e) && $e->isa(My::Exception::Base) ) { $h->{message} = $e->message; $h->{shortmess} = $e->{_shortmess}; } else { $h->{message} = $e } return $h;};
    117. 117. around BUILDARGS => sub { my $orig = shift; my $class = shift; my $h = $class->$orig(@_); my $e = $h->{exception} or return $h; if ( blessed($e) && $e->isa(My::Exception::Base) ) { $h->{message} = $e->message; $h->{shortmess} = $e->{_shortmess}; } else { $h->{message} = $e } return $h;};try { ... } catch { log_warning(exception => $_ ) That would now}; work !
    118. 118. Log::Message::Structured
    119. 119. Log::Message::Structured • Does the job
    120. 120. Log::Message::Structured • Does the job • Please contribute
    121. 121. Log::Message::Structured • Does the job • Please contribute • we need more components
    122. 122. Log::Message::Structured • Does the job • Please contribute • we need more components • we need more stringifiers
    123. 123. Log::Message::Structured • Does the job • Please contribute • we need more components • we need more stringifiers • compatibility with Moo
    124. 124. Log::Message::Structured • Does the job • Please contribute • we need more components • we need more stringifiers • compatibility with Moo • On CPAN / github
    125. 125. • We have all we need as JSON
    126. 126. • We have all we need as JSON• But thats not enough :(
    127. 127. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log
    128. 128. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface
    129. 129. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research
    130. 130. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research • thousands of files, grep not good enough
    131. 131. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research • thousands of files, grep not good enough • => need a powerful search engine
    132. 132. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research • thousands of files, grep not good enough • => need a powerful search engine• Also, what if you have different log sources
    133. 133. • We have all we need as JSON• But thats not enough :(• Your boss wants to look at the log • => provide a simple interface• You need to do complex lookups / research • thousands of files, grep not good enough • => need a powerful search engine• Also, what if you have different log sources • From DB events, Redis MQ, system logs ?
    134. 134. Web interfaces
    135. 135. Web interfaces• Two interfaces
    136. 136. Web interfaces• Two interfaces• First one is easy to use
    137. 137. Web interfaces• Two interfaces• First one is easy to use • Based on Dancer + MongoDB
    138. 138. Web interfaces• Two interfaces• First one is easy to use • Based on Dancer + MongoDB • Shows logs in natural manner, grouped by job, hostname, and presented by time
    139. 139. Web interfaces• Two interfaces• First one is easy to use • Based on Dancer + MongoDB • Shows logs in natural manner, grouped by job, hostname, and presented by time• Second one is more for data mining
    140. 140. Web interfaces• Two interfaces• First one is easy to use • Based on Dancer + MongoDB • Shows logs in natural manner, grouped by job, hostname, and presented by time• Second one is more for data mining • Elasticsearch: deep searching, filtering, etc
    141. 141. We need some kind of big stuff syslog Standard log filesRedis MQ logs $stuff Web interface Other logs Search Engine
    142. 142. We need some kind of big stuff syslog Standard log filesRedis MQ logs $stuff Web interface Other logs Search Engine $stuff = Message::Passing
    143. 143. Message::Passing
    144. 144. Message::Passing• Written by Tomas Doran
    145. 145. Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style
    146. 146. Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)
    147. 147. Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)• basic concepts: inputs, filters, outputs
    148. 148. Message::Passing• Written by Tomas Doran• Logstash-like but in Perl and with style• Its a daemon (AnyEvent based)• basic concepts: inputs, filters, outputs• run it as : command line, using the DSL, using classes directly.
    149. 149. Real Life
    150. 150. Real Life• My software runs on many machines
    151. 151. Real Life• My software runs on many machines• Connected to one syslog-ng server
    152. 152. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files
    153. 153. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng to output to a pipe
    154. 154. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng to output to a pipe• Branch M::P on this named pipe
    155. 155. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng to output to a pipe• Branch M::P on this named pipe• Web interface: Dancer + MongoDB
    156. 156. Real Life• My software runs on many machines• Connected to one syslog-ng server• Writing to log files• Configure syslog-ng to output to a pipe• Branch M::P on this named pipe• Web interface: Dancer + MongoDB• Search interface : Elasticsearch + plugin
    157. 157. syslog-ng many logsmachines standard log files
    158. 158. syslog-ng duplicated flow many logsmachines standard log files Message::PassingElasticSearch MongoDB Web GUI Web GUI
    159. 159. logs MQ Message::Passing manymachines In a better worldElasticSearch MongoDB Web GUI Web GUI
    160. 160. Lets use Message::Passing
    161. 161. Lets use Message::Passing • step 1 : configure & run M::P daemon
    162. 162. Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ...
    163. 163. Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ... • step 3 : PROFIT
    164. 164. Lets use Message::Passing • step 1 : configure & run M::P daemon • step 2 : ... • step 3 : PROFIT Remember the basic concepts? inputs, filters, outputs
    165. 165. use Message::Passing::DSL; use Moose;with Message::Passing::Role::Script;sub build_chain { message_chain { input filetail => ( class => FileTail, filename => ‘/path/to/logs’, output_to => cleanup_logs ); filter cleanup_logs => ( class => +My::Filter::DeJSON, output_to => [ elasticsearch, mongodb ] ); output elasticsearch_servers => ( class => ElasticSearch, elasticsearch_servers => localhost:9200 ); output mongodb => ( class => MongoDB, hostname => localhost, database => reporting, collection => logs, indexes => [ # specify fields to index ] );}}__PACKAGE__->start;
    166. 166. My::Filter::DeJSONpackage My::Filter::DeJSON;use Moose;sub consume { my ( $self, $message ) = @_; $message =~ m/(w+):JSON:(.*)/ or return; my $structure_log = from_json($2); $structure_log->{log_level} = $1; foreach my $output_to ( @{ $self->output_to } ) { $output_to->consume($structure_log); }}
    167. 167. Message::Passing::Output::MongoDBuse Moose; use MongoDB; use AnyEvent;with qw/    Message::Passing::Role::Output    Message::Passing::Role::HasUsernameAndPassword    Message::Passing::Role::HasHostnameAndPort/;# ... hostname, port, user/pass attributes ...has _db => (    is => ro, isa => MongoDB::Database, lazy => 1,    default => sub {        my $self = shift;        my $connection = MongoDB::Connection->new(            host => $self->hostname,            port => $self->port,        );        return $connection->get_database($self->database);    },);
    168. 168. Message::Passing::Output::MongoDB# ... _collection attributesub _build_logs_collection {    my ($self) = @_;    my $collection_name = $self->collection;    my $collection = $self->_db->$collection_name;     if ($self->_has_indexes) {        foreach my $index (@{$self->indexes}){            $collection->ensure_index(@$index);        }    }    return $collection;}
    169. 169. Message::Passing::Output::MongoDBsub consume {    my ($self, $data) = @_;    return unless $data;    my $date;    my $collection = $self->_collection;    $collection->insert($data)      or warn "Insertion failure: " . Dumper($data) . "n";    if ($self->verbose) {        $self->_inc_log_counter;        warn("Total " . $self->_log_counter . " records inserted in MongoDBn");    }} On CPAN, by Bin Shu
    170. 170. Dancer web GUI
    171. 171. Dancer web GUI• Very simple Dancer application
    172. 172. Dancer web GUI• Very simple Dancer application• Could have used Dancer::Plugin::Mongo
    173. 173. Dancer web GUI• Very simple Dancer application• Could have used Dancer::Plugin::Mongo• A screen to display log
    174. 174. Dancer web GUI• Very simple Dancer application• Could have used Dancer::Plugin::Mongo• A screen to display log• A screen to search (filter) logs
    175. 175. Dancer web GUI• Very simple Dancer application• Could have used Dancer::Plugin::Mongo• A screen to display log• A screen to search (filter) logs• Result page always show logs ordered by time
    176. 176. Dancer GUI
    177. 177. Dancer GUI
    178. 178. Dancer GUI routes
    179. 179. Dancer GUI routes• get / :
    180. 180. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result
    181. 181. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search :
    182. 182. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result
    183. 183. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result• get /result :
    184. 184. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result• get /result : • get search params from url, issue a find request
    185. 185. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result• get /result : • get search params from url, issue a find request • display in a table
    186. 186. Dancer GUI routes• get / : • a mongoDB group request per host, command, => /result• get /search : • a mapreduce to display asearch form, => /result• get /result : • get search params from url, issue a find request • display in a table• clicking on a result brings up a JS detailed result popup
    187. 187. Show me the video
    188. 188. Elasticsearch
    189. 189. Elasticsearch• Installation : trivial
    190. 190. Elasticsearch• Installation : trivial• Configuration : none
    191. 191. Elasticsearch• Installation : trivial• Configuration : none• Web GUI : elasticsearch-head
    192. 192. Elasticsearch• Installation : trivial• Configuration : none• Web GUI : elasticsearch-head• Code ? there is no code to show :)
    193. 193. Elasticsearch• Installation : trivial• Configuration : none• Web GUI : elasticsearch-head• Code ? there is no code to show :)• Usage : perform advanced search on massive backlog
    194. 194. Show me the video
    195. 195. Additional stuff• If there is an error in Message::Passing • You should use the error channel • To avoid loop of death
    196. 196. Conclusion• Thats all there is about it, really• Message::Passing is easy and work great• Log::Message::Structured : an awesome idea
    197. 197. Questions ? Thanks !Damien dams Krotkine

    ×