Logging and Analytics
Using Perl and Fluentd
Jason Crome
Technical Lead, All Around the World
June 21, 2016 Copyright 2016...
Who’s this for?
• People who want to know more about enterprise
logging (beginner and intermediate)
• People who want to k...
Who Am I?
• Project manager for Veure at AAW
• Formerly spent 18 years writing software for local
government
• Dancer core...
Veure’s Building Blocks
• Catalyst
• Template Toolkit
• DBIx::Class
• Moose
• PostgreSQL
Copyright 2016, Jason A. CromeJun...
– Colin Eberhardt, The Art of Logging
“Logging is the process of recording
application actions and state to a secondary
in...
Why Do We Log?
• Prevent cheating
• Help determine effectiveness of game content
• Find out what players are spending mone...
Why Fluentd?
• Log collector/aggregator (“Unified Logging Layer”)
• Log from any layer in the stack
• Asynchronous
• Cluste...
Why Fluentd?
• Supports structured logging
• Self-hosted
• Affordable/free
• Memory-friendly
• Built-in reliability
• Supe...
Why not Fluentd?
• Ruby :/
• Overkill for many projects
Copyright 2016, Jason A. CromeJune 21, 2016
Alternatives
• logstash
• Splunk
• Loggly
• Papertrail
Copyright 2016, Jason A. CromeJune 21, 2016
Why Log4Perl?
• Does what we need
• Has everything and the kitchen sink
• Easily integrates with Catalyst
• It already tal...
Where Do We Log From?
• DBIx::Class
• Other Moose-based classes
• Catalyst
• TT (this is rare)
• Stack (future)
Copyright ...
What Do We Log?
Copyright 2016, Jason A. CromeJune 21, 2016
Copyright 2016, Jason A. CromeJune 21, 2016
What Do We Log?
Copyright 2016, Jason A. CromeJune 21, 2016
What Do We Log?
Copyright 2016, Jason A. Crome
Everything
June 21, 2016
What Do We Log?
Copyright 2016, Jason A. Crome
Everything
(seriously)
June 21, 2016
What Do We Log?
• With an MMO of any sort, you must log all sorts of
things you might not otherwise care about
• None of u...
• Logins
• Access log
• Errors
• Warnings
• Character events
• Character actions
• Randomness
• Courses taken
• URL Spoofin...
Randomness
• Really?
• Things left up to chance (combat success/failure)
should be measured for effectiveness
• Veure::Rol...
Randomness (cont.)
• Allows us to easily modify the difficulty of all rolls
• Allows us to easily change the random number
...
How Do We Log?
• Everything runs through log4perl
• Log4perl uses the Fluentd appender
• Log methods in Catalyst routed to...
The Code!
• Based on Log::Message::Structured
• L::M::S automatically logs all attributes
• L::M::S does not let attribute...
The Solution
• Veure::Role::LogEvent
• Along with a trait, allows us to pass in an object
and ignore what we don’t want
• ...
Veure::Role::LogEvent::Character
has character => (
is => 'ro',
isa => 'Veure::Schema::Result::Character',
);
has ch => (
...
Randomness
sub attempt_to {
my ($self, $type, $chance) = @_;
my $rand = rand();
my $success = $rand < $chance ? 1 : 0;
$se...
Randomness
package Veure::LogEvent::RandomAttempt;
use Moose;
use namespace::autoclean;
with 'Veure::Role::LogEvent';
use ...
Exchanges
  my $exchange = $self->exchange(
       action      => 'purchase_visa',
       messages    => {
           succ...
Exchanges
package Veure::LogEvent::Exchange;
use Moose;
use namespace::autoclean;
with qw(
 Veure::Role::LogEvent
 Veure::...
Exchanges
has action => (
traits => ['Logged'],
is => 'ro',
lazy => 1,
default => sub { shift->exchange->action },
);
has ...
Access Logging
# End action of Veure::Controller::Root
if ($self->log->is_info && ($ENV{TEST_ACCESSLOG} || !$ENV{TESTING_V...
Access Logging
package Veure::LogEvent::Accesslog;
use Moose;
use namespace::autoclean;
extends 'Veure::LogEvent';
has [qw...
Verbosity
• Overall Verbosity vs. Individual Verbosity
• Can adjust the overall verbosity on different
subsystems within t...
Verbosity
# Set to 0 to disable logging for that LogEvent
<LogEvent>
Accesslog 2
Exchange 1
RandomAttempt 1
RandomValue 1
...
Where do the Logs Go?
• For a developer: local filesystem
• For a development or production server: Postgres
• Developers c...
Stringifying Output
• Use Tie::IxHash to order hashes
• Log::Message::Structured::Stringify::
AsJSON does the rest
Copyrig...
Stringifying Output
package Veure::Role::LogEvent;
use Moose::Role;
use namespace::autoclean;
with qw(
Log::Message::Struc...
PostgreSQL (JSON)
# Easiest
<match Veure.**>
type pgjson
host localhost
port 5432
sslmode require
database veure
table flu...
# Maps to this…
CREATE TABLE fluentd (
tag Text
,time Timestamptz
,record Json
);
Copyright 2016, Jason A. CromeJune 21, 2...
Text Output
Copyright 2016, Jason A. Crome
# Developer
2016-04-28 14:37:16 INFO Veure::Role::WithLogging:184 127.0.0.1 cro...
Testing
• Log messages easily testable, but log must be
turned on to be tested
• Test::Role::LogTesting used for testing l...
Problem (cont.)
• Couldn’t you just test the individual log events
rather than sending them through Test::Log4perl?
Yup.
•...
Problem
• Test::Log::Log4perl only lets us run one
regex against a log entry
• To solve this, we build up a big regex in p...
Code
# By default access logging is disabled in the test suite
local $ENV{TEST_ACCESSLOG} = 1;
# With Accesslog = 1 we sho...
How Do We Analyze?
• For the easy cases (logins, access log, etc), SQL
select record->>'distance' as distance
from fluentd...
Game Balance/Log Analysis
• For game balancing and other analysis, more is
needed. Using randomness as an example…
• For e...
Game Balance/Log Analysis
• If we need to adjust, we have easy/medium/hard
settings to tweak the success rate.
• These set...
Game Balance/Log Analysis
• Leveling subject to this too - intended to be slow,
but not too slow
• Tracking experience gai...
Economic Balance
• Economy = transfer of one resource (time, money,
item) for another (course, item, skill)
• Log every ty...
Economic Balance
Copyright 2016, Jason A. CromeJune 21, 2016
Log Analysis - Future
• Redshift (Amazon AWS - based on Postgres)
• keen.io
Copyright 2016, Jason A. CromeJune 21, 2016
Log Retention
• Having lots of data is nice, where to keep it all?
• Current: archive after 1 month
• Future: archive week...
The Future
• Overall Verbosity vs. Individual Verbosity
• A/B testing results
• Stack logging via fluentd
Copyright 2016, J...
Final Thoughts
• Logging is a process. Don’t ever expect to get it
right.
• Make something flexible enough to expand later....
Credits
• Cees Hek (AAW, Toronto.pm)
• Noel Maddy (AAW)
• https://bost.ocks.org/mike/sankey/ (Sankey
diagram)
Copyright 20...
Questions?
Copyright 2016, Jason A. CromeJune 21, 2016
Bonus Slides! (?)
Copyright 2016, Jason A. CromeJune 21, 2016
What’s Veure?
• Catalan verb for “to see”.
• Pet project/dream of Ovid
• Massively Multiplayer Online Bulletin Board Game
...
What’s Veure?
Copyright 2016, Jason A. CromeJune 21, 2016
Upcoming SlideShare
Loading in …5
×

Logging with Perl and Fluentd

244 views

Published on

Enterprise logging using Perl and fluentd.

Published in: Software
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
244
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Logging with Perl and Fluentd

  1. 1. Logging and Analytics Using Perl and Fluentd Jason Crome Technical Lead, All Around the World June 21, 2016 Copyright 2016, Jason A. Crome
  2. 2. Who’s this for? • People who want to know more about enterprise logging (beginner and intermediate) • People who want to know about structured logging • People who want to know a bit about fluentd • Some code, some setup, some discussion. Copyright 2016, Jason A. CromeJune 21, 2016
  3. 3. Who Am I? • Project manager for Veure at AAW • Formerly spent 18 years writing software for local government • Dancer core team member • Student pilot and hockey player (probably not relevant but fun to mention anyhow) Copyright 2016, Jason A. CromeJune 21, 2016
  4. 4. Veure’s Building Blocks • Catalyst • Template Toolkit • DBIx::Class • Moose • PostgreSQL Copyright 2016, Jason A. CromeJune 21, 2016
  5. 5. – Colin Eberhardt, The Art of Logging “Logging is the process of recording application actions and state to a secondary interface.” Copyright 2016, Jason A. CromeJune 21, 2016 What is Logging?
  6. 6. Why Do We Log? • Prevent cheating • Help determine effectiveness of game content • Find out what players are spending money on and why • Help us adjust difficulty of the game Copyright 2016, Jason A. CromeJune 21, 2016
  7. 7. Why Fluentd? • Log collector/aggregator (“Unified Logging Layer”) • Log from any layer in the stack • Asynchronous • Cluster-able • Expandable Copyright 2016, Jason A. CromeJune 21, 2016
  8. 8. Why Fluentd? • Supports structured logging • Self-hosted • Affordable/free • Memory-friendly • Built-in reliability • Super easy to set up Copyright 2016, Jason A. CromeJune 21, 2016
  9. 9. Why not Fluentd? • Ruby :/ • Overkill for many projects Copyright 2016, Jason A. CromeJune 21, 2016
  10. 10. Alternatives • logstash • Splunk • Loggly • Papertrail Copyright 2016, Jason A. CromeJune 21, 2016
  11. 11. Why Log4Perl? • Does what we need • Has everything and the kitchen sink • Easily integrates with Catalyst • It already talks to Fluentd (so we don’t have to build this ourselves) Copyright 2016, Jason A. CromeJune 21, 2016
  12. 12. Where Do We Log From? • DBIx::Class • Other Moose-based classes • Catalyst • TT (this is rare) • Stack (future) Copyright 2016, Jason A. CromeJune 21, 2016
  13. 13. What Do We Log? Copyright 2016, Jason A. CromeJune 21, 2016
  14. 14. Copyright 2016, Jason A. CromeJune 21, 2016
  15. 15. What Do We Log? Copyright 2016, Jason A. CromeJune 21, 2016
  16. 16. What Do We Log? Copyright 2016, Jason A. Crome Everything June 21, 2016
  17. 17. What Do We Log? Copyright 2016, Jason A. Crome Everything (seriously) June 21, 2016
  18. 18. What Do We Log? • With an MMO of any sort, you must log all sorts of things you might not otherwise care about • None of us are veteran game designers (some of us aren’t even game players!), and need all the information we can get from our game Copyright 2016, Jason A. CromeJune 21, 2016
  19. 19. • Logins • Access log • Errors • Warnings • Character events • Character actions • Randomness • Courses taken • URL Spoofing • Chat hacking • Combat • Item Transfers • Credit Transfers • Auction house results • Real-World Purchases • Monetization events • Security • A/B test results (future) Copyright 2016, Jason A. Crome What Do We Log? June 21, 2016
  20. 20. Randomness • Really? • Things left up to chance (combat success/failure) should be measured for effectiveness • Veure::Role::Random is used wherever randomness is needed • Right now, only collecting information. Insufficient data to get meaningful analysis. Copyright 2016, Jason A. CromeJune 21, 2016
  21. 21. Randomness (cont.) • Allows us to easily modify the difficulty of all rolls • Allows us to easily change the random number generator used • Running all random checks through one spot makes logging easy Copyright 2016, Jason A. CromeJune 21, 2016
  22. 22. How Do We Log? • Everything runs through log4perl • Log4perl uses the Fluentd appender • Log methods in Catalyst routed to Fluentd • Logs are structured data. Turned into JSON or stringified. Copyright 2016, Jason A. CromeJune 21, 2016
  23. 23. The Code! • Based on Log::Message::Structured • L::M::S automatically logs all attributes • L::M::S does not let attributes contain objects • Both of these are a problem for us Copyright 2016, Jason A. CromeJune 21, 2016
  24. 24. The Solution • Veure::Role::LogEvent • Along with a trait, allows us to pass in an object and ignore what we don’t want • Individual attributes can be set to only log at a specific verbosity level Copyright 2016, Jason A. CromeJune 21, 2016
  25. 25. Veure::Role::LogEvent::Character has character => ( is => 'ro', isa => 'Veure::Schema::Result::Character', ); has ch => ( traits => ['Logged'], is => 'ro', isa => 'Str', lazy => 1, default => sub { my $self = shift; if ( my $char = $self->character ) { return $char->slug; } }, ); Copyright 2016, Jason A. CromeJune 21, 2016
  26. 26. Randomness sub attempt_to { my ($self, $type, $chance) = @_; my $rand = rand(); my $success = $rand < $chance ? 1 : 0; $self->log_event('RandomAttempt', type => $type, success => $success, chance => $chance, value => $rand, ) if $type; return $success; } Copyright 2016, Jason A. CromeJune 21, 2016
  27. 27. Randomness package Veure::LogEvent::RandomAttempt; use Moose; use namespace::autoclean; with 'Veure::Role::LogEvent'; use Veure::LogEvent::Trait::Logged; has type => ( traits => ['Logged'], is => 'ro', isa => 'Str', ); has [qw(chance value)] => ( traits => ['Logged'], is => 'ro', isa => 'Num', ); has success => ( traits => ['Logged'], is => 'ro', isa => 'Bool', ); __PACKAGE__->meta->make_immutable; 1; Copyright 2016, Jason A. CromeJune 21, 2016
  28. 28. Exchanges   my $exchange = $self->exchange(        action      => 'purchase_visa',        messages    => {            success => 'You purchased a visa',            failure => 'You could not purchase a visa',        },        steps => [            [ Wallet => remove => $visa_price ],            [ Visa   => add    => $affiliation->affiliation_id => $visa_dur ],        ],    );    return $exchange->attempt; Copyright 2016, Jason A. CromeJune 21, 2016
  29. 29. Exchanges package Veure::LogEvent::Exchange; use Moose; use namespace::autoclean; with qw(  Veure::Role::LogEvent  Veure::Role::LogEvent::Character  Veure::Role::LogEvent::CharacterLocation  Veure::Role::LogEvent::CharacterVitals ); use Veure::LogEvent::Trait::Logged; has exchange => (    is       => 'ro',    isa      => 'Veure::Economy::Exchange',    required => 1, ); has '+character' => (    lazy => 1,    default => sub { shift->exchange->character }, ); has steps => (    traits  => ['Logged'],    is      => 'ro',    lazy    => 1,    default => sub { shift->exchange->steps }, ); Copyright 2016, Jason A. CromeJune 21, 2016
  30. 30. Exchanges has action => ( traits => ['Logged'], is => 'ro', lazy => 1, default => sub { shift->exchange->action }, ); has counterparty => ( traits => ['Logged'], is => 'ro', isa => 'Str', default => 'GAME', ); has success => ( traits => ['Logged'], is => 'ro', lazy => 1, default => sub { shift->exchange->success }, ); has details => ( traits => ['Logged'], is => 'ro', lazy => 1, default => sub { shift->exchange->_messages }, ); __PACKAGE__->meta->make_immutable; 1; Copyright 2016, Jason A. CromeJune 21, 2016
  31. 31. Access Logging # End action of Veure::Controller::Root if ($self->log->is_info && ($ENV{TEST_ACCESSLOG} || !$ENV{TESTING_VEURE})) { my $req = $c->request; my $elapsed = $c->use_stats ? $c->stats->elapsed : '??'; $self->log_event( 'Accesslog', uri => $req->uri->as_string, ua => $req->user_agent, method => $req->method, args => $req->uri->query, timing => $elapsed, ); } Copyright 2016, Jason A. CromeJune 21, 2016
  32. 32. Access Logging package Veure::LogEvent::Accesslog; use Moose; use namespace::autoclean; extends 'Veure::LogEvent'; has [qw(uri method args timing)] => ( is => 'ro', isa => 'Maybe[Str]', ); has ua => ( traits => ['Log'], log_level => 2, is => 'ro', isa => 'Maybe[Str]', ); __PACKAGE__->meta->make_immutable; 1; Copyright 2016, Jason A. CromeJune 21, 2016
  33. 33. Verbosity • Overall Verbosity vs. Individual Verbosity • Can adjust the overall verbosity on different subsystems within the game • With individual verbosity, we can log everything a player does (future) Copyright 2016, Jason A. CromeJune 21, 2016
  34. 34. Verbosity # Set to 0 to disable logging for that LogEvent <LogEvent> Accesslog 2 Exchange 1 RandomAttempt 1 RandomValue 1 Scavenge 0 Travel 1 </LogEvent> Copyright 2016, Jason A. CromeJune 21, 2016
  35. 35. Where do the Logs Go? • For a developer: local filesystem • For a development or production server: Postgres • Developers can also install local instances of fluentd if so inclined and do what they want with the log output • Fluentd lets you easily configure multiple outputs Copyright 2016, Jason A. CromeJune 21, 2016
  36. 36. Stringifying Output • Use Tie::IxHash to order hashes • Log::Message::Structured::Stringify:: AsJSON does the rest Copyright 2016, Jason A. CromeJune 21, 2016
  37. 37. Stringifying Output package Veure::Role::LogEvent; use Moose::Role; use namespace::autoclean; with qw( Log::Message::Structured Log::Message::Structured::Stringify::AsJSON ); use MooseX::ClassAttribute; use Veure::LogEvent::Trait::Logged; use Tie::IxHash; # …snip… sub as_hash { my ($self) = @_; my %hash; tie( %hash, 'Tie::IxHash' ); %hash = map { $_ => $self->$_ } @{ $self->logged_attrs }; return %hash; } 1; Copyright 2016, Jason A. CromeJune 21, 2016
  38. 38. PostgreSQL (JSON) # Easiest <match Veure.**> type pgjson host localhost port 5432 sslmode require database veure table fluentd user veure_user password abcdefg time_col time tag_col tag # Classname record_col record </match> Copyright 2016, Jason A. CromeJune 21, 2016
  39. 39. # Maps to this… CREATE TABLE fluentd ( tag Text ,time Timestamptz ,record Json ); Copyright 2016, Jason A. CromeJune 21, 2016 PostgreSQL (JSON)
  40. 40. Text Output Copyright 2016, Jason A. Crome # Developer 2016-04-28 14:37:16 INFO Veure::Role::WithLogging:184 127.0.0.1 cromedome> {"args":null,"class":"Veure::LogEvent::Accesslog","method":"GET","timing": 0.044844,"ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36”,"uri":"http://localhost:5000/combat"} 2016-04-28 14:37:16 INFO Veure::Role::WithLogging:184 127.0.0.1 cromedome> {"chance": 0.9,"class":"Veure::LogEvent::RandomAttempt","success":1,"type":"Flee","value": 0.743257306080512} 2016-04-28 14:37:16 INFO Veure::Role::WithLogging:184 127.0.0.1 cromedome> {"class":"Veure::LogEvent::RandomValue","max":5,"min":0,"step":null,"type":"Area","value":2} # Fluentd 2016-03-18T09:22:55-04:00 Veure.Model.DB.Character {"success":0,"stamina":0.9,"level": 1,"class":"Veure::LogEvent::Scavenge","item":"","character":"cees","reason":"Nothing found"} June 21, 2016
  41. 41. Testing • Log messages easily testable, but log must be turned on to be tested • Test::Role::LogTesting used for testing log output • Look for specific hash values in log message Copyright 2016, Jason A. CromeJune 21, 2016
  42. 42. Problem (cont.) • Couldn’t you just test the individual log events rather than sending them through Test::Log4perl? Yup. • So why didn’t you? Time. Copyright 2016, Jason A. CromeJune 21, 2016
  43. 43. Problem • Test::Log::Log4perl only lets us run one regex against a log entry • To solve this, we build up a big regex in parts and test for all hash entries we care about • Test::Role::LogTesting does the dirty work for us • It’s not pretty. Copyright 2016, Jason A. CromeJune 21, 2016
  44. 44. Code # By default access logging is disabled in the test suite local $ENV{TEST_ACCESSLOG} = 1; # With Accesslog = 1 we should see log record with no ua local config->{LogEvent}->{Accesslog} = 1; $test->log_expect('Veure.LogEvent.Accesslog', info => $test->log_event_check({ "args" => undef, "class" => "Veure::LogEvent::Accesslog", "method" => "GET", "timing" => "anything", "ua" => "missing", "uri" => “http://localhost/travel/area/ruins" }) ); $mech->get('/travel/area/ruins'); $test->logs_ok; Copyright 2016, Jason A. CromeJune 21, 2016
  45. 45. How Do We Analyze? • For the easy cases (logins, access log, etc), SQL select record->>'distance' as distance from fluentd where record->>'class' = ‘Veure::LogEvent::Travel'; select record->>'distance' as distance from fluentd where record::jsonb @> '{"class" : "Veure::LogEvent::Travel", "distance" : "1.74"}'; Copyright 2016, Jason A. CromeJune 21, 2016
  46. 46. Game Balance/Log Analysis • For game balancing and other analysis, more is needed. Using randomness as an example… • For each type of roll, track chance to succeed, roll, and success • Periodically (daily? weekly?) determine average roll, % of success, standard deviation, other stats • Are we too high? Too low? Copyright 2016, Jason A. CromeJune 21, 2016
  47. 47. Game Balance/Log Analysis • If we need to adjust, we have easy/medium/hard settings to tweak the success rate. • These settings adjust the success rate by a multiple of the standard deviation. • Still haven’t accumulated enough data for this. Copyright 2016, Jason A. CromeJune 21, 2016
  48. 48. Game Balance/Log Analysis • Leveling subject to this too - intended to be slow, but not too slow • Tracking experience gain, look at daily progression over time. Determine average gain, standard deviation, etc. • Based on the information we have collected, increase or decrease experience gain using standard deviation as a guide. Copyright 2016, Jason A. CromeJune 21, 2016
  49. 49. Economic Balance • Economy = transfer of one resource (time, money, item) for another (course, item, skill) • Log every type of transfer in game and absolute amount of each resource • Gives us both state and the flow of all resources in game. • Visualizing flow through Sankey diagrams and other tools Copyright 2016, Jason A. CromeJune 21, 2016
  50. 50. Economic Balance Copyright 2016, Jason A. CromeJune 21, 2016
  51. 51. Log Analysis - Future • Redshift (Amazon AWS - based on Postgres) • keen.io Copyright 2016, Jason A. CromeJune 21, 2016
  52. 52. Log Retention • Having lots of data is nice, where to keep it all? • Current: archive after 1 month • Future: archive weekly? • With an agile release cycle, anything older than one month becomes much less relevant Copyright 2016, Jason A. CromeJune 21, 2016
  53. 53. The Future • Overall Verbosity vs. Individual Verbosity • A/B testing results • Stack logging via fluentd Copyright 2016, Jason A. CromeJune 21, 2016
  54. 54. Final Thoughts • Logging is a process. Don’t ever expect to get it right. • Make something flexible enough to expand later. • Once you show people what information they can get, they will constantly ask you for more. Copyright 2016, Jason A. CromeJune 21, 2016
  55. 55. Credits • Cees Hek (AAW, Toronto.pm) • Noel Maddy (AAW) • https://bost.ocks.org/mike/sankey/ (Sankey diagram) Copyright 2016, Jason A. CromeJune 21, 2016
  56. 56. Questions? Copyright 2016, Jason A. CromeJune 21, 2016
  57. 57. Bonus Slides! (?) Copyright 2016, Jason A. CromeJune 21, 2016
  58. 58. What’s Veure? • Catalan verb for “to see”. • Pet project/dream of Ovid • Massively Multiplayer Online Bulletin Board Game (MMOBBG) • Set in dystopian future where humanity has colonized the galaxy, and eventually pushed to the brink of extinction • Think Mad Max meets Firefly Copyright 2016, Jason A. CromeJune 21, 2016
  59. 59. What’s Veure? Copyright 2016, Jason A. CromeJune 21, 2016

×