This story describes the development of the Apache::LogFormat::Compiler (ALFC) module by an operations engineer to optimize logging performance in a web application. The original PM::AccessLog module was identified as a performance bottleneck by profiling tools. Several optimizations were tried, including the PM::AxsLog middleware, but it only supported fixed log formats. The operations engineer then created ALFC to compile log formats to Perl code for improved performance. It allowed the AxsLog middleware to be updated, achieving a 5x performance gain in logging. This addressed the original developer's need to customize log formats and store additional fields in logs.

1. ALFCYAPC::Asia 2013 Tokyo
前夜祭 LT-thon
Masahiro Nagano @kazeburo

2. Me
• 長野雅広 Masahiro Nagano
• @kazeburo
• PAUSE: KAZEBURO
• Operations Engineer, Site Reliability
• LINE Corp.
Development support
on LINE Familly, livedoor

3. ALFC =

4. Apache::LogFormat::Compiler

5. Convert

6. ‘%h %l %u %t "%r" %>s %b’

7. to perl-code

8. my $code = <<EOF;
sub {
my ($env,$res,$length,$reqtime,$time) = @_;
$time = time() if ! defined $time;
my @lt = localtime($time);;
my $t = sprintf '%02d/%s/%04d:%02d:%02d:%02d %s', $lt[3],
$abbr[$lt[4]], $lt[5]+1900, $lt[2], $lt[1], $lt[0], $tz;
q!! . ($env->{REMOTE_ADDR} || '-')
. q! ! . '-'
. q! ! . ($env->{REMOTE_USER} || '-')
. q! ! . "[" . $t . "]"
. q! "! . _safe($env->{REQUEST_METHOD}) . " " . _safe($env-
>{REQUEST_URI}) . " " . $env->{SERVER_PROTOCOL}
. q!" ! . $res->[0]
. q! ! . (defined $length ? $length : '-')
. q!!
}
EOF

9. and compile

10. my $log_line = eval $code;
$log_line->($env,$res);

11. SYNOPSIS

12. my $log_handler = Apache::LogFormat::Compiler->new(
'%h %l %u %t "%r" %>s %b'
);
warn $log_handler->log_line($env,$res);

13. use Plack::Builder;
use Time::HiRes qw/gettimeofday tv_interval/;
use Apache::LogFormat::Compiler;
my $log_handler = Apache::LogFormat::Compiler->new(
'%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i" %D'
);
my $compile_log_app = builder {
enable sub {
my $app = shift;
sub {
my $env = shift;
my $t0 = [gettimeofday];
my $res = $app->();
my $reqtime = int(tv_interval($t0) * 1_000_000);
$env->{psgi.error}->print($log_handler->log_line(
$env,$res,6,$reqtime, $t0->[0]));
}
};
$app
};

14. OR

15. Plack::Middleware::AxsLog
yet another AccessLog Middleware

16. use Plack::Builder;
builder {
enable 'AxsLog',
combined => 1,
response_time => 1;
$app
};

17. OR

18. Plack::Middleware::AccessLog
Plack (1.0023~)

19. Benchmark

20. 0
10000
20000
30000
40000
50000
60000
51262
8826
Plack-1.0022 Plack-1.0029
Req/Sec

21. A

22. Story

23. of

24. ALFC
とある Ops と Dev のおはなし∼

25. I'm

26. tuning

27. up

28. this

29. web

30. application.

31.
NYTProf

32. said

33. that

34. PM::AccessLog

35. is

36. slow

37.
Yes,

38. I

39. know.

40. It's

41. slow

42. because..

43.
Ops Dev

44. Why PM::AccessLog
was slow?

45. my %char_handler = (
h = sub { $env-{REMOTE_ADDR} || '-' },
t = sub { [ . $strftime-('%d/%b/%Y:%H:%M:%S %z', localtime) . ] },
r = sub { _safe($env-{REQUEST_METHOD}) .
. _safe($env-{REQUEST_URI}) .
. $env-{SERVER_PROTOCOL} },
);
my $char_handler = sub {
my $char = shift;
exists $char_handler{$char}
? $char_handler{$char}-($char) : -;
};
$fmt =~ s!
(?:
%{(.+?)}([a-z]) |
%(?:[])?([a-zA-Z%])
)
! $1 ? $block_handler-($1, $2) : $char_handler-($3) !egx;

46. I

47. can't

48. live

49. with

50. PM::AccessLog!!
I

51. have

52. idea

53. for

54. speedup
Ops Dev

55. Solution: combine strings