1.
Turbo charge your logs

2.
Who?
●
●
●
●
●
Ex-pat Englishman, now living in
Southern Ontario.
Web developer for 5 years, mostly
PHP.
(Almost) senior software engineer at
TribeHR.
Co-organiser of Guelph PHP User
Group.
Ex-professional musician.

3.
Why logging?

4.
Your app is trying to talk to
you!
●
●
●
●
Logs are the way your app speaks
to you.
Ignore log messages at your peril...
Logging is the L in LUCID
development.
http://crisscott.com/2012/09/11/lucid-developme

5.
It's confessional time...

6.
Using log data is hard
●
Not in a human friendly format.
●
A lot of data.
●
Many log files.
●
Potentially many servers.

7.
Use your logs pro­actively
How can we stop using log
data reactively and start
using it pro-actively?

8.
The 'ideal' logging setup
●
●
Centralised.
Accepts logs from application code,
software and the OS.
●
Performant.
●
Scalable.
●
Easily searchable.
●
Alarms and alerting.

9.
RFC 5424 logging levels
●
Debug
●
Info
●
Notice
●
Warning
●
Error
●
Critical
●
Alert
●
Emergency

10.
Logging from your code

11.
What's wrong with
error_log?
●
Nothing at all but...
●
It's limited:
–
Have to format the message yourself.
–
Limited number of destinations.
–
Doesn't support logging levels defined
in RFC 5424.

12.
Introducing Monolog
●
PHP 5.3+ logging library by Jordi
Boggiano based on Python's Log
Book library.
●
PSR-3 compliant.
●
Supports RFC 5424.

13.
Installing Monolog
●
●
●
Symfony2, Laravel4, Silex and PPI
all come with Monolog.
CakePHP and Slim have have plugins to use it.
Most easily installed with
Composer:

14.
Monolog concepts
●
Channels.
●
Handlers.
●
Formatters.
●
Processors.

15.
Channels
●
●
●
A channel is a name or category for
a logger.
Each logger instance is given a
channel when instantiated.
Allows for multiple loggers, each
with a different channel.

16.
Handlers
●
●
●
●
Handlers write log messages to a
storage medium.
Multiple handlers can be attached to
each logger.
Set lowest level handler logs at and
if it 'bubbles'.
Many handlers available or you can
write your own.

17.
Example handlers
Files/Syslog
Notifications
●
Stream Handler
●
Mail handlers
●
Rotating File Handler
●
Pushover Handler
●
Syslog Handler
●
HipChat Handler
Debugging
Networked Logging
●
Socket Handler
●
AMQP Handler
●
Gelf Handler
●
Zend Monitor Handler
●
●
FirePHP Handler
ChromePHP
Handler

18.
Formatters
●
●
Processes a log message into the
appropriate format for a handler.
Each handler has a default
formatter to use but this can be
overridden.

19.
Simple example

20.
Using multiple handlers

21.
Leveraging bubbling

22.
Processors
●
●
●
Used to amend or add to the log
message.
PHP callable, called when a
message is logged.
Built in processors available:
–
IntrospectionProcessor
–
WebProcessor
–
MemoryUsageProcessor
–
MemoryPeakUsageProcessor
–
ProcessIdProcessor
–
UidProcessor

23.
Processor example

24.
Where does this get us
to?
●
●
Centralised. Maybe...
Accepts messages from application
code, software and the OS.
●
Performant. Maybe...
●
Scalable. Maybe...
●
Easily searchable.
●
Alarms and alerting. Yes but crude.

25.
We can do better!

26.
Leveraging Syslog

27.
Why Syslog?
●
●
Loggable events don't only happen
in code!
To get a full picture of what's going
on we need to monitor what's going
on in other services too.

28.
Syslog basics
●
●
●
●
OS daemon to process log
messages.
Messages are assigned a facility,
such as auth, authpriv, daemon or
cron or a custom one.
Messages are also assigned a
severity, defined in RFC 5424.
Messages can be sent to files,
console or a remote location.

29.
Which Syslog daemon
to use?
●
In part will depend on your OS.
●
Things to consider:
–
Syslog is the oldest with not as many
features.
–
Syslog-ng is produced under a dual
license.
–
Rsyslog fully featured and open
source.

30.
Introduction to Rsyslog
●
Fork of syslog by Rainer Gerhards.
●
Drop in replacement for syslog.
●
●
Many, many features including
plugin system for extending.
Default syslogger in Debian, can be
installed on other distros too.

31.
Remote logging with
Rsyslog
●
Rsyslog can be configured to work
in a client-server setup.
–
–
●
One or more machines are setup as
clients to forward log messages.
One machine is setup to receive and
store them.
Probably want to filter sender on the
receiving machine...

32.
Rsyslog client setup

33.
Rsyslog server setup

34.
Leveling up with Rsyslog
●
●
Apache can send all error logs to syslog
directly.
Rsyslog can also monitor other log files
using the Text File Input module.
–
Example of monitoring Apache access log at
https://gist.github.com/joseph12631/2580615

35.
Where does this get us?
●
●
Centralised. Yes.
Accepts messages from application
code, software and the OS.
Possibly.
●
Performant. Depends.
●
Scalable. Depends.
●
Easily searchable.
●
Alarms and alerting. Yes but crude.

36.
Taking it further with
Logstash

37.
What is Logstash?
●
●
●
Tool to collect, filter and output log
messages.
Built in web interface or richer web
interface project called Kibana
available.
Full information at
http://logstash.net/ and Kibana
demo at http://demo.logstash.net/

38.
Installing Logstash
●
●
Current release is 1.3.3 and can be
downloaded from here.
Run from cli, use supervisord or an
init.d/upstart script (cookbook entry
on how to do this at
http://cookbook.logstash.net/).

39.
Inputs, filters and outputs
Inputs
–
AMQP/RabbitMQ
–
Syslog
–
Varnishlog
Outputs
–
Statsd
–
XMPP
–
AMQP/RabbitMQ
–
Nagios
–
Graphite
Filters
–
Anonymize
–
Grok
–
Geoip
–
Mutate

40.
Logstash config
●
●
●
When starting specify the path to a
config file for Logstash to use.
Three main sections: input, filter
and output.
Each section may have multiple
instances of each type.

41.
Sample configuration file
input {
file {
path => "/var/log/apache2/*access.log"
type => "apache"
}
}
filter {
if [type] == "apache" {
grok {
pattern => "%{COMBINEDAPACHELOG}"
}
}
}
output {
redis { host => "10.0.0.5" data_type => "list" key => "logstash" }
}
●
See http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/

42.
Where does this get us?
●
●
Centralised. Yes.
Accepts messages from application
code, software and the OS. Yes.
●
Performant. Yes.
●
Scalable. Yes.
●
Easily searchable. Possibly.
●
Alarms and alerting. Yes.

43.
Introducing Graylog2

44.
What is Graylog2?
●
●
●
●
Log storage and search application.
Can accept thousands of messages
per second and store terabytes of
data.
Web interface for searching and
analytics.
Built in alerting and metrics.

45.
Installing Graylog2
●
Components:
–
–
Graylog2 server
–
●
MongoDb
–
●
Elasticsearch
Graylog2 web interface
Full info on installing at
http://support.torch.sh/help/kb
Live demo at
http://public-graylog2.taulia.com/login

46.
Getting log messages into
Graylog2
●
Can accept log messages in 3
ways:
–
–
Syslog via UDP or TCP.
–
●
Graylog Extended Log Format (GELF)
via UDP .
AMQP.
Multiple Graylog2 server instances
can be run in parallel.

47.
Graylog2 web interface
●
●
Main view shows recent log
messages and graphs of recent
message numbers.
Single message can be clicked on
to view all details for it.
●
Dashboard views.
●
Full search functionality.
●
Analytics dashboard and metrics.

48.
Web interface view

49.
Details of an individual
message

50.
Dashboard view

51.
Searches and streams
●
●
●
Web interface allows fine grained
searching by different fields.
Frequently used searches can be
saved as streams.
Streams can be marked as
favourites by users and can be
viewed as dashboards.

52.
Stream alarms
●
●
Alarms can be sent for a stream
with user defined sensitivity.
Plugins for sending alarms include:
–
–
PagerDuty
–
HipChat
–
Twilio SMS
–
●
Email
Jabber/XMPP
You can also write your own

53.
Where does this get to?
●
●
Centralised. Yes.
Accepts messages from application
code, software and the OS. Yes.
●
Performant. Yes.
●
Scalable. Yes.
●
Easily searchable. Yes.
●
Alarms and alerting. Yes.

54.
Thanks for listening
●
Contact me:
–
–
●
jeremycook0@gmail.com
@JCook21
Questions?

55.
Putting it all together
A few possible implementations.