Rsyslog vs Systemd Journal Presentation

rsyslog vs journal?
Rainer Gerhards

Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany
Me & the Talk
• Rainer Gerhards
▫ Data center guy
▫ Involved 15+ years in logging
▫ Founded rsyslog in 2003
• The talk
▫ Will rsyslog fight the journal?
▫ Some history on journal-like system
▫ Ways of integration
▫ How to do things the journal announcement
claimed as impossible

Does journal replace syslog?
• The initial announcement sounded a bit in that
way, or was at least interpreted by most
(including me) in that direction.
• Looking at how things have evolved
▫ There of course is overlap between both systems
▫ But there are also (large) regions that do not
overlap
• This is not a new situation, there is some history
lesson...

Windows Event Log!
• The Windows Event Log is in many ways similar
to systemd journal
▫ Binary database with rollover and fast access time
▫ uses a simple structured format that captures core
metadata items (like timestamps, user IDs, …)
▫ uses unique identifiers for different types of log
messages
▫ Files are especially secured by OS

Event Log History
• Introduced with Windows NT 3.1 in 1993
• Greatly enhanced in 2007, starting with
Windows Vista
• Originally single-computer only
• Now provides network functionality
▫ EventLog-to-EventLog push and pull
subscriptions
▫ Can be used to setup log forwarding in the
enterprise

So what does history tell us?
• If such a system can totally replace syslog, there
should be no syslog on Windows at all – and
never have been.
• Well... there are ample of applications
▫ WinSyslog (initial version by me, 1996)
▫ Kiwi Syslog (Solarwinds)
▫ EventReporter (first ever Windows-to-syslog tool,
1997)
▫ Snare
▫ and many more!

Obviously, there must be some
need to syslog technology...
• Face it: syslog is the lingua franca of network
event logging.
▫ If you want to process messages from different
sources, chances are high you will need it.
▫ Even if not syslog (protocol) is used, you usually
need some common denominator
 e.g. Linux does not understand native Windows
EventLog
 Windows neither does understand native journal

A key problem solved by syslog
• You want to integrate all of your systems into a
consolidated log
• This either means
▫ A common protocol
▫ A system that is capable of processing multiple
protocols and somehow “normalize” them
• Syslog is ubiquitous – because a basic client is
dumb easy to implement!

Window as a sender...
• Early days: missing network functionality was a
problem; brought up the idea of Event Log
forwarding
• Big customers quickly adopted that for
integration into their management system
• Today's hot topics:
▫ local filtering and preprocessing
▫ Ability to extract and properly express OS objects
▫ Support all Windows capabilities
▫ Secure protocol choices

Windows as a receiver...
• Windows acts as syslog server
• Messages are written to
▫ Local files
▫ Windows Event Log (!)
▫ Some other processing (like alerting)
• Typical deployment scenario for SOHO
• But some large Windows-only shops also use it
for integration of non-Windows sources

Why I am talking so much about
Windows?
• As I said, I see strong similarities between
journal and Windows Event Log
• Except that journal has much more quickly
gotten some network functionality
• So my best guess is that deployments and end-
user needs will evolve into mostly the same
directions

Journal vs. Syslog:
low end systems
• Usually users of these machines are not at all
interested in logging
• Journal is very convenient as a troubleshooting
tool
• Works perfect on personal desktop & notebook
• Rsyslog will be needed by some users to
integrate e.g. their DSL router's messages into
the journal

Journal vs. Rsyslog:
enterprise systems
• Impossible to manage without any syslog
• Journal integrated as another event source
▫ Journal-centric
 As much as possible is done with journal
 Integration happens at central head server(s)
▫ Syslog-centric
 Journal is used only as much as unavoidable
 Each machine runs rsyslog and forwards events
▫ Mode depends on end-user's philosophy

How did the journal affect the
rsyslog project?
• Obviously, we expect less presence on low-end
systems
• So we re-focussed the project
▫ Previously low-end and enterprise needs were
equal peers
▫ Now strong focus on enterprise
• The logging world at large got benefit as
suddenly everyone was interested in logging –
which also helps rsyslog!

What have we done to integrate
with the journal?
• Module omjournal
▫ Provides ability to store messages into the journal
▫ Traditional syslog, text files, ...
▫ Caters for the low-end use case
• Module imjournal
▫ Provides ability to pull messages off the journal,
just as another event source
▫ Contributed by Red Hat
▫ Caters for the enterprise use case

Integrating syslog Data into the
journal (SOHO env)
/* first, we make sure all necessary modules are present: */
module(load="imudp") # input module for UDP syslog
module(load="omjournal") # output module for journal
/* then, define the actual server that listens to the
* router. Note that 514 is the default port for UDP syslog.
*/
input(type="imudp" port="514" ruleset="writeToJournal")
/* inside that ruleset, we just write data to the journal: */
ruleset(name="writeToJournal") {
action(type="omjournal")
}

Integrating journal data into syslog
module(load="imjournal" PersistStateInterval="100" StateFile="/path/to/file")
module(load="mmjsonparse") #load mmjsonparse module for structured logs
$template CEETemplate,"%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-
json%n" #template for messages
*.* :mmjsonparse:
*.* /var/log/ceelog;CEETemplate
• Necessary to obtain extended journal properties
• If not needed, regular system log socket can be
used

Why is it simple to integrate the
journal?
• Rsyslog is actually
▫ A message router
▫ With dynamically loadable inputs and outputs
▫ Highly configurable
• So, journal support is as easy as adding some
new inputs and outputs!
• The rest of the plumbing is already there.

rsyslog Architecture
journal
Network
(e.g.TCP)
/dev/log file
Database
Remote
system
Parsers
For-
matter
Rules
&
Filters

“String” Handling in rsyslog
Message String
Parser
Dictionary of Name/Value Pairs
Formatter
Output String
“special”
Outputs
Modifi-
cation Mod

Now let's look at some
“impossible” things
• The original journal paper claimed that syslog is
▫ Seriously broken
▫ Cannot provide some important features
• I'll show how to do these “impossible” things
▫ Based on 2011 technology
▫ And on current one (v7.4)

Log File Manipulation Protection
• The traditional approach is to ship logs off the
machine, to a central and highly secured system
• Keeping them on a system that is “easily
compromised” is asking for trouble.
• Problem is that local secrets can always be
compromised
• In rsyslog 7.4, we address these problems via log
signatures and encryption...

Signed Log Records
• In 2011, there was no good solution (and
journal's solution was also not good)
• Things have evolved since them
▫ Journal got “forward secure sealing”
▫ Rsyslog
 got a crypto provider interface and a provider for
“Keyless Signature Infrastructure” (KSI)
 Hash chain for log record is created, and key hashes
are chained in a global hash chain, which provides
signature & timestamp

Signing via Hash Chains...
• Very rough sample (actually Merkle trees!)
• No local secret!
• Consider “chain layer” to be operated on a
schedule (timer ticks!)
Source: http://en.wikipedia.org/wiki/File:Hashlink_timestamping.svg

Activating Log Signing
action(type="omfile"
file="/var/log/logfile"
sig.provider="gt"
sig.keepTreeHashes="on"
sig.keepRecordHashes="on")
• Parameters except sig.provider are optional
• Writes
▫ regular log file
▫ plus signature file (*.gtsig)

Log File Encryption
• Crypto-Provider interface recently added
• As well as a libgcrypt-based crypto provider
• Symmetric cryptoraphy, all ciphers & modes
supported by libgcrypt
• Key can come from
▫ Config param (testing only, pls!)
▫ File
▫ Script (interface for advanced key exchange
options)

Activating Log Encryption
action(type="omfile"
file="/var/log/logfile"
cry.provider="gcry"
cry.keyprogram=”/path/to/binary”)
• Addtl Parameters for ciphers, etc...
• Writes
▫ regular log file, encrypted
▫ plus encryption info file (*.encinfo)
• Works in conjunction with signatures
• In 7.5 extended to rsyslog disk queues!

Syslog Network Processing
• Original Journal accouncement missed
improvments and talked only about UDP syslog
• We have
▫ TCP & TLS support (RFC5425, 2009)
▫ Mutual authentication & authorization
▫ Multiple hops
▫ Buffered send queues, even with disk buffers
▫ Rsyslog can utilize other protocols as well (RELP,
SNMP)!

Sample: TLS-encrypted for-
warding using a buffer queue
$DefaultNetstreamDriver gtls # make gtls driver the default
# certificate files
$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem
$DefaultNetstreamDriverCertFile /rsyslog/protected/ma-cert.pem
$DefaultNetstreamDriverKeyFile /rsyslog/protected/ma-key.pem
# authorization
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer central.example.net
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
# Queue
$ActionQueueType LinkedList # use own queue
$ActionQueueFileName fwq # set file name, enable disk mode
*.* @@central.example.net:10514 # forward to remote server

Indexed Store
• Core idea: connect to existing “life stores”, do
not push that part into the syslogd
▫ Actually one of the core requirements that started
rsyslog (MySQL, ~2004)!
• 2011: various relational databases
• Today also
▫ Document based databases (MongoDB,...)
▫ Elasticsearch
▫ Journal DB

Rate-Limiting
• 2011
▫ Repeated message reduction
($RepeatedMsgReduction on)
▫ Output throttling
• V7, additionally
▫ Repeated message processing on a per-input basis
▫ Object-based rate limiters (n messages within s
seconds)

Log compression
• Log files can be zip compressed by specifying the
“ZipLevel” parameter
• Of course, accessing compressed log records
requires more processing time.
• Today, we also have experimental code for log
transfer compression (directly built into the
protocol/output modules).

Message authentication and
Metadata availablility
• Rsyslog uses the same SCM_CREDENTIALS
facility that journal does
• And in both cases it can be faked – as journal
demonstrates when it actually fakes it on the
system log socket ;)
• The volume of metadata available has been
increased starting in 2012
• Total authenticity requires signatures at
the original originator level (each app),
what currently is impossible in the *nix
framework.

Free-Formedness of Log Records
• Traditional syslog messages are much like free-
form text
• Today, we see the same for typical journal
messages
• There are a couple of standardization efforts
underway to provided structured logging
• Project lumberjack (lead by Red Hat) provides
JSON-based structured logs

Unstructured Text Log Duality
• If a log format does not support freeform-text, it
is not used (at least not more than one can avoid
to...)
• If it supports freeform-text (among others), that
freeform-text will be abused
• → unstructured logs won't go away!
• We've seen this in Windows Event Log and looks
much the same for journal.

Converting Free-Text Messages
via mmnormalize
• Uses a “sample rule base”
▫ One sample for each expected message type
▫ Sample contains text (for matching) and property
descriptions (like IPv4 Address, char-matches, …)
▫ If sample matches, corresponding properties are
extracted
▫ Special parser for iptables
• Very fast algorithm (much faster than regex)
• Based on liblognorm (which can also be used in
other programs to gain this functionality!)

Timestamp format
• No year, no timezone
• ... just because distros turn it off.
• Remove “$ActionFileDefaultTemplate
RSYSLOG_TraditionalFileFormat” from
rsyslog.conf to get rsyslog's default high-
precision RFC5424 timestamp
• Some tools may have problems with that, but
can't be too bad – some Distros use the default
format

Questions?
• rgerhards@adiscon.com
• http://blog.gerhards.net
• http://www.rsyslog.com
• http://www.adiscon.com
• Associated paper is
available on SlideShare.

Rsyslog vs Systemd Journal Presentation

More Related Content

What's hot

Similar to Rsyslog vs Systemd Journal Presentation

More from Rainer Gerhards

Recently uploaded

Rsyslog vs Systemd Journal Presentation