Rsyslog vs Systemd Journal Presentation


Published on

Do rsyslog and the journal cooperate? If so, how? This is the presentation from the LinuxTag 2013 conference. It details the rsyslog team's current position on the journal, how it affected rsyslog, what is being done for integration and some notes about how to configure rsyslog to do things that the journal announcement claimed to be impossible.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Rsyslog vs Systemd Journal Presentation

  1. 1. rsyslog vs journal?Rainer Gerhards
  2. 2. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyMe & the Talk• Rainer Gerhards▫ Data center guy▫ Involved 15+ years in logging▫ Founded rsyslog in 2003• The talk▫ Will rsyslog fight the journal?▫ Some history on journal-like system▫ Ways of integration▫ How to do things the journal announcementclaimed as impossible
  3. 3. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyDoes journal replace syslog?• The initial announcement sounded a bit in thatway, or was at least interpreted by most(including me) in that direction.• Looking at how things have evolved▫ There of course is overlap between both systems▫ But there are also (large) regions that do notoverlap• This is not a new situation, there is some historylesson...
  4. 4. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyWindows Event Log!• The Windows Event Log is in many ways similarto systemd journal▫ Binary database with rollover and fast access time▫ uses a simple structured format that captures coremetadata items (like timestamps, user IDs, …)▫ uses unique identifiers for different types of logmessages▫ Files are especially secured by OS
  5. 5. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyEvent Log History• Introduced with Windows NT 3.1 in 1993• Greatly enhanced in 2007, starting withWindows Vista• Originally single-computer only• Now provides network functionality▫ EventLog-to-EventLog push and pullsubscriptions▫ Can be used to setup log forwarding in theenterprise
  6. 6. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanySo what does history tell us?• If such a system can totally replace syslog, thereshould be no syslog on Windows at all – andnever have been.• Well... there are ample of applications▫ WinSyslog (initial version by me, 1996)▫ Kiwi Syslog (Solarwinds)▫ EventReporter (first ever Windows-to-syslog tool,1997)▫ Snare▫ and many more!
  7. 7. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyObviously, there must be someneed to syslog technology...• Face it: syslog is the lingua franca of networkevent logging.▫ If you want to process messages from differentsources, chances are high you will need it.▫ Even if not syslog (protocol) is used, you usuallyneed some common denominator e.g. Linux does not understand native WindowsEventLog Windows neither does understand native journal
  8. 8. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyA key problem solved by syslog• You want to integrate all of your systems into aconsolidated log• This either means▫ A common protocol▫ A system that is capable of processing multipleprotocols and somehow “normalize” them• Syslog is ubiquitous – because a basic client isdumb easy to implement!
  9. 9. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyWindow as a sender...• Early days: missing network functionality was aproblem; brought up the idea of Event Logforwarding• Big customers quickly adopted that forintegration into their management system• Todays hot topics:▫ local filtering and preprocessing▫ Ability to extract and properly express OS objects▫ Support all Windows capabilities▫ Secure protocol choices
  10. 10. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyWindows as a receiver...• Windows acts as syslog server• Messages are written to▫ Local files▫ Windows Event Log (!)▫ Some other processing (like alerting)• Typical deployment scenario for SOHO• But some large Windows-only shops also use itfor integration of non-Windows sources
  11. 11. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyWhy I am talking so much aboutWindows?• As I said, I see strong similarities betweenjournal and Windows Event Log• Except that journal has much more quicklygotten some network functionality• So my best guess is that deployments and end-user needs will evolve into mostly the samedirections
  12. 12. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyJournal vs. Syslog:low end systems• Usually users of these machines are not at allinterested in logging• Journal is very convenient as a troubleshootingtool• Works perfect on personal desktop & notebook• Rsyslog will be needed by some users tointegrate e.g. their DSL routers messages intothe journal
  13. 13. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyJournal vs. Rsyslog:enterprise systems• Impossible to manage without any syslog• Journal integrated as another event source▫ Journal-centric As much as possible is done with journal Integration happens at central head server(s)▫ Syslog-centric Journal is used only as much as unavoidable Each machine runs rsyslog and forwards events▫ Mode depends on end-users philosophy
  14. 14. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyHow did the journal affect thersyslog project?• Obviously, we expect less presence on low-endsystems• So we re-focussed the project▫ Previously low-end and enterprise needs wereequal peers▫ Now strong focus on enterprise• The logging world at large got benefit assuddenly everyone was interested in logging –which also helps rsyslog!
  15. 15. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyWhat have we done to integratewith the journal?• Module omjournal▫ Provides ability to store messages into the journal▫ Traditional syslog, text files, ...▫ Caters for the low-end use case• Module imjournal▫ Provides ability to pull messages off the journal,just as another event source▫ Contributed by Red Hat▫ Caters for the enterprise use case
  16. 16. Rainer Gerhards * * LinuxTag 2013, Berlin, Germany
  17. 17. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyIntegrating syslog Data into thejournal (SOHO env)/* first, we make sure all necessary modules are present: */module(load="imudp") # input module for UDP syslogmodule(load="omjournal") # output module for journal/* then, define the actual server that listens to the* router. Note that 514 is the default port for UDP syslog.*/input(type="imudp" port="514" ruleset="writeToJournal")/* inside that ruleset, we just write data to the journal: */ruleset(name="writeToJournal") {action(type="omjournal")}
  18. 18. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyIntegrating journal data into syslogmodule(load="imjournal" PersistStateInterval="100" StateFile="/path/to/file")module(load="mmjsonparse") #load mmjsonparse module for structured logs$template CEETemplate,"%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-json%n" #template for messages*.* :mmjsonparse:*.* /var/log/ceelog;CEETemplate• Necessary to obtain extended journal properties• If not needed, regular system log socket can beused
  19. 19. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyWhy is it simple to integrate thejournal?• Rsyslog is actually▫ A message router▫ With dynamically loadable inputs and outputs▫ Highly configurable• So, journal support is as easy as adding somenew inputs and outputs!• The rest of the plumbing is already there.
  20. 20. Rainer Gerhards * * LinuxTag 2013, Berlin, Germanyrsyslog ArchitecturejournalNetwork(e.g.TCP)/dev/log fileDatabaseRemotesystemParsersFor-matterRules&Filters
  21. 21. Rainer Gerhards * * LinuxTag 2013, Berlin, Germany“String” Handling in rsyslogMessage StringParserDictionary of Name/Value PairsFormatterOutput String“special”OutputsModifi-cation Mod
  22. 22. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyNow lets look at some“impossible” things• The original journal paper claimed that syslog is▫ Seriously broken▫ Cannot provide some important features• Ill show how to do these “impossible” things▫ Based on 2011 technology▫ And on current one (v7.4)
  23. 23. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyLog File Manipulation Protection• The traditional approach is to ship logs off themachine, to a central and highly secured system• Keeping them on a system that is “easilycompromised” is asking for trouble.• Problem is that local secrets can always becompromised• In rsyslog 7.4, we address these problems via logsignatures and encryption...
  24. 24. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanySigned Log Records• In 2011, there was no good solution (andjournals solution was also not good)• Things have evolved since them▫ Journal got “forward secure sealing”▫ Rsyslog got a crypto provider interface and a provider for“Keyless Signature Infrastructure” (KSI) Hash chain for log record is created, and key hashesare chained in a global hash chain, which providessignature & timestamp
  25. 25. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanySigning via Hash Chains...• Very rough sample (actually Merkle trees!)• No local secret!• Consider “chain layer” to be operated on aschedule (timer ticks!)Source:
  26. 26. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyActivating Log Signingaction(type="omfile"file="/var/log/logfile"sig.provider="gt"sig.keepTreeHashes="on"sig.keepRecordHashes="on")• Parameters except sig.provider are optional• Writes▫ regular log file▫ plus signature file (*.gtsig)
  27. 27. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyLog File Encryption• Crypto-Provider interface recently added• As well as a libgcrypt-based crypto provider• Symmetric cryptoraphy, all ciphers & modessupported by libgcrypt• Key can come from▫ Config param (testing only, pls!)▫ File▫ Script (interface for advanced key exchangeoptions)
  28. 28. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyActivating Log Encryptionaction(type="omfile"file="/var/log/logfile"cry.provider="gcry"cry.keyprogram=”/path/to/binary”)• Addtl Parameters for ciphers, etc...• Writes▫ regular log file, encrypted▫ plus encryption info file (*.encinfo)• Works in conjunction with signatures• In 7.5 extended to rsyslog disk queues!
  29. 29. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanySyslog Network Processing• Original Journal accouncement missedimprovments and talked only about UDP syslog• We have▫ TCP & TLS support (RFC5425, 2009)▫ Mutual authentication & authorization▫ Multiple hops▫ Buffered send queues, even with disk buffers▫ Rsyslog can utilize other protocols as well (RELP,SNMP)!
  30. 30. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanySample: TLS-encrypted for-warding using a buffer queue$DefaultNetstreamDriver gtls # make gtls driver the default# certificate files$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem$DefaultNetstreamDriverCertFile /rsyslog/protected/ma-cert.pem$DefaultNetstreamDriverKeyFile /rsyslog/protected/ma-key.pem# authorization$ActionSendStreamDriverAuthMode x509/name$ActionSendStreamDriverPermittedPeer$ActionSendStreamDriverMode 1 # run driver in TLS-only mode# Queue$ActionQueueType LinkedList # use own queue$ActionQueueFileName fwq # set file name, enable disk mode*.* # forward to remote server
  31. 31. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyIndexed Store• Core idea: connect to existing “life stores”, donot push that part into the syslogd▫ Actually one of the core requirements that startedrsyslog (MySQL, ~2004)!• 2011: various relational databases• Today also▫ Document based databases (MongoDB,...)▫ Elasticsearch▫ Journal DB
  32. 32. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyRate-Limiting• 2011▫ Repeated message reduction($RepeatedMsgReduction on)▫ Output throttling• V7, additionally▫ Repeated message processing on a per-input basis▫ Object-based rate limiters (n messages within sseconds)
  33. 33. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyLog compression• Log files can be zip compressed by specifying the“ZipLevel” parameter• Of course, accessing compressed log recordsrequires more processing time.• Today, we also have experimental code for logtransfer compression (directly built into theprotocol/output modules).
  34. 34. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyMessage authentication andMetadata availablility• Rsyslog uses the same SCM_CREDENTIALSfacility that journal does• And in both cases it can be faked – as journaldemonstrates when it actually fakes it on thesystem log socket ;)• The volume of metadata available has beenincreased starting in 2012• Total authenticity requires signatures atthe original originator level (each app),what currently is impossible in the *nixframework.
  35. 35. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyFree-Formedness of Log Records• Traditional syslog messages are much like free-form text• Today, we see the same for typical journalmessages• There are a couple of standardization effortsunderway to provided structured logging• Project lumberjack (lead by Red Hat) providesJSON-based structured logs
  36. 36. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyUnstructured Text Log Duality• If a log format does not support freeform-text, itis not used (at least not more than one can avoidto...)• If it supports freeform-text (among others), thatfreeform-text will be abused• → unstructured logs wont go away!• Weve seen this in Windows Event Log and looksmuch the same for journal.
  37. 37. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyConverting Free-Text Messagesvia mmnormalize• Uses a “sample rule base”▫ One sample for each expected message type▫ Sample contains text (for matching) and propertydescriptions (like IPv4 Address, char-matches, …)▫ If sample matches, corresponding properties areextracted▫ Special parser for iptables• Very fast algorithm (much faster than regex)• Based on liblognorm (which can also be used inother programs to gain this functionality!)
  38. 38. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyTimestamp format• No year, no timezone• ... just because distros turn it off.• Remove “$ActionFileDefaultTemplateRSYSLOG_TraditionalFileFormat” fromrsyslog.conf to get rsyslogs default high-precision RFC5424 timestamp• Some tools may have problems with that, butcant be too bad – some Distros use the defaultformat
  39. 39. Rainer Gerhards * * LinuxTag 2013, Berlin, GermanyQuestions?••••• Associated paper isavailable on SlideShare.