State of the art logging Syslog-ng, journal, CEE/Lumberjack and ELSA Péter Czanik community managerCopyright 2013 BalaBit IT Security Ltd.
Topics • No, it is not about cutting trees :-) • What is syslog? And syslog-ng? • Free-form messages against name-value pairs • The new buzzword: journal • Standardization efforts: CEE/Lumberjack • Name-value pairs at work: ELSACopyright 2013 BalaBit IT Security Ltd.
What is syslog? • Logging: recording events • Syslog: - Application: collecting events - Protocol: forwarding eventsCopyright 2013 BalaBit IT Security Ltd.
What is syslog-ng? • “Next Generation” syslog server • “Swiss army knife” of logging • More input sources (files, sockets, and so on) • Better filtering (not only priority, facility) • Processing (rewrite, normalize, correlate, and so on) • More destinations (databases, encrypted network, and so on)Copyright 2013 BalaBit IT Security Ltd.
What is new since 2.0 • 2.0 is best known, but EOL • Most important new features since 2.0: - PatternDB and CSV message parsing - Correlation - SQL and MongoDB destinations - JSON formatting - Modularization - Multi-threading • Next: 3.4 - JSON parsing - More flexible configurationCopyright 2013 BalaBit IT Security Ltd.
Free form log messages • Most logs are in /var/log • Most are from syslog (but also wtmp, apache, and so on) • Most are: date + hostname + text Mar 11 13:37:56 linux-6965 sshd: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 46048 ssh2 • Text = English sentence with some variable parts • Easy to readCopyright 2013 BalaBit IT Security Ltd.
Why it does not scale? • Few logs (workstation) → easy to find information • Many logs (server) → difficult to find information • Relevant information is presented differently by each application • Difficult to process them with scripts • Answer: structured logging - Events represented as name value pairsCopyright 2013 BalaBit IT Security Ltd.
Solution from syslog-ng: PatternDB • Most messages are static texts with some variable parts embedded • PatternDB parser: - Can extract useful information into name-value pairs - Add status fields based on message text • Example: - user=root - action=login - status=failure • It requires patterns • syslog-ng: name-value pairs insideCopyright 2013 BalaBit IT Security Ltd.
Journal • The logging component of systemd • Name-value pairs inside: - Message - Trusted properties - Any additional name-value pairs • Native support for name-value pair storageCopyright 2013 BalaBit IT Security Ltd.
Journal: the enemy? • FAQ: Q: is journal the enemy? A: No! • Journal is limited to Linux/systemd (syslog-ng: all Linux/BSD/UNIX) • Journal is local only (syslog-ng: client – server) • Journal does not filter or process log messages • Journal + syslog-ng complement each other • Logs forwarded to syslog-ng through: /run/systemd/journal/syslog • syslog-ng can filter, process and forward logs to many different destinations (one day also to journal)Copyright 2013 BalaBit IT Security Ltd.
CEE • Journal, syslog-ng, Windows eventlog, rsyslog, auditd, and so on are based on name- value pairs • All use different field names • Standardization is a must: CEE → Common Event Expression • Events: name-value pairs instead of free-form text - Taxonomy: name-value pairs to describe events (example: status) - Dictionary: name-value pairs for event parameters (example: user) • PatternDB can turn free-form messages into CEECopyright 2013 BalaBit IT Security Ltd.
Lumberjack • Make CEE happen → implementation • Coordinated by RedHat - CEE (Mitre), syslog-ng, rsyslog, and so on - Open, with high traffic mailing list - https://fedorahosted.org/lumberjack/ • API(s) to make structured logging easier • Work on dictionary, taxonomy, transport issuesCopyright 2013 BalaBit IT Security Ltd.
Name-value pairs in action: ELSA • ELSA: Enterprise Log Search and Archive • Based on syslog-ng, PatternDB and MySQL • Simple and powerful web GUI • Extreme scalability • Patterns focused on network security (Cisco, Snort, HTTP, Bro, and so on)Copyright 2013 BalaBit IT Security Ltd.
Some logsCopyright 2013 BalaBit IT Security Ltd.
DiagramCopyright 2013 BalaBit IT Security Ltd.
A few extrasCopyright 2013 BalaBit IT Security Ltd.
Questions? • Questions?Copyright 2013 BalaBit IT Security Ltd.
Thank You! Péter Czanik community manager firstname.lastname@example.orgCopyright 2013 BalaBit IT Security Ltd.