State of the art logging

638 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
638
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

State of the art logging

  1. 1. State of the art logging Syslog-ng, journal, CEE/Lumberjack and ELSA Péter Czanik community managerCopyright 2013 BalaBit IT Security Ltd.
  2. 2. Topics • No, it is not about cutting trees :-) • What is syslog? And syslog-ng? • Free-form messages against name-value pairs • The new buzzword: journal • Standardization efforts: CEE/Lumberjack • Name-value pairs at work: ELSACopyright 2013 BalaBit IT Security Ltd.
  3. 3. What is syslog? • Logging: recording events • Syslog: - Application: collecting events - Protocol: forwarding eventsCopyright 2013 BalaBit IT Security Ltd.
  4. 4. What is syslog-ng? • “Next Generation” syslog server • “Swiss army knife” of logging • More input sources (files, sockets, and so on) • Better filtering (not only priority, facility) • Processing (rewrite, normalize, correlate, and so on) • More destinations (databases, encrypted network, and so on)Copyright 2013 BalaBit IT Security Ltd.
  5. 5. What is new since 2.0 • 2.0 is best known, but EOL • Most important new features since 2.0: - PatternDB and CSV message parsing - Correlation - SQL and MongoDB destinations - JSON formatting - Modularization - Multi-threading • Next: 3.4 - JSON parsing - More flexible configurationCopyright 2013 BalaBit IT Security Ltd.
  6. 6. Free form log messages • Most logs are in /var/log • Most are from syslog (but also wtmp, apache, and so on) • Most are: date + hostname + text Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 46048 ssh2 • Text = English sentence with some variable parts • Easy to readCopyright 2013 BalaBit IT Security Ltd.
  7. 7. Why it does not scale? • Few logs (workstation) → easy to find information • Many logs (server) → difficult to find information • Relevant information is presented differently by each application • Difficult to process them with scripts • Answer: structured logging - Events represented as name value pairsCopyright 2013 BalaBit IT Security Ltd.
  8. 8. Solution from syslog-ng: PatternDB • Most messages are static texts with some variable parts embedded • PatternDB parser: - Can extract useful information into name-value pairs - Add status fields based on message text • Example: - user=root - action=login - status=failure • It requires patterns • syslog-ng: name-value pairs insideCopyright 2013 BalaBit IT Security Ltd.
  9. 9. Journal • The logging component of systemd • Name-value pairs inside: - Message - Trusted properties - Any additional name-value pairs • Native support for name-value pair storageCopyright 2013 BalaBit IT Security Ltd.
  10. 10. Journal: the enemy? • FAQ: Q: is journal the enemy? A: No! • Journal is limited to Linux/systemd (syslog-ng: all Linux/BSD/UNIX) • Journal is local only (syslog-ng: client – server) • Journal does not filter or process log messages • Journal + syslog-ng complement each other • Logs forwarded to syslog-ng through: /run/systemd/journal/syslog • syslog-ng can filter, process and forward logs to many different destinations (one day also to journal)Copyright 2013 BalaBit IT Security Ltd.
  11. 11. CEE • Journal, syslog-ng, Windows eventlog, rsyslog, auditd, and so on are based on name- value pairs • All use different field names • Standardization is a must: CEE → Common Event Expression • Events: name-value pairs instead of free-form text - Taxonomy: name-value pairs to describe events (example: status) - Dictionary: name-value pairs for event parameters (example: user) • PatternDB can turn free-form messages into CEECopyright 2013 BalaBit IT Security Ltd.
  12. 12. Lumberjack • Make CEE happen → implementation • Coordinated by RedHat - CEE (Mitre), syslog-ng, rsyslog, and so on - Open, with high traffic mailing list - https://fedorahosted.org/lumberjack/ • API(s) to make structured logging easier • Work on dictionary, taxonomy, transport issuesCopyright 2013 BalaBit IT Security Ltd.
  13. 13. Name-value pairs in action: ELSA • ELSA: Enterprise Log Search and Archive • Based on syslog-ng, PatternDB and MySQL • Simple and powerful web GUI • Extreme scalability • Patterns focused on network security (Cisco, Snort, HTTP, Bro, and so on)Copyright 2013 BalaBit IT Security Ltd.
  14. 14. Some logsCopyright 2013 BalaBit IT Security Ltd.
  15. 15. DiagramCopyright 2013 BalaBit IT Security Ltd.
  16. 16. A few extrasCopyright 2013 BalaBit IT Security Ltd.
  17. 17. Questions? • Questions?Copyright 2013 BalaBit IT Security Ltd.
  18. 18. Thank You! Péter Czanik community manager peter.czanik@balabit.comCopyright 2013 BalaBit IT Security Ltd.

×