1. Why proper logging is important...
...in all phases of development?
Péter Czanik
community manager
Copyright 2013 BalaBit IT Security Ltd.
2. About me
• Peter Czanik from Hungary
• community manager at BalaBit: syslog-ng upstream
• BalaBit is an IT security company with HQ in Budapest,
Hungary with 100+ developers
• part of the openSUSE testing team
• openSUSE syslog-ng package maintainer
Copyright 2013 BalaBit IT Security Ltd.
3. Topics
• no, it is not about cutting trees :-)
• what is syslog? and syslog-ng?
• who uses syslog-ng?
• what to log?
• free-form messages against name-value pairs
• the new buzzword: journal
• standardization efforts: CEE/Lumberjack
• name-value pairs at work: ELSA
Copyright 2013 BalaBit IT Security Ltd.
4. What is syslog?
• logging: recording events
• syslog:
- application: collecting events
- data: the actual log messages
- protocol: forwarding events
• history:
- originally developed as a logging tool for sendmail
- quickly many other apps started to use it
• syslog-ng: “next generation” syslog server
- since 1997
- focus on central log collection
Copyright 2013 BalaBit IT Security Ltd.
5. What is syslog-ng
• “Swiss army knife” of logging
• OSE vs. PE
• high performance
• more input sources (files, programs, and so on)
• more destinations (databases, encrypted net, etc.)
• better filtering (not only priority, facility)
• processing (rewrite, parse, correlate, and so on)
• JSON output and parser
• AMQP
Copyright 2013 BalaBit IT Security Ltd.
6. Who uses syslog-ng?
• syslog-ng is the default logging solution in SLES since
SLES 10
- Uses 2.0, an ancient version
• syslog-ng is the default logging solution in Gentoo
• syslog-ng is available in openSUSE (package is
maintained by me :-) )
• ...and?
Copyright 2013 BalaBit IT Security Ltd.
8. What to log?
• what: everything :-)
• in more detail: SANS Top5 log reports:
- authentication, change, resource access, etc.
• during development logging is often an afterthought
- some/many of the above is missing
- aids just coding
- difficult to debug or audit in production
• logging should be an integral part of development
- think also about production :-)
- consult with operators → DEVOPS!!!
- use a similar logging environment, as in production
Copyright 2013 BalaBit IT Security Ltd.
9. How to log?
• short answer: centrally
• long: centrally, because:
- ease of use: one place to check instead of many
- availability: even if the sender machine is down
- security: logs are available even if sender machine is compromised
Copyright 2013 BalaBit IT Security Ltd.
10. Free form log messages
• most log messages are: date + hostname + text
Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted
keyboard-interactive/pam for root from 127.0.0.1
port 46048 ssh2
• text = English sentence with some variable parts
• easy to read by a human
Copyright 2013 BalaBit IT Security Ltd.
11. Why it does not scale?
• few logs (workstation) → easy to find information
• many logs (server) → difficult to find information
• information is presented differently by each application
• difficult to process them with scripts
• answer: structured logging
- Events represented as name value pairs
Copyright 2013 BalaBit IT Security Ltd.
12. Solution from syslog-ng: PatternDB
• syslog-ng: name-value pairs inside
- date, facility, priority, program name, pid, etc.
• PatternDB parser:
- can extract useful information into name-value pairs
- add status fields based on message text
- message classification
• example: an ssh login failure:
- user=root, action=login, status=failure
- classified as “violation”
Copyright 2013 BalaBit IT Security Ltd.
13. Journal
• the logging component of system
• name-value pairs inside:
- message
- trusted properties
- sny additional name-value pairs
• native support for name-value pair storage
• persistent log storage can be disabled
• logs can be forwarded to syslog-ng through a socket
• syslog-ng can filter, process logs and forward them to
central log server
Copyright 2013 BalaBit IT Security Ltd.
14. Journal: the enemy?
• FAQ: Q: is journal the enemy? A: No!
• Journal is local only (syslog-ng: client – server)
• Journal does not filter or process log messages
• Journal is limited to Linux/systemd (syslog-ng: all
Linux/BSD/UNIX)
Copyright 2013 BalaBit IT Security Ltd.
15. CEE
• Journal, syslog-ng, Windows eventlog, rsyslog, auditd,
and so on are based on name-value pairs
• All use different field names
• Standardization is a must: CEE → Common Event
Expression
• Events: name-value pairs instead of free-form text
- Taxonomy: name-value pairs to describe events (example: status)
- Dictionary: name-value pairs for event parameters (example: user)
• PatternDB can turn free-form messages into CEE
Copyright 2013 BalaBit IT Security Ltd.
16. Name-value pairs in action: ELSA
• ELSA: Enterprise Log Search and Archive
• based on syslog-ng, PatternDB and MySQL
• simple and powerful web GUI
• extreme scalability
• patterns focused on network security:
- firewalls: Cisco, iptables
- IDS: Snort, Suricata, Bro
- HTTP, Windows logs, etc.
Copyright 2013 BalaBit IT Security Ltd.
20. So, why syslog-ng?
• 15 years of open source development
• high performance log management
• flexible configuration
• excellent documentation
• PatternDB message parsing
Copyright 2013 BalaBit IT Security Ltd.
21. Questions? (and some answers)
• Questions?
• Some useful syslog-ng resources:
- Syslog-ng: http://www.balabit.com/network-security/syslog-ng
- SANS top5 essential log reports extended:
http://chuvakin.blogspot.hu/2010/08/updated-with-community-
feedback-sans_06.html
- Many books at http://oreilly.com/
- ELSA: http://code.google.com/p/enterprise-log-search-and-archive/
- My blog: http://czanik.blogs.balabit.com/
Copyright 2013 BalaBit IT Security Ltd.
22. Thank You!
Péter Czanik
community manager
peter.czanik@balabit.com
Copyright 2013 BalaBit IT Security Ltd.
