Why proper logging is important... ...in all phases of development? Péter Czanik community managerCopyright 2013 BalaBit IT Security Ltd.
About me • Peter Czanik from Hungary • community manager at BalaBit: syslog-ng upstream • BalaBit is an IT security company with HQ in Budapest, Hungary with 100+ developers • part of the openSUSE testing team • openSUSE syslog-ng package maintainerCopyright 2013 BalaBit IT Security Ltd.
Topics • no, it is not about cutting trees :-) • what is syslog? and syslog-ng? • who uses syslog-ng? • what to log? • free-form messages against name-value pairs • the new buzzword: journal • standardization efforts: CEE/Lumberjack • name-value pairs at work: ELSACopyright 2013 BalaBit IT Security Ltd.
What is syslog? • logging: recording events • syslog: - application: collecting events - data: the actual log messages - protocol: forwarding events • history: - originally developed as a logging tool for sendmail - quickly many other apps started to use it • syslog-ng: “next generation” syslog server - since 1997 - focus on central log collectionCopyright 2013 BalaBit IT Security Ltd.
What is syslog-ng • “Swiss army knife” of logging • OSE vs. PE • high performance • more input sources (files, programs, and so on) • more destinations (databases, encrypted net, etc.) • better filtering (not only priority, facility) • processing (rewrite, parse, correlate, and so on) • JSON output and parser • AMQPCopyright 2013 BalaBit IT Security Ltd.
Who uses syslog-ng? • syslog-ng is the default logging solution in SLES since SLES 10 - Uses 2.0, an ancient version • syslog-ng is the default logging solution in Gentoo • syslog-ng is available in openSUSE (package is maintained by me :-) ) • ...and?Copyright 2013 BalaBit IT Security Ltd.
Who uses syslog-ng?Copyright 2013 BalaBit IT Security Ltd.
What to log? • what: everything :-) • in more detail: SANS Top5 log reports: - authentication, change, resource access, etc. • during development logging is often an afterthought - some/many of the above is missing - aids just coding - difficult to debug or audit in production • logging should be an integral part of development - think also about production :-) - consult with operators → DEVOPS!!! - use a similar logging environment, as in productionCopyright 2013 BalaBit IT Security Ltd.
How to log? • short answer: centrally • long: centrally, because: - ease of use: one place to check instead of many - availability: even if the sender machine is down - security: logs are available even if sender machine is compromisedCopyright 2013 BalaBit IT Security Ltd.
Free form log messages • most log messages are: date + hostname + text Mar 11 13:37:56 linux-6965 sshd: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 46048 ssh2 • text = English sentence with some variable parts • easy to read by a humanCopyright 2013 BalaBit IT Security Ltd.
Why it does not scale? • few logs (workstation) → easy to find information • many logs (server) → difficult to find information • information is presented differently by each application • difficult to process them with scripts • answer: structured logging - Events represented as name value pairsCopyright 2013 BalaBit IT Security Ltd.
Solution from syslog-ng: PatternDB • syslog-ng: name-value pairs inside - date, facility, priority, program name, pid, etc. • PatternDB parser: - can extract useful information into name-value pairs - add status fields based on message text - message classification • example: an ssh login failure: - user=root, action=login, status=failure - classified as “violation”Copyright 2013 BalaBit IT Security Ltd.
Journal • the logging component of system • name-value pairs inside: - message - trusted properties - sny additional name-value pairs • native support for name-value pair storage • persistent log storage can be disabled • logs can be forwarded to syslog-ng through a socket • syslog-ng can filter, process logs and forward them to central log serverCopyright 2013 BalaBit IT Security Ltd.
Journal: the enemy? • FAQ: Q: is journal the enemy? A: No! • Journal is local only (syslog-ng: client – server) • Journal does not filter or process log messages • Journal is limited to Linux/systemd (syslog-ng: all Linux/BSD/UNIX)Copyright 2013 BalaBit IT Security Ltd.
CEE • Journal, syslog-ng, Windows eventlog, rsyslog, auditd, and so on are based on name-value pairs • All use different field names • Standardization is a must: CEE → Common Event Expression • Events: name-value pairs instead of free-form text - Taxonomy: name-value pairs to describe events (example: status) - Dictionary: name-value pairs for event parameters (example: user) • PatternDB can turn free-form messages into CEECopyright 2013 BalaBit IT Security Ltd.
Name-value pairs in action: ELSA • ELSA: Enterprise Log Search and Archive • based on syslog-ng, PatternDB and MySQL • simple and powerful web GUI • extreme scalability • patterns focused on network security: - firewalls: Cisco, iptables - IDS: Snort, Suricata, Bro - HTTP, Windows logs, etc.Copyright 2013 BalaBit IT Security Ltd.
So, why syslog-ng? • 15 years of open source development • high performance log management • flexible configuration • excellent documentation • PatternDB message parsingCopyright 2013 BalaBit IT Security Ltd.
Questions? (and some answers) • Questions? • Some useful syslog-ng resources: - Syslog-ng: http://www.balabit.com/network-security/syslog-ng - SANS top5 essential log reports extended: http://chuvakin.blogspot.hu/2010/08/updated-with-community- feedback-sans_06.html - Many books at http://oreilly.com/ - ELSA: http://code.google.com/p/enterprise-log-search-and-archive/ - My blog: http://czanik.blogs.balabit.com/Copyright 2013 BalaBit IT Security Ltd.
Thank You! Péter Czanik community manager email@example.comCopyright 2013 BalaBit IT Security Ltd.