Centralized Syslog with Syslog-ng and Logmuncher Russell Adams [email_address]
About the Author <ul><li>Proprietor Adams Information Services </li></ul><ul><ul><li>http://adamsinfoserv.com/ </li></ul><...
About the Author <ul><li>Linux since 1995 </li></ul><ul><li>Distribution Preferences </li></ul><ul><ul><li>Debian for serv...
Today's Discussion - Logs <ul><li>Proactive Monitoring </li></ul><ul><li>Software Recommendations </li></ul><ul><li>Tutori...
Proactive Monitoring <ul><li>Regular review of system logs can catch problems before they cause an outage. </li></ul><ul><...
System Log Contents <ul><li>System Events </li></ul><ul><li>User Activity </li></ul><ul><li>Network Connections </li></ul>...
Dark Side of Logs <ul><li>Verbose </li></ul><ul><li>Repetitive </li></ul><ul><li>Host Specific </li></ul><ul><li>Accumulat...
Overcoming Apathy <ul><li>Automation </li></ul><ul><li>Centralization </li></ul><ul><li>Filtering </li></ul><ul><li>Report...
Benefits of Centralized Logging <ul><li>Prevent Tampering </li></ul><ul><li>Single Point of Administration </li></ul><ul><...
Software <ul><li>Today's Tutorial uses </li></ul><ul><ul><li>Syslog-NG </li></ul></ul><ul><ul><ul><li>http://www.balabit.c...
Software <ul><li>Other Packages </li></ul><ul><ul><li>Commercial Packages </li></ul></ul><ul><ul><ul><li>Kiwi Enterprises ...
Best Practices <ul><li>Synchronize time via NTP </li></ul><ul><li>Version control configuration files, dictionaries </li><...
Syslog-ng <ul><li>Written by Balazs Scheidler </li></ul><ul><li>Supports any syslog client </li></ul><ul><ul><li>Linux, *n...
Syslog-ng <ul><li>Using a hierarchy to organize logs </li></ul><ul><ul><li>Logs stored in a directory tree </li></ul></ul>...
Syslog-ng <ul><li>Benefits of using a hierarchy over monolithic log files </li></ul><ul><ul><li>Automatic rotation within ...
Logmuncher <ul><li>Written by Geoff Kuenning </li></ul><ul><li>Shell script  </li></ul><ul><ul><li>Uses standard UNIX Comm...
Logmuncher <ul><li>Flexible configuration language </li></ul><ul><ul><li>Variable substitution </li></ul></ul><ul><ul><ul>...
Logmuncher <ul><li>Dictionaries </li></ul><ul><ul><li>Contain single line regular expressions which match syslog messages ...
Logmuncher <ul><li>Recommended Configuration </li></ul><ul><ul><li>Use Ignore Dictionaries! </li></ul></ul><ul><ul><ul><li...
Centralized Log Flow
Syslog-ng Server Illustration Example: /var/log/HOSTS/hosta/2007/08/08/kernel20070808 /var/log/HOSTS/hostb/2007/08/08/mail...
Logmuncher Concept Chart
Syslog-ng Server Configuration Sample <ul><li>options {  </li></ul><ul><li>long_hostnames(off); </li></ul><ul><li>sync(0);...
Syslog-ng Client Configuration Sample <ul><li>options {  </li></ul><ul><li>long_hostnames(off); </li></ul><ul><li>sync(0);...
Logmuncher Report Sample <ul><li>Date: Sat, 17 May 2003 09:25:11 -0500 </li></ul><ul><li>From: root@logmuncher.mydomain.co...
Logmuncher Host Configuration <ul><li>subject ns2 %d %t Logmuncher Report </li></ul><ul><li>header ********** ns2 Log Entr...
Logmuncher Dictionary Sample <ul><li>CROND.*bin.*CMD.*/usr/local/sbin/amdump </li></ul><ul><li>CROND.*bin.*CMD.*/usr/local...
Q&A <ul><li>Come to the Wednesday night workshop, 6 PM weekly! </li></ul><ul><li>Thank you! </li></ul>
Upcoming SlideShare
Loading in …5
×

Centralized Syslog

1,923 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,923
On SlideShare
0
From Embeds
0
Number of Embeds
45
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Centralized Syslog

  1. 1. Centralized Syslog with Syslog-ng and Logmuncher Russell Adams [email_address]
  2. 2. About the Author <ul><li>Proprietor Adams Information Services </li></ul><ul><ul><li>http://adamsinfoserv.com/ </li></ul></ul><ul><ul><li>Open Source Monitoring & Consulting </li></ul></ul><ul><ul><li>Author of NACE </li></ul></ul><ul><ul><ul><li>http://adamsinfoserv.com/AISTWiki/bin/view/AIS/NACE </li></ul></ul></ul><ul><li>IBM CATE </li></ul><ul><li>Mission critical large enterprise systems </li></ul><ul><ul><li>AIX / Linux </li></ul></ul><ul><ul><li>High Availability Clustering </li></ul></ul>
  3. 3. About the Author <ul><li>Linux since 1995 </li></ul><ul><li>Distribution Preferences </li></ul><ul><ul><li>Debian for servers </li></ul></ul><ul><ul><li>Gentoo for personal workstation </li></ul></ul>
  4. 4. Today's Discussion - Logs <ul><li>Proactive Monitoring </li></ul><ul><li>Software Recommendations </li></ul><ul><li>Tutorial / Sample </li></ul><ul><li>Discuss Log Management </li></ul>
  5. 5. Proactive Monitoring <ul><li>Regular review of system logs can catch problems before they cause an outage. </li></ul><ul><ul><li>Failing Drives </li></ul></ul><ul><ul><li>Temperature Problems </li></ul></ul><ul><ul><li>Full filesystems </li></ul></ul><ul><ul><li>Dysfunctional Software </li></ul></ul>
  6. 6. System Log Contents <ul><li>System Events </li></ul><ul><li>User Activity </li></ul><ul><li>Network Connections </li></ul><ul><li>Authentication Failures </li></ul><ul><li>Device Errors </li></ul><ul><li>Kernel Messages </li></ul><ul><li>Firewall Rule Violations </li></ul><ul><li>Database Warnings </li></ul>
  7. 7. Dark Side of Logs <ul><li>Verbose </li></ul><ul><li>Repetitive </li></ul><ul><li>Host Specific </li></ul><ul><li>Accumulate Rapidly </li></ul><ul><li>Cumbersome Archival </li></ul><ul><li>Low Signal to Noise Ratio </li></ul><ul><li>Easily Tampered With </li></ul>
  8. 8. Overcoming Apathy <ul><li>Automation </li></ul><ul><li>Centralization </li></ul><ul><li>Filtering </li></ul><ul><li>Reporting </li></ul><ul><li>Rotation </li></ul><ul><li>Archival </li></ul>
  9. 9. Benefits of Centralized Logging <ul><li>Prevent Tampering </li></ul><ul><li>Single Point of Administration </li></ul><ul><li>Correlate Events between Hosts </li></ul><ul><li>Simplify Reporting / Filtering </li></ul><ul><li>Organized by Host, Date, Type </li></ul>
  10. 10. Software <ul><li>Today's Tutorial uses </li></ul><ul><ul><li>Syslog-NG </li></ul></ul><ul><ul><ul><li>http://www.balabit.com/network-security/syslog-ng/ </li></ul></ul></ul><ul><ul><li>Logmuncher </li></ul></ul><ul><ul><ul><li>http://lasr.cs.ucla.edu/geoff/logmuncher.html </li></ul></ul></ul>
  11. 11. Software <ul><li>Other Packages </li></ul><ul><ul><li>Commercial Packages </li></ul></ul><ul><ul><ul><li>Kiwi Enterprises </li></ul></ul></ul><ul><ul><ul><ul><li>http://kiwisyslog.com/ </li></ul></ul></ul></ul><ul><ul><ul><li>Splunk </li></ul></ul></ul><ul><ul><ul><ul><li>http://www.splunk.com/ </li></ul></ul></ul></ul><ul><ul><li>OSS Alternatives </li></ul></ul><ul><ul><ul><li>Swatch </li></ul></ul></ul><ul><ul><ul><ul><li>http://swatch.sourceforge.net/ </li></ul></ul></ul></ul><ul><ul><ul><li>Prelude </li></ul></ul></ul><ul><ul><ul><ul><li>http://www.prelude-ids.org/ </li></ul></ul></ul></ul>
  12. 12. Best Practices <ul><li>Synchronize time via NTP </li></ul><ul><li>Version control configuration files, dictionaries </li></ul><ul><li>Use multiple methods of notification </li></ul><ul><li>Share common dictionaries </li></ul>
  13. 13. Syslog-ng <ul><li>Written by Balazs Scheidler </li></ul><ul><li>Supports any syslog client </li></ul><ul><ul><li>Linux, *nix, Cisco, Windows... </li></ul></ul><ul><li>Logging over TCP & UDP </li></ul><ul><ul><li>Optional support for Stunnel </li></ul></ul><ul><li>Future Plans for Signed Logs and Encryption </li></ul>
  14. 14. Syslog-ng <ul><li>Using a hierarchy to organize logs </li></ul><ul><ul><li>Logs stored in a directory tree </li></ul></ul><ul><ul><ul><li>Host </li></ul></ul></ul><ul><ul><ul><li>Date (Year, Month, Day) </li></ul></ul></ul><ul><ul><ul><li>Facility </li></ul></ul></ul><ul><ul><ul><li>/var/log/HOSTS/host/YYYY/MM/DD/facilityYYYYMMDD </li></ul></ul></ul><ul><ul><li>Kudos to the Syslog-ng FAQ at http://www.campin.net/ </li></ul></ul>
  15. 15. Syslog-ng <ul><li>Benefits of using a hierarchy over monolithic log files </li></ul><ul><ul><li>Automatic rotation within hierarchy </li></ul></ul><ul><ul><li>Easy cleanup </li></ul></ul><ul><ul><li>Hosts, times, facilities separated for easy access </li></ul></ul>
  16. 16. Logmuncher <ul><li>Written by Geoff Kuenning </li></ul><ul><li>Shell script </li></ul><ul><ul><li>Uses standard UNIX Commands </li></ul></ul><ul><ul><li>Grep / Sed / Awk </li></ul></ul><ul><li>Notification Options </li></ul><ul><ul><li>E-mail </li></ul></ul><ul><ul><li>Pager </li></ul></ul><ul><ul><li>External Script </li></ul></ul>
  17. 17. Logmuncher <ul><li>Flexible configuration language </li></ul><ul><ul><li>Variable substitution </li></ul></ul><ul><ul><ul><li>Common config file can apply to multiple hosts using %C </li></ul></ul></ul><ul><ul><ul><li>Reporting against different dictionaries with different actions (ie: warning vs critical) </li></ul></ul></ul><ul><li>Works with hierarchies of log files! </li></ul>
  18. 18. Logmuncher <ul><li>Dictionaries </li></ul><ul><ul><li>Contain single line regular expressions which match syslog messages </li></ul></ul><ul><ul><li>Common dictionaries </li></ul></ul><ul><ul><ul><li>Ignore </li></ul></ul></ul><ul><ul><ul><li>Warn </li></ul></ul></ul><ul><ul><ul><li>Critical </li></ul></ul></ul><ul><ul><ul><li>Platform specific </li></ul></ul></ul><ul><ul><ul><li>Node specific </li></ul></ul></ul><ul><ul><li>Configuration File specifies which dictionary to use and action to take </li></ul></ul>
  19. 19. Logmuncher <ul><li>Recommended Configuration </li></ul><ul><ul><li>Use Ignore Dictionaries! </li></ul></ul><ul><ul><ul><li>Unknown messages are escalated </li></ul></ul></ul><ul><ul><li>Known messages are OK to ignore </li></ul></ul><ul><ul><li>Can optionally match specific critical messages with separate behavior </li></ul></ul><ul><ul><ul><li>Critical matches result in page </li></ul></ul></ul><ul><ul><ul><li>Warning matches and default use email </li></ul></ul></ul><ul><ul><li>Gracefully Handle Unknown Log Messages </li></ul></ul>
  20. 20. Centralized Log Flow
  21. 21. Syslog-ng Server Illustration Example: /var/log/HOSTS/hosta/2007/08/08/kernel20070808 /var/log/HOSTS/hostb/2007/08/08/mail20070808
  22. 22. Logmuncher Concept Chart
  23. 23. Syslog-ng Server Configuration Sample <ul><li>options { </li></ul><ul><li>long_hostnames(off); </li></ul><ul><li>sync(0); </li></ul><ul><li>}; </li></ul><ul><li>source src { </li></ul><ul><li>unix-stream(&quot;/dev/log&quot;); </li></ul><ul><li>internal(); </li></ul><ul><li>file(&quot;/proc/kmsg&quot;); </li></ul><ul><li>tcp( max-connections(100) ); </li></ul><ul><li>udp(); </li></ul><ul><li>}; </li></ul><ul><li>destination hosts { </li></ul><ul><li>file( &quot;/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY$YEAR$MONTH$DAY&quot; </li></ul><ul><li>owner(root) </li></ul><ul><li>group(root) </li></ul><ul><li>perm(0600) </li></ul><ul><li>dir_perm(0700) </li></ul><ul><li>create_dirs(yes) </li></ul><ul><li>); </li></ul><ul><li>}; </li></ul><ul><li>log { </li></ul><ul><li>source(src); </li></ul><ul><li>destination(hosts); </li></ul><ul><li>}; </li></ul>
  24. 24. Syslog-ng Client Configuration Sample <ul><li>options { </li></ul><ul><li>long_hostnames(off); </li></ul><ul><li>sync(0); </li></ul><ul><li>log_fifo_size(1000); </li></ul><ul><li>}; </li></ul><ul><li>source src { </li></ul><ul><li>unix-stream(&quot;/dev/log&quot;); </li></ul><ul><li>internal(); </li></ul><ul><li>file(&quot;/proc/kmsg&quot;); </li></ul><ul><li>}; </li></ul><ul><li>destination loghost { tcp(&quot;mysyslogserver.mydomain.com.&quot;); }; </li></ul><ul><li>log { </li></ul><ul><li>source(src); </li></ul><ul><li>destination(loghost); </li></ul><ul><li>}; </li></ul>
  25. 25. Logmuncher Report Sample <ul><li>Date: Sat, 17 May 2003 09:25:11 -0500 </li></ul><ul><li>From: root@logmuncher.mydomain.com </li></ul><ul><li>Subject: lids1 05/17/03 09:25:01 Logmuncher Report </li></ul><ul><li>********** lids1 Log Entries ********** </li></ul><ul><li>May 17 09:10:24 lids1 snort: [1:553:4] POLICY FTP </li></ul><ul><li>anonymous login attempt [Classification: Misc activity] </li></ul><ul><li>[Priority: 3]: {TCP} 80.26.139.84:1558 -> 198.42.129.2:21 </li></ul><ul><li>May 17 09:10:24 lids1 snort: [1:553:4] POLICY FTP </li></ul><ul><li>anonymous login attempt [Classification: Misc activity] </li></ul><ul><li>[Priority: 3]: {TCP} 80.26.139.84:1559 -> 198.42.129.3:21 </li></ul><ul><li>May 17 09:10:24 lids1 snort: [1:553:4] POLICY FTP </li></ul><ul><li>anonymous login attempt [Classification: Misc activity] </li></ul><ul><li>[Priority: 3]: {TCP} 80.26.139.84:1565 -> 198.42.129.9:21 </li></ul>
  26. 26. Logmuncher Host Configuration <ul><li>subject ns2 %d %t Logmuncher Report </li></ul><ul><li>header ********** ns2 Log Entries ********** </li></ul><ul><li>mtailfile /var/log/HOSTS/ns2/*/*/*/* </li></ul><ul><li>re-ignore /etc/logmuncher/patterns/common </li></ul><ul><li>re-ignore /etc/logmuncher/patterns/ns2 </li></ul><ul><li>send-report [email_address] </li></ul>Note that when using a directory of configuration files, you can use a single common configuration file, and symlink it to each host name. Use %C in place of “ns2” above, and the filename will be used as the host name. The mtailfile directive is used with the hierarchy of files written in this tutorial. Static filenames can be used with the tailfile directive, to watch items like apache logs and application logs.
  27. 27. Logmuncher Dictionary Sample <ul><li>CROND.*bin.*CMD.*/usr/local/sbin/amdump </li></ul><ul><li>CROND.*bin.*CMD.*/usr/local/sbin/amcheck </li></ul><ul><li>CROND.*cricket.*CMD.*/home/cricket/cricket/collect-subtrees </li></ul><ul><li>CROND.*cricket.*CMD.*/usr/bin/find </li></ul><ul><li>ftpd.*incoming </li></ul><ul><li>ftpd.*FTP session closed </li></ul><ul><li>httpd.*No Local authentication done </li></ul><ul><li>httpd.*pam_smb.*Configuration Data </li></ul><ul><li>httpd.*pam_smb.*Correct NT username/password pair </li></ul><ul><li>sendmail.*stat=Sent </li></ul><ul><li>sendmail.*relay=.*@localhost </li></ul><ul><li>sendmail.*cricket.*forward </li></ul><ul><li>sendmail.*relay=ks119is01mail1.ksnet.com </li></ul><ul><li>sshd.*Generating new 768 bit RSA key </li></ul><ul><li>sshd.*RSA key generation complete </li></ul><ul><li>xinetd.*EXIT.*ftp </li></ul><ul><li>xinetd.*START.*amanda.*from=192.168.1.12 </li></ul><ul><li>xinetd.*START.*ftp.*from=192.168.1.13 </li></ul>
  28. 28. Q&A <ul><li>Come to the Wednesday night workshop, 6 PM weekly! </li></ul><ul><li>Thank you! </li></ul>

×