Centralized Logging with syslog

Slide 1: Building Centralized Logging: Syslog Steven “Maniac” McGrath

Slide 2: Syslog? • logging service • UNIX based • Networkable

Slide 3: Wait a Sec...Network? • UDP port 514 • Typically limited to 1024bytes

Slide 4: One more thing... • FIFO Buffers • First In First Out • Rolling View of Logs • Type of Named Pipe

Slide 5: FIFO...Tasty *chomp* 3 Line FIFO Buffer Item 5 Item 4 Item 3 Item 2 Item 1

Slide 6: Getting Started... • Ubuntu 6.06 Server • Base Install

Slide 7: Installing Syslog... • Update The Repository

Slide 8: Upgrade the OS • We need to upgrade the OS to current.

Slide 9: Install Syslog-NG • Syslog-NG will remove klogd, this is normal.

Slide 10: Reconfiguring Syslog-ng • Configuration depends on network environment. • Windows Hosts • Cisco Devices • Linux Hosts • Other Devices and Gear

Slide 11: First off...Global! /etc/syslog-ng/syslog-ng.conf options { chain_hostnames(0); time_reopen(10); time_reap(360); log_fifo_size(2048); create_dirs(yes); group(admin); perm(0640); dir_perm(0755); use_dns(no); stats_freq(0); }; • Disable Hostname Chaining • Time to wait before re-establishing a dead connection • Time to wait before an idle file is closed • FIFO Buffer size • Create Directories • Permissions • Disable DNS • Disable Statistic Logging

Slide 12: Next, The Source /etc/syslog-ng/syslog-ng.conf source s_all { internal(); unix-stream(\"/dev/log\"); file(\"/proc/kmsg\" log_prefix(\"kernel: \")); udp(); };

Slide 13: Defining Filters • Windows Filter • Cisco Filter

Slide 14: Windows Filter /etc/syslog-ng/syslog-ng.conf filter f_windows { program(MSWinEventLog); };

Slide 15: Cisco Filter /etc/syslog-ng/syslog-ng.conf filter f_cisco_pix { host(IP.OF.PIX.DEVICE); };

Slide 16: General Filter /etc/syslog-ng/syslog-ng.conf filter f_not_others { not host(IP.OF.PIX.DEVICE) and not program(MSWinEventLog); };

Slide 17: Destinations • FIFO Buffers • One Large File

Slide 18: Windows FIFO /etc/syslog-ng/syslog-ng.conf destination d_windows { pipe(“/var/log/buffers/windows”); };

Slide 19: Cisco FIFO /etc/syslog-ng/syslog-ng.conf destination d_cisco { pipe(“/var/log/buffers/cisco”); };

Slide 20: General FIFO /etc/syslog-ng/syslog-ng.conf destination d_gen_fifo { pipe(“/var/log/buffers/syslog”); };

Slide 21: ...And the Archive /etc/syslog-ng/syslog-ng.conf destination d_all { file(“/var/log/arch/$MONTH$DAY$YEAR”); };

Slide 22: Tying it all Together! • Now we tell syslog to handle the configs. ;)

Slide 23: Windows Log /etc/syslog-ng/syslog-ng.conf log { source(s_all); filter(f_windows); destination(d_windows); };

Slide 24: Cisco Log /etc/syslog-ng/syslog-ng.conf log { source(s_all); filter(f_cisco_pix); destination(d_cisco); };

Slide 25: General FIFO /etc/syslog-ng/syslog-ng.conf log { source(s_all); filter(f_not_others); destination(d_gen_fifo); };

Slide 26: Archive Log /etc/syslog-ng/syslog-ng.conf log { source(s_all); destination(d_all); };

Slide 27: Finishing up... • Making the FIFO buffers • Creating the directory structure

Slide 28: Run me :) $ sudo mkdir /var/log/arch $ sudo mkdir /var/log/buffers $ sudo mkfifo /var/log/buffers/windows $ sudo mkfifo /var/log/buffers/cisco $ sudo mkfifo /var/log/buffers/syslog

Slide 29: Restart Syslog-ng $ sudo /etc/init.d/syslog-ng restart

Slide 30: Is it working? • Check your Logfiles (/var/log/arch/*) • Check your FIFO Buffers • cat /var/log/buffers/windows • cat /var/log/buffers/cisco • cat /var/log/buffers/syslog

Slide 31: Awsome! Wait.... • How are we gonna view this data?

Slide 32: splunk> • Web-based Interface • Indexes arbitrary data • Searchable • Reporting

Slide 33: splunk> • No, I don’t work for them...I just really like their product.

Slide 34: Installing splunk> • Download The latest version (3.0b3 as of writing) • Extract the tarball • Run the application • Make it startup with a system boot

Slide 35: Installing splunk> $ wget 'http://www.splunk.com/index.php/ download_track?file=/3.0b3/linux/ splunk-3.0b3-20872-Linux- i686.tgz&ac=&wget=true&name=wget' $ sudo mkdir /opt;cd /opt $ sudo tar xzvf ~/splunk-3.0b3-20872-Linux- i686.tgz $ sudo /opt/splunk/bin

Slide 36: Configuring splunk>

Slide 37: Configuring splunk>

Slide 38: Configuring splunk>

Slide 39: Configuring splunk>

Slide 40: Configuring splunk>

Slide 41: splunk>

Slide 42: Syslog Agents • Windows Agents • UNIX Agents • Other Devices

Slide 43: Windows Logs? • SNARE Agent • Converts Event Logs to Syslog • Free

Slide 44: UNIX Agents • Use the syslog service! • *.* @Syslog Server

Slide 45: Other Devices • Various systems can be configured • Cisco, Juniper, Lotus Domino, Apache, IIS, etc. are just a few examples.

Slide 46: Recap • What is Syslog • What is FIFO • Installing and Configuring Syslog-NG • Installing and Configuring Splunk • Agents

Slide 47: Questions?