The document discusses setting up a centralized logging system using syslog and syslog-ng to collect logs from various systems into FIFO buffers, and then configuring Splunk for log analysis and searching. Instructions are provided on installing and configuring syslog-ng to define sources, filters, and destinations for different types of logs, as well as installing and configuring Splunk for log ingestion and a web-based interface for searching logs.

1. Building Centralized
Logging: Syslog
Steven “Maniac” McGrath

2. Syslog?
• logging service
• UNIX based
• Networkable

3. Wait a Sec...Network?
• UDP port 514
• Typically limited to 1024bytes

4. One more thing...
• FIFO Buffers
• First In First Out
• Rolling View of Logs
• Type of Named Pipe

5. FIFO...Tasty *chomp*
3 Line FIFO Buffer
Item 5
Item 4
Item 3
Item 2
Item 1

6. Getting Started...
• Ubuntu 6.06 Server
• Base Install

7. Installing Syslog...
• Update The Repository

8. Upgrade the OS
• We need to upgrade the OS to current.

9. Install Syslog-NG
• Syslog-NG will remove klogd, this is normal.

10. Reconfiguring Syslog-ng
• Configuration depends on network
environment.
• Windows Hosts
• Cisco Devices
• Linux Hosts
• Other Devices and Gear

11. First off...Global!
/etc/syslog-ng/syslog-ng.conf
options {
chain_hostnames(0);
time_reopen(10);
time_reap(360);
log_fifo_size(2048);
create_dirs(yes);
group(admin);
perm(0640);
dir_perm(0755);
use_dns(no);
stats_freq(0);
};
• Disable Hostname Chaining
• Time to wait before re-establishing a dead connection
• Time to wait before an idle file is closed
• FIFO Buffer size
• Create Directories
• Permissions
• Disable DNS
• Disable Statistic Logging

12. Next, The Source
/etc/syslog-ng/syslog-ng.conf
source s_all {
internal();
unix-stream(quot;/dev/logquot;);
file(quot;/proc/kmsgquot; log_prefix(quot;kernel: quot;));
udp();
};

13. Defining Filters
• Windows Filter
• Cisco Filter

14. Windows Filter
/etc/syslog-ng/syslog-ng.conf
filter f_windows {
program(MSWinEventLog);
};

15. Cisco Filter
/etc/syslog-ng/syslog-ng.conf
filter f_cisco_pix {
host(IP.OF.PIX.DEVICE);
};

16. General Filter
/etc/syslog-ng/syslog-ng.conf
filter f_not_others {
not host(IP.OF.PIX.DEVICE)
and not program(MSWinEventLog);
};

17. Destinations
• FIFO Buffers
• One Large File

18. Windows FIFO
/etc/syslog-ng/syslog-ng.conf
destination d_windows {
pipe(“/var/log/buffers/windows”);
};

19. Cisco FIFO
/etc/syslog-ng/syslog-ng.conf
destination d_cisco {
pipe(“/var/log/buffers/cisco”);
};

20. General FIFO
/etc/syslog-ng/syslog-ng.conf
destination d_gen_fifo {
pipe(“/var/log/buffers/syslog”);
};

21. ...And the Archive
/etc/syslog-ng/syslog-ng.conf
destination d_all {
file(“/var/log/arch/$MONTH$DAY$YEAR”);
};

22. Tying it all Together!
• Now we tell syslog to handle the configs. ;)

23. Windows Log
/etc/syslog-ng/syslog-ng.conf
log {
source(s_all);
filter(f_windows);
destination(d_windows);
};

24. Cisco Log
/etc/syslog-ng/syslog-ng.conf
log {
source(s_all);
filter(f_cisco_pix);
destination(d_cisco);
};

25. General FIFO
/etc/syslog-ng/syslog-ng.conf
log {
source(s_all);
filter(f_not_others);
destination(d_gen_fifo);
};

26. Archive Log
/etc/syslog-ng/syslog-ng.conf
log {
source(s_all);
destination(d_all);
};

27. Finishing up...
• Making the FIFO buffers
• Creating the directory structure

28. Run me :)
$ sudo mkdir /var/log/arch
$ sudo mkdir /var/log/buffers
$ sudo mkfifo /var/log/buffers/windows
$ sudo mkfifo /var/log/buffers/cisco
$ sudo mkfifo /var/log/buffers/syslog

29. Restart Syslog-ng
$ sudo /etc/init.d/syslog-ng restart

30. Is it working?
• Check your Logfiles (/var/log/arch/*)
• Check your FIFO Buffers
• cat /var/log/buffers/windows
• cat /var/log/buffers/cisco
• cat /var/log/buffers/syslog

31. Awsome! Wait....
• How are we gonna view this data?

32. splunk>
• Web-based Interface
• Indexes arbitrary data
• Searchable
• Reporting

33. splunk>
• No, I don’t work for them...I just really like
their product.

34. Installing splunk>
• Download The latest version (3.0b3 as of
writing)
• Extract the tarball
• Run the application
• Make it startup with a system boot

35. Installing splunk>
$ wget 'http://www.splunk.com/index.php/
download_track?file=/3.0b3/linux/
splunk-3.0b3-20872-Linux-
i686.tgz&ac=&wget=true&name=wget'
$ sudo mkdir /opt;cd /opt
$ sudo tar xzvf ~/splunk-3.0b3-20872-Linux-
i686.tgz
$ sudo /opt/splunk/bin

36. Configuring splunk>

37. Configuring splunk>

38. Configuring splunk>

39. Configuring splunk>

40. Configuring splunk>

41. splunk>

42. Syslog Agents
• Windows Agents
• UNIX Agents
• Other Devices

43. Windows Logs?
• SNARE Agent
• Converts Event Logs to Syslog
• Free

44. UNIX Agents
• Use the syslog service!
• *.* @Syslog Server

45. Other Devices
• Various systems can be configured
• Cisco, Juniper, Lotus Domino, Apache, IIS,
etc. are just a few examples.

46. Recap
• What is Syslog
• What is FIFO
• Installing and Configuring Syslog-NG
• Installing and Configuring Splunk
• Agents

47. Questions?