In this presentation we discuss gathering data with syslog-ng in order to properly feed your SIEM system such as ArcSight ESM. This presentation is from HP/ArcSight Protect 2011.

1. CSN08: You Can't Correlate
What You Don't Have
Scott Carlson & Rick Yetter
Apollo Group Inc.

2. Apollo Group
The Apollo Group Challenge
Apollo Group is a publicly traded parent company that
owns the University of Phoenix and a number of other
subsidiaries in the education arena. With 300 physical
locations in six countries, 500,000 students, 50,000
faculty and 22,000 employees, Apollo Group has a
formidable challenge in securing all its systems, data
and endpoints.
Reference:
http://www.arcsight.com/collateral/case_studies/ArcSight_CaseStudy_Apollo.pdf
2

3. 3

4. What’s this about?
• ArcSight products are awesome, as long as you send them
information. The products can’t do anything unless you send
them as much data as possible!
• Normal steps to implementation
1.
2.
3.
4.
5.
Define Use Cases
Send Logs
Build Correlation Rules in ArcSight ESM
…
Profit!!!
4

5. Our Environment
• 4,500 Servers
– Oracle Enterprise Linux, Red Hat Linux
– Windows 2000, 2003, 2008
– Solaris 9,10
•
•
•
•
•
•
•
60% Virtualized on VMWARE
Multiple international locations & data centers
Firewalls (Cisco, Juniper, Checkpoint)
Proxy (BlueCoat)
IDS (SourceFire)
AV/HIPS/DLP (McAfee)
….
5

6. It’s the Logs that matter
“The powerful correlation engine of ArcSight ESM sifts through MILLIONS OF LOG
RECORDS to find the critical incidents that matter” (www.arcsight.com)
Server Stuff
 Security Events
 Change Monitoring
 Failure Events
 Application Logs
 Web Logs
 Host Firewalls
 Active Directory Activity
Network Stuff
 Firewalls
 Proxies
 Intrusion Detection
 Antivirus
 Data Loss Prevention
 Email Traffic & Alerts
 Wireless
 Network Change Monitoring
6

7. Where to store the logs
• Long term storage is critical for large companies
– Determine retention requirements (30 days, 1 year, infinite)
• Determine who may need the logs, do you need them online?
– SysAdmin, Forensics, InfoSec, Legal
• Do you need non-repudiation?
• Determine Storage method
– Splunk
– ArcSight Logger
- Filesystem(s) full of Raw Log files
- Alternate Logging Product
• If you build your own
– SAN versus NAS versus Local JBOD. You need to log even if things
break!
7

8. Syslog Relay
• Red Hat Linux
• Syslog-ng v4 running on multiple ports
– For receiving logs from multiple sources with unique filters
• Local JBOD w/12TB configured as RAID-5
– Make sure you can log even if your SAN is borked!
• Additional security of SAMHAIN, tripwire, Solid Core to protect
your files from modification
• 64GB of ram
• Lots of processors
2

9. How to Get the Logs
•
•
•
•
•
Built In Syslog
Syslog-NG
SNARE or Epilog agent, kiwi
File-Reader ArcSight Connector
Something entirely custom, just put it in a FILE!
– Syslog format or CEF Format, you pick.
You found something without logs???? Well, Ask the developer or
company to add logging!!!!
9

10. Single Endpoint
• For smaller environments, or environments with fewer layer-2
boundaries
• Should configure server with redundancy in mind in case of
failure
• Can use file reader connector to read from local logs
• Single destination, easy to script
• May not scale
• Limited to small number of
networks unless you traverse
firewalls
2

11. Single Endpoint with DR Site
• Makes a copy of all logs to an alternate site
• Saves you in case of catastrophic failure
• Adds bandwidth to the WAN or remote site link
Data Center
Alternate Facility
Server(s)
Relay
2

12. Local Collect & Forward
• Individual Syslog collection in each major network block or
international location
12

13. How to send the Logs
• Configure syslog
– *.debug @loghost.mydomain.com
– *.*
@loghost
(Solaris)
(Linux)
• Configure SNARE
– Destination Snare Server address
– Destination Port
– Enable SYSLOG Header
= loghost
= 514
= Selected
• Read the Fine Manual of your product to enable logging with a
remote destination. If that’s not there, write to a file!
13

14. Decision points
• What’s your Double-Send point?
– Host
• Not available in all “free logging tools”
• Some things cannot double send (network gear, appliances)
– Relay
• Adds cross-data center traffic times # Relay
– Central
• Easy to control flow, exposure is at this point in each DC
• Blind to logs if central server is gone
• What about things that don’t have syslog?
– File Reader to multiple ArcSight ESM Targets is a possibility
14

15. Redundancy and Double-Sending
• Fail-over scenarios in use for Apollo Group using Syslog-ng
– Redundancy at the Syslog Relay level
• Logs are sent from Snare agents on Windows or by Syslog to relays
– Each Syslog relay has a VM hot standby in case of a hardware
failure.
• Each Syslog relay is configured to send all information received to
multiple central servers for redundancy and fault tolerance.
• Each Syslog relay retains all logs received for a period of 30 days
before being rotated out.
15

16. Syslog-NG Configuration
• Syslog-ng configuration (4 simple steps)
– Simple Configuration
• Source
– Where are the logs coming from? UDP, TCP, File
• Destination
– Where are you going to send the logs? Disk, output to TCP or
UDP?
» Can you handle the TCP Overhead?
• Filters
– Keep what you want, discard the rest!
• Log
– Log the source, process it, send it to the destination.
• Encrypted communications must use TCP
16

17. Syslog-NG Configuration Sample
•
•
•
•
•
•
•
source s_local {
internal();
unix-stream("/dev/log");
file("/proc/kmsg" program_override("kernel:"));
udp(ip(0.0.0.0) port(514) flags(store-legacy-msghdr));
udp(ip(10.11.12.13) port(514) flags(store-legacy-msghdr));
};
•
•
•
•
•
•
•
•
•
•
•
•
•
•
#
# Local filters
#
filter f_boot { facility(local1); };
filter f_messages { level(info..emerg); };
filter f_secure { facility(authpriv); };
filter f_mail { facility(mail); };
filter f_cron { facility(cron); };
filter f_emerg { level(emerg); };
filter f_spooler { level(crit..emerg) and facility(uucp, news); };
filter f_local7 { facility(local7); };
# Snare *NIX Filters
#
#filter f_filter_nix { match(":") and not match("snmp") and not match("printd") and not match("-6-302013") and
not match("-6-302015") and not match("kernel") and not match("lpstat") and not match("Application") and not
match("System") and not host("10.29.10.100") and not match("dhcpd") and not match("xinetd") and not
match("puppetmasterd") and not match("crond") and not match("multipathd") and not match("modprobe"); };
17

18. Syslog-ng destinations (local)
•
•
•
•
•
•
•
•
•
•
•
#
# Local destinations
#
destination d_messages { file("/u01/log/messages"); };
destination d_secure { file("/u01/log/secure"); };
destination d_maillog { file("/u01/log/maillog"); };
destination d_cron { file("/u01/log/cron"); };
destination d_console { usertty("root"); };
destination d_spooler { file("/u01/log/spooler"); };
destination d_bootlog { file("/u01/log/boot.log"); };
#
18

19. Syslog-ng destinations (remote)
•
•
•
•
•
•
•
•
•
•
•
•
•
•
# Remote Destinations
#
destination d_forward { udp("10.3.4.5" port(514) keep_alive(no)); };
#
# Local logs
#
log { source(s_local); filter(f_emerg); destination(d_console); };
log { source(s_local); filter(f_secure); destination(d_secure); };
log { source(s_local); filter(f_mail); destination(d_maillog); flags(final); };
log { source(s_local); filter(f_cron); destination(d_cron); flags(final); };
log { source(s_local); filter(f_spooler); destination(d_spooler); };
log { source(s_local); filter(f_boot); destination(d_bootlog); };
log { source(s_local); destination(d_messages); };
log { source(s_local); destination(d_forward); };
19

20. Crazy Use Case #1
Get the unique customer # out of a sub-url string within the debug
log of a firewall in order to perform tracking/troubleshooting
• Debug Logging ON on firewall (LOTS OF TRAFFIC!)
• Logs send to Syslog
• Syslog filter or external program called to trim out the customer
number and write it to a separate file
20

21. Crazy Use Case #2
Forward non-security events directly to your NOC Console, email
queue, or whatever
• Syslog filter or external program called to grab the events you’re
interested in, and send them to external mailer (mail –s “alert”)
or a syslog-ng filter
• Don’t forget the System Administrators
• IMHO 90% of problems are misconfigured systems
21

22. Crazy Use Case #3
Gather logs from a proxy server, at 5 minute intervals, and make
sure that they’re going to your DR Site with minimal delay, add a
filter to find naughty surfing.
•
•
•
•
Proxy server sends logs via SCP to syslog relay
Syslog relay writes file to local JBOD
Syslog-ng or local script scrapes naughtiness from file
Cron job runs at 5 minute intervals to SCP completed files to DR
Watch out for incomplete files!
Make sure your formatting is good!!!
22

23. Multiple ArcSight ESM Instances
• Double sending all logs allows you have two independent
ArcSight ESM instances, in multiple data centers capable of
performing your SOC duties at a moments notice.
23

24. Q&A
24