Web Proxy Log Analysis and Management 2007

Web Proxy Log Management Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Chief Logging Evangelist Mitigating Risk. Automating Compliance.

Outline
What is a Web proxy?
Overview of proxy logging
What are the logs? What is in the logs?
Why look at proxy logs?
Use cases: from security to compliance
Proxy log management “best practices”
Proxy log analysis techniques
Proxy logs correlation with other log sources

What is a Web proxy?
Web proxy stores, passes, blocks, authenticates, and secures web traffic
Examples : Squid, BlueCoat, NetCache, ISA, etc
Why use them?
Security
Efficiency and control
Compliance
User activity audit and monitoring

What is in proxy logs: big picture
Users’ activities on the web
Applications HTTP activity
Web-enabled malware traffic
Proxy performance metrics

What is in proxy logs: details
Typical proxy log contains:
Time stamp
Source IP and possibly user name
Browser type (“User-agent”) and OS (indirectly)
Destination URL and sometimes its category
HTTP method and response code
Proxy actions (blocked, proxied, passed, etc)
Example : 2006-05-08 16:15:01 2 192.168.1.3 Mary - authentication_redirect_from_virtual_host DENIED "Search Engines/Portals" - 307 TCP_AUTH_REDIRECT GET - http www.comcast.net 80 /home.html - html "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)" 192.168.1.2 970 425 - - none - -

What Are They Good For?
Security – compliance - operations
Web access policy violations
User activity monitoring
Internal spyware and malware tracking
Web client attack detection
Server attacks by hackers from inside
IP theft and information leakage detection
Proxy performance measurement

Example 1: Look for Spyware in Proxy Logs!
How?
Search for unusual “User-Agent” strings, or
View a report summarized by “User-Agent”
What ?
Look for log records containing unusual ( not Mozilla, etc) or “known bad” (e.g. Gator , FunWeb , etc) names
Investigate the machines with source addresses that produce such traffic

Proxy logs and compliance
No direct mention of proxy logs … however:
Proxy monitoring is part of the overall control and governance (thus SOX, HIPAA, GLBA, etc)
Legal requirements to have audit trails (thus HIPAA, PCI)
Breach disclosure laws also impact (SB1386 and others)
WARNING : privacy laws might mandate the opposite !
Also : legal liability is a major “compliance-like” driver for proxy log collection, storage and analysis.

Example 2: Monitor Users’ Surfing
How?
Search for a specific user name
View a report summarized by a user, and also filtered by group or business unit
What ?
Look for sites and site categories visited
Discipline the users or take other HR action incase of policy violations

Getting the logs
Usually , file-based logs (not syslog or event log)
Can be downloaded from the proxy (FTP, SCP, HTTP, etc) or uploaded by the proxy to a log management tool (HTTP)
Proxy log collection issues:
Volume – proxies are chatty!
Time sync – needed to trust log timestamps
Real time ? Probably not needed

Storing the logs
Any regulations that mandate storage of proxy logs? Not directly .
Typical operational storage requirements:
90 days online (with quick access and fast reports)
1-3 years long-term storage
Other considerations
Incident response, especially for IP theft cases, does point towards longer retention times
Privacy laws (outside the US) point towards shorter retention times
Use the same system for all logs, not just proxy logs!

Example 3: Correlated Log Investigation
How?
Search for an IP address in server, firewall and proxy logs
What ?
Look for a complete activity trail of a user as he performs various tasks, connects to various servers, runs tasks on his machine, accesses the web
Recover the sequence of events based on correlated logs (assumed time sync !)

Analyzing the logs
Near real-time analysis / alerts
Proxy failures
Major policy violations (by category)
High-risk spyware infections
Stored log analysis / reports and searches
Reports and summaries: covered in Part II
Investigative log searches
User name, URL/site, IP address
Analytic searching
Rare response codes, HTTP methods, user-agents, etc

Example 4: File Uploads Analytic Search
How?
Search for POST requests AND specific document content-types (e.g. msword, powerpoint, etc)
What ?
Look for a uploads to unusual sites (especially with unresolved IPs), web mail, or for sensitive document names
Especially, look for uploads to unusual ports
More details on LogBlog (Tip #12) or
Security Warrior Blog (Tip #12)

Conclusions
Collect, store and analyze proxy logs for security, operations and compliance
Use a single log management tool to collect proxy logs in combination with other logs (system, security, network, application, etc)
Proxy logs are a source of critical user behavior information

Thank You for Attending!
Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA
Chief Logging Evangelist
LogLogic, Inc
[email_address]
http://www.chuvakin.org
See www.info-secure.org for my papers, books, reviews
and other security and logging resources

http://amnuay.blogspot.com	19
http://www.slideshare.net	8
http://translate.googleusercontent.com	1

Web Proxy Log Analysis and Management 2007

by Anton Chuvakin on Sep 18, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 28

Statistics

Web Proxy Log Analysis and Management 2007 — Presentation Transcript