Web Proxy Log Analysis and Management 2007

Slide 1: Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Chief Logging Evangelist Web Proxy Log Management Mitigating Risk. Automating Compliance. 1 LogLogic Confidential Tuesday, September 18, 2007

Slide 2: Outline What is a Web proxy?  Overview of proxy logging  What are the logs? What is in the logs? – Why look at proxy logs?  Use cases: from security to compliance – Proxy log management “best practices”  Proxy log analysis techniques  Proxy logs correlation with other log sources  Mitigating Risk. Automating Compliance. 2 Tuesday, September 18, 2007 Confidential |

Slide 3: What is a Web proxy? Web proxy stores, passes, blocks, authenticates,  and secures web traffic Examples: Squid, BlueCoat, NetCache, ISA, etc  Why use them?  1. Security 2. Efficiency and control 3. Compliance 4. User activity audit and monitoring Mitigating Risk. Automating Compliance. 3 Tuesday, September 18, 2007 Confidential |

Slide 4: What is in proxy logs: big picture  Users’ activities on the web Applications HTTP activity  malware traffic  Web-enabled  Proxy performance metrics Mitigating Risk. Automating Compliance. 4 Tuesday, September 18, 2007 Confidential |

Slide 5: What is in proxy logs: details Typical proxy log contains:  Time stamp – Source IP and possibly user name – Browser type (“User-agent”) and OS (indirectly) – Destination URL and sometimes its category – HTTP method and response code – Proxy actions (blocked, proxied, passed, etc) – Example: 2006-05-08 16:15:01 2 192.168.1.3 Mary -  authentication_redirect_from_virtual_host DENIED \"Search Engines/Portals\" - 307 TCP_AUTH_REDIRECT GET - http www.comcast.net 80 /home.html - html \"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)\" 192.168.1.2 970 425 - - none - - Mitigating Risk. Automating Compliance. 5 Tuesday, September 18, 2007 Confidential |

Slide 6: What Are They Good For? Security – compliance - operations Web access policy violations  User activity monitoring  Internal spyware and malware tracking  Web client attack detection  Server attacks by hackers from inside  IP theft and information leakage detection  Proxy performance measurement  Mitigating Risk. Automating Compliance. 6 Tuesday, September 18, 2007 Confidential |

Slide 7: Example 1: Look for Spyware in Proxy Logs! How? Search for unusual “User-Agent” strings, or  View a report summarized by “User-Agent”  What? Look for log records containing unusual (not Mozilla,  etc) or “known bad” (e.g. Gator, FunWeb, etc) names Investigate the machines with source addresses that  produce such traffic Mitigating Risk. Automating Compliance. 7 Tuesday, September 18, 2007 Confidential |

Slide 8: Proxy logs and compliance No direct mention of proxy logs … however: Proxy monitoring is part of the overall control and governance  (thus SOX, HIPAA, GLBA, etc) Legal requirements to have audit trails (thus HIPAA, PCI)  Breach disclosure laws also impact (SB1386 and others)  WARNING: privacy laws might mandate the opposite!  Also: legal liability is a major “compliance-like” driver for proxy  log collection, storage and analysis. Mitigating Risk. Automating Compliance. 8 Tuesday, September 18, 2007 Confidential |

Slide 9: Example 2: Monitor Users’ Surfing How? Search for a specific user name  View a report summarized by a user, and also filtered  by group or business unit What? Look for sites and site categories visited  Discipline the users or take other HR action incase of  policy violations Mitigating Risk. Automating Compliance. 9 Tuesday, September 18, 2007 Confidential |

Slide 10: Getting the logs Usually, file-based logs (not syslog or event log)  Can be downloaded from the proxy (FTP, SCP, HTTP,  etc) or uploaded by the proxy to a log management tool (HTTP) Proxy log collection issues:  Volume – proxies are chatty!  Time sync – needed to trust log timestamps  Real time? Probably not needed Mitigating Risk. Automating Compliance. 10 Tuesday, September 18, 2007 Confidential |

Slide 11: Storing the logs Any regulations that mandate storage of proxy logs?  Not directly. Typical operational storage requirements:  90 days online (with quick access and fast reports) – 1-3 years long-term storage – Other considerations  Incident response, especially for IP theft cases, does point – towards longer retention times Privacy laws (outside the US) point towards shorter retention – times Use the same system for all logs, not just proxy logs!  Mitigating Risk. Automating Compliance. 11 Tuesday, September 18, 2007 Confidential |

Slide 12: Example 3: Correlated Log Investigation How? Search for an IP address in server, firewall and proxy  logs What? Look for a complete activity trail of a user as he  performs various tasks, connects to various servers, runs tasks on his machine, accesses the web Recover the sequence of events based on correlated  logs (assumed time sync!) Mitigating Risk. Automating Compliance. 12 Tuesday, September 18, 2007 Confidential |

Slide 13: Analyzing the logs Near real-time analysis / alerts  Proxy failures – Major policy violations (by category) – High-risk spyware infections – Stored log analysis / reports and searches  Reports and summaries: covered in Part II – Investigative log searches – User name, URL/site, IP address – Analytic searching – Rare response codes, HTTP methods, user-agents, etc – Mitigating Risk. Automating Compliance. 13 Tuesday, September 18, 2007 Confidential |

Slide 14: Example 4: File Uploads Analytic Search How?  Search for POST requests AND specific document content-types (e.g. msword, powerpoint, etc) What?  Look for a uploads to unusual sites (especially with unresolved IPs), web mail, or for sensitive document names  Especially, look for uploads to unusual ports More details on LogBlog (Tip #12) or Security Warrior Blog (Tip #12) Mitigating Risk. Automating Compliance. 14 Tuesday, September 18, 2007 Confidential |

Slide 15: Conclusions Collect, store and analyze proxy logs for security,  operations and compliance Use a single log management tool to collect proxy  logs in combination with other logs (system, security, network, application, etc) Proxy logs are a source of critical user behavior  information Mitigating Risk. Automating Compliance. 15 Tuesday, September 18, 2007 Confidential |

Slide 16: Thank You for Attending! Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Chief Logging Evangelist LogLogic, Inc anton@chuvakin.org http://www.chuvakin.org See www.info-secure.org for my papers, books, reviews and other security and logging resources Mitigating Risk. Automating Compliance. 16 Tuesday, September 18, 2007 Confidential |