Web Proxy  Log Management  Anton Chuvakin, Ph.D.,  GCIA, GCIH, GCFA Chief Logging Evangelist Mitigating Risk. Automating C...
Outline <ul><li>What is a Web proxy? </li></ul><ul><li>Overview of proxy logging </li></ul><ul><ul><li>What are the logs? ...
What is a Web proxy? <ul><li>Web proxy stores, passes, blocks, authenticates, and secures web traffic </li></ul><ul><li>Ex...
What is in proxy logs: big picture <ul><li>Users’  activities on the web </li></ul><ul><li>Applications  HTTP activity </l...
What is in proxy logs: details <ul><li>Typical proxy log contains: </li></ul><ul><ul><li>Time stamp </li></ul></ul><ul><ul...
What Are They Good For? <ul><li>Security – compliance - operations </li></ul><ul><li>Web access policy violations </li></u...
Example 1: Look for Spyware in Proxy Logs! <ul><li>How? </li></ul><ul><li>Search  for unusual “User-Agent” strings, or  </...
Proxy logs and compliance <ul><li>No  direct  mention of proxy logs … however: </li></ul><ul><li>Proxy monitoring is part ...
Example 2: Monitor Users’ Surfing <ul><li>How? </li></ul><ul><li>Search  for a specific user name  </li></ul><ul><li>View ...
Getting the logs <ul><li>Usually , file-based logs  (not syslog or event log) </li></ul><ul><li>Can be  downloaded  from t...
Storing the logs <ul><li>Any regulations that  mandate  storage of proxy logs? Not  directly . </li></ul><ul><li>Typical o...
Example 3: Correlated Log Investigation <ul><li>How? </li></ul><ul><li>Search  for an  IP address  in server, firewall and...
Analyzing the logs <ul><li>Near real-time  analysis / alerts </li></ul><ul><ul><li>Proxy  failures </li></ul></ul><ul><ul>...
Example 4: File Uploads Analytic Search <ul><li>How? </li></ul><ul><li>Search  for POST requests  AND  specific document c...
Conclusions <ul><li>Collect, store and analyze proxy logs  for security, operations and compliance </li></ul><ul><li>Use a...
Thank You for Attending! <ul><li>Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA </li></ul><ul><li>Chief Logging Evangelist </li><...
Upcoming SlideShare
Loading in …5
×

Web Proxy Log Analysis and Management 2007

37,553 views

Published on

Web proxy logging, log analysis and log management: tips, guidance, techniques.

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
37,553
On SlideShare
0
From Embeds
0
Number of Embeds
68
Actions
Shares
0
Downloads
0
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide
  • Full Blog Post: Following the new &amp;quot;tradition&amp;quot; of posting a tip of the week (mentioned here , here ; SANS jumped in as well ), I decided to follow along and join the initiative. One of the bloggers called it &amp;quot;pay it forward &amp;quot; to the community. So, Anton Logging Tip of the Day #12:  Proxy Log Fun - Proxy Logs vs Information Leakage You probably know that web proxies (such as Squid, BlueCoat SG, BlueCoat Netcache and others) produce a lot of fun logs. Indeed, they are fun since they can be used for a whole range of things, from routine monitoring for AUP compliance to malware detection as well as possibly looking for the security scourge of 2007 - web client attacks. Specifically, in this tip we will learn how proxy logs can be used for detection of file uploads and other outbound information transfers vie the web. First, think what is the legitimate use of file upload functionality for your web users. If web mail is allowed, then sending an attachment will include an upload. What else? The rest will be considered at least suspicious... In addition to file uploads, some spyware application will also use similar methods to steal data. Looking for methods and content-type in combination with either known suspicious URL  or user-agent (i.e. web client type) can often reveal spyware infections, actively collecting data. Admittedly, a well-written spyware can certainly fake the user-agent field so it is clearly not reliable, but still useful to add to our query above.  Here are some of the criteria we will use to look for uploads in Squid and BlueCoat SG proxy logs: HTTP method (logged as &amp;quot;cs-method&amp;quot; by BlueCoat) = POST   (as opposed to the usual GET, used to retrieve web content). For information uploads: content type (logged as &amp;quot;RS(content-type)&amp;quot; by BlueCoat) = anything but &amp;quot;html/text&amp;quot; (which is the type used for uploading  web form contents) - especially try content types  &amp;quot; application/octet-stream &amp;quot;, &amp;quot; application/msword &amp;quot;, &amp;quot; application/powerpoint &amp;quot;, &amp;quot; application/vnd.ms-excel &amp;quot;, &amp;quot;application/pdf&amp;quot;  and a few others to look for common file uploads For spyware and application data transfers: user-agent set to anything but the common ones (i.e. not Mozilla, iTunes, LiveUpdate, etc) or even to &amp;quot;unknown.&amp;quot; One can also try user-agent containing your favorite messaging app (e.g. &amp;quot;MSN Messenger&amp;quot;, etc) (if you feel adventurous, other interesting content-types to try are &amp;quot;application/x-javascript&amp;quot; and &amp;quot;text/javascript&amp;quot;) Here are the examples of the above, including some &amp;quot;classics&amp;quot; (while spyware specimen are a bit dated, this method of  detecting them via logs is relevant): 1124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports. hotbar .com/reports/hotbar/4.0/HbRpt.dll 1124392388.975 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc. cometsystems .com/dss/cc.2_0_0.report_u 2007-05-19 03:55:12 160 10.1.1.3 - - - OBSERVED &amp;quot;Spyware/Malware Sources;Spyware Effects;Web Advertisements&amp;quot; - 200 TCP_NC_MISS POST text/html;%20charset=utf-8 http bis. 180solutions .com 80 /versionconfig.aspx ?did=5342&amp;ver=1.0 aspx - 10.1.1.2 273 175 - - none - - 2007-05-21 03:10:40 4 10.1.1.3 Joanna- authentication_redirect_to_virtual_host PROXIED &amp;quot;Search Engines/Portals&amp;quot; - 307 TCP_AUTH_REDIRECT POST - http storage.msn.com 80 /storageservice/schematizedstore.asmx - asmx &amp;quot;Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSN Messenger 7.5.0324)&amp;quot; 10.1.1.2 791 2566 - - none - - Here are some other signs that will make the above log entry extra-suspicious is: A dead giveaway: upload happens to a &amp;quot;known bad&amp;quot; URL (e.g containing &amp;quot; gator &amp;quot; and others above)  Upload happens to an unresolved IP address (do a &amp;quot;whois&amp;quot; on it!) Uploads happens to a port not equal to 80 (i.e. the URL contains a port such as http://10.1.10.10:31337 ) Upload has confidential file name in the log entry (e.g. somebody dumb emailing a sensitive file to himself - as discussed here ) Overall, this log analysis method is good for casting a broad net to catch not just spyware-infected systems, but also unauthorized applications (e.g. method= POST and user-agent= iTunes ), instant messaging (e.g. method= POST   and then by user-agent, content or URL), simple forms of data theft and document handling policy violations (emailing files to self via web mail: method= POST and sensitive file name present in the entry; also content type set to popular file types) as well as other abuses of web access. As a result, proxy logs provide an extremely rich AND readily available source of data about threats that users face! To top it off, one promising direction of future research is using web proxy logs to detect client-side exploits by malicious web servers (more on this in the near future!) Possibly related posts: Anton Security Tip of the Day #6: The Other Web Log All my past security and logging tips Also, I am tagging all the tips on my del.icio.us feed . Here is the link: All Tips of the Day . Technorati tags: security , logging , log management , tips , chuvakin
  • Web Proxy Log Analysis and Management 2007

    1. 1. Web Proxy Log Management Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Chief Logging Evangelist Mitigating Risk. Automating Compliance.
    2. 2. Outline <ul><li>What is a Web proxy? </li></ul><ul><li>Overview of proxy logging </li></ul><ul><ul><li>What are the logs? What is in the logs? </li></ul></ul><ul><li>Why look at proxy logs? </li></ul><ul><ul><li>Use cases: from security to compliance </li></ul></ul><ul><li>Proxy log management “best practices” </li></ul><ul><li>Proxy log analysis techniques </li></ul><ul><li>Proxy logs correlation with other log sources </li></ul>
    3. 3. What is a Web proxy? <ul><li>Web proxy stores, passes, blocks, authenticates, and secures web traffic </li></ul><ul><li>Examples : Squid, BlueCoat, NetCache, ISA, etc </li></ul><ul><li>Why use them? </li></ul><ul><ul><li>Security </li></ul></ul><ul><ul><li>Efficiency and control </li></ul></ul><ul><ul><li>Compliance </li></ul></ul><ul><ul><li>User activity audit and monitoring </li></ul></ul>
    4. 4. What is in proxy logs: big picture <ul><li>Users’ activities on the web </li></ul><ul><li>Applications HTTP activity </li></ul><ul><li>Web-enabled malware traffic </li></ul><ul><li>Proxy performance metrics </li></ul>
    5. 5. What is in proxy logs: details <ul><li>Typical proxy log contains: </li></ul><ul><ul><li>Time stamp </li></ul></ul><ul><ul><li>Source IP and possibly user name </li></ul></ul><ul><ul><li>Browser type (“User-agent”) and OS (indirectly) </li></ul></ul><ul><ul><li>Destination URL and sometimes its category </li></ul></ul><ul><ul><li>HTTP method and response code </li></ul></ul><ul><ul><li>Proxy actions (blocked, proxied, passed, etc) </li></ul></ul><ul><li>Example : 2006-05-08 16:15:01 2 192.168.1.3 Mary - authentication_redirect_from_virtual_host DENIED &quot;Search Engines/Portals&quot; - 307 TCP_AUTH_REDIRECT GET - http www.comcast.net 80 /home.html - html &quot;Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)&quot; 192.168.1.2 970 425 - - none - - </li></ul>
    6. 6. What Are They Good For? <ul><li>Security – compliance - operations </li></ul><ul><li>Web access policy violations </li></ul><ul><li>User activity monitoring </li></ul><ul><li>Internal spyware and malware tracking </li></ul><ul><li>Web client attack detection </li></ul><ul><li>Server attacks by hackers from inside </li></ul><ul><li>IP theft and information leakage detection </li></ul><ul><li>Proxy performance measurement </li></ul>
    7. 7. Example 1: Look for Spyware in Proxy Logs! <ul><li>How? </li></ul><ul><li>Search for unusual “User-Agent” strings, or </li></ul><ul><li>View a report summarized by “User-Agent” </li></ul><ul><li>What ? </li></ul><ul><li>Look for log records containing unusual ( not Mozilla, etc) or “known bad” (e.g. Gator , FunWeb , etc) names </li></ul><ul><li>Investigate the machines with source addresses that produce such traffic </li></ul>
    8. 8. Proxy logs and compliance <ul><li>No direct mention of proxy logs … however: </li></ul><ul><li>Proxy monitoring is part of the overall control and governance (thus SOX, HIPAA, GLBA, etc) </li></ul><ul><li>Legal requirements to have audit trails (thus HIPAA, PCI) </li></ul><ul><li>Breach disclosure laws also impact (SB1386 and others) </li></ul><ul><li>WARNING : privacy laws might mandate the opposite ! </li></ul><ul><li>Also : legal liability is a major “compliance-like” driver for proxy log collection, storage and analysis. </li></ul>
    9. 9. Example 2: Monitor Users’ Surfing <ul><li>How? </li></ul><ul><li>Search for a specific user name </li></ul><ul><li>View a report summarized by a user, and also filtered by group or business unit </li></ul><ul><li>What ? </li></ul><ul><li>Look for sites and site categories visited </li></ul><ul><li>Discipline the users or take other HR action incase of policy violations </li></ul>
    10. 10. Getting the logs <ul><li>Usually , file-based logs (not syslog or event log) </li></ul><ul><li>Can be downloaded from the proxy (FTP, SCP, HTTP, etc) or uploaded by the proxy to a log management tool (HTTP) </li></ul><ul><li>Proxy log collection issues: </li></ul><ul><li>Volume – proxies are chatty! </li></ul><ul><li>Time sync – needed to trust log timestamps </li></ul><ul><li>Real time ? Probably not needed </li></ul>
    11. 11. Storing the logs <ul><li>Any regulations that mandate storage of proxy logs? Not directly . </li></ul><ul><li>Typical operational storage requirements: </li></ul><ul><ul><li>90 days online (with quick access and fast reports) </li></ul></ul><ul><ul><li>1-3 years long-term storage </li></ul></ul><ul><li>Other considerations </li></ul><ul><ul><li>Incident response, especially for IP theft cases, does point towards longer retention times </li></ul></ul><ul><ul><li>Privacy laws (outside the US) point towards shorter retention times </li></ul></ul><ul><li>Use the same system for all logs, not just proxy logs! </li></ul>
    12. 12. Example 3: Correlated Log Investigation <ul><li>How? </li></ul><ul><li>Search for an IP address in server, firewall and proxy logs </li></ul><ul><li>What ? </li></ul><ul><li>Look for a complete activity trail of a user as he performs various tasks, connects to various servers, runs tasks on his machine, accesses the web </li></ul><ul><li>Recover the sequence of events based on correlated logs (assumed time sync !) </li></ul>
    13. 13. Analyzing the logs <ul><li>Near real-time analysis / alerts </li></ul><ul><ul><li>Proxy failures </li></ul></ul><ul><ul><li>Major policy violations (by category) </li></ul></ul><ul><ul><li>High-risk spyware infections </li></ul></ul><ul><li>Stored log analysis / reports and searches </li></ul><ul><ul><li>Reports and summaries: covered in Part II </li></ul></ul><ul><ul><li>Investigative log searches </li></ul></ul><ul><ul><ul><li>User name, URL/site, IP address </li></ul></ul></ul><ul><ul><li>Analytic searching </li></ul></ul><ul><ul><ul><li>Rare response codes, HTTP methods, user-agents, etc </li></ul></ul></ul>
    14. 14. Example 4: File Uploads Analytic Search <ul><li>How? </li></ul><ul><li>Search for POST requests AND specific document content-types (e.g. msword, powerpoint, etc) </li></ul><ul><li>What ? </li></ul><ul><li>Look for a uploads to unusual sites (especially with unresolved IPs), web mail, or for sensitive document names </li></ul><ul><li>Especially, look for uploads to unusual ports </li></ul><ul><li>More details on LogBlog (Tip #12) or </li></ul><ul><li>Security Warrior Blog (Tip #12) </li></ul>
    15. 15. Conclusions <ul><li>Collect, store and analyze proxy logs for security, operations and compliance </li></ul><ul><li>Use a single log management tool to collect proxy logs in combination with other logs (system, security, network, application, etc) </li></ul><ul><li>Proxy logs are a source of critical user behavior information </li></ul>
    16. 16. Thank You for Attending! <ul><li>Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA </li></ul><ul><li>Chief Logging Evangelist </li></ul><ul><li>LogLogic, Inc </li></ul><ul><li>[email_address] </li></ul><ul><li>http://www.chuvakin.org </li></ul><ul><li>See www.info-secure.org for my papers, books, reviews </li></ul><ul><li>and other security and logging resources </li></ul>

    ×