Web Proxy Log Analysis and Management 2007
•
8 likes•36,124 views
Web proxy logging, log analysis and log management: tips, guidance, techniques.
![Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (13)
Similar to Web Proxy Log Analysis and Management 2007
Similar to Web Proxy Log Analysis and Management 2007 (20)
More from Anton Chuvakin
More from Anton Chuvakin (20)
Recently uploaded
Recently uploaded (20)
Web Proxy Log Analysis and Management 2007
- 1. Web Proxy Log Management Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Chief Logging Evangelist Mitigating Risk. Automating Compliance.
Editor's Notes
- Full Blog Post: Following the new "tradition" of posting a tip of the week (mentioned here , here ; SANS jumped in as well ), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward " to the community. So, Anton Logging Tip of the Day #12: Proxy Log Fun - Proxy Logs vs Information Leakage You probably know that web proxies (such as Squid, BlueCoat SG, BlueCoat Netcache and others) produce a lot of fun logs. Indeed, they are fun since they can be used for a whole range of things, from routine monitoring for AUP compliance to malware detection as well as possibly looking for the security scourge of 2007 - web client attacks. Specifically, in this tip we will learn how proxy logs can be used for detection of file uploads and other outbound information transfers vie the web. First, think what is the legitimate use of file upload functionality for your web users. If web mail is allowed, then sending an attachment will include an upload. What else? The rest will be considered at least suspicious... In addition to file uploads, some spyware application will also use similar methods to steal data. Looking for methods and content-type in combination with either known suspicious URL or user-agent (i.e. web client type) can often reveal spyware infections, actively collecting data. Admittedly, a well-written spyware can certainly fake the user-agent field so it is clearly not reliable, but still useful to add to our query above. Here are some of the criteria we will use to look for uploads in Squid and BlueCoat SG proxy logs: HTTP method (logged as "cs-method" by BlueCoat) = POST (as opposed to the usual GET, used to retrieve web content). For information uploads: content type (logged as "RS(content-type)" by BlueCoat) = anything but "html/text" (which is the type used for uploading web form contents) - especially try content types " application/octet-stream ", " application/msword ", " application/powerpoint ", " application/vnd.ms-excel ", "application/pdf" and a few others to look for common file uploads For spyware and application data transfers: user-agent set to anything but the common ones (i.e. not Mozilla, iTunes, LiveUpdate, etc) or even to "unknown." One can also try user-agent containing your favorite messaging app (e.g. "MSN Messenger", etc) (if you feel adventurous, other interesting content-types to try are "application/x-javascript" and "text/javascript") Here are the examples of the above, including some "classics" (while spyware specimen are a bit dated, this method of detecting them via logs is relevant): 1124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports. hotbar .com/reports/hotbar/4.0/HbRpt.dll 1124392388.975 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc. cometsystems .com/dss/cc.2_0_0.report_u 2007-05-19 03:55:12 160 10.1.1.3 - - - OBSERVED "Spyware/Malware Sources;Spyware Effects;Web Advertisements" - 200 TCP_NC_MISS POST text/html;%20charset=utf-8 http bis. 180solutions .com 80 /versionconfig.aspx ?did=5342&ver=1.0 aspx - 10.1.1.2 273 175 - - none - - 2007-05-21 03:10:40 4 10.1.1.3 Joanna- authentication_redirect_to_virtual_host PROXIED "Search Engines/Portals" - 307 TCP_AUTH_REDIRECT POST - http storage.msn.com 80 /storageservice/schematizedstore.asmx - asmx "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSN Messenger 7.5.0324)" 10.1.1.2 791 2566 - - none - - Here are some other signs that will make the above log entry extra-suspicious is: A dead giveaway: upload happens to a "known bad" URL (e.g containing " gator " and others above) Upload happens to an unresolved IP address (do a "whois" on it!) Uploads happens to a port not equal to 80 (i.e. the URL contains a port such as http://10.1.10.10:31337 ) Upload has confidential file name in the log entry (e.g. somebody dumb emailing a sensitive file to himself - as discussed here ) Overall, this log analysis method is good for casting a broad net to catch not just spyware-infected systems, but also unauthorized applications (e.g. method= POST and user-agent= iTunes ), instant messaging (e.g. method= POST and then by user-agent, content or URL), simple forms of data theft and document handling policy violations (emailing files to self via web mail: method= POST and sensitive file name present in the entry; also content type set to popular file types) as well as other abuses of web access. As a result, proxy logs provide an extremely rich AND readily available source of data about threats that users face! To top it off, one promising direction of future research is using web proxy logs to detect client-side exploits by malicious web servers (more on this in the near future!) Possibly related posts: Anton Security Tip of the Day #6: The Other Web Log All my past security and logging tips Also, I am tagging all the tips on my del.icio.us feed . Here is the link: All Tips of the Day . Technorati tags: security , logging , log management , tips , chuvakin