Search

Web Proxy Log Analysis and Management 2007

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    Full Blog Post: Following the new "tradition" of posting a tip of the week (mentioned here , here ; SANS jumped in as well ), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward " to the community. So, Anton Logging Tip of the Day #12:  Proxy Log Fun - Proxy Logs vs Information Leakage You probably know that web proxies (such as Squid, BlueCoat SG, BlueCoat Netcache and others) produce a lot of fun logs. Indeed, they are fun since they can be used for a whole range of things, from routine monitoring for AUP compliance to malware detection as well as possibly looking for the security scourge of 2007 - web client attacks. Specifically, in this tip we will learn how proxy logs can be used for detection of file uploads and other outbound information transfers vie the web. First, think what is the legitimate use of file upload functionality for your web users. If web mail is allowed, then sending an attachment will include an upload. What else? The rest will be considered at least suspicious... In addition to file uploads, some spyware application will also use similar methods to steal data. Looking for methods and content-type in combination with either known suspicious URL  or user-agent (i.e. web client type) can often reveal spyware infections, actively collecting data. Admittedly, a well-written spyware can certainly fake the user-agent field so it is clearly not reliable, but still useful to add to our query above.  Here are some of the criteria we will use to look for uploads in Squid and BlueCoat SG proxy logs: HTTP method (logged as "cs-method" by BlueCoat) = POST   (as opposed to the usual GET, used to retrieve web content). For information uploads: content type (logged as "RS(content-type)" by BlueCoat) = anything but "html/text" (which is the type used for uploading  web form contents) - especially try content types  " application/octet-stream ", " application/msword ", " application/powerpoint ", " application/vnd.ms-excel ", "application/pdf"  and a few others to look for common file uploads For spyware and application data transfers: user-agent set to anything but the common ones (i.e. not Mozilla, iTunes, LiveUpdate, etc) or even to "unknown." One can also try user-agent containing your favorite messaging app (e.g. "MSN Messenger", etc) (if you feel adventurous, other interesting content-types to try are "application/x-javascript" and "text/javascript") Here are the examples of the above, including some "classics" (while spyware specimen are a bit dated, this method of  detecting them via logs is relevant): 1124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports. hotbar .com/reports/hotbar/4.0/HbRpt.dll 1124392388.975 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc. cometsystems .com/dss/cc.2_0_0.report_u 2007-05-19 03:55:12 160 10.1.1.3 - - - OBSERVED "Spyware/Malware Sources;Spyware Effects;Web Advertisements" - 200 TCP_NC_MISS POST text/html;%20charset=utf-8 http bis. 180solutions .com 80 /versionconfig.aspx ?did=5342&ver=1.0 aspx - 10.1.1.2 273 175 - - none - - 2007-05-21 03:10:40 4 10.1.1.3 Joanna- authentication_redirect_to_virtual_host PROXIED "Search Engines/Portals" - 307 TCP_AUTH_REDIRECT POST - http storage.msn.com 80 /storageservice/schematizedstore.asmx - asmx "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSN Messenger 7.5.0324)" 10.1.1.2 791 2566 - - none - - Here are some other signs that will make the above log entry extra-suspicious is: A dead giveaway: upload happens to a "known bad" URL (e.g containing " gator " and others above)  Upload happens to an unresolved IP address (do a "whois" on it!) Uploads happens to a port not equal to 80 (i.e. the URL contains a port such as http://10.1.10.10:31337 ) Upload has confidential file name in the log entry (e.g. somebody dumb emailing a sensitive file to himself - as discussed here ) Overall, this log analysis method is good for casting a broad net to catch not just spyware-infected systems, but also unauthorized applications (e.g. method= POST and user-agent= iTunes ), instant messaging (e.g. method= POST   and then by user-agent, content or URL), simple forms of data theft and document handling policy violations (emailing files to self via web mail: method= POST and sensitive file name present in the entry; also content type set to popular file types) as well as other abuses of web access. As a result, proxy logs provide an extremely rich AND readily available source of data about threats that users face! To top it off, one promising direction of future research is using web proxy logs to detect client-side exploits by malicious web servers (more on this in the near future!) Possibly related posts: Anton Security Tip of the Day #6: The Other Web Log All my past security and logging tips Also, I am tagging all the tips on my del.icio.us feed . Here is the link: All Tips of the Day . Technorati tags: security , logging , log management , tips , chuvakin

    2 Favorites

    Web Proxy Log Analysis and Management 2007 - Presentation Transcript

    1. Web Proxy Log Management Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Chief Logging Evangelist Mitigating Risk. Automating Compliance.
    2. Outline
      • What is a Web proxy?
      • Overview of proxy logging
        • What are the logs? What is in the logs?
      • Why look at proxy logs?
        • Use cases: from security to compliance
      • Proxy log management “best practices”
      • Proxy log analysis techniques
      • Proxy logs correlation with other log sources
    3. What is a Web proxy?
      • Web proxy stores, passes, blocks, authenticates, and secures web traffic
      • Examples : Squid, BlueCoat, NetCache, ISA, etc
      • Why use them?
        • Security
        • Efficiency and control
        • Compliance
        • User activity audit and monitoring
    4. What is in proxy logs: big picture
      • Users’ activities on the web
      • Applications HTTP activity
      • Web-enabled malware traffic
      • Proxy performance metrics
    5. What is in proxy logs: details
      • Typical proxy log contains:
        • Time stamp
        • Source IP and possibly user name
        • Browser type (“User-agent”) and OS (indirectly)
        • Destination URL and sometimes its category
        • HTTP method and response code
        • Proxy actions (blocked, proxied, passed, etc)
      • Example : 2006-05-08 16:15:01 2 192.168.1.3 Mary - authentication_redirect_from_virtual_host DENIED "Search Engines/Portals" - 307 TCP_AUTH_REDIRECT GET - http www.comcast.net 80 /home.html - html "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)" 192.168.1.2 970 425 - - none - -
    6. What Are They Good For?
      • Security – compliance - operations
      • Web access policy violations
      • User activity monitoring
      • Internal spyware and malware tracking
      • Web client attack detection
      • Server attacks by hackers from inside
      • IP theft and information leakage detection
      • Proxy performance measurement
    7. Example 1: Look for Spyware in Proxy Logs!
      • How?
      • Search for unusual “User-Agent” strings, or
      • View a report summarized by “User-Agent”
      • What ?
      • Look for log records containing unusual ( not Mozilla, etc) or “known bad” (e.g. Gator , FunWeb , etc) names
      • Investigate the machines with source addresses that produce such traffic
    8. Proxy logs and compliance
      • No direct mention of proxy logs … however:
      • Proxy monitoring is part of the overall control and governance (thus SOX, HIPAA, GLBA, etc)
      • Legal requirements to have audit trails (thus HIPAA, PCI)
      • Breach disclosure laws also impact (SB1386 and others)
      • WARNING : privacy laws might mandate the opposite !
      • Also : legal liability is a major “compliance-like” driver for proxy log collection, storage and analysis.
    9. Example 2: Monitor Users’ Surfing
      • How?
      • Search for a specific user name
      • View a report summarized by a user, and also filtered by group or business unit
      • What ?
      • Look for sites and site categories visited
      • Discipline the users or take other HR action incase of policy violations
    10. Getting the logs
      • Usually , file-based logs (not syslog or event log)
      • Can be downloaded from the proxy (FTP, SCP, HTTP, etc) or uploaded by the proxy to a log management tool (HTTP)
      • Proxy log collection issues:
      • Volume – proxies are chatty!
      • Time sync – needed to trust log timestamps
      • Real time ? Probably not needed
    11. Storing the logs
      • Any regulations that mandate storage of proxy logs? Not directly .
      • Typical operational storage requirements:
        • 90 days online (with quick access and fast reports)
        • 1-3 years long-term storage
      • Other considerations
        • Incident response, especially for IP theft cases, does point towards longer retention times
        • Privacy laws (outside the US) point towards shorter retention times
      • Use the same system for all logs, not just proxy logs!
    12. Example 3: Correlated Log Investigation
      • How?
      • Search for an IP address in server, firewall and proxy logs
      • What ?
      • Look for a complete activity trail of a user as he performs various tasks, connects to various servers, runs tasks on his machine, accesses the web
      • Recover the sequence of events based on correlated logs (assumed time sync !)
    13. Analyzing the logs
      • Near real-time analysis / alerts
        • Proxy failures
        • Major policy violations (by category)
        • High-risk spyware infections
      • Stored log analysis / reports and searches
        • Reports and summaries: covered in Part II
        • Investigative log searches
          • User name, URL/site, IP address
        • Analytic searching
          • Rare response codes, HTTP methods, user-agents, etc
    14. Example 4: File Uploads Analytic Search
      • How?
      • Search for POST requests AND specific document content-types (e.g. msword, powerpoint, etc)
      • What ?
      • Look for a uploads to unusual sites (especially with unresolved IPs), web mail, or for sensitive document names
      • Especially, look for uploads to unusual ports
      • More details on LogBlog (Tip #12) or
      • Security Warrior Blog (Tip #12)
    15. Conclusions
      • Collect, store and analyze proxy logs for security, operations and compliance
      • Use a single log management tool to collect proxy logs in combination with other logs (system, security, network, application, etc)
      • Proxy logs are a source of critical user behavior information
    16. Thank You for Attending!
      • Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA
      • Chief Logging Evangelist
      • LogLogic, Inc
      • [email_address]
      • http://www.chuvakin.org
      • See www.info-secure.org for my papers, books, reviews
      • and other security and logging resources

    + Anton ChuvakinAnton Chuvakin, 3 years ago

    custom

    5036 views, 2 favs, 1 embeds more stats

    Web proxy logging, log analysis and log management: more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 5036
      • 5025 on SlideShare
      • 11 from embeds
    • Comments 0
    • Favorites 2
    • Downloads 0
    Most viewed embeds
    • 11 views on http://amnuay.blogspot.com

    more

    All embeds
    • 11 views on http://amnuay.blogspot.com

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved