OWASP AppSec 2004 Presentation
Upcoming SlideShare
Loading in...5
×
 

OWASP AppSec 2004 Presentation

on

  • 2,013 views

 

Statistics

Views

Total Views
2,013
Views on SlideShare
2,012
Embed Views
1

Actions

Likes
0
Downloads
18
Comments
0

1 Embed 1

http://www.slideshare.net 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

OWASP AppSec 2004 Presentation OWASP AppSec 2004 Presentation Presentation Transcript

  • ::Software Security Quality:: Testing Taxonomy & Testing Tools Classification Arian J. Evans, OWASP Tools Project Random Security Title, FishNet Security [email_address] 913.710.7085 [mobile]
  • OWASP AppSec DC NIST 2005
    • ::Software Security Quality::
    • Testing Taxonomy & Testing Tools Classification
    Subtitle: What is Software Security, how do we evaluate it, and what do we use to evaluate it?
  • --OR--
    • ::Software Security Quality::
    • Why does every web-based Document Management Solution Suck so Badly?
    View slide
  • Testing Taxonomy & Tools Project
    • This project, like many other OWASP projects (e.g.-WebScarab, .NET Tools), is:
    • Performed mostly nights and weekends
    • A personal labor of love (or dislike of FUD)
    • Performed by a team of one
    • As objective and vendor-agnostic as possible
    • Owes thanks to FishNet for letting me use their lab and encouraging vendors to participate
    • Results are free and worth every penny
    • So don’t complain too much about the results. 
    • Emphasis has been on “web app scanners” since that’s where the predominant market interest seems to lie right now.
    View slide
  • Who
    • Arian J. Evans
    • Tester, Teacher, Researcher, Decoder
    • Application Security Services
    • FishNet Security
    • FishNet Security
    • Security Consulting Company focused only on Infosec
    • Work in AppSec Group
    • They pay me to test and help people fix things
    • (blah, blah)
  • Outline:
    • Introduction (done)
    • Chapter One: In the Beginning was FUD.
      • Problems
      • About the OWASP Testing Taxonomy & Tools Project
    • Chapter Two: What is Application Security?
      • Distinctions
      • Definition
    • Chapter Three: Clarification and Goals
    • Chapter Four: Taxonomy of Testing
    • Chapter Five: Automation Tools
    • Chapter Seven: Q&A (Shoot the Messenger)
  • Chapter One:
    • Problems with understanding application security, vendor FUD, appsec testing tools, and testing techniques.
    In the Beginning was the FUD
  • Webappsec Vendor Marketing Hype
    • I. The Problem of Particulars
      • XSS, XST, XDS, XDF, XBS: why particulars can hurt more than they help (reference: Issues slide)
      • Selling managers/auditors bullet-point friendly products
      • Classification of Threat/Risk/Issue: Lack of distinction between Category/Class/Particular—Does anyone take formal logic classes anymore?
    • Immature Metrics
      • What is *really* being exploited? (This is not what the OWASP Top-10 or WASC taxonomy addresses.)
      • *What* exploits result in real loss? (Not what the threads on webappsec@sf are talking about.)
      • No reward in securing without demonstrable risk.
  • The Problems:
    • Different ways to “test” applications for security quality.
    • Different tools to “test” applications for security quality.
    • No categorization of differences or metrics to measure pluses and minuses of testing techniques/tools.
    • No single source of information on what tools are available for testing applications
    • No independent source of information on testing tools that are available.
    • Some tool vendors do a terrible job of educating the consumer on what they do and don’t do, and how they relate to their competition.
  • The Problems:
    • Unclear and immature terminology
    • Few Taxonomies, Categorizations, Metrics and Models on AppSec
    • No Business Rule/Risk Taxonomy
    • No mature Threat Models/Taxonomy
    • No accurate Attack Models/Taxonomy
    • Other Taxonomies needed
    • We need a Taxonomy of Taxonomies
  • This Presentation is About:
    • The OWASP Tools Project Version 1.2 which aims to:
    • Categorize the different ways to “test” an application for security.
    • Categorize the variety of tools that may be used “test” applications for security.
    • Provide basic insight into features and functionality of “software security tools”.
    • Offset the active dissemination of misinformation about what tools can and cannot provide.
    • I bit off more than I can chew here ( why are there so many tools out there for this this??? )
  • This Project is Headed:
    • The OWASP Tools Project Version 2.0 which aims to:
    • Categorize & Classify tools that may be used “test” applications for security.
    • Work with groups like OWASP, WASC, NIST, and Mitre/CVE
    • Evaluate the value of & possibly provide checklist charts of features and functionality in specific versions of appsec tools.
    • I have concerns about above because for example not all XSS checks are equal, and the quality can and does change overnight in current tools.
    • In non-COTS software metrics tend to lack pragmatic value as software continues to increase size, complexity, extensibility, and develop emergent behaviors as disparate technologies are bolted together and new software evolves.
  • Limited Information; Full-on Equivocation
    • I . Things Commercial Tool Vendors like to say:
      • Sanctum: “Oh, those are *network* scanners”
      • Compuware: “the only ones in the world who do this”
      • Cenzic: “We’ve automated webapp pen testing”
      • Several: “Oh, no, that’s a CVE scanner. It doesn’t find anything on our sample application which has “real” vulnerabilities. See here, we’ll just install our sample application we wrote right here and show you…”
      • Tautology: The Scanner and Firewall Vendors
    • Goings-on in the AppSec Industry:
      • No Standards or Standards-bodies
      • No Metrics
      • No Watchdogs
      • Vendors provide the vast bulk of “Factual Literature”
  • Chapter II: The Definition
    • What is Application Security?
    • Why do you care about AppSec Testing Tools?
    • When will the tools be mature enough that I just click “Scan” and get my report?
    Why and What is Application Security?:
    • Because you lack expertise
    • Because you lack subject information.
    • Because you lack resources.
    • Because you lack time.
    • Because of the desire for automation.
    • Because of the need for reliable, repeatable tests that produce accurate quantifiable information about software defects with security implications.
    Why do you care about AppSec Tools?
  • Never. There are things that we can automate and things that we cannot automate. There are issues that are easy to automate reliable detection and issues that are not: - business logic flaws - workflow bypasses - human eyeball parsable errors. Terminology: Functional versus Technical Issues When can I click ‘Scan’?
  • Functional versus Technical Issues
    • There are two main types of application security issues:
    • Technical Vulnerabilities
      • Lack of Input Validation, resulting in:
      • XSS
      • SQL Injection
      • Parameter Tampering, etc.
    • Functional or Logical Vulnerabilities
      • Click two or more buttons at the same time results in unexpected behavior.
      • Password change form disclosing detailed errors.
      • Session-idle deconstruction not consistent with Policies
      • Spend deposit before deposit funds are validated (B/L)
  • Functional versus Technical Issues
    • These two types of security issues must be tested differently:
    • Technical Vulnerabilities
      • Usually about Data Handling
      • Dealing with Strong Typing, Encoding, and Validation of Data that the application handles
      • Can be tested fairly effectively by automated tools (at least, in theory, as tools mature)
    • Functional or Logical Vulnerabilities
      • Deals with issues allowed by design, but not foreseen by the designers (or understood to be a risk)
      • Deals with eyeball issues—does the error message text provide a brain-decipherable “secret message”?
  • Two Items to Reiterate
    • Restated:
    • Automation Tools
      • Testing: useful; use daily
      • Protection: maybe useful, maybe not (is your issue XSS? Do you know what your issues are?)
      • Anecdote of client who solved 90% of their issues even though they didn’t know what they were.
      • Anecdote of WAF vendor who “successfully deployed in under an hour” for the java applet communicating on TCP/5000 with an encrypted binary protocol.
    • Examples of Vendor Challenges
      • Sanctum Support Issues: XSS vs. Phishing debate
      • Other vendors “Phishing” “XSS” “blahfoo” feature FUD
  • What is “Application Security”?
    • I. Things that Application Security is not:
      • It is not a product
      • It is not a firewall with “AI” (app intelligence)
      • It is not a NIDS/NIPS with app signatures
      • It is not any “perimeter access control”.
      • It is not a “web app firewall” widget.
      • For today and tomorrow’s web applications the concept of a perimeter is dead.
  • What is “Application Security”?
    • II. Things that Application Security is not:
      • It is not a process (it is possible, however unlikely, to have secure applications with or without a process)
      • It is not “application hacking school” or “secure coding school” for your developers
      • It is not the BPPT (Big Production Penetration Test) you have at the end of your BUFD app lifecycle.
  • What is “Application Security”?
    • III. Things that Application Security IS:
      • It is one possible emergent Quality of a well designed application.
      • It is about predictability, dependability, reliability. (Business speak !=“security”)
      • Rob’s Report must Run Right (bugs are not difference of kind but difference of degree and the issue is always the same)
  • How do we evaluate applications for Security?
    • First and most important application security principle:
    • The Application Must Defend Itself
  • Presentation Outline Updated
    • Introduction (done)
    • Chapter One: The Problem (done)
    • Chapter Two: The Definition (done)
    • Chapter Three: The Clarification and Goals
    • Chapter Four: Taxonomy of Testing
    • Chapter Five: Tools Tools Tools of Automation
    • Chapter Six: Q&A (Shoot the Messenger)
  • Chapter III
    • Clarification and Goals
  • The Categorization Basics:
    • Software flaws with Security Implications
    • Category
    • Class
    • Particular
    • -Example-
    • Category (Coding Error)
    • Class (Output Encoding)
    • Particular (XSS (Cross-Site Scripting))
  • Are we trying to Categorize and Classify?
    • Clarification of Terms
    • Categorization of Issues
    • Abstract fixation from particulars
    • Needs: Category, Class, Particular
    • Needs Risk, Threat, Attack, Weakness, Vulnerability
    • Needs account for Severity, Attackability (Ease), Prevalence, Priority
  • The Classification Basics:
    • Risk ((Loss|Impact) : Financial, Legal, Market)
    • Threat (Likelihood, Severity, Ease, Implication)
    • Attack (Exploit Vector)
    • Weakness (Fundamental Issue)
    • Vulnerability (Particular Issue/Flaw Instance)
  • The Classification Breakdown:
    • -Example One-
    • Risk
      • Financial Loss, Repudiation, Market Goodwill
    • Threat
      • Elevation of Privilege; Forged Transaction
    • Attack (A: Spoofing of User Identity)
      • A 1 Session Fixation
        • A 1-2 Session TokenCookie set a priori user authentication
    • Weakness
      • Insecure Authorization Mechanism; Insecure Session Handling
    • Vulnerability
      • Harvest Session Cookie on Login Form
  • The Classification Breakdown:
    • -Example Two-
    • Risk
      • Financial Loss(???), Market Goodwill
    • Threat
      • Elevation of Privilege; Sensitive Information Disclosure
    • Attack (A: Arbitrary Script Injection)
      • A 1 XSS/Cross-Site Scripting
        • A 1-2 Embedded/Persisted Malicious Javascript
    • Weakness
      • Insecure Output Encoding
    • Vulnerability
      • Support forum page allows embedded linking to external javascript/flash/actionscript/etc.
  • Example Problems with Existing Threat Models:
    • Microsoft: STRIDE and DREAD
    • Spoofing of User Identity (Threat or Attack?)
    • Tampering with Data (Attack?)
    • Repudiation (Risk or Threat?)
    • Denial of Service (Threat or Attack?)
    • Elevation of Privilege (Threat)
    • STRIDE, DREAD and others mix of Threats, Attacks, Vulnerabilities, disparate elements. New TRIKE methodology shows promise.
  • More on Threat Modeling:
    • Threat Modeling is very important; I use it extensively particularly in communicating to business owners or stakeholders of an application
    • For more see the OWASP Testing presentation
    • The OWASP Testing GUIDE
    • STRIDE & DREAD in MS Threat Modeling Book
    • TRIKE is a new Threat Modeling Whitepaper
    • Visit the Threats and Countermeasures Website by Foundstone
  • How do we evaluate applications for Security?
    • First and most important application security principle:
    • The Application Must Defend Itself
  • Conclusion to Chapter III
    • The previous material is presented to provide a common set of terms and framework for understanding what we can and can’t do with automated tools that discover software flaws with security implications.
    • Security is not a “feature” you implement in code
    • Security is an emergent Quality of well-designed and well-built software, much like speed and reliability
    • Categorical difference between technical and functional issues and how you discover them
    • Tools tend to flag everything as “vulnerabilities”
    • Tools really find a mix of threats, attacks, and vulnerabilities (XSS is an attack, not a vulnerability)
    • Tools tend not to indicate systemic weaknesses, like improper output encoding techniques, which is what we really need to be concerned with if our focus is *fixing* the application issues.
  • Chapter IV
    • Taxonomy of Testing
  • Taxonomy of Testing Categories
    • Vulnerability Scanning
    • Fault-Injection Testing (Blackboxing)
    • Fault-Injection Testing with Sandboxing
    • Binary Analysis
    • Source Code Analysis
    • Threat Modeling/Architectural Analysis
    • Rootkit and Trojan Analysis
    • (What about runtime analysis and runtime patching?)
  • Taxonomy of Testing Categories
    • Vulnerability Scanning
      • regex matches on version numbers.
    • Fault-Injection Testing (Blackboxing)
      • Inject all possible faults and parse responses.
    • Fault-Injection Testing with Sandboxing
      • Inject all possible faults and analyze data storage, manipulation, and local entry and exit points, and fuzz/fault-inject those points.
    • Binary Analysis
      • Analyze static binary or perform runtime analysis and look for breakpoints.
  • Taxonomy of Testing Categories
    • Source Code Analysis
      • Mostly static signature matching, but some tools claim to walk codepath, follow nested loops and other regression through objects
    • Threat Modeling/Architectural Analysis
      • Detailed, logical modeling. Time consuming, thorough; avoid wasted man hours of broken feature-implementation up front. Or find obvious design flaws quickly.
    • Rootkit and Trojan Analysis
      • Why not? The NCUA/OCCU mandates it.
    • What about runtime analysis and runtime patching? Another idea…
  • What Follows The next series of slides is a short summary of common and well-known applications that fall into these categories. It is not intended to be comprehensive (later slides will be more comprehensive). It is not intended to indicate quality or imply any form of advocacy for the listed software security tools. If you are looking for product advocacy there are plenty of sales people and “sales engineers” that can help you there.  Most importantly: These tools are constantly evolving. Consider that the data presented may already be outdated. Use as a guide only and evaluate any selection you make Yourself , on Your Applications .
  • Vulnerability Scanning
    • ISS Internet Security Scanner (the old basic VA tool)
    • eEye Retina (CGI Scanning?)
    • Qualys Qualysguard (XSS, limited SQLi)
    • NGS Typhon (claims full JS Parsing)
    • McAfee Foundscan (DNE for webapps)
    • nCircle IP360 (similar to Qualysguard)
    • Known-File Scanners e.g.-Whisker, Nikto, Nessus+Nikto, Nstalker Nstealth
    • These tools do a regex version match (e.g. 12.8.1.0 diff vuln-database) or simply match a filename (e.g. “login.asp”…Oh No!, not a Login!)
  • Fault Injection Testing
    • SPI Dynamics WebInspect 5.5
    • Watchfire AppScan 5.5
    • Acunetix WVS 2.x
    • Cenzic Hailstorm 2.6
    • NTO Spider x?
    • Compuware Devpartner x?
    • NGS Typhon x?
    • Parasoft Webking X?
    • SPIKE (while using binaries with attached debuggers to monitor breakpoints like any traditional fuzzing)
    • Peach Fuzzer
    • These tools inject a fault into the application and in some cases evaluate the response.
  • Fault Injection with Sandboxing
    • Sandboxing Tools:
    • Security Innovation’s Holodeck
    • Sysinternals filemon, regmon, procmon
    • Any sandboxing malware-analysis tools
  • Binary Analysis
    • IDA Pro
    • @stake SmartRisk Analyzers
    • Fortify Application Risk Analyzer
    • Halvar Flake’s Sabre-Security projects
      • BinNavi
      • BinDiff
      • BinAudit
    • WiSA
  • Source Code Analysis
    • Compuware
    • Coverity
    • Fortify Software
    • Ounce Labs
    • Parasoft
    • Secure Software
    • Lots O’ Free & Open Source Stuff
  • Threat Modeling / Architectural Analysis
    • Microsoft Threat Modeling Tool
    • SecuriTree
    • PTA Technologies
    • TRIKE TM tool (to be released)
  • Rootkit/Trojan/Backdoor Analysis
    • Checking for Rootkits, Trojans, and Backdoors is another interesting issue that is recommended in the “Secure Coding” pamphlet sold as a book by O’Reilly, and recommended by the NCUA 2004-03 Guidance letter “must review source code for backdoors, trojans, etc. etc.”
    • Rootkit and Binary analysis tools for Backdoors & Trojans
  • Runtime Analysis and Patching
    • Tools that are designed to analyze behavior and code at runtime and perform runtime limitation/security enforcement and/or “virtual” patching of userspace coded and kernel code.
    • Immunix kernel & stack protection.
    • Some new virtual patching product for Linux that I accidentally deleted the info on and can’t find the emails of the kind folks who contacted me originally.
  • Chapter V
    • Automation Tools
    • (or manual human eyeball supplicants)
    • YMMV
    • Key:
    • MA (Mature)
    • HP (Has Promise)
    • DNE (Did Not Evaluate)
    • RE (Refused Evaluation or ignored repeated requests)
    • TOY (Amusing little widget)
  • Vulnerability Scanning
    • ISS Internet Security Scanner (???)
    • eEye Retina (CGI Scanning?)
    • Qualys Qualysguard (XSS, limited SQLI)
    • McAfee Foundstone Enterprise
    • nCircle IP360
    • Rapid 7 Nexus
    • NGS Typhon (full JS Parsing; XSS & SQLI)
    • Saint
    • Known-File Scanners e.g.-Whisker, Nikto, Nessus+Nikto, Nstalker Nstealth
  • Commercial Fault Injection Test Tools
    • SPI Dynamics WebInspect (5.5, MA)
    • Sanctum now Watchfire AppScan (5.5 MA but may become a ‘dashboard’)
    • Cenzic Hailstorm (HP…ways to go, version 2.6 DNE)
    • NT Objectives NTOSpider (HP, DNE new GUI version)
    • Acunetix Web Vulnerability Scanner 2 (TOY, no scheduling, state, form-fill, etc.)
    • Compuware DevPartner Fault Simulator (RE)
    • Whitehats Security (hosted application, ‘dashboard’ approach, think ‘Qualys’)
    • Fortify Pen Testing Team Tool (RE, unknown; secretive)
    • Burp Intruder (HP, nice fuzzer)
    • MaxPatrol 7 (strange phone-home to Russia traffic observed, stopped eval)
    • Syhunt Sandcat Scanner & Miner (evolving)
    • TrustSecurityConsulting HTTPExplorer (TOY, limited)
    • Ecyware BlueGreen Inspector (TOY, limited)
    • NGS Typhon (DNE, claims full parsing/testing capability)
    • Parasoft WebKing (DNE, positive references from credible sources)
    • Kavado Scando (dead, IP purchased, may show up again, CVE-type scanner)
    • AppSecInc AppDetective for Web Apps (dead, believed permanently)
    • @stake Web Proxy 2.0 (dead or suffocating at Symantec)
    • Sandsprite Web Sleuth (HP, but development End-of-Life?)
  • Commercial Fault Injection Test Tools
    • Data on FI Tools:
    • Data includes version number as many of these tools have new features/improvements since time of testing:
    • Tools tested from March to August of 2005
    • Tools tested on sample applications like OWASP WebGoat
    • Tools tested on multiple real-world applications
    • Tools tested on same versions of applications
    • Tools tested on “default” settings and also attempted to customize tools as best possible to provide optimal use-case scenario for final result data
    • Many tools were configured with help from or onsite support from the vendors, all installed on clean, new boxes, so should represent the best the vendor could provide at that time.
    • I have been guilty of RTFM errors before, but for this testing this is why vendor configuration support was solicited.
  • Commercial Fault Injection Test Tools
    • SPI Dynamics WebInspect 5.5
    • Pros:
    • Best manual testing tools of the FI market (Proxy, Cookie Tester)
    • Excellent javascript parsing
    • Most powerful tools for writing custom checks
    • Best API integration with other QA tools
    • Have removed a lot of the marketing crap that other tools still report like detecting “31 Flavors of XSS” and
    • Cons:
    • No ability to look under the hood on adaptive agents
    • No ability to customize default checks (to protect “IP”)
    • Come on guys, I can run a sniffer and see what you are doing anyway
    • Not the fastest
  • Commercial Fault Injection Test Tools
    • Watchfire AppScan 5.5
    • Pros:
    • Fast on smaller applications
    • Best XSS testing
    • Most powerful right out of the box with no tweaking
    • Excellent reporting
    • New free manual testing toolkit; ‘PowerTools’
    • Cons:
    • No ability to look under the hood or customize effectively
    • Unnecessary XSS variants, and made-up vuln terminology
    • Watchfire is a dashboard company and this is not a effective way to evaluate/measure application security
    • Limited ability to tweak to handle dynamic session affinity and complex authentication variants
  • Commercial Fault Injection Test Tools
    • Cenzic Hailstorm 2.3
    • Pros:
    • Allow you to get under the hood and customize everything.
    • Most potential for automating complex custom checks.
    • Nice drag-and-drop configuration in GUI.
    • Definitely a contender to keep an eye on.
    • Cons:
    • Custom checks simply didn’t work right
    • Slowest tool tested
    • No ability to configure operational/engine parameters like threading, bandwidth, scope, etc.
    • At time of testing was long on hype, highest priced, low on delivery of results
  • Commercial Fault Injection Test Tools
    • NT Objectives NTOSpider
    • 1.something CLI-only beta version
    • Pros:
    • Fast. Wow, this thing is fast.
    • Most effective XSS checks behind WI and AS.
    • Only tool with effective automated session token testing.
    • Auto-state detection and auto-login abilities.
    • Small, lightweight, simple configuration.
    • Cons:
    • No CVE-style checks.
    • No ability to look under the hood or customize tests easily.
    • Roughly half the accuracy of WI or AS on XSS and SQLi.
    • No GUI at time of testing.
  • Commercial Fault Injection Test Tools
    • Acunetix WVS
    • Pros:
    • Nice, pretty GUI
    • Appears fast, but it doesn’t do much so I guess it’s easy to be fast at that.
    • Cons:
    • Limited authentication support
    • No scripting or scheduling capabilities
    • Utter lack of session/state handling
    • No manual analysis tools
    • False positive centric
    • Marketing appears targeted at sub-100 IQ demographic
    • Included due to massive advertising blitz recently
  • Commercial Fault Injection Test Tools
    • Compuware DevPartner Fault Simulator
    • Pros:
    • Built with the Developer in mind
    • IDE integration
    • Integrates with Source-Code analysis
    • May be one to keep an eye on but looks like not intended to be “best of breed” solution.
    • Cons:
    • Vendor has no clue about competitive landscape.
    • Vendor appears to have limited security knowledge.
    • Did Webex eval only.
    • Got tool but not license key; vendor promised follow up repeatedly but never did.
    • Vendor to provide data for the OWASP Tools project but never followed through in providing this.
  • Commercial Fault Injection Test Tools
    • WhiteHat Security (the Seattle Company)
    • Hosted FI Tool; think Qualys of WebAppSec
    • Pros:
    • DNE (Did Not Evaluate)
    • Some smart people building this
    • Hosting approach nice for repeated testing/change management
    • Cost effective
    • Cons:
    • No replacement for manual testing
    • DNE
  • Commercial Fault Injection Test Tools
    • NGS Typhon
    • Pros:
    • DNE for Web Application Checks
    • Experience with NGS tools is that they are very fast, fairly accurate, lack ability to customize/configure much through the GUI and have terrible reporting. 
    • Will include in next release of this project.
    • Cons:
    • DNE.
  • Commercial Fault Injection Test Tools
    • Parasoft WebKing
    • Pros:
    • DNE
    • Have heard has is less mature but some nice features
    • Will include in next version of project
    • Cons:
    • DNE
  • Commercial Fault Injection Test Tools
    • SandSprite Web Sleuth
    • Pros:
    • Inexpensive
    • Simple easy-to-use Windows GUI
    • Nice code markup and views
    • Custom scripts for crawling, cookie testing, etc.
    • Cons:
    • Geared towards all manual testing
    • No default pre-built checks
    • Doesn’t appear under active development
  • Commercial Fault Injection Test Tools
    • OWASP WebScarab v.20051012 
    • Pros:
    • Many advanced testing features geared towards experienced application security professionals.
    • Free & Open Source.
    • Session Token entropy analysis.
    • Static string fuzzer.
    • User-extensible with custom agents.
    • Web Services testing interface for complex types
    • Cons:
    • Non-intuitive, complex interface.
    • No “scan” or pre-built checks like CVE issues.
    • Tends to hang and need restarted.
    • No vulnerability reporting mechanism or management style reporting a’la Watchfire.
  • Fault Injection Test Tools Anecdotes
    • Similar to diet fads
    • My Book “ Body of Anecdotal Evidence: Portion-sized Marketing ” out soon
    • That was a “ Body for Life ” joke for you not from the US
    • Again, there is no book for those of you that asked me after the presentation.
    • YMMV
    • Anecdotes are subjective
    • Anecdotes are not logically valid proofs
    • Repeat Disclaimer: YMMV
  • Fault Injection Tool Experiences
    • The following metrics were gathered from real-world assessments performed by myself along with up to four other testers.
    • These were commercial assessments on a mix of COTS software and custom applications developed by clients ranging from Fortune 100 to smaller ISVs that authorized benchmarking of these tools during the course of software security assessment.
  • Fault Injection Tool Experiences
    • Application #1:
    • 150+ pages, Complex workflow
    • Dynamically built javascript menu/functions
    • Application contained 1 trivial XSS, 1 Workflow Bypass (WFB), 1 Brute Force (BF)
    • The tools in this test were run with default configurations; essentially clicked ‘scan’ as many people unfortunately use these tools in RealLife™.
    • WebInspect 5.0: 147 pages, no XSS, WFB, BF
    • AppScan 5.5: 38 pages, 1 XSS, no WFB, BF
    • Acunetix 2: 6 pages, no valid positive results
    • NTO Spider: 80+ pages, no XSS, WFB, BF
    • Paros 2.something: 0 (zero) (loop)
    • Cenzic Hailstorm: Couldn’t get working
    • (The security defects detected in the applications were discovered during the course of manual analysis)
  • Fault Injection Tool Experiences
    • Application #2:
    • 30+ templates, Dynamically generated content
    • 15k+ actual pages, Long forms, simple workflow
    • 32 XSS from URL parameters to multi-form <--!-->
    • Weak Session Handling; Auto-scripting attacks (“Session Riding” or “Web Trojans”)
    • Tests run as ‘scan’ with minimal tweaking
    • WebInspect 5.0: 31 XSS, no SH or SR issues
    • AppScan 5.5: 18k+ pages, slowed to 1 rec/6 seconds
    • Acunetix 2: No Results (couldn’t automate effectively)
    • NTO Spider: 18 XSS, weak SH, no SR, fast! (CLI version)
    • Hailstorm 2.2: 3 days, 4 people, Cenzic onsite, nothing
    • Paros 3.0.2: 18 pages, no results, would hang on audit
  • Fault Injection Tool Experiences
    • Application #3:
    • 100’s of pages/forms
    • Strong UI, Input Validation, Output Encoding
    • Web Server packages XML POST string
    • Passes to routing servlet for authorization
    • Function Servlet name/route in Hidden FF
    • Create XML String and POST directly to DST servlet
    • XML Abstraction Layer has no authentication/authorization
    • ASMQ does not have crypto enabled; auth is function group code
    • Point of this example is that there are holes you could drive a Mac Truck through and every FI Scanner failed to detect them. Some things require humans.
    • WebInspect 5.5: Nothing
    • AppScan 5.5: Nothing; WF PowerTools Proxy: failed
    • Acunetix 2: PHP issue, some other issues, false positives (200 codes)
    • NTO Spider: Nothing (CLI version); no proxy
    • Paros 3.0.2: Nothing; Proxy-mode: failed
    • Proxies: SPI Proxy (only proxy to handle various cert-auth)
  • Fault Injection Tool Experiences
    • Application #4:
    • AAWBDMS (Another Awful Web-Based DMS)
    • XSS, Parameter Tampering, etc. (everything you can imagine controlled in client-side hidden form-field parameters)
    • URL Parameters passed directly to compiled C code (DLLs, EXEs) running as Domain Admin with traditional buffer overflows.
    • Dual-Factor Authentication
    • Authentication Two-step Workflow
    • First page UserID & PRNG Token number in HTML Form
    • Second Page identical UserID and Password in HTML Form
    • All responses 200; everything returns one of two login forms
    • Every darn tool: NADT
    • NADT: Not a Darn Thing
    • Nobody could handle dual-auth with custom error pages identical to the login pages.
  • Fault Injection Tool Experiences
    • XSS attacks should be fully automated, but as of yet are not. We find XSS in almost every application we test that is not found by any automated tool; some are trivially devastating. Examples:
    • Form value expects email address ( [email_address] )
    • Form value validated by sloppy regex; Output not encoded
    • Form error returns no data unless you bypass regex filter
    • Tools submit the following types of checks:
      • <script>alert(‘somethingclever’);<script>
      • <script>alert(‘somethingclever’);<script>silly@domain.com
      • silly@domain.com<script>alert(‘somethingclever’);<script>
      • some replace ‘silly’ and ‘com’ with <script>alert();<script> check
      • some add escape characters like ‘> , ‘>> , ‘) , and so on
    • Actual XSS by silly@<script>alert()<script>.com
    • Other attack vectors tools fail include
      • encoding attack strings (reference OWASP Guide 2.0.1 or RSnake’s XSS page)
      • partial encoding of attack string elements
      • using specific HTML tags like body elements and background
  • Open Source or Freeware Fault Injection Test Tools
    • WebScarab (DNE current version, re: HTTPush, Exodus)
    • Paros Proxy (HP)
    • Watchfire PowerTools (HP, free, proxy nice GUI, rest so-so)
    • Burp Spider (HP, free)
    • Burp Proxy (HP, free)
    • Snark HTTP Proxy & Tabby Tunnel (DNE)
    • Peach Fuzzer (HP, free, Python fuzzer)
    • SPIKE Proxy (Horrible interface, free, limited fuzzing)
    • SPIKE Fuzzer (dev EOL? Dave pointing people at Peach)
    • Achilles Proxy (TOY, first Windows proxy, free, old)
    • Odysseus Proxy (some cool features, GUI needs work)
    • Webstretch Proxy (DNE)
    • Absinthe 1.1 (formerly SQLSqueal) (HP, free)
    • Sensepost E-Or, Wikto (Google hacking utility), other webappsec and pen testing tools
      • (everything Sensepost is working on is worth looking at)
  • Open Source or Freeware Fault Injection Test Tools
    • And your Basic Web Browsers…don’t forget them:
    • Internet Explorer + HTMLBar DLL extension (buggy)
    • Firefox + LiveHTTP Headers, Tamper Data, Developer Tools,
        • and many, many, more useful extensions….however,
        • keep in mind Firefox encoding is often different, more
        • restrictive, sometimes more secure than IE so if the
        • application client population includes/is predominantly
        • IE MAKE SURE you test encoding/format in IE as well.
    • Foundstone Free Tools:
      • SiteDigger (Google Hacking Utility)
      • SSLDigger
      • Probably others I haven’t looked at yet
      • SiteGenerator (coming soon, may be useful for testing Fault-Injection tools for javascript parsing, etc.)
  • Fault Injection with Sandboxing
    • Sandboxing Tools (Commercial & Free):
    • Security Innovation’s Holodeck (HP)
    • Sysinternals filemon, regmon, procmon can be combined with VMWare
    • Any sandboxing malware-analysis tools available from various forensics sites and possibly from the AV vendors.
  • Binary Analysis
    • IDA Pro + scripts from Halvar & iDefense
    • Grammatech CodeSonar
    • Paradyn Project (claims superiority to IDA Pro when C++ variables present in object)
    • FX Dumbug
    • @Stake SmartRisk Analyzers (gone/dead)
    • Fortify Application Risk Analyzer (marketing freebie)
    • WiSA (Wisconsin Safety Analyzer)
    • SABRE BinDiff (IDA Pro extension)
    • SABRE BinNavi (Graphical flowgraphs, playback of codepaths, Win/Linux debuggers)
    • SABRE BinAudit (Full Binary Analysis; beta project; Halvar restarted this again)
    • Secure Software CodeAssure Auditor (Binary Analysis)
    • Brute Force Binary Checker ( http://bfbtester.sourceforge.org )
  • Commercial Source Code Analysis
    • Compuware DevPartner SecurityChecker .NET Languages
    • Coverity SWAT C/C++
    • Escher Technologies Eschertech
    • Fortify Software Suite (analysis, workbench, metrics & trending console, customization module, won’t show me anything) C/C++,Java,JSP,HTML,Java bytecode,(PL/SQL),C#,VB.NET,MSIL, …
    • Gimple PC and Flexe-Lint C/C++ (we use, simple, cheap, functional)
    • Grammatech CodeSurfer C/C++
    • Ounce Labs Prexis C/C++, JSP, Java (nice, rough UI, recursion abilities?)
    • Parasoft JTest Java (? NE)
    • Reasoning SecurityInspect “Service”
    • Secure Software CodeAssure Workbench C/C++, Java
    • Security-Assessments CodeScan ? (April 01 05) won’t give specifics
    • Note if you are in sales: My name is not Ariel, Arial, Adrian, Aruin, etc
    • WiSA C/C++
  • Open Source, Free, or Bundled Source Code Analysis
    • OWASP .NET (Reference Dinis Cruz presentations on OWASP website about OWASP .NET tools)
    • Foundstone .NET Tools (same story as above; DNE but heard these were coming out soon)
    • The Next Slide has All The Other Stuff
  • Open Source, Free, or Bundled Source Code Analysis
    • Bunch http://serg.cs.drexel.edu/bunch
    • Cigital ITS4 (no longer supported) C/C++
    • CQual http://www.cs.berkeley.edu/~jfoster/cqual
    • David Wagner’s BOON C http://www.cs.berkeley.edu/~daw/boon/
    • David Wheeler’s Flawfinder C/C++ www.dwheeler.com
    • Reference/Credit David Wheeler’s Page
    • Microsoft PREFix (full codepath execution, recursion), C families
    • Microsoft PREFast (stripped down PREfix) only static signatures for performance reasons, shipping with Visual Team Studio 2005, C families
    • Microsoft FXCop (even more limited signature tool) C/C++ only
    • MOPS http://www.cs.berkeley.edu/~daw/mops
    • PScan http://www.striker.ottawa.on.ca/~aland/pscan
    • RAT C/C++, Java, Perl, Python, whatever other random signatures written
    • ShareFuzz
    • SPLINT http://splint.org/
  • Threat Modeling / Architectural Analysis
    • Microsoft Threat Modeling Tool
      • (non-intuitive for beginners)
    • SecuriTree
        • (clunky Java tool, not appsec focused)
    • PTA Technologies
        • (DNE, have had several references)
    • TRIKE Threat-Modeling Tool
        • (DNE, in development stage, unsure release
        • date, but idea has promise)
    • Need more tools here!
  • Rootkit/Trojan/Backdoor Analysis
    • Checking for Rootkits, Trojans, and Backdoors is another interesting issue that is recommended in the “Secure Coding” pamphlet sold as a book by O’Reilly, and recommended by the NCUA 2004-03 Guidance letter “must review source code for backdoors, trojans, etc. etc.”
    • Rootkit and Binary analysis tools for Backdoors & Trojans
    • rootkits.org and rootkit.nl + many Forensic Distributions have these tools
  • Recommended Motto for Vendors
    • Ne Supra Crepidum Suter Judicaret
  • Chapter 6
    • Shoot the Messenger
    • (he may duck and run)
    • Arian J. Evans
    • Does Application Security Stuff
    • FishNet Security
    • 913.710.7085 [mobile]
    • [email_address]
    • OWASP Tools Project
    • OWASP Chapter Leader, Kansas City
    • Feel free to contact me with questions
    • Vendor updates, corrections, confusions, or complaints welcome
    • Email me repeatedly if I miss your first one