The 90 minute sessions with three people is what Microsoft reports as being effective. After 90 minutes, productivity drops sharply. With too many reviewers, productivity also drops. Code reviews and fixes must be timed at the appropriate stage of the development lifecycle. Waiting until code complete will likely frustrate QA and product management.
An example of scenario testing is the recent 802.11 wireless vulnerability announced by the Australians (RTS DoS). It usually starts by saying &quot;What if I do this instead of what's expected?&quot; What if I try to enter more numbers than a PIN is supposed to have? What if I keep punching keys on the ATM while it is counting money? What if I insert an oversized card or envelope, or insert them sideways, or insert something made of another material than the expected plastic (e.g. to jam and disable a competitor's ATMs)? Do the requirements deal with or expect these situations (abuse cases)? If I'm designing a network, what if a user sends a ping to a broadcast address? What if I send SYN packets without completing the connection? etc... Most of the 802.11 and TCP/IP vulnerabilities were found by scenario testing, just using the researchers' imagination with the high-level description of the mechanism or protocol. The requirements didn't capture or address malicious users; benevolent users, or users incapable of interacting with the networks at those levels, were assumed.
Specifications testing is what the Finland Codenomicon/OUSPG/PROTOS specialize in; their formal work resulted in the flood of vulnerability announcements for SNMP, etc... Some findings from specifications testing are usually less obvious than those of scenario testing. They overlap -- specifications testing should a) report all the vulnerabilities found in scenario testing and b) be comprehensive and exhaustive.
Definition of invariant: an assertion that must be true both before and after the execution of each operation (e.g., application use case, class method). See also postcondition and precondition. An assertion that should always be true for a specified segment or at a specified point of a computer program. From [IEEE90].
Symantec calls this ET for external testing. There are tiers of customers, Tier1 ET customers promise to interact at a much higher level than Tier 3 testers. We expect low rates of feedback from most testers because they are busy people with their own agendas and time pressures.
Course 1: Overview of Secure Programming, Section 4
Pascal Meunier, Ph.D., M.Sc., CISSP
May 2004; updated August 12, 2004
Developed thanks to support and contributions from Symantec Corporation, support from the NSF SFS Capacity Building Program (Award Number 0113725) and the Purdue e-Enterprise Center
Copyright (2004) Purdue Research Foundation. All rights reserved.
Correcting security problems after implementation has known security limitations
Cost and time
Some corrections would require rewriting everything
Superficial patch is usually applied instead
"Products claiming security that are created from previous versions without security cannot achieve high trust because they lack the fundamental and structural concepts required for high assurance" (Sullivan 2003)