Destabilized Server Load Testing


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Destabilized Server Load Testing

  1. 1. Destabilized Server Load Testing Written for Software Test & Performance Peter Jenney VP of Strategy Security Innovation Security Innovation, Inc. 187 Ballardvale Street, Suite A170 Wilmington, MA 01887 +1.978.694.1008
  2. 2. Destabilized Server Load Testing T here are four degrees of freedom in the software development process—time, resources, features and quality. Product management gets to define any three, and engineering always gets to define the fourth. In a typical time boxed development schedule the challenge is to squeeze as much into the available time as possible while maintaining appropriate levels of quality. When applied to load testing, fault injection tools allow testers to uncover more defects, especially gnarlier and more significant defects, in the same amount of time; providing managers with the additional information necessary to decide which features or functionality will ship, and which will not. Fault-Injection Modern application software, whether it is Java or .NET, consumes huge amounts of code from hundreds or thousands of libraries provided by hundreds of vendors, and none of it with any guarantee of quality or stability. To compensate, developers maintain a belief system enabling them to use unfamiliar libraries and remain sane. This system assumes: • Resources their applications consume are always available and correct. • Their applications own all the resources on the system. • Nobody can monkey with the system resources they consume. Worse, as managers we enable these belief systems by imposing unrealistic time constraints, staffing and feature loads on the try development team. Experiencing “urgency” in the development { newThingy = new criticalResource(); process forces developers to take shortcuts and make } assumptions regarding how an application will be used when catch deployed- which leads to simple code problems like that in Figure 1 { that have dramatic effects. It is this belief system and the urgency // TODO: figure out what to do if we get here imposed by management that leads to fragile and insecure } applications, and it’s not likely that anything will come along that will relieve the market pressures that lead to it. FIGURE 1 The answer is to improve the depth of testing that can be applied to an application without increasing the duration of the testing process. The simplest way to do this is to apply fault injection. Fault injection is a method of forcing applications to execute code paths meant to handle error recovery, and increasing an application’s test surface. By simulating the conditions that one would expect to force errors, such as no disk space, no memory, missing DLLs, corrupted data files or broken network controllers, defects that would be missed in normal testing are exposed. By applying fault injection to the FIGURE 2 normal automated functional testing process it is possible to uncover not only more defects, but more interesting ones, and typically those which are much harder to find. Additionally this technique will expose defects not only in the development teams own code, but those in the libraries and resources it consumes - destroying the belief system upon which development depends and encouraging better code and more appropriate management behavior. 2/4
  3. 3. Destabilized Server Load Testing Load Testing The benefits of fault injection in functional testing are obvious and the same holds true in the load and performance space. Load testing applications in an otherwise stable environment is a useful test to understand scalability. There is also no such thing as stability in today's IT environment. All deployed applications are subject to variable I/O bandwidth, sporadic remote system availability, disk fragmentation, resource corruption, and performing a load test without taking these things into consideration reduces the value of the test dramatically. After all, what good is it being able to support 50,000 users if the system falls over when it gets a bad packet, can't read a file or runs low on memory? Truly testing an application means that the application has been run in an environment like that in which it will live out its life, and survived. Anything short of this is only half- tested and not acceptable. From a fragility standpoint, server applications that crash are just plain unreliable, and people won't buy them. From a security standpoint it’s worse. Making servers crash by doing things like fuzzing protocols is a classic attack strategy that may result in the application leaving unencrypted and exploitable data behind. This is not something that any application should do. Testing with the goal of forcing crashes is simple to do with an appropriate fault injection tool and is easily combined with other existing test processes. Some great examples of tests that one might apply are: • Linear Memory Availability Cycling – The application under test will be Variableexposed to varying RAM availability with the change being a fixed increment from maximum to minimum and back again. • Random Memory Cycling – The application under test will be exposed to varying RAM availability with the change being a random increment between fixed maximum and minimum values. • Linear Network Up and Down Bandwidth Availability – The application under test will be exposed to parallel bandwidth degradation with the change being a fixed increment from maximum to minimum and back again; and exposed to crossover degradation where the up and down bandwidth are opposite. • Network Packet Fuzzing – Randomly corrupting network packet I/O to watch for unstable behavior, or the lack of unstable behavior. In security testing it is often the fact that an 1 application which doesn’t fail the one that is most interesting Each of these tests should cause interesting behavior in applications under load. Starving applications of bandwidth or limiting system memory are great ways to force systems to thrash and uncover all kinds of interesting performance problems, and will likely uncover functional problems too. Combining fault injection during the load testing phase makes a lot of sense for the following reasons: • It can expose functional defects that only occur under load. • It expands the test surface without adding testing time. • It can expose functional defects that would normally go unfound. • It can expose security defects that would not normally be found Successful software development teams understand that there’s little chance that ship dates can be extended and that they have to cram as much testing into a given time box as possible. Teams are also aware that the types of defects that are interesting is increasing, and includes security as a primary concern. Fault injection 1 See “How to Break Software Security”, 2004 by Whittaker & Thompson and “The Software Vulnerability Guide”, 2005 by Thompson & Chase 3/4
  4. 4. Destabilized Server Load Testing tooling and black box load testing are effective methods of exposing defects that lead to failure in deployment. They give the teams the confidence their application will stand up to the rigors of deployment and heavy use. About the Author Mr. Jenney is a seasoned veteran in the software testing space and brings more than 20 years of high- tech experience to Security Innovation, where he plans and directs the company’s commercial technology development, marketing and strategic partnership activities. About Security Innovation Security Innovation is the authority on application security and a leading provider of risk analysis, risk mitigation and education services to mid-size and Fortune 500 companies. Global technology vendors and enterprise IT organizations rely on our services to identify security risks in their software systems and development processes and facilitate the changes needed to mitigate them. The company is headquarted in Wilmington, MA and has offices in Seattle, WA and Amsterdam, the Netherlands. 4/4