Agile                          Flexibility and

Performance                          Iterative Processes
           gile software development involves iterations, open collaboration and process adaptability throughout a pro-
eral large tests, provides minimal infor-    resists testing early. While Gunther’s              device. This is not a sta...
ance engineering approach starting early                                                       Is the hardware the same?
times automatic script generation from        Oracle products, and we’ve found that a        response time will be almost ...
need not only department names but               erly and the results will be meaningful. In   ing—testers look for bugs a...
accurately matches the queuing model
  FIG. 2: RESPONSE TIME                                                              ...
Upcoming SlideShare
Loading in …5

Agile Performance Testing


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Agile Performance Testing

  1. 1. Agile Flexibility and Performance Iterative Processes Are Key to Keeping Testing Tests on Track 14 • Software Test & Performance NOVEMBER/DECEMBER 2009
  2. 2. A gile software development involves iterations, open collaboration and process adaptability throughout a pro- ject’s life cycle. The same approaches are fully applicable to performance testing projects. So you need a plan, but it has to be malleable. It becomes an iterative process involving tuning and troubleshooting in close coop- eration with developers, system administrators, database administrators and other experts. You run a test and get a lot of information to the requirements. By Alexander Podelko about the system. To be efficient you need to • Involve the development team if require- analyze the feedback you get, make modifica- ments are missed. tions and adjust your plans if necessary. Let’s At first glance, the waterfall approach to per- say, for example, you plan to run 20 different formance testing appears to be a well-estab- tests, but after executing the first test you dis- lished, mature process. But there are many cover a bottleneck (for instance, the number of potential—and serious—problems. For example: Web server threads). Unless you eliminate the • The waterfall approach assumes that the bottleneck, there’s no point running the other entire system—or at least all the functional 19 tests if they all use the Web server. To iden- components involved—is ready for the per- tify the bottleneck, you may need to change the formance test. This means the testing can’t be test scenario. done until very late in the development cycle, at Even if the project scope is limited to pre- which point even small fixes would be expen- production performance testing, an agile, itera- sive. It’s not feasible to perform such full-scope tive approach helps you meet your goals faster testing early in the development life cycle. and more efficiently, and learn more about the Earlier performance testing requires a more system along the way. After we prepare a test agile, explorative process. script (or generate workload some other way), • The scripts used to create the system we can run single- or multi-user tests, analyze load for performance tests are themselves soft- results and sort out errors. The source of errors ware. Record/playback load testing tools may can vary–you can experience script errors, func- give the tester the false impression that creat- tional errors and errors caused directly by per- ing scripts is quick and easy, but correlation, formance bottlenecks—and it doesn’t make parameterization, debugging and verification sense to add load until you determine the can be extremely challenging. Running a script specifics. Even a single script allows you to for a single user that doesn’t yield any errors locate many problems and tune the system at doesn’t prove much. I’ve seen large-scale cor- least partially. Running scripts separately also porate performance testing where none of the lets you see the amount of resources used by script executions made it through logon (single each type of load so you can build a system sign-on token wasn’t correlated), yet perform- “model” accordingly (more on that later). ance testing was declared successful and the The word “agile” in this article doesn’t results were reported to management. refer to any specific development process or • Running all scripts simultaneously makes methodology; performance testing for agile it difficult to tune and troubleshoot. It usually development projects is a separate topic not becomes a good illustration of the shot-in-the- covered in this paper. Rather, it’s used as an dark antipattern—“the best efforts of a team application of the agile principles to perform- attempting to correct a poorly performing appli- ance engineering. cation without the benefit of truly understanding why things are as they are” (http://www.kirk WHY THE ‘WATERFALL’ APPROACH DOESN’T WORK the_dark.htm). Or you need to go back and The “waterfall” approach to software develop- deconstruct tests to find exactly which part is ment is a sequential process in which develop- causing the problems. Moreover, tuning and per- ment flows steadily downward (hence the formance troubleshooting are iterative process- name) through the stages of requirements es, which are difficult to place inside the “water- analysis, design, implementation, testing, inte- fall.” And in most cases, you can’t do them gration and maintenance. Performance testing offline—you need to tune the system and fix the typically includes these steps: major problems before the results make sense. • Prepare the system. • Running a single large test, or even sev- • Develop requested scripts. • Run scripts in the requested combin- Alexander Podelko has specialized in performance ations. engineering for 12 years. Currently he is a • Compare results with the requirements Consulting Member of Technical Staff at Oracle, responsible for performance testing and tuning of provided. the Hyperion product family. • Allow some percentage of errors according NOVEMBER/DECEMBER 2009 • 15
  3. 3. eral large tests, provides minimal infor- resists testing early. While Gunther’s device. This is not a standard practice, mation about system behavior. It doesn’t book presents a broader perspective but it should be. The later in the devel- allow you to build any kind of model, for- on capacity planning, the methodology opment cycle, the more costly and diffi- mal or informal, or to identify any rela- discussed, including the guerrilla cult it becomes to make changes, so tionship between workload and system approach, is highly applicable to per- why wait until the entire system is behavior. In most cases, the workload formance engineering. assembled to start performance testing? used in performance tests is only an Gunther says there’s a set of We don’t wait in functional testing. The educated guess, so you need to know unspoken assumptions behind the predeployment performance test is an how stable the system would be and resistance to performance-related activ- analog of system or integration tests, how consistent the results would be if ities. Some that are particularly relevant but it’s usually conducted without any real workload varied. to performance engineering: “unit testing” of performance. Using the waterfall approach does- • The schedule is the main measure The main obstacle is that many sys- n’t change the nature of performance of success. tems are somewhat monolithic; the testing; it just means you’ll probably do • Product production is more impor- parts, or components, don’t make much a lot of extra work and end up back at tant than product performance. sense by themselves. But there may be the same point, performance tuning and • We build product first and then significant advantages to test-driven troubleshooting, much later in the cycle. tune performance. development. If you can decompose the Not to mention that large tests involving • Hardware is not expensive; we can system into components in such a way multiple-use cases are usually a bad just add more of it if necessary. that you may test them separately for point to start performance tuning and • There are plenty of commercial performance, you’ll only need to fix inte- troubleshooting, because symptoms tools that can do it. gration problems when you put the sys- you see may be a cumulative effect of It may be best to accept that many tem together. Another problem is that multiple issues. project schedules don’t allow sufficient many large corporations use a lot of Using an agile, iterative approach time and resources for performance third-party products in which the system doesn’t mean redefining the software engineering activities and proceed in appears as a “black box” that’s not eas- development process; rather, it means “guerrilla” fashion: Conduct perform- ily understood, making it tougher to test finding new opportunities inside existing ance tests that are less resource-inten- effectively. processes to increase efficiency overall. In sive, even starting by asking just a few During unit testing, variables such fact, most good perform- key questions and expand- as load, security configuration and ance engineers are already ing as time and money amount of data can be reviewed to doing performance testing permit. determine their impact on performance. in an agile way but just pre- The software perform- Most test cases are simpler and tests senting it as “waterfall” to management. In most cases, once you present [Why wait ance engineering approach to development of software systems to meet perform- are shorter in unit performance testing. There are typically fewer tests with limit- ed scope—for example, fewer variable and get management until the entire ance requirements has long combinations than in a full stress or per- approval on a waterfall-like been advocated by Dr. formance test. plan, you’re free to do what- system is Connie Smith and Dr. Lloyd We shouldn’t underestimate the ever’s necessary to test the Williams (see, for example, power of the single-user performance system properly inside the their book Performance test: If the system doesn’t perform well scheduled time frame and assembled Solutions, Addison-Wesley, for a single user, it certainly won’t per- scope. If opportunities 2001). While their method- form well for multiple users. Single-user exist, performance engi- to start ology doesn’t focus on testing is conducted throughout the neering may be extended testing initiatives, it can’t be application development life cycle, dur- further, for example, to early successfully implemented ing functional testing and user accept- performance checkpoints performance without some preliminary ance testing, and gathering performance or even full software per- testing and data collection data can be extremely helpful during formance engineering. TEST EARLY testing? ] to determine both model inputs and parameters, and to validate model results. these stages. In fact, single-user per- formance tests may facilitate earlier detection of performance problems and Although I’ve never read Whether you’re consider- indicate which business functions and or heard of anybody argu- ing a full-blown perform- application code need to be investigated ing against testing early, it rarely hap- ance engineering or guerrilla-style further. pens in practice. Usually there are some approach, you still need to obtain baseline So while early performance engi- project-specific reasons—tight sched- measurements on which to build your cal- neering is definitely the best approach (at ules or budgets, for instance—prevent- culations. Early performance testing at least for product development) and has ing such activities (if somebody thought any level of detail can be very valuable at long been advocated, it’s still far from about them at all). this point. commonplace. The main problem here is Dr. Neil Gunther, in his book A rarely discussed aspect of early that the mindset should change from a Guerrilla Capacity Planning (Springer, performance testing is unit performance simplistic “record/playback” perform- 2007), describes the reasons manage- testing. The unit here may be any part of ance testing occurring late in the product ment (consciously or unconsciously) the system—a component, service or life cycle to a more robust, true perform- 16 • Software Test & Performance NOVEMBER/DECEMBER 2009
  4. 4. ance engineering approach starting early Is the hardware the same? in the product life cycle. You need to It’s essential to get the test envi- translate “business functions” per- ronment as close as possible to the pro- formed by the end user into compo- duction environment, but performance nent/unit-level usage, end-user require- testing workload will never match pro- ments into component/unit-level require- duction workload exactly. In “real life,” ments and so on. You need to go from the workload changes constantly, includ- the record/playback approach to using ing user actions nobody could ever programming skills to generate the work- anticipate. load and create stubs to isolate the com- Indeed, performance testing isn’t an ponent from other parts of the system. exact science. It’s a way to decrease risk, You need to go from “black box” per- not to eliminate it. Results are only as formance testing to “gray box.” meaningful as the test and environment If you’re involved from the begin- you created. Performance testing typical- ning of the project, a few guerrilla-style ly involves limited functional coverage, actions early on can save you (and the and no emulation of unexpected events. project) a lot of time and resources later. Both the environment and the data are But if you’re called in later, as is so often often scaled down. All of these factors the case, you’ll still need to do the best confound the straightforward approach to performance testing possible before the performance testing, which states that product goes live. The following sec- we simply test X users simulating test tions discuss how to make the most of cases A and B. This way, we leave aside limited test time. a lot of questions: How many users can the system handle? What happens if we THE IMPORTANCE OF add other test cases? Do ratios of use WORKLOAD GENERATION cases matter? What if some administra- The title of Andy Grove’s book Only the tive activities happen in parallel? All of Paranoid Survive may relate even better these questions and more require some to performance engineers than to exec- investigation. utives! It’s hard to imagine a good per- The gathered requirements should Perhaps you even need to investi- formance engineer without this trait. be projected onto the system architec- gate the system before you start creat- When it comes to performance testing, ture because it’s important to know if ing performance test plans. Performance it pays to be concerned about every part included test cases add value by testing engineers sometimes have system of the process, from test design to different sets of functionality or different insights nobody else has; for example: results reporting. components of the system. It’s also • Internal communication between Be a performance test architect. important to make sure we have test client and server if recording used The sets of issues discussed below cases for every component (or, if we • Timing of every transaction (which require architect-level expertise. don’t, to know why). may be detailed to the point of 1) Gathering and validating all 2) Making sure the system under specific requests and sets of requirements (workload definition, first test is configured properly and the parameters if needed) and foremost), and projecting them onto results obtained may be used (or at • Resource consumption used by a the system architecture: least projected) for the production specific transaction or a set of Too many testers consider all infor- system: transactions mation they obtain from the business Environment and setup considera- This information is additional side (workload descriptions, scenarios, tions can have a dramatic effect. For input—often the original test design is use cases, etc.) as the “holy scripture.” instance: based on incorrect assumptions and But while businesspeople know the • What data is used? Is it real pro- must be corrected based on the first business, they rarely know much about duction data, artificially generated data results. performance engineering. So obtaining or just a few random records? Does the Be a script writer. Very few sys- requirements is an iterative process, volume of data match the volume fore- tems today are stateless systems with and every requirement submitted should casted for production? If not, what’s the static content using plain HTML—the be evaluated and, if possible, validated. difference? kind of systems that lend themselves to Sometimes performance requirements • How are users defined? Do you a simple “record/playback” approach. are based on solid data; sometimes have an account set with the proper In most cases there are many obstacles they’re just a guess. It’s important to security rights for each virtual user or do to creating a proper workload. If it’s the know how reliable they are. you plan to re-use a single administrator first time you see the system, there’s Scrutinize system load carefully as ID? absolutely no guarantee you can quickly well. Workload is an input to testing, • What are the differences between record and play back scripts to create while response times are output. You the production and test environments? If the workload, if at all. may decide if response times are your test system is just a subset of your Creating performance testing acceptable even after the test, but you production system, can you simulate the scripts and other objects is, in essence, must define workload beforehand. entire load or just a portion of that load? a software development project. Some- NOVEMBER/DECEMBER 2009 • 17
  5. 5. times automatic script generation from Oracle products, and we’ve found that a response time will be almost instanta- recording is mistakenly interpreted as few scripting challenges exist for almost neous, while in reality the consolidation the entire process of script creation, but every product. Nothing exceptional— may take many minutes or even hours. If it’s only the beginning. they’re usually easily iden- we have several iterations in the script, Automatic generation pro- tified and resolved—but we’ll submit several consolidations, vides ready scripts in very time after time we’re which continue to work in the back- simple cases, but in most nontrivial cases it’s just a first step. You need to cor- [ Don’t called on to save problem- atic performance testing projects only to discover ground, competing for the same data, while we report subsecond response times. relate and parametize assume the serious problems with Consolidation scripts require cre- scripts (i.e., get dynamic scripts and scenarios that ation of an explicit loop around GET- variables from the server system works make test results mean- CONSOLSTATUS to catch the end of and use different data for ingless. Even experienced the consolidation: different users). testers stumble, but many correctly just web_custom_request(“XMLDataGrid.asp_7”, “URL={URL}/Data/XMLDataGrid.asp?Action= After the script is cre- problems could be avoid- EXECUTE&TaskID=1024&RowStart=1&ColStart= ated, it should be evaluat- ed if more time were ed for a single user, multi- because the spent analyzing the situa- 2&RowEnd=1&ColEnd=2&SelType=0&Format= ple users and with different tion. JavaScript”, LAST); script was do { data. Don’t assume the Consider the follow- system works correctly ing examples, which are just because the script executed typical challenges you can sleep(3000); was executed without face with modern Web- web_reg_find(“Text=1”,”SaveCount=abc_count”, errors. Workload validation without errors. is critical: We have to be sure the applied workload ] based applications: 1) Some operations, like financial consolida- LAST); web_custom_request(“XMLDataGrid.asp_8”, “URL={URL}/Data/XMLDataGrid.asp?Action= is doing what it’s supposed tion, can take a long time. to do and that all errors are The client starts the oper- GETCONSOLSTATUS”, LAST); caught and logged. This can be done ation on the server, then waits for it to directly, by analyzing server responses finish, as a progress bar shows on } while (str- cmp(lr_eval_string(“{abc_count}”),”1”)==0); or, in cases where that’s impossible, indi- screen. When recorded, the script looks rectly—for example, by analyzing the like (in LoadRunner pseudocode): web_custom_request(“XMLDataGrid.asp_7”, application log or database for the exis- Here, the loop simulates the inter- “URL={URL}/Data/XMLDataGrid.asp?Action= tence of particular entries. nal logic of the system, sending GET- Many tools provide some way to EXECUTE&TaskID=1024&RowStart=1&ColStart= CONSOLSTATUS requests every three verify workload and check errors, but a 2&RowEnd=1&ColEnd=2&SelType=0&Format= seconds until the consolidation is com- JavaScript”, LAST); complete understanding of what exactly plete. Without such a loop, the script is happening is necessary. For example, web_custom_request(“XMLDataGrid.asp_8”, just checks the status and finishes the HP LoadRunner reports only HTTP “URL={URL}/Data/XMLDataGrid.asp?Action= iteration while the consolidation contin- errors for Web scripts by default (500 GETCONSOLSTATUS”, ues long after that. LAST); “Internal Server Error,” for example). If 2) Web forms are used to enter and we rely on the default diagnostics, we web_custom_request(“XMLDataGrid.asp_9”, save data. For example, one form can might still believe that everything is “URL={URL}/Data/XMLDataGrid.asp?Action= be used to submit all income-related GETCONSOLSTATUS”, LAST); going well when we get “out of memo- data for a department for a month. Such ry” errors instead of the requested a form would probably be a Web page reports. To catch such errors, we should web_custom_request(“XMLDataGrid.asp_9”, with two drop-down lists (one for depart- add special commands to our script to “URL={URL}/Data/XMLDataGrid.asp?Action= ments and one for months) on the top GETCONSOLSTATUS”, LAST); check the content of HTML pages and a table to enter data underneath returned by the server. them. You choose a department and a When a script is parameterized, it’s Each request’s activity is defined by month on the top of the form, then enter good to test it with all possible data. For the ?Action= part. The number of GET- data for the specified department and example, if we use different users, a few CONSOLSTATUS requests recorded month. If you leave the department and of them might not be set up properly. If we depends on the processing time. month in the script hardcoded as record- use different departments, some could be In the example above, the request ed, the script would be formally correct, mistyped or contain special symbols that was recorded three times, which means but the test won’t make sense at all— must be properly encoded. These prob- the consolidation was done by the each virtual user will try to overwrite lems are easy to spot early on, when moment the third GETCONSOLSTATUS exactly the same data for the same you’re just debugging a particular script. request was sent to the server. If you play department and the same month. To But if you wait until the final, all-script back this script, it will work this way: The make it meaningful, the script should be tests, they muddy the entire picture and script submits the consolidation in the parameterized to save data in different make it difficult to see the real problems. EXECUTE request and then calls GET- data intersections. For example, differ- My group specializes in perform- CONSOLSTATUS three times. If we ent departments may be used by each ance testing of the Hyperion line of have a timer around these requests, the user. To parameterize the script, we 18 • Software Test & Performance NOVEMBER/DECEMBER 2009
  6. 6. need not only department names but erly and the results will be meaningful. In ing—testers look for bugs and log them also department IDs (which are internal most cases, if a performance problem is into a defect tracking system, then the representations not visible to users that found, it should be diagnosed further, up defects are prioritized and fixed inde- should be extracted from the metadata to the point when it’s clear how to han- pendently by the developers—doesn’t repository). Below is a sample of correct dle it. Generally speaking, performance work well for performance testing. First, LoadRunner pseudocode (where values testing, tuning, diagnostics and capacity a reliability or performance problem between { and } are parameters that may planning are quite different processes, often blocks further performance testing be generated from a file): and excluding any one of them from the until the problem is fixed or a web_submit_data(“WebFormGenerated.asp”, test plan (if they’re assumed) will make workaround is found. Second, usually “Action= the test unrealistic from the beginning. the full setup, which tends to be very bFormGenerated.asp?FormName=Tax+QFP&call Both performance tuning and trou- sophisticated, should be used to repro- er=GlobalNav&iscontained=Yes”, bleshooting are iterative processes duce the problem. Keeping the full setup ITEMDATA, “Name=SubmitType”, “Value=1”, where you make the change, run the for a long time can be expensive or even ENDITEM, test, analyze the results and repeat the impossible. Third, debugging perform- “Name=FormPOV”, “Value=TaxQFP”, process based on the findings. The ance problems is a sophisticated diag- ENDITEM, advantage of performance testing is that nostic process usually requiring close “Name=FormPOV”, “Value=2007”, ENDITEM, you apply the same synthetic load, so collaboration between a performance “Name=FormPOV”, “Value=[Year]”, you can accurately quantify the impact engineer running tests and analyzing the ENDITEM, of the change that was made. That results and a developer profiling and “Name=FormPOV”, “Value=Periodic”, ENDITEM, makes it much simpler to find problems altering code. Special tools may be nec- “Name=FormPOV”, “Value= during performance testing than to wait essary; many tools, such as debuggers, {department_name}”, ENDITEM, until they happen in production, when work fine in a single-user environment “Name=FormPOV”, “Value=<Entity workload is changing all the time. Still, but do not work in the multi-user envi- Currency>“, ENDITEM, “Name=FormPOV”, even in the test environment, tuning and ronment, due to huge performance over- “Value=NET_INCOME_LEGAL”, ENDITEM, performance troubleshooting are quite heads. What’s usually required is the “Name=FormPOV”, “Value=[ICP Top]”, ENDITEM, FIG. 1: THROUGHPUT “Name=MODVAL_19.2007.50331648.1. {department_id}.14.407.2130706432. ”, “Value=<1.7e+3>;;”, ENDITEM, “Name=MODVAL_19.2007.50331648.1. {department_id}.14.409.2130706432. ”, “Value=<1.7e+2>;;”, ENDITEM, LAST); If department name is parameter- ized but department ID isn’t, the script won’t work properly. You won’t get an error, but the information won’t be saved. This is an example of a situation that never can happen in real life—users working through GUIs would choose department name from a drop-down box (so it always would be correct) and matching ID would be found automati- cally. Incorrect parameterization leads to sophisticated diagnostic processes usu- synchronized work of performance engi- sending impossible combinations of data ally requiring close collaboration neering and development to fix the prob- to the server with unpredictable results. between a performance engineer run- lems and complete performance testing. To validate this information, we would ning tests and developers and/or sys- check what’s saved after the test—if tem administrators making changes. In BUILD A MODEL you see your data there, you know the most cases, it’s impossible to predict Creating a model of the system under script works. how many test iterations will be neces- test significantly increases the value of sary. Sometimes it makes sense to cre- performance testing. First, it’s one more TUNE AND TROUBLESHOOT ate a shorter, simpler test still exposing way to validate test correctness and help Usually, when people talk about per- the problem under investigation. to identify system problems—deviations formance testing, they don’t separate it Running a complex, “real-life” test on from expected behavior might signal from tuning, diagnostics or capacity each tuning or troubleshooting iteration issues with the system or with the way planning. “Pure” performance testing is can make the whole process very long you create workload. Second, it allows possible only in rare cases when the and the problem less evident because of you to answer questions about the sizing system and all optimal settings are well- different effects the problem may have and capacity planning of the system. known. Some tuning activities are typi- on different workloads. Most performance testing doesn’t cally necessary at the beginning of test- An asynchronous process to fixing require a formal model created by a ing to be sure the system is tuned prop- defects, often used in functional test- sophisticated modeling tool—it may NOVEMBER/DECEMBER 2009 • 19
  7. 7. accurately matches the queuing model FIG. 2: RESPONSE TIME through step 6, where the system CPU usage is 87 percent. Most IT shops don’t want systems loaded more than 70 percent to 80 percent. That doesn’t mean we need to dis- card queuing theory and sophisticated modeling tools; we need them when systems are more complex or where more detailed analysis is required. But in the middle of a short-term performance engineering project, it may be better to build a simple, back-of-the-envelope type of model to see if the system behaves as expected. Running all scripts simultaneously makes it difficult to build a model. While you still can make some predictions for scaling the overall workload proportional- involve just simple observations of the (less so for single-processor machines). ly, it won’t be easy to find out where the amount of resources used by each sys- If there are no bottlenecks, throughput problem is if something doesn’t behave tem component for the specific work- (the number of requests per unit of time), as expected. The value of modeling load. For example, workload A creates as well as processor usage, should increases drastically when your test envi- significant CPU usage on server X while increase proportionally to the workload ronment differs from the production envi- server Y is hardly touched. This means (for example, the number of users) while ronment. In that case, it’s important to that if you increase workload A, the lack response time should grow insignificant- document how the model projects test- of CPU resources on server X will cre- ate a bottleneck. As you run increasing- FIG. 3: CPU USAGE ly complex tests, you verify results you get against your “model”—your under- standing of how the system behaves. If they don’t match, you need to figure out what’s wrong. Modeling often is associated with queuing theory and other sophisticated mathematical constructs. While queuing theory is a great mechanism to build sophisticated computer system models, it’s not required in simple cases. Most good performance engineers and ana- lysts build their models subconsciously, without even using such words or any for- mal efforts. While they don’t describe or document their models in any way, they take note of unusual system behavior— i.e., when system behavior doesn’t match ly. If we don’t see this, it means there’s a ing results onto the production system. the model—and can make some simple bottleneck somewhere in the system and predictions (“It looks like we’ll need X we need to discover where it is. THE PAYOFF additional resources to handle X users,” For example, let’s look at a simple The ultimate goal of applying agile princi- for example). queuing model I built using TeamQuest’s ples to software performance engineer- The best way to understand the modeling tool for a specific workload ing is to improve efficiency, ensuring system is to run independent tests for executing on a four-way server. It was better results under tight deadlines each business function to generate a simulated at eight different load levels and budgets. And indeed, one of the workload resource usage profile. The (step 1 through step 8, where step 1 tenets of the “Manifesto for Agile load should not be too light (so resource represents a 100-user workload and 200 Software Development” (http://agile usage will be steady and won’t be dis- users are added for each step there- is that responsiveness torted by noise) or too heavy (so it won’t after, so that step 8 represents 1,500 to change should take priority over fol- be distorted by nonlinear effects). users). Figures 1 through 3 show lowing a plan. With this in mind, we can Considering the working range of throughput, response time and CPU take performance testing of today’s processor usage, linear models often usage from the modeling effort. increasingly complex software to new lev- can be used instead of queuing models An analysis of the queuing model els that will pay off not just for testers and for the modern multiprocessor machines results shows that the linear model engineers but for all stakeholders. 20 • Software Test & Performance NOVEMBER/DECEMBER 2009