Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Early Performance Testing of Distributed Software Applications


Published on

  • Be the first to comment

  • Be the first to like this

Early Performance Testing of Distributed Software Applications

  1. 1. Early Performance Testing of Distributed Software Applications∗ Giovanni Denaro† Andrea Polini† Wolfgang Emmerich Universita di Milano Bicocca ´ Scuola St. Anna Superiore University College London Dipartimento di Informatica, Piazza Martiri della Liberta 1 Dept. of Computer Science Sistemistica e Comunicazione 56127 Pisa, Italy WC1E 6BT, London, UK I-20126, Milano, Italy ABSTRACT 1. INTRODUCTION Performance characteristics, such as response time, through- Various commercial trends have lead to an increasing de- put and scalability, are key quality attributes of distributed mand for distributed applications. Firstly, the number of applications. Current practice, however, rarely applies sys- mergers between companies is increasing. The different di- tematic techniques to evaluate performance characteristics. visions of a newly merged company have to deliver unified We argue that evaluation of performance is particularly cru- services to their customers and this usually demands an in- cial in early development stages, when important architec- tegration of their IT systems. The time available for deliv- tural choices are made. At first glance, this contradicts ery of such an integration is often so short that building a the use of testing techniques, which are usually applied to- new system is not an option and therefore existing system wards the end of a project. In this paper, we assume that components have to be integrated into a distributed system many distributed systems are built with middleware tech- that appears as an integrating computing facility. Secondly, nologies, such as the Java 2 Enterprise Edition (J2EE) or the the time available for providing new services are decreasing. Common Object Request Broker Architecture (CORBA). Often this can only be achieved if components are procured These provide services and facilities whose implementations off-the-shelf and then integrated into a system rather than are available when architectures are defined. We also note built from scratch. Components to be integrated may have that it is the middleware functionality, such as transaction incompatible requirements for their hardware and operat- and persistence services, remote communication primitives ing system platforms; they have to be deployed on different and threading policy primitives, that dominate distributed hosts, forcing the resulting system to be distributed. Fi- system performance. Drawing on these observations, this nally, the Internet provides new opportunities to offer prod- paper presents a novel approach to performance testing of ucts and services to a vast number of potential customers. distributed applications. We propose to derive application- The required scalability of e-commerce or e-government sites specific test cases from architecture designs so that perfor- cannot usually be achieved by centralized or client-server ar- mance of a distributed application can be tested using the chitectures but demand the use of distributed software ar- middleware software at early stages of a development pro- chitectures. cess. We report empirical results that support the viability of the approach. Given the growing importance of distributed systems, we are interested in devising systematic ways to ascertain that Keywords a given distributed software architecture meets performance Middleware, Software Testing, Distributed Software Archi- requirements of their users. Performance can be charac- tecture, Software Performance, Performance Analysis Mod- terised in several different ways. Latency typically describes els the delay between request and completion of an operation. Throughput denotes the number of operations that can be † The research was conducted while this author was on study completed in a given period of time. Scalability identifies leave at UCL the dependency between the number of distributed system ∗This work has been partially supported by the European resources that can be used by a distributed application (typ- Union through the Research Training Network SeGraVis. ically number of hosts or processors) and latency or through- put. Despite the practical significance of these various as- pects it is still not adequately understood how to test the performance of distributed applications. Weyuker and Vokolos reported on the weakness of the pub- lished scientific literature on software performance testing in [24]. To this date no significant scientific advances have been made on performance testing. Furthermore the set of tools available for software performance testing is fairly limited. The most widely used tools are workload genera-
  2. 2. tors and performance profilers that provide support for test databases usually contain the software for transaction and execution and debugging, but they do not solve many un- persistence management, remote communication primitives clear aspects of the process of performance testing. In par- and threading policies, which have great impact on the dif- ticular, researchers and practitioners agree that the most ferent aspects of performance of distributed systems. How- critical performance problems depend on decisions made in ever, we note that only the coupling between the middle- the very early stages of the development lifecycle, such as ware and the application architecture determines the actual architectural choices. Even though iterative and incremen- performance. The same middleware may perform very dif- tal development has been widely promoted [18, 11, 12], the ferently in the context of different applications. Based on testing techniques developed so far are very much focused these observations, we propose using architecture designs to on the end of the development process. derive application-specific performance test cases that can be executed on the early available middleware platform a As a consequence of the need for early evaluation of soft- distributed application is built with. We argue that this ware performance and the weakness of testing, the great allows empirical measurements of performance to be suc- majority of research has focused on performance analysis cessfully done in the very early stages of the development models [1, 20, 19, 2, 3, 9] rather than testing techniques. process. Furthermore, we envision an interesting set of prac- This research shares in general the approach of translating tical applications of this approach, that is: evaluation and architecture designs, mostly given in the Unified Modeling selection of middleware for specific applications; evaluation Language (UML [5]), to models suitable for analyzing per- and selection of off-the-shelf components; empirical evalua- formance, such as, Layered Queuing Networks (e.g. [19]), tion and comparison of possible architectural choices; early Stocastic Petri Nets (e.g. [2]) or stocastic process algebras configuration of applications; evaluation of the impact of (e.g. [20]). Estimates of performance are used to reveal flaws new components on the evolution of existing applications. in the original architecture or to compare different archi- tectures and architectural choices. Models may give use- The paper is further structured as follows. Section 2 dis- ful hints of the performance and help identify bottlenecks, cusses related work and highlights the original aspects of however they tend to be rather inaccurate. Firstly, models our research. Section 3 gives details of our approach to per- generally ignore important details of the deployment envi- formance testing. Section 4 reports about the results of an ronment. For example, performance differences may be sig- empirical evaluation of the main hypothesis of our research, nificant when different databases or operating systems are i.e., that the performance of distributed application can be used, but the complex characteristics of specific databases successfully measured based on the early-available compo- and operating systems are very seldom included in the mod- nents. Finally, Section 5 summarizes the contributions of els. Secondly, models often have to be tuned manually. For the paper and sketches our future research agenda. example, in the case of Layered Queued Networks, solving contention of CPU(s) requires, as input, the number of CPU 2. RELATED WORK cycles that each operation is expected to use. Tuning of this In this section, we briefly review related work in the areas type of parameters is usually guessed by experience and as of performance testing of distributed applications and stud- a result it is not easy to obtain precise models. ies on the relationships between software architecture and middleware. With the recent advances in distributed component tech- nologies, such as J2EE [22] and CORBA [17], distributed systems are no longer built from scratch [7]. Modern dis- Performance testing of distributed applications Some authors exploited empirical testing for studying the tributed applications often integrate both off-the-shelf and performance of middleware products. Gorton and Liu com- legacy components, use services provided by third-parties, pare the performance of six different J2EE-based middle- such as real-time market data provided by Bloomberg or wares [10]. They use a benchmark application that stresses Reuters, and rely on commercial databases to manage per- the middleware infrastructure, the transaction and directory sistent data. Moreover, they are built on top of middleware services and the load balancing mechanisms. The compar- products (hereafter referred to as middlewares), i.e., middle- ison is based on the empirical measurement of throughput tier software that provides facilities and services to simplify per increasing number of clients. Similarly, Avritzer et al. distributed assembly of components, e.g., communication, compare the performance of different ORB (Object Request synchronization, threading and load balancing facilities and Broker) implementations that adhere to the CORBA Com- transaction and security management services [8]. As a re- ponent Model [13]. Liu and al. investigate the suitability sult of this trend, we have a class of distributed applications of micro-benchmarks, i.e., light-weight test cases focused on for which a considerable part of their implementation is al- specific facilities of the middlewares, such as, directory ser- ready available when the architecture is defined (e.g. during vice, transaction management and persistence and security the Elaboration phase of the Unified Process). In this pa- support [14]. per, we argue that this enables performance testing to be successfully applied at these early stages. This work suggests the suitability of empirical measurement for middleware selection, i.e, for making decisions on which The main contribution of this paper is the description and middleware will best satisfy the performance requirements evaluation of a method for testing performance of distributed of a distributed application. However, as Liu et al. remark software in the early development stages. The method is in the conclusions of their paper ([14]), “how to incorpo- based on the observation that the middleware used to build rate application-specific behavior in the equations and how a distributed application often determines the overall per- far the results can be generalized across different hardware formance of the application. For example, middlewares and platforms, databases and operating systems, are still open
  3. 3. problems.” Our research tackles these problems. We study 2. Mapping of the selected use-cases to the actual deploy- application-specific test cases for early performance evalu- ment technology and platform. ation (or also comparing) the performance of distributed 3. Generation of stubs of components that are not avail- applications in specific deployment environments, which in- able in the early stages of the development life cycle, clude middlewares, databases, operating systems and other but are needed to implement the use cases. off-the-shelf components. 4. Execution of the test, which in turn includes: deploy- Weyuker and Vokolos report on the industrial experience ment of the Application Under Test (AUT), creation of testing the performance of a distributed telecommunica- of workload generators, initialization of the persistent tion application at AT&T [24]. They stress that, given the data and reporting of performance measurements. lack of historical data on the usage of the target system, the architecture is key to identify software processes and input parameters (and realistic representative values) that We now discuss the research problems and our approach will most significantly influence the performance. to solving them for each of the above phases of the testing process. Our work extends this consideration to a wider set of dis- tributed applications, i.e., distributed component-based soft- Selecting performance use-cases ware in general. Moreover, we aim to provide a systematic The design of test suites for performance testing is radically approach to test-definition, implementation and deployment different from the case of functional testing. (This has been that are not covered in the work of Weyuker and Vokolos. noticed by many authors, e.g. [24].) In performance testing, the functional details of the test cases, i.e., the actual val- ues of the inputs, are generally of little importance. Table 1 Software architecture and middleware classifies the main parameters relevant to performance test- Medvidovic, Dashofy and Taylor state the idea of coupling ing of distributed applications. First, important concerns the modeling power of software architectures with the im- are traditionally associated with workloads and physical re- plementation support provided by middleware [15]. They sources, e.g., the number of users, the frequencies of inputs, notice that “architectures and middleware address similar the duration of the test, the characteristics of the disks, the problems, that is large-scale component-based development, network bandwidth and the number and speed of CPU(s). but at different stages of the development lifecycle.” They Next, it is important to consider the middleware configura- propose to investigate the possibility of defining systematic tion, for which the table reports parameters in the case of mappings between architectures and middlewares. To this J2EE-based middlewares. Here, we do not comment further end, they study the suitability of a particular element of soft- on workload, physical resource and middleware parameters, ware architecture, the software connector. Metha, Phadke which are extensively discussed in the literature (e.g., [24, and Medvidovic himself moreover present an interesting tax- 23, 14]). onomy of software connectors [16]. Workload Number of clients Although they draw on similar assumptions (i.e., the re- Client request frequency lationships between architecture and middleware), our re- Client request arrival rate search and that of Medvidovic et al. have different goals: Duration of the test We aim at measuring performance attributes of an architec- Physical Number and speed of CPU(s) ture based on the early available implementation support resources Speed of disks (which the middleware is a significant part of); Medvidovic Network bandwidth and colleagues aim at building implementation topologies Middleware Thread pool size (e.g., bridging of middlewares) that preserve the properties configuration Database connection pool size of the original architecture. However, the results of pre- Application component cache size vious studies on software connectors and the possibility of JVM heap size mapping architectures on middlewares may be important for Message queue buffer size engineering our approach. Message queue persistence Application Interactions with the middleware 3. EARLY TESTING OF PERFORMANCE specific - use of transaction management In this section, we introduce our approach to early perfor- - use of the security service mance testing of distributed component-based applications. - component replication We also focus on the aspects of the problem that need fur- - component migration ther investigation. Our long-term goal is to realize a soft- Interactions among components ware tool that supports the application of the approach we - remote method calls describe below. - asynchronous message deliveries Interactions with persistent data Our approach comprises a performance testing process that - database accesses consists of the following phases: Table 1: Performance parameters 1. Selection of the use-case scenarios (hereafter referred to simply as use-cases) that relevant to performance, Other important parameters of performance testing in dis- given a set of architecture designs. tributed settings can be associated with interactions among
  4. 4. distributed components and resources. Different ways of To address the mapping of abstract use-cases to specific de- using facilities, services and resources of middlewares and ployment technologies and platforms, we intend to leverage deployment environments likely correspond to different per- recent studies on software connectors [16, 15]. Well charac- formance results. Performance will differ if the database is terized software connectors may be associated with deploy- accessed many times or never. A given middleware may per- ment topologies that preserve the properties of the original form adequately for applications that stress persistence and architecture. Also, possible implementation alternatives can quite badly for transactions. In some cases, middlewares be defined in advance as characteristics of the software con- may perform well or badly for different usage patterns of nectors. For instance, with reference to example above, we the same service. Application-specific cases for performance would like to define a component creation connector that should be given such that the most relevant interactions provides properties for the case showed in Figure 1(a). Then, specifically triggered by the AUT are covered. we would like to associate with this connector, the J2EE de- ployment topology given in Figure 1(b) and the list of the The last row of Table 1 classifies the relevant interactions possible transaction behaviors. Deployment topologies and in distributed settings according to whether they take place alternatives might vary for different deployment technologies between the middleware and the components, among the and platforms. Previous studies on software connectors are components themselves1 or to access persistent data in a precious references in this direction, but further work is still database. This taxonomy is far from complete and we are needed to understand the many dimensions and species of conducting further studies and empirical assessments in this software connectors and their relationships with the possible direction. However, we believe that a taxonomy of dis- deployment technologies and platforms. tributed interactions is key for using our approach. Based on such a taxonomy, it will be possible to evaluate the Generating stubs performance-relevance of the available use-cases according So far, we have suggested that early test cases of perfor- to the interactions that they trigger. This will allow the se- mance can be derived from instances of architecture use- lection of sets of use-cases adequately representative for the cases and that software connectors can be exploited as a performance of the target application. means to establish the correspondence between the abstract views provided by the use-cases and the concrete instances. Mapping use-cases to middleware However, to actually implement the test cases, we must also In the initial stages of a software process, software architec- solve the problem that not all the application components tures are generally defined at a very abstract level. Often, that participate in the use-cases are available in the early they just describe the business logic and abstract many de- stages of the development lifecycle. For example, the com- tails of deployment platforms and technologies. One of the ponents that implement the business logic are seldom avail- expected strengths of our approach is indeed the possibility able, although they participate in most of the use-cases. Our of driving software engineers through the intricate web of approach uses stubs in place of the missing components. architectural choices, technologies and deployment options, keeping the focus on the performance of the final product. Stubs are fake versions of components that can be used in- To this end, however, we need to understand how abstract stead of the corresponding components for instantiating the use-cases are mapped to possible deployment technologies abstract use-cases. In our approach, stubs are specifically and platforms. adjusted to use-cases, i.e., different use-cases will require different stubs of the same component. Stubs will only take Let us for example consider the use-case fragment in Fig- care that the distributed interactions happen as specified ure 1(a), which represents in UML the creation of a compo- and the other components are coherently exercised. Our nent C of type Component issued by a client with the mes- idea of the engineered approach is that the needed stubs sage create. The deployment of such use-case on a J2EE- are automatically generated based on the information con- based middleware will likely map on a component (namely, tained in use-cases elaborations and software connectors. Of an Enterprise Java Bean) with the interactions shown in course, many functional details of the missing components Figure 1(b). The message create from the client will be ac- are not known and cannot be implemented in the stubs. This tually delivered to a component container that is part of the results in discrepancies between execution times within the middleware. The middleware will then dispatch the request stubs and the actual components that they simulate. to the actual component C with two messages, create and post create, sent respectively before and after the initial- The main hypothesis of our work is that performance mea- ization of the support structures for C within the middle- surements in the presence of the stubs are good enough ap- ware. Optionally, the client will use the directory service proximations of the actual performance of the final appli- provided by the middleware, through the Java Naming and cation. This descends from the observation that the avail- Directory Interface (JNDI), to retrieve the component fac- able components, e.g., middlewares and databases, embed tory responsible for creating C. Furthermore, we ought to the software that mainly impact performance. The coupling specify the transactional behavior, e.g., whether the method between such implementation support and the application- will execute in the context of the client transaction or a specific behavior can be extracted from the use-cases, while newly created (nested) transaction. the implementation details of the business components re- 1 main negligible. In other words, we expect that the discrep- Although interactions among distributed components map on interactions that take actually place at the middleware ancies of execution times within the stubs are orders of mag- level, they are elicited at a different abstraction level and nitude less than the impact of the interactions facilitated by thus they are considered as a different category in our clas- middleware and persistence technology, such as databases. sification. We report a first empirical assessment of this hypothesis in
  5. 5. Container C:Component JDNI C:Component create lookup factory create create post_create (a) (b) Figure 1: An abstract use-case (a) and its deployment view (b) Section 4 of this paper, but are aware that further empirical EJB Container studies are needed. Session Entity Database Beans Beans Table Executing the test Accounts Account Bean Accounts Web Controller Building the support to test execution shall mostly involve Client technical rather than scientific problems, at least once the research questions stated above have been answered. Part of the work consists of engineering the activities of select- Customers Customer Bean Customers Controller ing the use-cases, mapping them to deployment technologies and platforms, and generating the stubs to replace missing J2EE Application components. Also, we must automate deployment and im- Client plementation of workload generators, initialization of per- Transactions Transaction Bean Transactions sistent data, execution of measurements and reporting of Controller results. No relevant aspects for research are directly con- nected with this. Figure 2: The Duke’s Bank application. 4. PRELIMINARY ASSESSMENT This section empirically evaluates the core hypothesis of our research, i.e., that the performance of a distributed appli- consider the Duke’s bank application to be adequately rep- cation can be successfully tested based on the middleware resentative of medium-size component-based distributed ap- and/or off-the-shelf components that are available in the plications. The Duke’s bank application is referred to as early stages of the software process. To this end, we con- DBApp in the remainder of this paper. ducted an experiment in a controlled environment. First, we considered a sample distributed application for which we The organization of the DBApp is given in Figure 2 (bor- had the whole implementation available. Then, we selected rowed from [4]). The application can be accessed by both an abstract use-case of the application and implemented it Web and application clients. It consists of six EJB (Enter- as a test case based on the approach described in Section 3. prise Java Beans [22]) components that handle operations Finally, we executed the performance test (with different issued by the users of a hypothetic bank. The six compo- amounts of application clients) on the early available compo- nents can be associated with classes of operations that are nents and compared the results with the performance mea- related to bank accounts, customers and transactions, re- sured on the actual application. spectively. For each of these classes of operations a pair of session bean and entity bean is provided. Session beans are responsible for the interface towards the users and the en- Experiment setting tity beans handle the mapping of stateful components to un- As for the target application, we considered the Duke’s Bank derlying database table. The arrows represent the possible application presented in the J2EE tutorial [4]. This appli- interaction patterns among the components. All EJBs are cation is distributed by Sun under a public license, thus deployed in a single container within the application server we were able to obtain the full implementation easily. The (which is part of the middleware). For the experiment we Duke’s bank application consists of 6,000 lines of Java code used the JBoss application server and the MySql database that is meant to exemplify all the main features of the J2EE engine, running on the same machine. platform, including the use of transactions and security. We
  6. 6. Finally, we implemented a workload generator and initial- Duke’s Bank Database ized the persistent data in the database. The workload gen- erator is able to activate a number of clients at the same time and takes care of measuring the average response time. For Transfer(account_from, the persistent data, we instantiated the case in which each account_to, Update account_from amount) client withdraws money from its own account (i.e., there Update account_to exists a bank account for each client) and deposits the cor- responding amount to the account of a third part, which is supposed to be the same for all clients. This simulates the Log transaction of account_from recurrent case in which a group of people is paying the same authority over the Internet. Incidentally, we notice that, in Log transaction of account_to an automated test environment, initialization of persistent data would only require to specify the performance sensi- ble part of the information, while the actual values in the database tables are generally of little importance. For ex- Figure 3: A sample use-case for the Duke’s Bank. ample, in our case, only the number of elements in each table and the relationships with the instanced use-case, i.e., whether all clients access the same or a different table row, Then, we selected a sample use-case that describes the trans- are the real concerns. fer of funds between two bank accounts. Figure 3 illustrates the selected use-case in UML. A client application uses the With reference to the performance parameters of Table 1, service Transfer provided by the DBApp. This service re- we generated a workload, to test both DBApp and DBTest, quires three input parameters, representing the two accounts with increasing numbers of clients starting from one to one and the amount of money, respectively involved in the trans- hundred. The two applications were deployed on a JBoss 3.0 fer. The business components of the DBApp realize the ser- application server running on a PC equipped with a Pentium vice using the database for storing the persistent data: the III CPU at 1 G-Hz, 512 M-Byte of RAM memory and the database is invoked four times, for updating the balances Linux operating system. To generate the workload we run of the two accounts and recording the details of the corre- the clients on a Sun Fire 880 equipped with 4 Spark CPU at sponding transactions. We assume that the database engine 850 MHz and 8G-Byte of RAM memory. This two machine is software that is available early in the software process. were linked to a controlled local area network at 100 M-Bit. Thus, for the test, we used the same database engine, ta- Moreover, in order to avoid influences among the experi- ble structure and SQL code than in the original application. ments that could be caused by the contemporary existence This is why we represented the database as a shadowed box of a lot of active sessions beans, we restarted the applica- in the figure. Differently from the database, the business tion server between two successive experiments. JBoss has components of the DBApp are assumed to be not available, been used running the default configuration. Finally the thus we had to generate corresponding stubs. specific setting concerning the particular use case, has al- ready discussed in the previous paragraphs, foresaw the use For implementing the stubs, we had to map the abstract use- of remote method calls between the components and the use case on the selected deployment technology, namely J2EE. of the transaction management service, in order to handle, We already commented on the role that software connectors consistently, the data shared by the various bean. may play in the mapping. However, our research is not yet mature enough to exploit software connectors for this pur- pose. Presently, we just manually augmented the use-case Experiment results We have executed both DBApp and DBTest for increasing with the necessary information as follows. As for the inter- numbers of clients and measured the latency for the test action between the clients and the DBApp, we specified that case. We repeated each single experiment 10 times and av- the service Transfer is invoked as a synchronous call and eraged the results. The platform for the experiment was a starts a new transaction. As for the interaction between the PC equipped with a Pentium III CPU at 1 G-Hz, 512 M- DBApp and the database, we specified that: the four invoca- Byte of RAM memory and the Linux operating system. We tions are synchronous calls that take place in the context of also used a private network at 100 M-Bit to avoid the bias a single transaction and embed the available SQL code; the of the network traffic. database connection is initialized for each call; the DBApp uses entity beans and bean managed persistence2 to handle We repeated each single experiment 15 times and measured the interactions with the database tables. Based on this in- the average latency time. Figure 4 shows the results of the formation, we implemented the stubs as needed to realize experiments. It plots the latency time of both DBApp and the interactions in the considered use-case and we deployed DBTest against the number of clients, for all the repetitions the test version of the DBApp (referred to as DBTest) on of the experiment. We can see that the two curves are very the JBoss application server. near to each other. The average difference accounts for the 2 Entity beans are J2EE components that represent persis- 9.3% of the response time. The experiments also showed a tent data within an application. Each database table is as- low value for the standard deviation. The ratio between σ sociated to an entity bean. The data in the entity bean are and the expectation results, in fact, definitively lower of the taken synchronized with the database. In the case of bean managed persistence the synchronization code is embedded 0.15, both for the DBApp and for the DBTest. in the entity bean. The results of this experiment suggest the viability of our
  7. 7. finally reported on experiments that show as the actual per- formance of a sample distributed application is well approx- imated by measurements based only on its early available components, thus supporting the main hypothesis of our re- search. Software performance testing of distributed applications has not been thoroughly investigated so far. The reason for this is, we believe, that testing techniques have traditionally been applied at the end of the software process. Conversely, the most critical performance faults are often injected very early, because of wrong architectural choices. Our research tack- les this problem suggesting a method and a class of appli- cations such that software performance can be tested in the very early stages of development. In the long term and as far as the early evaluation of software architectures and ar- chitectural choices are concerned, we believe that empirical testing may outperform performance estimation models, be- Figure 4: Latency of DBApp and DBTest for in- ing the former more precise and easier to use. Moreover, we creasing numbers of clients envision the application of our ideas to a set of interesting practical cases: research because they witness that the performance of the DBApp in a specific use-case is well approximated by the Middleware selection: The possibility of evaluating and DBTest, which is made out of the early-available compo- selecting the best middleware for the performance of nents. However, although the first results are encouraging, a specific application is reckoned important by many we are aware that a single experiment cannot be generalized. authors, as we already pointed out in Section 2 of this We are now working on other experiments to cover the large paper. To this end, our approach provides a valuable set of alternatives of component-based distributed applica- support. Based on the abstract architecture designs, tions. We plan to experiment with different use-cases, sets it allows to measure and compare the performance of a of use-cases for the same test case, different management specific application for different middleware and mid- schemas for transactions and performance, different commu- dleware technologies. nication mechanisms such as asynchronous call, J2EE-based COTS selection: A central assumption of traditional test- application server other than JBoss, CORBA-based middle- ing techniques is that testers have complete knowledge wares, other commercial databases and in the presence of of the software under test as well as of its requirements other early-available components. and execution environment. This is not the case for components off-the-shelf (COTS) that are produced 5. CONCLUSIONS AND FUTURE WORK independently and then deployed in environments not Distributed component technologies enforce the use of mid- known in advance. Producers may fail in identifying dleware, commercial databases and other off-the-shelf com- all possible usage profiles of a component and there- ponents and services. The software that implements these fore testing of the component in isolation (performed is available in the initial stages of a software process and by producers) is generally not enough [21]. Limited to moreover it generally embeds the software structures, mech- the performance concerns, our approach allows to test anisms and services that mostly impact the performance in off-the-shelf components in the context of a specific ap- distributed settings. This paper proposed to exploit the plication that is being developed. Thus, it can be used early availability of such software to accomplish empirical to complement the testing done by COTS providers measurement of performance of distributed applications at and thus assist in selecting among several off-the-shelf architecture-definition-time. To the best of our knowledge, components. the approach proposed in this paper is novel in software performance engineering. Iterative development: Modern software processes pre- scribe iterative and incremental development in order This paper fulfilled several goals. It discussed the published to control risks linked to architectural choices (see e.g., scientific works related to ours, thus positioning our ideas the Unified Process [6]). Applications are incremen- in the current research landscape. It described a novel ap- tally developed in a number of iterations. During an proach to performance testing that is based on selecting per- iteration, a subset of the user requirements is fully im- formance relevant use-cases from the architecture designs, plemented. This results in a working slice of the appli- and instantiating and executing them as test cases on the cation that can be presently evaluated and, in the next early available software. It indicated important research di- iteration, extended to cover another part of the miss- rections towards engineering such approach, i.e.: The classi- ing functionality. At the beginning of each iteration, fication of performance-relevant distributed interactions as new architectural decisions are generally made whose a base to select architecture use-cases; The investigation of impact must be evaluated with respect to the current software connectors as a mean to instantiate abstract use- application slice. For performance concerns, our ap- cases on actual deployment technologies and platforms. It proach can be used when the life cycle architecture is
  8. 8. established during the elaboration phase, because it [11] I. Jacobson, G. Booch, and J. Rumbaugh. The Unified allows to test the expected performance of a new soft- Software Development Process. Addison Wesley ware architecture based on the software that is initially Longman, Reading, MA, USA, 1999. available. [12] P. Kruchten. The Rational Unified Process: An Introduction. Addison Wesley Longman, 2000. [13] C. Lin, A. Avritzer, E. Weyuker, and L. Sai-Lai. We are now continuing the experiments for augmenting the Issues in interoperability and performance verification empirical evidence of the viability of our approach, providing in a multi-orb telecommunications environment. In a wider coverage of the possible alternatives of component- Proceedings of the International Conference on based distributed applications. We are also working for engi- Dependable Systems and Networks (DSN 2000), pages neering the approach, starting from the study of the research 567–575, 2000. problems outlined in this paper. [14] Y. Liu, I. Gorton, A. Liu, N. Jiang, and S. Chen. Designing a test suite for empirically-based 6. ACKNOWLEDGMENTS middleware performance prediction. In Fortieth The authors would like to thank James Skene for his con- International Conference on Technology of tribution during the preliminary definition of the research Object-Oriented Languages and Systems (TOOLS ideas presented in this paper. Pacific 2002), Sydney, Australia, 2002. ACS. [15] N. Medvidovic, E. Dashofy, and R. Taylor. On the role 7. REFERENCES of middleware in architecture-based software [1] S. Balsamo, P. Inverardi, and C. Mangano. An development. International Journal of Software approach to performance evaluation of software Engineering and Knowledge Engineering, 13(4), 2003. architectures. In Proceedings of the First International To appear. Workshop on Software and Performance, pages [16] N. Mehta, N. Medvidovic, and S. Phadke. Towards a 178–190, 1998. taxonomy of software connectors. In Proceedings of the [2] S. Bernardi, S. Donatelli, and J. Merseguer. From 22nd International Conference on Software UML sequence diagrams and statecharts to analysable Engineering (ICSE-00), pages 178–187. ACM Press, Petri Nets models. In Proceedings of the 3rd 2000. International Workshop on Software and [17] P. Merle. CORBA 3.0 new components chapters. Performance(WOSP02), pages 35–45, 2002. Technical report, TC Document ptc/2001-11-03, [3] A. Bertolino, E. Marchetti, and R. Mirandola. Object Management Group, 2001. Real-time UML-based performance engineering to aid [18] H. D. Mills. Top-Down Programming in Large manager’s decisions in multi-project planning. In Systems. In R. Ruskin, editor, Debugging Techniques Proceedings of the 3rd International Workshop on in Large Systems. Prentice Hall, 1971. Software and Performance (WOSP-02), pages [19] D. Petriu, C. Shousha, and A. Jalnapurkar. 251–261, New York, July 24–26 2002. ACM Press. Architecture-based performance analysis applied to a [4] S. Bodoff et al. The J2EE Tutorial. Addison-Wesley, telecommunication system. IEEE Transactions on 2002. Software Engineering, 26(11):1049–1065, 2000. [5] G. Booch, J. Rumbaugh, and I. Jacobson. The Unified [20] R. Pooley. Using UML to derive stocastic process Modeling Language User Guide. Addison-Wesley, 1999. algebra models. In Proceedings of the 15th UK [6] G. Booch, J. Rumbaugh, and I. Jacobson. The Unified Performance Engineering Workshop (UKPEW), pages Software Development Process. Addison-Wesley, 1999. 23–34, 1999. [7] W. Emmerich. Software engineering and middleware. [21] D. Rosenblum. Challenges in exploiting architectural In Proceedings of the 22th International Conference on models for software testing. In Proceedings of the Software Engineering (ICSE-00), pages 117–132. ACM International Workshop on the Role of Software Press, 2000. Architecture in Testing and Analysis (ROSATEA), [8] W. Emmerich. Distributed component technologies 1998. and their software engineering implications. In [22] B. Shannon. Java 2 platform enterprise edition Proceedings of the 24th International Conference on specification, 1.4 - proposed final draft 2. Technical Software Engineering (ICSE-02), pages 537–546. ACM report, Sun Microsystems, Inc., 2002. Press, 2002. [23] B. Subraya and S. Subrahmanya. Object driven [9] W. Emmerich and J. Skene. Model driven performance testing of Web applications. In performance analysis of enterprise information Proceedings of the First Asia-Pacific Conference on systems. In Proceedings of the International Workshop Quality Software (APAQS’00), 2000. on Testing and Analysis of Component-Based [24] E. Weyuker and F. Vokolos. Experience with Systems(TACOS’03), 2003. performance testing of software systems: issues, an [10] I. Gorton and A. Liu. Software component quality approach, and case study. IEEE Transactions on assessment in practice: successes and practical Software Engineering, 26(12):1147–1156, 2000. impediments. In Proceedings of the 24th International Conference on Software Engineering (ICSE-02), pages 555–558, New York, 2002. ACM Press.