Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Performance Prediction for Software                           Architectures                        Evgeni Eskenazi, Alexan...
software performance prediction is a promising                evaluation of software performance. In [6], an approachdirec...
A performance prediction model is created by means             and its performance, it is also advisable to construct aof ...
1. Use cases definition                                                           6. Use cases definition (new )          ...
D. Virtual Service Platform identification                   statistics, and the cross-validation “leave-one-out”  The abs...
relevant input/diversity parameters and the number of                                   REFERENCESperformance relevant cal...
Upcoming SlideShare
Loading in …5

Performancepredictionforsoftwarearchitectures 100810045752-phpapp02


Published on

  • Be the first to comment

  • Be the first to like this

Performancepredictionforsoftwarearchitectures 100810045752-phpapp02

  1. 1. Performance Prediction for Software Architectures Evgeni Eskenazi, Alexandre Fioukov, Dieter K. Hammer Department of Mathematics and Computing Science, Eindhoven University of Technology, Postbox 513, 5600 MB Eindhoven, The Netherlands Phone: +31 (0) 40 – 247 E-mail: { e.m.eskenazi, a.v.fioukov, d.k.hammer } Abstract— The quantitative evaluation of certain quality i.e. decomposition of the architecture into smaller partsattributes— performance, timeliness, and reliability— is and reasoning about the necessary quality aspects,important for component-based embedded systems. We starting from the lowest level. We started bypropose an approach for the performance estimation of investigating how far we could push the limits of state-component-based software that forms a product family.The proposed approach, Analysis and Prediction of of-the-art analytic methods. Since we did thisPerformance for Evolving Architectures (APPEAR), investigation in an industrial setting, we quickly ranemploys both structural and stochastic modeling against the following basic limitations:techniques. The former are used to reason about the • Analytical methods are based on an analysis of allproperties of components, while the latter allow one to possible execution traces. The high complexity ofabstract from irrelevant details of the execution software for product families, with hundreds ofarchitecture. The method consists of two main parts: (1)calibrating a statistical regression model by measuring parameters influencing the software qualities,existing applications and (2) using the calibrated model to causes these approaches to fail. Accounting for allpredict the performance of new applications. Both parts performance critical details of the software andare based on a model of the application in order to attempting to reason in an analytical way leads todescribe relevant execution properties in terms of a so- a combinatorial explosion.called signature. A predictor that is determined bystatistical regression techniques is used to relate the values • Analytical methods are mainly suitable forof the signature to the observed or predicted performance investigating the WCET/BCET (Worst-Case/Best-measures. APPEAR supports the flexible choice of Case Execution Time). Especially the WCET issoftware parts that need structural modeling and ones that important for safety-critical systems. However,statistical modeling. Thereby it is assumed that the latter for a typical consumer electronics or professionalare not seriously modified during the software evolution. system, the architects are usually more interested The suggested approach is being validated with two in the average performance, because the limitsindustrial case studies in the Consumer Electronics andProfessional Systems domain. might not be representative at all. • Analytical methods rely on models of the Keywords— Performance prediction; Embedded hardware resources to estimate the WCET/BCET.Systems; Software architecture Due to the non-determinism in modern computing facilities— caches, pipelines, and branch predictors— the analytical methods often result in I. INTRODUCTION over-pessimistic estimates. During the past years, the complexity and the amount One of the possible solutions for the aforementionedof software, especially within product families for problems can be the use of the statistical techniques likeembedded systems, has grown significantly. regression analysis. Such an analysis allows one toUnfortunately, many existing approaches turned out to construct a statistical predictor, based on thebe not suitable for the evaluation of the quality attributes measurements on the existing parts of the software, and(e.g. performance) of the entire software system. use it for the prediction of the quality attributes of newly This also holds for the popular analytical approaches, developed parts. The use of regression techniques for
  2. 2. software performance prediction is a promising evaluation of software performance. In [6], an approachdirection, since less and less software is created from for the generation of Petri nets from UML collaboration-scratch. There always exists an initial software stack and statechart-diagrams is proposed. These Petri nets are(reusable components, previous versions, etc.) that can then used to estimate different performancebe used for measurements and predictor training. characteristics. The statistical approach abstracts from the details of An example for the use of the regression techniques isthe system. However, this abstraction can cause other presented in [4]. In this approach, the results of softwareproblems like decreased accuracy of the prediction and profiling are used for the prediction of softwareexcessive time for measurements and construction of the reliability.predictor. Thus, the relevant details should be includedexplicitly into the approach to shorten the predictor III. REQUIREMENTSconstruction time and to raise the accuracy. The aim of the APPEAR method is the support of As a compromise, the mix of analytical and statistical architects in analyzing the performance of newtechniques for the performance evaluation is considered. applications during the early phases of productThis approach is based on the knowledge of the development.application structure and the use of statistical methods in In this paper, the performance is considered in termsorder to abstract from irrelevant architectural details. of CPU utilization and end-to-end response time of an The paper is structured as follows. Section 2 application for different use cases.summarizes related work. In Section 3, the requirements The essential requirements for the APPEAR methodfor the APPEAR method are given. Section 4 describes are the following:the basic constituents and essential steps of the method. 1. Allow performance prediction of a new softwareSections 5 presents the results of building the part or of a complete software system where someperformance prediction model for a part of a Medical parts are added or modified.Imaging software system. Finally, Section 6 concludes 2. Allow the localization of performance bottlenecksthe paper and sketches future work. by giving insight into the execution architecture of the software. II. RELATED WORK 3. Ensure a reasonable level of accuracy for During the past decade, significant research effort was performance prediction. The accuracy level isput into the performance-engineering domain. The main product family dependent. A survey revealed thatinvestigations were aimed at the development of architects consider an accuracy of 50% to 80% asmethods for the performance estimation of software- a definite improvement with respect to theintensive systems and defining the theoretical basis for presently used performance engineering [7]. One of the most 4. The method should be much faster thancritical issues in software architecting is early implementation and subsequent measurements ofperformance estimation, based on architectural the system.descriptions or executable prototypes. The classical approaches [7] use queuing network IV. APPEAR METHODmodels, derived from the structural description of the This section sketches the APPEAR method and drawsarchitecture, and performance-critical use cases. A some assumptions that enable its application.similar approach that also includes specific architecture A. Notion of signaturedescription styles is presented in [1]. A remarkable toolfor the transformation of software architecture The signature of an application is a set of parametersdescriptions into queuing networks and the subsequent that provide sufficient information for performanceperformance analysis is described in [8]. estimation. An interesting approach is proposed in [5]. The We treat the performance as a function over theexecutable prototype (a simulation model) generates signature:traces that are expressed in a specific syntax (angio- P:S →C .traces). These traces are used for building performance In this formula, S = {S1 , S 2 , , S N } is a signatureprediction models, based on layered queuing networks. vector with parameters Si , and C is a performance Stochastic Petri nets are also widely used for the metric, like response time.
  3. 3. A performance prediction model is created by means and its performance, it is also advisable to construct aof statistical regression analysis techniques (see e.g. [2] high-level executable model of an application. Such aand [3]). These techniques define the relation between a simulation model must capture relevant executionsignature of an application and a performance estimate properties of an application. Relevant executionby discovering the correlation between these two properties are those that have a significant impact on theentities. Subsequently, this correlation can be used to performance, e.g. the most time consuming service callsextrapolate to new signature values and to predict the and important input/diversity parameters. Theseperformance of new applications. execution properties are said to form the signature of an Indentifying the signature needs answering the application.following questions: App lication 1 App licatio n 2 App lication 3 1. Which of the hundreds of parameters have the Variable strongest impact on the performance? 2. What is the quantitative dependency between S = {S 1 , S 2 , , S5 } S i − service i these parameters and the performance? S1 S2 S3 S4 S5 Answering the first question helps to reduce the Virtu al Service Platform S tableparameter space and to concentrate on the criticalparameters only, while answering the second questionallows one to predict the performance based on the Figure 1. Applications and Virtual Service Platform (VSP).experimental data. An example of the signature of a hypothetical The proposed method includes two main parts: (1)software application could look as follows: calibrating the predictor on the existing applications and S = {Number of memory allocation calls, Number of (2) applying it to a new application to obtain an estimatedisk calls, Number of network calls} of its performance. A signature typically includes the types of calls that The steps of the APPEAR method are describedseriously influence the response time of an application. below (see also Figure 2):Note that it is important to distinguish between the Step 0, Virtual Service Platform identification. Thesignature type (see above) and a signature instance that software is divided into two parts: a stable VSP andcontains actual values for a concrete execution scenario, variable applications (see Figure 1). The guideline fore.g. S = {99, 66, 33}. VSP selection are sketched in section D. Usually, a signature is built in an iterative way: after Step 1, Definition of use cases for the existingeach step overlooked parameters are added, and applications. The relevant use cases for measuring thesuperfluous parameters are excluded. performance of the existent applications are defined. Step 2, Collection of measurements. The definedB. Method essence use cases are executed (with different parameters) and The APPEAR method assumes that the software stack the corresponding performance values are measured.of a product family consists of two parts: (1) Step 3, Construction of a simulation model. A high-applications and (2) a Virtual Service Platform (VSP). level simulation model of the execution architecture isThe former consist of components specific for different built to gain insight into the performance of anproducts, while the latter comprises a relatively stable application. This supports the extraction of a signatureset of services and does not seriously evolve during the in step lifecycle. This is shown in Figure 1. Step 4, Signature extraction for the existing The stability of the VSP allows one to use the applications. The simulation model is executed (the realinformation about its performance for estimating the system was already executed in step 2) in order toperformance of applications that are built upon it. The extract the signature, i.e. to obtain the values of thesignature of both already existing and not yet signature parameters.implemented applications can be described in terms of Step 5, Construction of a prediction model. Basedservice calls to the VSP. By extrapolating the relation on the statistics gained in step 2 and 4, it is possible tobetween the measured performance of the existing build and calibrate a predictor that translates a signatureapplications and their signature S = {S1 , S2 , , S N } , it is vector into a performance measure. Such a predictorpossible to predict the performance of new applications. may be constructed by employing (linear or non-linear) To get more insight into the execution architecture statistical regression analysis techniques.
  4. 4. 1. Use cases definition 6. Use cases definition (new ) 3. M odel construction Existing applications Abstract sim ulation m odel New application 2. Measurem ents 4. Signature extraction 7. Signature extraction (new ) Signature 5. Training S1 S2 S3 S4 S5 param eters 8. Prediction PredictorFigure 2. The steps of the APPEAR method. Step 6, Definition of the use cases for new other.applications. After having the predictor calibrated, it is 2. The services of the VSP are independent. Sincepossible to use it for assessing the performance of new the service calls are used as input for theapplications. Possibly a new set of use cases has to be prediction model, there should be no interactionsdetermined for these applications, e.g. if new features that significantly influence the performance, e.g.are defined. via exclusive access to shared resources. Step 7, Signature extraction for the new 3. The order of service calls does not matter.applications. The model of the execution architecture of 4. The application performance can be abstractedthe new applications is simulated with the new use cases with a number of VSP service calls or anotherin order to extract the corresponding signature vector. similar metric. It should be possible to obtain the Step 8, Performance prediction for the new application signature from its simulation modelapplications. Provided that the newly obtained and to use this signature to predict performance.signature agrees1 with the statistics used for calibrating This means that the application must not performthe predictor, it can be used to estimate the performance CPU-intensive internal calculations. Thisof the new application. condition is usually met for embedded control Notice that the proposed method benefits from an systems, the class of systems we are interested in.important property: during the evolution of a product 5. Gradual product (family) evolution. During thefamily, the statistics upon which the predictor is evolution of a product family, a significant part ofcalibrated continuously grows. This enhances the the software remains unchanged. If the newprediction quality and increases the coverage of the applications are completely new and independentstatistics with respect to the signature space. from the existing parts, the prediction can fail because of the lack of statistics.C. Assumptions 6. A large set of applications for training the The following assumptions must be fulfilled to apply predictor is available.the APPEAR method: 1. Applications are independent. The applications interact only with the VSP, but not with each 1 In principle, a newly obtained signature may lie too far away from the signature space on which the predictor was calibrated. In this case, we have to deal with so-called outliers.
  5. 5. D. Virtual Service Platform identification statistics, and the cross-validation “leave-one-out” The abstraction level of the VSP can be selected strategy was applied to them. This resulted in theaccording to the following criteria: distribution of the relative prediction error shown in • Within a product family, there is always a Figure 3 relatively stable part and parts that are frequently 0,4 modified or added. The parts that are likely to 0,35 change should be modeled, while the stable parts 0,3 Relative error 0,25 are captured by a predictor. For “new” parts, the 0,2 performance estimation is important at the early 0,15 architecting phases when only high-level 0,1 0,05 descriptions or models are available. The stable 0 part is treated as a “black-box”, addressed by a 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 Point number statistical predictor. • For the variable part, insight into the performance relevant parts of the execution architecture is Figure 3. The relative prediction error. needed. This means that a model for the In this figure, one can distinguish three parts: one part performance critical components must be built. with high prediction accuracy (points 11 to 24) and two Interactions between these components, their parts with lower accuracy (remaining points). So far, modification and substitution with other ones can two possible reasons for the occurrence of these “low influence the application performance. accuracy intervals” can be considered: • To extract the signature, it must be possible to • There were not enough statistics in the relate this model to a number of relevant service neighborhood of these points because the points calls, input parameters, etc. were actually outliers. The construction of the formula was dominated by the statistics from the V. METHOD APPLICATION TO A MEDICAL IMAGING intervals containing much more points. SOFTWARE SYSTEM Consequently, the intervals with larger amount of This section describes our experience in building a points have higher accuracy.prediction model for the response time for a part of a • An improper set of basis functions was used forMedical Imaging software system. This experiment aims the construction of the formula. This set canat validating the statistical part of the APPEAR method. handle only the points within a certain intervalIn parallel, similar experiments are being performed in and fails for the rest of the points. Probably, linearthe Consumer Electronics domain: a prediction model is approximation is not suitable here, and the shapesbuilt for assessing the CPU utilization of TV software. of basis functions have to be changed.Finalizing these experiments will allow checking theapplicability of the APPEAR method to the Consumer VI. CONCLUSIONElectronics domain. Because the experiments in the Our experiences in applying pure analyticalConsumer Electronics domain are still running, they are approaches to assessment the performance of industrialnot described here. scale software failed due to combinatorial explosion of The performance prediction model for the Medical too many parameters. Thus, we decided to choose for aImaging software was created and then calibrated with mix of analytical and statistical techniques.different values of the signature vector as inputs and The APPEAR (Analysis and Prediction of theapplication response times (from the traces) as outputs. Performance of Evolving Architectures) method forThis model is intended to predict the response times of performance prediction of software applications duringthe new applications, given their signatures. the architecting phase was suggested. This method The collected statistics was used as input for a tool presumes that an application can be subdivided into twoimplementing a Multivariate Adaptive Regressive parts: variable and stable (application and VSP). TheSplines (MARS) algorithm [3]. This tool determines an method includes an analytical part for the explicitapproximation formula for the prediction model. As an description of the execution architecture and a statisticalinitial iteration, linear basis functions were used. part for the performance prediction. The execution Thirty points were randomly selected from the architecture is described in terms of performance
  6. 6. relevant input/diversity parameters and the number of REFERENCESperformance relevant calls to the underlying VSP. It is [1] F. Aquilani, S. Balsamo and P. Inverardi, "An Approach toused to determine the signature. Performance Performance Evaluation of Software Architectures", Research Report, CS-2000-3, Dipartimento di Informatica Universita Cameasurements, collected during the execution of Foscari di Venezia, Italy, March 2000.existing applications, together with the signature can be [2] G. Bontempi, “Local Learning Techniques for Modeling,used for calibrating the predictor. For a new application, Prediction and Control”, PhD thesis, IRIDIA- Universite’ Librea model of the execution architecture is constructed in de Bruxelles, Belgium, 1999. [3] J.H. Friedman, “Multivariate Adaptive Regression Splines”,order to obtain its signature. This signature is taken as Tech. Report 102, Department of Statistics, Stanford University,an input for the predictor to get the performance USA, August 1990.estimation for the new application. [4] K. Goseva-Popstojanova and K.S. Trivedi, "Architecture Based Criteria for choosing the abstraction level of the VSP Approach to Reliability Assessment of Software Systems", Performance Evaluation, Vol.45/2-3, June 2001.were suggested. [5] C.E. Hrischuk, C.M. Woodside and J.A. Rolia, "Trace Based A simple case study, performed for Medical Imaging Load Characterization for Generating Software Performancesoftware, resulted in relative prediction error inferior to Models", IEEE Trans. on Software Engineering, Vol. 25, Nr. 1, pp 122-135, Jan. 1999.35%. This means that the level of prediction accuracy is [6] P. King and R. Pooley, “Derivation of Petri Net Performanceconsiderably high with respect to the requirements given Models from UML Specifications of Communicationsin section 3. Software”, Proc. 11th Int. Conf. on Tools and Techniques for However, there is still not enough experimental Computer Performance Evaluation (TOOLS), Schaumburg, Illinois, USA, 2000.evidence to ensure that the method will work on a [7] C. Smith and L. Williams, “Performance Solutions: A Practicalbroader range of software applications. Also, the Guide to Creating Responsive, Scalable Software”, Addison-predictor reliability with respect to outliers was not Wesley, 2001. [8] B. Spitznagel and D. Garlan, “Architecture-based performancechecked because of the lack of data. analysis”, in Yi Deng and Mark Gerken (editors), Proc. 10th The future work on the APPEAR method will be International Conference on Software Engineering andperformed in the following directions: Knowledge Engineering, pp 146—151, Knowledge Systems Institute, 1998. • Identification of the reasons for the varying prediction accuracy (possible reasons are given in section V). • Building a model of the execution architecture of the applications to validate the structural part of the method and to automate the signature extraction process. • Tackling the compositionality problem in order to be able to derive the performance of a component- based architecture from the performance of its components. This is, however, not a trivial task because of the involvement of statistics. • Construction of execution architecture models and predictive models for more use cases of the Medical Imaging application. • Construction of the execution architecture models and prediction model for an application in the Consumer Electronics domain (TV software). VII. ACKNOWLEDGEMENTS We thank Wim van der Linden for providing us withall necessary information on statistical methods andtools. We want to express gratitude to STW that fundedthe presented work within the AIMES project(EWI.4877).