1. The 2008 International Conference on Embedded Software and Systems (ICESS2008)
Performance Testing Based on Time Complexity Analysis for Embedded
Software
Hu Jin1, Liang-Yin Chen2, Ling-Ming Zeng1, Bao-Lin Li2
1
Computer Science Department, Chengdu University of Information Technology,
Chengdu610225, China
2
College of Computer Science, Sichuan University, Chengdu 610065, Sichuan, China
cn_jh@cuit.edu.cn chenliangyin@cs.scu.edu.cn
Abstract de-efficiency of software and even make system
function failure.
Architecture design and software implementation Performance testing [1] is an efficient method for
both contribute to the correctness of ultimate software detecting benign faults. It is the incarnation of both
products. So performance testing is very helpful to test design and implementation. Software architecture
the inconsistence between design and implementation design describes software systems at a high level of
since it is the mix reflection of both sides. Especially in abstraction and provides a basis for course-grain static
embedded system, with limited resources, execution analysis [2]. However, at implementation level, there
time is more likely to expose hidden defects. This exist risks such as misunderstanding or error coding.
research presents a time performance analysis method Then, performance is not only affected by the internal
for software testing. Firstly, software modules were design attributes but also by the implementation detail.
divided according to functionality in architecture view, Performance testing can help to find out inefficient
and their time complexity were computed in static; software modules cost much fulfill time which could
Secondly, testing activities were designed to track be improved with more design consideration. Also, it
running time cost of those modules; Next, expected can help to detect the inconsistence of design and
time complexity was compared with actual running implementation by figuring out abnormal time
time to figure out abnormal function modules defects consuming modules. This research shows performance
resident. Lastly, experiments were conducted in an testing is especially helpful to test those problems not
embedded software project. The results showed time fatal to the software.
performance testing is an efficient way to find out some This issue is organized as follows. Section 2
kinds of defects concerned with the inconsistence of describes how to divide software modules for static
design and implementation. time complexity analysis at architecture view. Section
3 then uses dynamic testing to analyze the reasonable
Keywords: Software Testing, Embedded System, time cost of software execution time on the basis of the
Time Complexity, Architecture Design previous time complexity model. Section 4 does
experiments in WinCE to verify the effectiveness of
1. Introduction performance testing method. Last section summarizes
the work of this paper.
Software testing is one of the most important ways
to assure the software products quality. Conventional 2. Static time complexity analysis
testing methods take more care about the usability and
functional correctness but often ignore those benign Software architecture describes system at high
defects which may not lead to devastation disaster. levels of abstraction. As the size and complexity of
These trivial defects may not lead to much trouble software systems increase, the design and specification
when running environment is favorable. But with much of overall system structure become a significant issue
more tough circumstance, such as in embedded system [3]. People have long advocated a component-based
environment with much more limited resources, those approach to software construction as a way of
benign defects will contribute much to the performance simplifying the design and maintenance of large
software system [4]. All components are rather
978-0-7695-3287-5/08 $25.00 © 2008 IEEE 243
DOI 10.1109/ICESS.2008.90
Authorized licensed use limited to: King Mongkuts Institute of Technology Ladkrabang. Downloaded on July 17,2010 at 08:23:31 UTC from IEEE Xplore. Restrictions apply.

2. independent modules to fulfill specific functions, and component, ni is for how many times instruction im
m
integrated to make the software work as a whole.
should be executed. It is mainly an internal attribute of
Components are composed of limited number of
the program design. The total execution time of
instructions, and the components can be expressed as C
= <is, I, ie>. I is the set of instructions, and is, ie is the component will be influenced by the input variables
corresponding entry and exit instruction of component with e( Input , ni ) , if the global variables are not
m
[5]. Between is and ie are the instructions sequenced by considered. TM is thus the time complexity of the
the control flow. There is only one pair of is and ie for corresponding component. The relationship between
each component, which guarantee the component to be TM and input variables is as a function: TM:
closed. No intersection instructions can exist in two or InputVar time.
more different components, Ci ∩Cj = Φ, i≠j. The The program now can be represented using such
software is P =∪Ci with all components are referenced. component model. All the components are associated
Connections amongst different components are deeply with actual control flow path with structural analysis
influenced by the architecture being applied. For a [7]. A control flow path could be thought of as the
most often used pipes and filters architecture style, the composition of components arranged in sequence and
main interaction factors include input variables to the i i i
be represented as vectors: CFPi = ( M 0 , M 1 ,..., M n ) ,
component and global variables of the program. Next abbreviated as Pi. So, for all control flow paths in
figure 1 shows this type of component model.
program, they are vectors set form: ( P1T , P2T ,..., PkT ) .
These control flow paths can be figured out
Is automatically by many typical technologies such as:
Input Var. traversing abstract syntax tree [8] or adjacent matrix.
Global Var. Now, the control flow paths can be expressed as the
Filter product of coefficient matrix and component matrix:
C1
Ie [ P1T , P2T ,..., Pm ] = Am×n × Diag(M0,M1,…,Mn)
T
Element aij of matrix A is the execution times of the
Is Is jth component in the ith control flow path.
Input Var. Input Var.
Global Var. Global Var. A C1 F C2
C2 C3
Ie Ie Ie G
B
Is Is H I J
C D
Input Var. Input Var. K
Global Var. Global Var.
E
C4 C4 L
Ie Ie
Pipes
F C3 F C4
Fig. 1 Pipes and filters architecture styles G
With the software components deployment in H + I J
design, time complexity of the component could be
computed out respectively. The expected total time K
cost of the component is the sum of desired execution
time of all instructions in it, its time complexity can be L L
represented as: TM = ∑t(im)×e(Input, ni ) [6]. im is
m Fig. 2 Different granularity for component
the abstract instruction type for all programming
languages, and t(im) is the execution time of that The component deployment is by testers’ decision
instruction for specific implementation language. For a according to the granularity of design view. Just like
figure 2, C2 is subdivided into small components.
244
Authorized licensed use limited to: King Mongkuts Institute of Technology Ladkrabang. Downloaded on July 17,2010 at 08:23:31 UTC from IEEE Xplore. Restrictions apply.

3. On the basis of previous analysis, the expected From the previous description, the components’
theoretical execution time of designed components execution time is only influenced by the input variables.
along some specified control flow path could be That time can be looked upon as the dependant
computed out. By reasons of implementation or coding, variable of input invariables: f:(V1,V2,…,Vn) T. It is
the actual program execution time may be far away possible to use this execution time function to
from the theoretical values, which may indicate some accomplish testing work. If actual execution time can
benign errors. Next, research was done to find out comply with their static time complexity, then their
those defects. execution time could be thought as normal. Else, there
probably defects existed. Usually it is impractical to
3. Dynamic Performance Testing analyze such time function with more than tree input
variables. Also, not all but some of variables are
Program executes with specific input data and attractive for testers. This paper just discusses one
attains corresponding results. The Input-Output pair independent input variable case. Regression [12]
with associate execution conditions is called as test method obviously is the most proper solution for this
cases [9]. Test cases are used to verify program work.
compliance with specific test requirements, and they Linear regression is the simplest form of regression.
reflect the dynamic characters of program [10]. Bivariate linear regression models a random variable Y
According to executed control flow paths, test data as linear function of another independent variable X,
space are subdivided into different subsets. For each of that is, Y = α+βX. α, β are regression coefficients, α
subset, the elements in itself are all equivalent in the specifies Y-intercept and β is the slope of the line.
sense of executing the same control flow path [11]. Take the execution time as Y and input variable as X.
All the test cases in one subset means performing a Then, the dynamic testing data in equivalent set can be
testing task, driven by which, sequential software collected for regression analysis. Coefficient α and β
components are executed in order. The testing task is can be computed out with the next formula.
∑
s
expressed as: TAi = [pi1, pi2, …, pin | vi1, vi2,…, vik]. In ( xi − x)( yi − y )
this presentation, two parts are included. One is the β= i =1
∑
s
executed control flow path. The other is the associate
i =1
( xi − x ) 2
test cases or input variable set. The execution control
flow path can be substituted with the multiplication of α = y−βx
coefficient matrix and component matrix. That is very
convenient for performance testing analysis: t(TAi) = where x , y is the average of testing data x1, x2, …,
[ai1 × t ( M i1 ), ai 2 × t (M i2 ),..., ain × t (M in ) | vi1 , vi 2 ,..., vik ] . xs, and y1, y2, …, ys respectively. Once the regression
line is determined, it would like to evaluate the
Now, the actual execution time should be in effectiveness of the fitted line. A measure that helps to
accordance with the components’ time complexity answer this question is provided by formula: SSE =
computed out in static. Therefore, the testing tasks n 2
∧ ∧
driven by equivalent test cases set have the same
coefficients. Time costs on the same testing task could ∑ (Yi − Yi ) , where Y i = α + βX i . If SSE = 0, the
i =1
only depend on the components’ time complexity with
regression line fits perfectly. As the fit gets worse, SSE
different input variables in equivalent test case set. It is
gets larger. This can be used to detect the abnormal
then a very useful conclusion for time performance
points which have large deviations.
testing: t ij (M s ) : t ik (M s ) = TM sj : TM sk . i specifies the This linear regression relationship with only one
ith testing task. j and k are for two different test cases independent variable is easy to be tested. Polynomial
in an equivalent test case set. s is for the components regression can also be applied for analyzing none
along with the testing task. If program is tested with an linear dependence. By applying transformations to the
equivalent test case set, time consumed on the same variables, the none-linear regression model still could
component will only depend on the input variables be converted into a linear one. Since not all of the input
assignment: e(InputVar, ni ). To some extent, the variables need to be cared about according to specific
m
testing purpose, we can just isolate those input
equivalent test cases set will also guarantee the same variables interested to delivery component time testing.
external execution environment, thus making global Here summarizes the testing procedure briefly.
variables in the component constant when execution. Program should firstly be executed with equivalent test
That is also the reason why it can not simply use cases as input. Then, those interested components
component alone to decide its execution time. running time are to be recorded for performance testing
245
Authorized licensed use limited to: King Mongkuts Institute of Technology Ladkrabang. Downloaded on July 17,2010 at 08:23:31 UTC from IEEE Xplore. Restrictions apply.

4. analysis. The ration of the execution time and input ms Switching time diagram
variable is arranged for linear regression. Thus, those 2000
Switch Time
350
1800
abnormal data is selected out for further investigation 1600
300
Drawing Object Number
to find out inconsistency of design and implementation. 1400 250
1200
200
4. Experiments 1000
800 150
600 100
The experiment platform is a embedded system. 400
50
Hardware is a touch screen with CPU ARM9 2440 200
400MHz and displaying acceleration chip SM501. The 0
oil trans. dryer sys. dryer room heating sys.cleaner sys. purify sys.
0
system memory size is 32M. WinCE 4.0 is the installed
OS which is a pre-empty schedule system. A HMI Fig. 3 Switch Time and object numbers of
project is to be tested. Its main function is to monitor different animation windows
attached control nodes information in real time. Data
collected from distributed nodes are used for After more careful investigation, an implementation
presenting simulation in center touch screen. Every defect was found out. The drawing cut region is not
monitored scene can be switched on from one to properly set when animation. Then, more unnecessary
another when needed. redrawing time was spent to do this work. Because
The primary displaying module of the system can such a defect is context related, all except at the oil
be divided into three parts. (1) Initialization module to transferring scene sight can work well. Next figure is
do some common works such as memory preparing, the results after defect revised.
protocol initialization and drawing context setting. (2)
Data collection from all net nodes being monitored and ms Switching time diagram
1000 350
decoding according to protocol. (3) Animation and
900 Switch time
displaying corresponding scene of node information 800
300
when needed. These parts still can be further divided 700
Drawing Object Number
250
into smaller modules. Main trouble of the system is 600
that it is too slow to response when switching from one 500
200
scene to another. How to evaluate the impaction to the 400
150
inefficiency of the system is rather a more difficult 300 100
problem than what it looks like from the apparent. 200
There are two obstacles must be tackled for the 100
50
traditional software testing method. One is that pre- 0 0
empty strategy makes the lowest priority displaying oil trans. dryer sys. dryer room heating sys. cleaner sys. purify sys.
module more difficult to trace. Another one is that
users’ interference induces more uncertainty to the Fig.4 Switch Time after implementation
system response time. revising
There are six scene sighs to be monitored in this
project. Different execution time was spent to animate After removal of this bug in drawing component,
those scenes that contribute much for switching time. figure 4 is more reasonable. Switch time cost is in
Next figures show the irrational execution time of the accord with the drawing object numbers. Certainly,
animation module indicating further investigation is switching time is not strictly conformance to drawing
needed. objects numbers because different objects may need
In figure 3, left vertical coordination is to specify different drawing time. But it is still a very good
switching time needed for displaying corresponding indicator for time cost. Now, all the switching time is
scene sight. Right vertical coordination is to show confined in 1000ms, and the operator can get a rather
drawing object numbers in displaying scenery. more comfortable response time.
Obviously, animation time of the oil transferring is
abnormal for its switching time is not accord with its 5. Conclusions
contained object count. It spent almost twice time cost In summary, we have proposed a time performance
as any other scenery. By applying linear regression, it testing model for embedded software testing. The
can be figured out two coefficient are α = 400, and β = proposed approach takes advantage of comparability of
1.298. execution time driven by equivalent test cases. By
comparing or analyzing the abnormal execution time
246
Authorized licensed use limited to: King Mongkuts Institute of Technology Ladkrabang. Downloaded on July 17,2010 at 08:23:31 UTC from IEEE Xplore. Restrictions apply.

5. cost with regression method, it is helpful to find out the [4] Rosenblum D S, “Adequate Testing of Component-Based
deficiency of design or defects introduced by Software”, Department of Information and Computer
implementation. Future works include finding out more Science University of California, Irvine, CA 92697-3425,
effective method for subdividing the software August 1997.
[5] Ren-Jie Zheng, Computer Software Testing Technology,
components, and combining more independent input
Tsinghua University Press, 1991.
variables for directing time performance testing. [6] Xiao-Dong Wang, The Design and Analysis of
Algorithms, Tsinghua University Press, 2003.
Acknowledgement [7] Jorgensen P C, A Craftsman’s Approach, CRC Press,
This work was supported by the Sichuan province Inc., 2002.
science and technology tackle key problems project [8] Kenneth C. Louden, Compiler Construction: Principles
under grant No 05GG0321, the National scientific and and Practice, Thomson Learning, 2002.
[9] [IEEE Std 610.12-1990] IEEE Standard Glossary of
technological middle or small sized enterprise
Software Engineering Terminology Standards, 1999.
innovative fund under grant No 06C26225101730. [10] Christoph C.Michael, “Genetic Algorithms for
Dynamic Test Data Generation”, Proceedings of the
References 12th international conference on Automated Software
Engineering, pp.307-308, 1997.
[1] Glenford J. Myers, The Art of Software Testing, John [11] Hu Jin, Zhi-Shu Li, “Using Genetic Algorithm to
Wiley & Sons, Inc., 1979. Generating Test Cases Automatically and Path-
[2] Stafford J A, Richardson D J, Wolf A L, “Architecture- Oriented”, Computer Engineering, Vol.33, No.3, pp.21-
level Dependence Analysis for Software Systems”, 23, 2007.
International Journal of Software Engineering and [12] Jia-wei Han, Micheline Kamber, Data Mining:
Knowledge Engineering, Vol. 11, No. 4, pp. 431-453, Concepts and Techniques, Higher Education Press,
August 2001. 2001.
[3] Mary Shaw, David Garlan., Software Architecture:
perspectives on an emerging discipline, Prentice Hall,
Inc., 1996.
247
Authorized licensed use limited to: King Mongkuts Institute of Technology Ladkrabang. Downloaded on July 17,2010 at 08:23:31 UTC from IEEE Xplore. Restrictions apply.