3. 3
Abstract
The very complex engines for the vehicle industry of today demand advanced controlling systems
to control a large number of engine functions. It is very important that these engine controllers
are tested thoroughly to ensure high service reliability, but also to guarantee that the controlling
logic and software does not mean security risks for the driver or other road-users.
At Scania, heavy trucks and buses are developed and built. These vehicles are powered by strong
engines, which are controlled by electronic controllers with self-developed software. In order to
continuously test newly developed program logic and new versions of existing controller
software, a mix of manual and automated tests are used. Automated tests have a number of
advantages, of which most important are that they are time efficient, and that they make exact
reiterations of test sequences possible.
This thesis has two purposes, where the first is to develop automated tests for the starting motor
logic in the engine controller. The second purpose is to develop an automated system for building
software for the engine controller from the latest available source files, and to run all tests
available for the present engine type.
Five different tests for the starting motor logic were developed in this thesis and are accounted
for in this report. An automated system that can be used to verify new software functionality in
the engine controller software was also constructed. The system can be used on a daily basis and
the results of the tests are e-mailed to all concerned persons.
4. 4
Sammanfattning
Dagens mycket komplexa motorer för fordonsindustrin kräver avancerade styrsystem för att
kontrollera en stor mängd motorfunktioner. Det är mycket viktigt att dessa motorstyrsystem testas
noggrant för att säkerställa hög driftsäkerhet, men också för att garantera att styrlogiken och
programvaran inte innebär säkerhetsrisker för föraren eller andra trafikanter.
På Scania utvecklas tunga lastbilar och bussar vilka drivs av kraftfulla motorer, vilka styrs med
hjälp av elektroniska styrsystem med egenutvecklad programvara. För att kontinuerligt kunna
testa nyutvecklad programlogik och nya versioner av existerande motorstyrsystemsprogramvara
används en blandning av manuella och automatiserade tester. Automatiserade tester har en rad
fördelar, där de främsta är att de är tidseffektiva och att de är möjliggör exakta upprepningar av
testsekvenser.
Detta examensarbete har två syften, där det första är att utveckla automatiserade tester för
startmotorlogiken i styrenheten. Det andra syftet med examensarbetet är att utveckla ett
automatiserat system för att varje natt bygga programvara för motorstyrenheten med hjälp av de
senast tillgängliga källfilerna, för att sedan köra de automatiserade tester som finns tillgängliga
för aktuell motortyp.
Fem olika tests för startmotorlogiken skapades inom detta examensarbete, vilka redogörs för i
den här rapporten. Ett automatiserat system för verifiering av ny mjukvarufunktionalitet i
motorstyrenheten utvecklades också. Systemet kan användas dagligen och resultaten från
genomförda tester e-postas till berörda personer.
5. 5
Acknowledgements
I would like to take the opportunity to thank my supervisors at Scania, Olof Lundström and
Håkan Bengtsson for their contribution and help during this thesis, and for always being patiently
helpful and being jolly good fellows. I also want to thank my supervisor at the Royal Institute of
Technology, Ingo Sander, for his assistance during the work with this thesis and during the
writing of this report.
I would also like to say thank you to everyone else who in different ways have
helped me with my work. Another thank you to the other thesis workers and all employees at the
department for verification and tools (NEV) at Scania, for fruitful and funny discussions during
the coffee brakes. Finally, I would like to direct thanks to Emma, for proof-reading, her patience,
and the support she always gives me.
6. 6
List of Abbreviations
ACC Automatic Climate Control
ACS Articulation Control System
BCS Bus (Body) Chassis System
BMS Brake Management System
BWS Body Work System
CAN Controller Area Network
CMOL Central Monitoring Logic
COO Coordinator system
CPU Central Processing Unit
CSMA/CA Carrier Sense Multiple Access with Collision Avoidance
DTC Diagnostic Trouble Code
ECU Electronic Control Unit
EEPROM Electrically-Erasable Programmable Read-Only Memory
EGR Exhaust Gas Recirculation
EMS Engine Management System
FLIM Fuel limitation
FMEA Failure mode effect analysis
FPU Floating Point Unit
GMS Gearbox Management System
HIL Hardware-in-the-loop
HILS Hardware-in-the-loop Simulation
HLP Higher Layer Protocol
HPI High Pressure Injection
KTH Royal Institute of Technology
NE Department for Powertrain Control System Development (Scania)
NEV Group for Verification and Tools (dep. NE at Scania)
NGS New Generation Scania
OBD On Board Diagnostics
7. 7
PDE Pump Düse Einheit
SESAMM Scania Electrical System Anno 2000
UC User Case
UDS Urea Dosing System
UF User Function
VBScript Visual Basic Script Edition
9. 9
Contents
1. INTRODUCTION..................................................................................................................................................13
2. PROBLEM DESCRIPTION.................................................................................................................................15
2.1 PROBLEM STATEMENT ............................................................................................................................................15
2.2 BACKGROUND.........................................................................................................................................................15
2.3 GOALS ....................................................................................................................................................................16
2.4 PURPOSES ...............................................................................................................................................................17
2.5 METHODS................................................................................................................................................................17
3. SOFTWARE DEVELOPMENT AND SOFTWARE VERIFICATION...........................................................19
3.1 INTRODUCTION .......................................................................................................................................................19
3.2 SOFTWARE DEVELOPMENT METHODS......................................................................................................................20
3.2.1 Waterfall-model ..................................................................................................................................................20
3.2.2 The V-model........................................................................................................................................................21
3.2.3 The W-model .......................................................................................................................................................23
3.2.4 Spiral model........................................................................................................................................................25
3.2.5 Agile Methods and Extreme Programming.........................................................................................................26
3.3 THE IMPORTANCE OF TESTING................................................................................................................................27
3.4 TESTING AT DIFFERENT LEVELS .............................................................................................................................27
3.4.1 Unit testing..........................................................................................................................................................27
3.4.2 Component testing...............................................................................................................................................28
3.4.3 Integration testing...............................................................................................................................................28
3.4.4 System Testing.....................................................................................................................................................29
3.4.5 Acceptance Testing..............................................................................................................................................29
3.4.6 Regression Testing ..............................................................................................................................................29
3.4.7 Automatic Testing ...............................................................................................................................................30
3.5 WHEN TO STOP TESTING.........................................................................................................................................31
3.5.1 Defect Density.....................................................................................................................................................32
3.5.2 Defect Pooling.....................................................................................................................................................33
3.5.3 Defect Seeding.....................................................................................................................................................33
3.6 TESTING EMBEDDED SYSTEMS ................................................................................................................................34
3.7 TESTING THE ENGINE ECU AT SCANIA....................................................................................................................35
3.7.1 The Test System...................................................................................................................................................35
3.7.2 Testing at Different Levels ..................................................................................................................................36
3.7.3 The engine management system..........................................................................................................................37
3.7.4 Hardware-in-the-loop-testing .............................................................................................................................39
10. 10
3.7.5 Fault code testing................................................................................................................................................40
4. BUILDING A SYSTEM FOR DAILY BUILD ...................................................................................................42
4.1 WINDOWS TASK SCHEDULER..................................................................................................................................42
4.2 BATCH SCRIPTING...................................................................................................................................................43
4.3 VBSCRIPT...............................................................................................................................................................43
4.4 PROGRAMMING IN CVI ...........................................................................................................................................44
4.5 THE CAN-NETWORK ..............................................................................................................................................45
5. IMPLEMENTATION AND RESULTS...............................................................................................................48
5.1 CREATING AUTOMATED TESTS FOR THE ENGINE ECU.............................................................................................48
5.1.1 Introduction.........................................................................................................................................................48
5.1.2 Testing the ECU Functions for the Starting motor .............................................................................................48
5.1.3 The Test Scripts...................................................................................................................................................49
5.2 CREATING A SYSTEM FOR DAILY BUILDS..............................................................................................50
5.2.1 Introduction.........................................................................................................................................................50
5.2.2 Checking Out Source Files and Building the ECU Software..............................................................................51
5.2.3 Flashing ..............................................................................................................................................................52
5.2.4 The Start Script ...................................................................................................................................................53
5.2.5 The Test Software................................................................................................................................................53
5.2.6 Running of Tests..................................................................................................................................................54
5.2.7 The daily build report..........................................................................................................................................54
6 DISCUSSIONS........................................................................................................................................................56
APPENDIX 1..............................................................................................................................................................64
List of Figures
FIGURE 1. THE REQUESTED DAILY BUILD SYSTEM ......................................................................................................15
FIGURE 2. THE WATERFALL MODEL .............................................................................................................................21
FIGURE 3. THE V-MODEL ..............................................................................................................................................22
FIGURE 4. THE W-MODEL .............................................................................................................................................24
FIGURE 5. THE SPIRAL MODEL. ADAPTED FROM [7].....................................................................................................26
FIGURE 6. THE S6 ECU.................................................................................................................................................38
FIGURE 7. THE HARDWARE-IN-THE-LOOP SIMULATION SYSTEM AT NEV, SCANIA.....................................................40
FIGURE 8. SCREENSHOT FROM WINDOWS TASK MANAGER..........................................................................................43
11. 11
FIGURE 9. LABWINDOWS/CVI FROM NATIONAL INSTRUMENTS...................................................................................45
FIGURE 10. THE IMPLEMENTED DAILY BUILD SYSTEM ................................................................................................51
FIGURE 11. A SAMPLE TEST REPORT............................................................................................................................55
List of Formulas
FORMULA 1. DEFECT DENSITY .....................................................................................................................................32
FORMULA 2. DEFECT POOLING - NUMBER OF UNIQUE DEFECTS....................................................................................33
FORMULA 3. DEFECT POOLING - ESTIMATED TOTAL NUMBER OF DEFECTS...................................................................33
FORMULA 4. DEFECT SEEDING......................................................................................................................................34
13. 13
1. Introduction
This report is written as a part of a master’s thesis done at the Department of Microelectronics
and Information Technology at the Royal Institute of Technology (KTH), within the subject of
computer science. The work that underlies this report was performed at the department of
Powertrain Control System Development. My supervisors have been associate professor Ingo
Sander (KTH), development engineer Olof Lundström (Scania), and development engineer
Håkan Bengtsson (Scania).
At the department for verification and tools, NEV, at the department for Powertrain Control
System Development, NE, at Scania, self-developed soft- and hardware for testing of the engine
controllers that are developed at the department, is used. The work that has been implemented in
this thesis can be said to be a further elaboration of this self-developed test software. The aim was
to make the testing of the engine controllers easier by providing a tool for automatic building and
testing of controller software on a daily basis. Another task in the thesis was to compose test
cases to test the engine controller’s functionality for the starting motor.
15. 15
2. Problem description
2.1 Problem Statement
This thesis consisted of two main parts, which to their content were quite different. One was
clearly focusing on implementation, while the other had its focus on investigating strategies for
software development with its main emphasis on testing and verification.
The first task was to construct a test case for the engine controller for engines developed at
Scania. The second task was to create a system that would get the latest source files from a
version management system, compile these files and build software for the electronic control unit
for the engine management systems. The system should also be able to set certain parameters in
the software, depending on which engine type the software is intended for. When this was
completed, the system should program the unit mounted in a test rig. After programming the
electronic control unit, the system should run all automatic test scripts available for the specific
mounted unit. When the system has completed these tasks, it should report the results of the tests,
by means of a homepage or emails. The requested steps in the daily build system are illustrated in
figure 1.
Autostart at
midnight
Fetch source files
from the version
management
system
Compile source
files
Set parameters for
specific motor type
Flasha the
mounted ECU with
the new software
Run available
tests
Present the result
(Web or email)
Start the Daily Build
system at a time specified,
when no one is logged on
to the test computer.
Fetch the latest source
files from the version
management system and
build software for all
engine types
Apply the bin-file for a
specific engine type and
program the ECU mounted
in the rig with it.
Run all tests that are
available for the specific
engine type and present
the result in an appropriate
way.
Autostart Compile Apply & program Auto tests
1
5 6
2 3 4
Figure 1. The Requested Daily Build System
2.2 Background
The modern vehicles of today demand more and more advanced controlling units in order to
control a vast number of complex vehicle functions. An Electronic Control Unit, ECU is an
electronic controller where the main purpose is to control a complicated device or system, such as
16. 16
a combustion engine, brake system or perhaps an industrial robot. These units are found in most
modern vehicles, such as cars, heavy trucks and buses. There are usually different ECU’s for
different parts of the vehicle, one for the engine management system, one for the gear
management system, one for the brake management system, and so on.
Engines for heavy trucks, buses, industry and marine purposes developed at Scania have in
time to an increasing degree become more dependent on electronic controllers. The complexity in
these embedded systems result in increased demands on testing and verification. At NEV at
Scania, the engineers are responsible for the testing and verification of the engine and gearbox
controlling units, among other things. A test system exists for running automated tests for these
units, which means that no one has to be present when the tests are running. These systems
consist of both hardware and software. However, there are some steps in this process that have to
be made manually. There was a wish by the engineers at NEV to develop this system even
further, making it even more automated. In order to get a very quick response to the developers of
the controller software, it would be desirable to test the latest available software each night.
2.3 Goals
The main goal during this work was to develop an automated system, for building and testing
software for the engine controller, which would run every night, or as often as possible. The
system should, in order:
• check out the latest source files for each ECU system from a version management system
• compile these source files and build software for all ECU types
• set certain parameters in the binary software file to be used, depending on which ECU
that is currently mounted in the rig
• load the suitable software onto the ECU
• run all automated test scripts available for this particular ECU
• report the results of all steps by means of a webpage or by email.
Except for this, the system should be able to be started at a certain time, when no user is logged
on to the operating system on the computer, on which the program is running. Another goal was
that the system was constructed as simple as possible; demand little or no maintenance, but to just
17. 17
run every night. The implementation was to be constructed and run in a PC system with
Microsoft Windows 2000 as operating system.
The other goal in this thesis was to compose a series of tests for the engine controller’s
functionality for the starting motor. This test should also be implemented and integrated into the
daily build system.
2.4 Purposes
The software used in the embedded controlling systems used on modern commercial vehicles is
supposed to be able to handle many different tasks. Various parts of the applications are done by
different developers. It is therefore very important that these engineers can have their pieces of
functionality tested as soon as possible in the development process, so that any errors are detected
at an early stage. With continuous testing every night, engineers can receive feedback from
changes made to the software very soon.
The expectation was accordingly that this system would shorten the total time for
development and modification of the engine ECU software, foremost by detecting faults and bugs
at an early stage, which increases the opportunities to quickly locate the erroneous code or idea.
Further on the system is expected to make it easier for the engineers that today have to start the
automatic tests manually. Another effect is that the test hardware, the rig, is used more
effectively, as it can be used at night when it before usually wasn’t utilized at all.
2.5 Methods
The work consisted of two main parts, for which the methods have been different. To complete
the task concerned with creating tests for the ECU, many documents on how the engine ECU
functions were thoroughly read, and interviews were held. In addition to this other documentation
on how to write tests were read, and finally other tests written for the ECU were studied.
To solve the second task, concerned with the daily build system, interviews were held with
the people who wanted to have the system. This was done to clarify what they were looking for,
and also to get to know what resources that were available. A lot of information was also gone
through as the work progressed. Most of this information was sought on the internet. As problems
and questions have arisen, people with knowledge within the different areas have been consulted.
19. 19
3. Software Development and Software Verification
3.1 Introduction
According to [1] everyone writes bugs, and therefore it is essential to test. The need and cost for
testing can vary substantially. For a very simple consumer software product, the cost for testing
might be as low as 2 % of the total developing cost, while it for space aircraft software can be as
high as 80 %.
In software development, the largest cost usually lies on bugs [2]. This cost can be split up
in the costs to find bugs, correct or remove them, create tests to find them and the cost to run the
tests. The least costly with testing is to prevent bugs, a lot cheaper than to discover and correct
errors [1].
When developing software, it can be the developer himself that carries out the testing, or it
could be someone outside the developing group. According to [1], it is a great advantage if the
tester does not know in great detail how the system is constructed, but can test it on the basis of
what it is intended to do. This way the tester is able to come up with tests that the developer
might not have thought of since he usually on forehand knows exactly what works, but maybe not
what might not function completely as planned. It is important that these things are tested too,
since they are possibly used and thereby tested by an end user. These tests are called negative
tests. An independent tester knows only what the system is supposed to be able to do, and can
thereby test more general functions, which are possibly used by the end user.
However, it is a vast difference between testing a software completely unprejudiced and
being well familiar with the system. The tester should know enough to make it possible to test
everything that needs to be, and is possible to be tested. If the tester can not predict the
consequences of a test in advance, then the tester does not fully understand what the software is
supposed to be able to do.
It is very important that the testing begins at an early stage of the development. The cost,
counted both in time and in money, increases rapidly as the development of the software moves
on [1]. At Scania, as well as within most other industries worldwide, different parts of a project
are developed simultaneously, and thereby it is very important to test the different parts as soon
as possible. It is a lot cheaper to detect a bug or an error before a project module is merged
20. 20
together with other parts. In the case of the engine ECU, it has to be tested as soon as there is a
working prototype. But, as mentioned in section 3.2.3, there are ways of testing software even
before a prototype is available.
3.2 Software development methods
Developing software is fairly easy for small projects, but when these grow, and perhaps comprise
more than just one developer, a carefully thought-out and planed work is usually demanded for
best result. It is important to determine how to plan the work, how to verify that requirements are
followed, how to test the system built, set up time plans, and so on. Various models for software
development exist. They all have the same goal, to guide a software development team through
the steps of creating good software. However, they all emphasize different parts of the
development process differently, and they also allot testing and verification different amounts of
time and resources. A few of them will be mentioned here, and the focus lies on how the models
advocate testing.
3.2.1 Waterfall-model
The classic and probably most common method for software development is the waterfall model.
This method usually starts with the idea, followed by a requirements analysis, design,
implementation, testing and then presumably maintenance. In this model, the different phases are
carried out in a sequence, with one phase following the other. Each activity in the software
development must be completed before the next can begin. It is not possible to go back to a
preceding phase once a phase is started, even if problems with earlier phases are discovered.
However, it exist versions of the waterfall model where steps to the previous phase are allowed.
It is of great importance that everything within a phase is done with great accuracy before the
following phase is initiated. This fact could have a negative effect on the project time line. The
ambition to deliver everything as error free as possible from one phase to another could result in
exceeded phase time limits.
21. 21
Figure 2. The Waterfall model
The waterfall method is very intuitive and easy to understand, if however a bit naïve. The model
conveys a number of risks. If the maintenance is excluded from the software development period,
testing is found as the last phase of the project, after the coding. As the time limits in waterfall
model projects are often exceeded, there is sometimes very little time left for testing, if any.
Faults discovered in the test phase are usually very costly to repair, as they are found very late in
the project. Another weakness with the model is that the testers of the project, if separated from
the developers, get into the project very late, and thereby have limited knowledge of the system.
One problem with the model is that it is not iterative. Things that are laid down at the start
of the project are never changed. The requirements, for example, that are stated in the beginning
are never changed, even though the reality usually shows that the real demands are changed
during the development. Changes to the requirements are never made through the project. [2]
3.2.2 The V-model
The V-model is another method for software development. This model is also pretty commonly
used. One benefit with this model is that it, just like the waterfall method, is very easy to
understand. The development usually starts with a requirements analysis, followed by
22. 22
specification, architectural design, detailed design, and then the actual coding. Unlike in the
waterfall model, the planning of the test work starts as soon as the project starts.
Figure 3. The V-model
The V contains the planning part on one side and the implementation and testing part on the
other. The time moves from left to right in the illustration. Just like the waterfall model, the
actual testing starts at first after the actual coding. However, in this model, some of the test work
starts as soon as the project starts. Each phase on the left hand side of the V has a corresponding
test phase, which is planned as the step on the left is performed.
The requirements specification contains very few details about the system, but the main
purpose and tasks are specified. This phase corresponds to the acceptance testing phase, where
the system is tested to determine if it does what it is supposed to do. These tests are often
23. 23
performed by the customer. At Scania, the acceptance tests for the engine management system
are made through field tests, in real trucks.
In the function specification phase, the functionality of the system is specified. Specific functions
such as a function to start an engine in an engine management system or a function for booking a
room in a hotel reservation system. The system test phase matches to this phase. In this test
phase, the functions are tested. Is it possible to start the engine? Can a room be booked? As many
as possible of these functions are tested, one by one.
The design specification specifies the design of the system, and the functionality. How the
different functions should be designed for example. During this phase, the integration test phase
is planned. The integration tests test that the different modules, code components, and subsystems
can be merged together and communicate. Various strategies for integration testing exist, such as
big bang where all modules are merged together at once, and bottom-up where low-level
components are integrated first.
The component specification deals with the detailed design of the system. The component
testing is usually carried out by the developers themselves. Typical things that should be tested in
this stage is that the component follows the company’s name and coding standard, that the
components handle errors and exceptions in a satisfactory way, that the component does not leak
memory, that component documentation exists, that the user interface is designed according to
the company standard, that buttons and menus work properly, and of course that the code is well
written. The last test activity is preferably done by a co-developer. It is important that not only
positive tests, i.e. tests where the system is tested with proper, and expected values, are
performed, but also negative tests, where the systems is fed with values outside the accepted
input range, or even without any input. [4]
3.2.3 The W-model
This model was introduced to even further emphasize testing. The W-model is based on the V-
model, but here a second V is added, which together with the first V forms a W. According to [5]
the models presented above are deficient in a number of ways. For example, the test activities
does not start until after the implementation and the connection between the test stages and the
24. 24
basis for the test is not obvious, and there is no clear connection between test, debug and change
tasks. The W-model was invented to deal with these weaknesses.
According to the W-model advocates, the W-model is a further developed version of the
V-model, but should not be seen as just a development of its predecessor; rather it should replace
it. The new model has many similarities with its ancestor, but has removed the deficiencies with
it. One of the shortcomings with the preceding model is that the actual testing starts at first after
the actual coding of the system. For example, the V-model says almost nothing about static
testing.
Static testing can be things such as reading the different specification documents created in
the various phases of planning, and look for inaccuracies. Inaccuracies are typically things that
are not testable, and functions that are not completely distinct. Functionality that is not distinct in
the requirements specification is hard to implement, and forces the programmer to design the
software as he construes the requirements specification, whereupon the tester has problems
knowing how to test the functionality.
Figure 4. The W-model
25. 25
Another weakness with the V-model, according to [6] is that the connection between the various
test stages and the basis for the test is not very clear. It is unusual that one specification phase
alone stand as a base for the test phase. In the W-model it is accepted that a phase on the left hand
side of the model is not the only basis for the testing phase. In order to plan the system tests for
example, not only the functional specification must be used, but also parts of the requirement and
the design phases.
In software testing there is a close connection between testing, debugging, changing and re-
testing. The W-model takes note of this fact, and is illustrated in the figure 4 with circular arrows.
It is very important to start testing early, preferably at the start of a project, and then
continuously test the system throughout the whole development of the systems. To emphasize
this, the model adds a second V to the V-model, where testing is allotted one of the two Vs. The
model shows that testing should constitute about half of the project time and effort. However, the
exact relation between time for development and testing should not be considered as fixed. Nor
should the different phases during the development have equal amounts of resources, such as
time, personnel, and money. All phases require different amounts of time, and the time required
for the various phases may vary from project to project. In an error-sensitive software
development project, it is likely that testing is allotted a larger share of the project resources.
One of the advantages with the W-model is that both the developers and the testers start
working from the initiation of the project, and are given equal resources. The model advocates a
tight collaboration between developers and testers. This reduces conflicts between the groups, a
thing that is pretty common otherwise, where testers are considered as just a destructive part of
the development process.
3.2.4 Spiral model
The Spiral model is an iterative software development model where software is developed in
series of incremental releases, each in principal following the waterfall model. Early releases
sometimes consist of simple prototypes or just paper models. Each phase starts with a design
goal, and ends with the client. The model is often used with larger projects, where requirements
often change during the development process [7]. One advantage of the spiral model is that it
maintains a systematic stepwise approach, like the waterfall model, but that it incorporates it into
an iterative model, where changed requirements are accepted [7].
26. 26
Figure 5. The Spiral Model. Adapted from [7]
3.2.5 Agile Methods and Extreme Programming
Agile methods tries to overcome the inflexibility of producing complete, well defined
specifications from the beginning of the development. [8, b9]. Agile techniques are often simple,
and considered common knowledge; however, they are not always followed [8]. The core rules of
Agile programming methods are defined in an agile software development manifesto:
• Individuals and Interaction over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan
Agile methods are said to be adaptive, rather than predictive, as for example the waterfall model.
Adaptive methods attach great importance to adapting to changing realities, whereas predictive
methods focus on planning the future in detail. Adaptive methods are often iterative, but in
27. 27
contrast to other iterative methods, the iterations are much shorter, measured in weeks rather than
months. [10]
One of the most known and popular agile software development methods is called eXtreme
Programming, or just XP. It is based on the principles of agile principles. It consists of a number
of steps describing planning, designing, coding, and testing. On of the main principles of
development in XP is that it is test driven. That is, tests are written before the code is written.
That means that all produced code is verified by these tests. Another principle of XP is that all
programming should be done in pairs, with two persons in front of each computer screen, which
is supposed to increase the quality of the produced code [11].
3.3 The Importance of Testing
Unless a constructed system is thoroughly tested, one cannot be certain that it actually does what
it was built to do. Nor is it definite that the system does not do unexpected things. By testing the
software it is made sure that the software meets the requirements stipulated in the beginning of
the development. The product can be said to reach a certain level of quality. In most commercial
line of businesses, the quality of a company’s products is extremely important for a company, and
must be very high; otherwise the company will find it hard to survive on the market. A company
that produces and sells software in one form or the other that does not reach the markets
expectations, in terms of quality, rarely survives. The line between releasing poor quality
software early and high quality software late can be treacherous [1].
For security critical applications, such as the space aircraft mentioned earlier, testing is of
course essential. For many of these security critical applications, a bug in the software of the
embedded systems could result in heavy injury and damage on both person and material.
3.4 Testing at Different Levels
Testing is traditionally divided into four different levels, unit or component testing, integration
testing, system testing, and acceptance testing. The methods for testing usually differ between the
different test levels, and different errors are revealed.
3.4.1 Unit testing
A unit is the smallest piece of software that can be compiled, linked, and loaded. It is usually just
one function or method. It consists of a very limited amount of code, some hundred lines of code,
28. 28
or less [1]. It is typically the work of one programmer, and it does not contain any called sub-
components.
As the name implies, these tests try to verify that the small units i.e. methods or functions,
of a software project behaves exactly as they are expected to do. It is a good idea to isolate the
units to be tested from the rest of the source code, to ensure that the result of the test is not
affected by other functions [12]. Unit testing is only concerned with the various software units
alone, and do not test how they interact with each other. This is done with integration testing. In
unit testing, called or communicating components are replaced with stubs (pieces of code
emulating called functions), simulators, or trusted components. Calling components are replaced
with drivers or super-components [13].
Unit testing is one of the corner stones in extreme programming (XP), where unit test are
written before the actual code is written [14]. XP implies that the developer creates unit tests for
all written code. This means a lot of tests. There exists, however, a lot of unit testing framework
for different languages, which supports and makes it easier to create unit tests.
3.4.2 Component testing
A component is a small piece of software that consists of one or more software units. It is always
one conceptually atomic unit, i.e. a set of units that provide certain functionality [15]. Just as with
unit testing; components should be tested isolated from the rest of the code. Tests should try to
discover components that do not meet its functional specification. Tests on the unit and
component level are usually carried out by the development team themselves [1].
3.4.3 Integration testing
Integration tests are concerned with two or more units or components that are merged into one
larger component. All included units should have been tested individually before being merged
together into larger components [1]. Starting with only two units merged together into one
component is a good idea, and then to expand the component with more units. The test usually
goes through several real-word scenarios to confirm that the system can successfully complete
workflow tasks [16]. The integration testing can be distributed between the developers and the
testers [1].
By only including units that are already tested, with unit testing, it is likely that eventual
errors found during integration testing are likely to originate from problems with the interfaces
29. 29
between units. Knowing this is a great advantage when trying to locate the source of the error
[17].
Three strategies for integration testing exist. The bottom-up, top-down, and the umbrella
strategy. The bottom up approach suggests that the lowest level components are integrated and
tested first. These lower level modules are often referred to as utility modules. The top down
approach requires the top level components to be integrated first [17]. The umbrella, or the
middle-out, strategy performs testing along functional data and control-flow paths. Other
strategies exist, of which Big bang is one. The Big bang strategy suggests that the complete
system is integrated and tested at once.
3.4.4 System Testing
System testing aims at testing the system as a whole, rather than the behaviour of individual
components, verifying that the software meets its requirements. System testing observes things
such as performance, throughput, security, recovery, and resource loss transaction in a
characteristic working environment [16].
A system could be seen as a big component [1] and many of the tests at this level may well
be performed at the integration testing level.
3.4.5 Acceptance Testing
Acceptance testing is done when the system is ready and is often done by the customer of the
developed application. The testing is done so that the customer can determine that the software is
as good as agreed upon, and that it meets stipulated requirements [18]. Other terms used for this
testing level is Beta Testing and End User Testing.
3.4.6 Regression Testing
Regression testing is when software is retested after modifications have been made to the code. A
regression bug is something that occurs when software functionality that previously worked stops
to work or stops to work in the same way it did before functionality was added, removed or
modified [19]. The testing is done to make sure the software hasn't regressed, or taken steps back
[20]. The two main goals of regression testing are to assure system stability. This is assured by
rerunning of test cases for all modified, or possibly affected requirement attributes. The other
30. 30
goal is to achieve confidence in the quality of the software by setting an appropriate pre-defined
coverage target [21].
3.4.7 Automatic Testing
Testing an embedded system, or ordinary software for that matter, with automated tests often
saves a lot of time, especially when the amount of tests grow, even if creating suitable tests takes
a lot of time. In addition to saving time, automatic testing enables the exact same tests to run
several times, which can be used to re-verify software when modifications have been made, i.e.
regression testing. Automated tests can also often be reused when new software versions are
created. On the other hand, when the software a test is supposed to verify or test, changes, it is
possible that the test must be rewritten [22].
However, sometimes manual tests are required. As many automatic tests are used to verify
the correctness of a piece of software, most errors are detected when manual tests are made. It is
not the number of times a test is made that determines the rate of bugs found, but rather the
quality of the development of the test [23]. Creating, and performing a manual test is sometimes
faster than creating an automated test, and it must be determined whether or not it is worth the
effort of writing an automated test. Writing an automated test that is run only once is usually not
economic defensible. Before writing automated test, the life time of the automated test must be
estimated. [22].
Berner et al. mentions in [23] four common mistakes with automated test generation. Their
assertions are based on observation of five companies where the automatic testing was not
completely successful.
Wrong expectations
Expectations were that test could be written in the beginning of the test phase, and then left as
they were. The expectation was that automated tests would save a lot of money and time. But,
automated tests must be maintained regularly; otherwise they will be useless after a while.
31. 31
Misplaced or forgotten
Tests were run at the wrong level; unit testing at system level is usually hard to compose.
Another thing discovered was that many of the examined companies did not test at all levels. For
example, only unit and system tests were made, but no real integration testing.
Missing diversification
The main goal for many companies that start with automated tests is usually to save time, money
and to cut down on testing resources. This means that many of these companies start to automate
all things that were previously tested manuall. Starting to automate a subset of the existing
graphical user interface (GUI) based test is therefore common. However, creating automated tests
that interacts with a GUI is very time consuming. Graphical user interfaces tend to change often,
which means that the automated tests have to be rewritten. Testing and verification of the system
response and state may also be difficult, as the system state is not always visible.
Tool Usage is Restricted to Test Execution
It is often only test execution that is considered for automation, even if it sometimes can be more
effective to automate other processes in the test lab, such as installation and configuration
procedures. Automated procedures for creating automated tests can also be considered, but as up
to 80 % of all errors in software development are found during creation of automated tests [9], it
is important that such a procedure generates high-quality tests.
3.5 When to Stop Testing
It is hard to know when enough testing is done [24]. There is no testing method that is so
extensive and perfect that it can declare a piece of software totally bug free. There is no way to
prove that a piece of software does not contain errors. The absence of errors does not prove that
that program does not contain any bugs or defects, but simply just that no errors were discovered
with the tests used.
In reality, testing is often a trade-off between budget, time and quality. If the software
product is never released, the producing company soon runs out of business, and if a program
with lots of errors is released, the result is usually the same. The testing phase is often allotted a
certain amount of time, at the end of the development, and testing ends when this period of time
32. 32
has elapsed. This is not a good method, as it does not encourage the tester to create and run
complicated, destructive tests [25].
There are some measures that can be used to approximate the total number of errors in a
piece of software, which can be used to determine when enough testing is done. Three common
strategies [1] that can be used will be mentioned below. All three strategies are good in
themselves, but an even better approximation can be obtained if they are used in combination [1].
3.5.1 Defect Density
This is a technique where the total number of errors per line of written source code is measured.
When this value is calculated for several applications, the mean value can be used to approximate
the total number of errors in new software. The more programs used to calculate an expected
value of newly developed software, the better the approximation. The defect density is calculated
with the following formula:
Density = Errors / Lines of code
Formula 1. Defect Density
If a program A consists of 100,000 lines of code, and the total number of detected defects is 700,
then it is said that that program A has a defect density of 7 errors per thousand lines of code
(KLOC). Assume that another program, B, developed by the same development team, has a
defect density of 10 errors per KLOC. Then another, not yet tested program, C, can be expected
to have something between 7 and 10 errors per KLOC. This means, that if the company has as a
quality goal that 95 percent of all pre-release defects are removed before shipping, then
something between 665 to 950 defects should be removed.
If, when debugging program C, 600 errors have been detected, it is possible that it has been better
coded, but if 20 previously released programs has an average defect density of 8,4 defects per
KLOC with only small variations, then it is probable that program C needs more testing.
33. 33
3.5.2 Defect Pooling
With this defect prediction technique, all detected errors are divided into two different pools; say
pool A and pool B. The distinction between the two pools is arbitrary. For example, the test team
could be divided into two groups, where errors found by one group are sorted into pool A and
errors found by the other group are sorted into pool B. Another distinction between the pools
could be to separate defects found on certain weekdays into one pool, and defects found on the
other weekdays into the other pool. Both test teams test the same code, and by keeping track of
the errors detected in both pools, the number of unique errors reported could be calculated, as
well as an approximation of the total number of errors in the program. The number of unique
defects detected at any time can be calculated with the following simple formula:
DefectsUnique = DefectsA + DefectsB – DefectsA&B
Formula 2. Defect Pooling - Number of unique defects
An approximation of the total number of defects can be calculated with the following formula:
DefectsTotal = (DefectsA * DefectsB) / DefectsA&B
Formula 3. Defect Pooling - Estimated total number of defects
If a program has 400 detected defects in pool A after testing, 350 detected defects in pool B, and
150 of the defects in both pools, then the number of unique defects would be 600. The
approximated number of defects would be 933. This means that there would be approximately
333 defects yet to be detected.
3.5.3 Defect Seeding
This technique for approximating defects means that errors are intentionally inserted into a
program by someone, for detection by someone else. By measuring the ratio of intentionally
34. 34
inserted errors that are detected, the total number of unseeded errors can be approximated. The
following formula can be used:
RealDefectsTotal = (SeededDefectsPlanted / SeededDefectsDetected) * RealDefectsDetected
Formula 4. Defect Seeding
For best result, the seeded errors should be spread over the full breadth of the programs
functionality and severities. Errors that lead to program crashes as well as cosmetic errors should
be inserted.
Suppose a program is intentionally seeded with 50 errors before it is tested. Then suppose that the
number of detected seeded defects, at a point where the program is expected to be fully tested, is
31, and the number of other defects is 600. The total number of real, or indigenous, defects can
then be estimated to be 967. Accordingly, there are approximately about 350 defects that are still
undiscovered.
A problem that is common with this technique is that the group or person that intentionally
inserted defects forgets where they were inserted, or that removing the seeded errors introduces
new errors.
3.6 Testing embedded systems
What an embedded system actually is can be hard to define. However, it could be described as a
special-purpose computer system, with specific requirements, that performs predefined tasks. All
embedded systems contain a microprocessor that is programmed to perform a usually limited
number of functions. [27]. When testing embedded systems, ordinary testing techniques can
usually be used. However, there are a few things that must be kept in mind when testing these
systems.
A term that is commonly used interchangeably is real-time. That is because most, if not all,
embedded systems are real-time systems. A real-time system is one where the result of a
calculation does not only lie in the exactness of the calculated outcome, but also that the outcome
was obtained within a specified time limit. Not met time constraints means system failure. For
some systems, such as a real time embedded control system for a sky rocket, the system is safety
35. 35
critical, and failure is not an option. [28]. This means that testing timing constraints, in addition
to testing the functional behaviour, is very important when testing embedded systems.
Another thing the tester must be aware of is that the application development platform and
the execution platform usually are separated. This means that functionality that works on the
development platform, might not work in the future execution environment.
One other thing that is important to test, especially for many embedded systems for the
vehicle industry, is that the system can handle large amounts of input. The engine controller on
Scania trucks and buses is connected to many other controllers, and must be able to handle
requests by these, in addition to the information it gets regularly from its connected sensors. This
is also very important to test.
Yet another thing, important to test for embedded systems are different software
configurations of the same software version. For the engine controller at Scania, a number of
parameters differ, depending on the engine type the system is supposed to control. It is very
important that all different variations of the parameter setting are tested.
3.7 Testing the engine ECU at Scania.
The current system for testing the engine ECUs at the NEV department at Scania is a PC-based
system for simulating the surrounding of the engine ECU. The system consists of a test rig, a
computer connected to it, and test software. Apart from offering support for manual testing, it
also enables the engineers to run automatic test sequences and simultaneously read the reactions
from the tested controller.
Most of the testing is concerned with fault codes, which will be mentioned later. These
fault code tests are mostly automated, as well as the diagnosis tests that are regularly run. These
two test types are most often run in the test rig, but some tests are also run directly in trucks,
where it is verified that new versions of the controller software is not security dangerous.
3.7.1 The Test System
The test rig consists of different control and measurement cards from National Instrument. These
cards provide support for input and output of different kind and are used to enable
communication between the test computer and the test object, the engine ECU. The cards have
different functions, such as digital and analogue in- and output, digital to analogue converters,
timers, CAN-communication, and other. They can be used to simulate power failures, simulate
36. 36
circuit shortenings, generating and measuring pulse width modulation, send CAN messages, and
other. The cards are mounted in a chassis, which provides the cards with electricity and cooling,
and which also contains a 10 MHz clock that can be used for synchronisation of the different
modules.
Other hardware found in the test rig for testing and verification of the engine ECU are
relays, power limitation units, real actuators, and other. The same test rig can be used to test
various engine controller systems, and also other controller systems.
The test software is developed in the programming environment CVI, which provides good
support for controlling and measurement, as well as for graphical user interfaces. It has inbuilt
support for control of the various I/O-cards from National Instruments, that are used in the test
rig,
The developed test software contains a graphical user interface from which various
functions in the ECU can be controlled and simulated. Diagnosis variables and fault codes can
also be read and visualized directly in the test system. The software part of the test system also
contains a model of a truck. The software supports automated tests, written as scripts files
describing the various steps in each test.
3.7.2 Testing at Different Levels
Most of the testing of the engine controllers is undertaken by NEV, the department for
Verification and Tools. The software in the controllers is written by another department, the
department for engine software and some of the testing is thereby done by the engineers at that
department.
Unit testing, which is the lowest level of software testing is done by the engineers writing
the software code for the ECU. The code is written by different persons, and they all test their
software themselves, making sure new or modified functions works as planned.
Integration testing is a quite wide notion, and is sometimes divided into integration testing
in the small and integration testing in the large. The integration testing in the small, referring to
the small pieces of code merged together into larger components in done by the engineers that
produce the software. The other integration testing is done at NEV, where the engine ECU is
integrated with the rest of the truck, if however often simulated. It is tested how the ECU
37. 37
integrates with the rest of the truck systems and that it is able to work in this environment,
without changing its intended behaviour.
The system testing is done by NEV. Here, the system is the ECU, and the whole range of
functionality is tested as a whole. The line between integration testing in the large and system
testing is quite thin in this case.
The acceptance testing is to some extent also done by NEV. In the case of the engine ECU
at Scania, it is the company itself that is the customer, and it NEV that verifies that stipulated
requirements are fulfilled. Another form of acceptance testing is the security tests that are done
on a test course with real trucks. It is then verified that the ECU can handle security critical
failures without affecting the security. Other acceptance testing is done by partner road carrier
companies. Then the acceptance testing is done through field tests, where real trucks are
equipped with the ECU and tested on roads worldwide.
Regression testing is also done by NEV, and mainly constitutes of the different automated
tests. These tests verify that new or modified functionality in the ECU software has not affected
other functionality. My work in this thesis will improve the regression testing, by providing a
system that gives the developers of the software very fast response on changed or added software.
3.7.3 The engine management system
The ECU is the central part of the EMS, the Engine Management System. The EMS also
comprises all the sensors and actuators connected to the ECU. The engine ECU used in
production on current Scania heavy trucks is called S6. It is composed of both hardware and
software. The software contains the actual controlling logic and lies on a small flash memory in
the unit. This controller is mounted directly on the engine it controls.
38. 38
Figure 6. The S6 ECU
To control the engine properly, the ECU software holds about 3 000 engine specific
parameters. The S6 communicates with other ECU’s on the truck by means of a Controller Area
Network bus (CAN-bus). The ECU controls things such as the starter engine, EGR (Exhaust Gas
Recirculation System), the cooling fan, exhaust braking, and perhaps most importantly the fuel
injection system. Typical input data originate from things such as boost temperature, oil pressure,
engine temperature and speed.
The controller is constantly connected to the truck’s battery. It is supplied with power, and
starts as the signal from the ignition goes logic high, that is, when the driver turns the key.
The ECU functions in broad outline as a computer and is also constructed as one, with an
internal central processing unit, CPU, some memory and firmware.
The engine ECU is usually the most powerful ECU of all the ECUs on modern vehicles. It
uses closed-loop control, where the system is continuously fed with sensor data about real engine
conditions in real-time. It uses sensor data obtained from sensors in the engine to perform
complex calculations millions of times each second. These calculations then underlie the
regulating of different parts of the engine.
39. 39
History of the engine ECU
One of the foremost reasons as to why the vehicle industry once had to abandon the carburettor as
a supplier of fuel to the engine, and start to use electronic controlling units instead, was to meet
new and increasingly stricter emission laws and requirements. By controlling, for example, the
fuel and air mixture that is used during the combustion, a great deal of the exhaust pollution can
be reduced. Other main benefits with an ECU are, in addition to the reduced emissions, that it
enables improved fuel consumption, and an extended life for the engine.
3.7.4 Hardware-in-the-loop-testing
When testing of the engine ECU is done in the test rig, the various parts that are connected to the
ECU are simulated. This is called Hardware in the loop simulation, or just HILS. With HILS, real
hardware is tested against simulated hardware. This means that a real ECU is tested against
simulated sensor signals and other messages transferred over the CAN bus.
At Scania, both closed loop and open loop simulation testing is used. Closed loop means that the
test system is closed. The simulated hardware receives information from the tested hardware, the
ECU, and simulated models manages it, before it in its part, sends simulated data back to the
ECU. The ultimate variant of closed loop testing of the engine ECU is when testing is done
directly in the truck, which however, cannot be called hardware in the loop simulation, as nothing
is simulated. With open loop simulation the surrounding hardware is still simulated, but no
models, managing the signals from the tested hardware, are used. The signals sent to the tested
hardware are sent directly.
40. 40
Figure 7. The Hardware-In-The-Loop Simulation System at NEV, Scania
Figure 6 illustrates the hardware in the loop system at Scania. Everything is initiated from a test
case, which sends instructions to a dynamic model, which manages this data and sends input to
the ECU being tested. The ECU manages the data, and reacts in some way. All output from the
ECU is returned to the dynamic model as input, which is managed and used for new input to the
ECU, and so it goes. All steps in the simulation are observed as state data, which is compared to
expected values, defined in the different test scripts. All results are then inserted into a report that
contains information about all steps in the test sequence and the outcome of each step.
3.7.5 Fault code testing.
Most of the testing at the NEV department at Scania is concerned with fault code testing. The
engine ECU produces fault codes that are stored in memory when it considers something wrong.
This could be for example that it at some point registers the engine temperature to be higher than
expected or that it discovers some electrical failure on some actuator or some other connected
device.
41. 41
These fault codes can be read, and it can be determined if the fault is active, how many
times it has appeared, and so on. This information can be used when testing. Tests are constructed
so that the controller should produce certain fault codes, and then the test verifies that the correct
error codes have been produced.
Fault codes are common in the vehicle industry, and can be used to locate errors in
vehicles. Mechanics have instruments to read fault codes from the different controllers on the
vehicle. In the car industry, the term diagnostic trouble code (DTC) is commonly used.
42. 42
4. Building a System for Daily Build
A daily build system is one where software is build out of the latest version of a program each
day so that it can be tested, in order to be verified that no new bugs are introduced [29]. Having a
daily build system is often important for larger software development systems, where the
software system is written by many different programmers, each responsible for their own part. It
is important that the developers can receive feedback on how their modified or added code is
affecting other code.
The sections in this chapter are about things with relevance to the daily build system
created in this thesis.
4.1 Windows Task Scheduler
Running things automatically in Windows 2000 is fairly easy, depending on what is scheduled to
run, of course. However, the principles are simple. With the Windows task manager, you just
specify which program, or script for that matter, to run, and when to run it [30]. You specify
which user the program should be run by, which Windows user account to use. This means that
no one has to be logged on to the computer, or that any one could be logged on, the scheduled
task will still run, as long as the user that “runs” the program is authorized to do so.
However, there are a few things that make it harder to run without anyone being logged on
to the computer. That is that paths in the file system that are possibly used by the running
application are not defined the same way as if the user was logged on to the system. That means
that network drives, for example, must be specified with an absolute path, and not with a letter,
such as O:, which is a mapped path, specific for each user.
43. 43
Figure 8. Screenshot from Windows Task Manager
4.2 Batch Scripting
Batch scripting is very straight forward. A batch script file, with the extension .bat or .cmd is a
file containing nothing but commands intended to be executed by the command interpreter in
sequential order. That means MSDOS commands in an unformatted text file. There exists no
more than eight DOS commands, and some miscellaneous things such as replaceable parameters,
environment variables, labels, double “== “signs, and @ signs [31].
Limited as it might seem, batch scripting is very powerful and can be used for many tasks,
such as replacing text in a text file, starting other programs and other. A batch script can also be
used to initiate a sequence of processes. Common programming logic, such as conditional clauses
and loops, can be used.
4.3 VBScript
VBScript, or Visual Basic Script Edition, with the file extension .vbs if used as a standalone
application, is, as the name implies, a script language, based on the programming language
Visual Basic. It is created by Microsoft, and the visual part refers to the window handling, used in
44. 44
the Windows operating system. VBScript significantly support Windows administrators with
need for automation tools more powerful than the previously mentioned batch-script. VBScripts
can be used to do a number of tasks, and a lot of scripts are available on the internet. Scripts for
replacing text in files, managing files and a lot more. They can also be used for automatic e-mail
sending, a fact that has made VBScript very popular among ill-disposed persons that use it to
compose harmful internet worms.
4.4 Programming in CVI
CVI is a programming environment from National Instrument, which features generation tools
and prototyping utilities for C code development. It contains libraries for many commonly used
programming tools, for interaction with other Windows applications, such as MS Word and MS
Excel, and good support for creating graphical user interfaces. However, the main benefit with
CVI is that it has a number of libraries developed for communication with various hardware
components from National Instrument.
CVI is used at Scania, and the test rig contains a lot of components from National
Instruments. The software for the engine ECU is written in C, and it is a good idea to have the
tests written in the same language, not least because the knowledge of C-programming is good at
the department.
45. 45
Figure 9. LabWindows/CVI from National Instruments
4.5 The CAN-Network
CAN or Controller Area Network is a serial bus system with real-time support and transmission
rates up to 1 Mbit/s for distances up to 40 meters. It is constructed to be stable in noisy
environments, such as in cars and trucks. All CAN messages contain only small amounts of data,
between one and eight bytes. The data coding is done with NRZ-L, Non Return to Zero-Level.
All nodes in a CAN network can send messages. The CAN protocol uses so called
broadcasting, where a message from one node is sent to all other nodes. It is up to the receiving
node to decide whether or not the message is relevant. All nodes in the network communicate
with the same bit rate. The CAN system provides a multi-master hierarchy, which means that if
one node in the network is defect, the communication between the remaining nodes is not
affected [32] .One of the major advantages of the CAN is the very high transfer reliability.
At Scania, CAN is used to build the network that connects the different electronic control
units, sensors, and actuators in Scania heavy trucks. The CAN bus use at Scania uses CSMA/CA,
46. 46
Carrier Sense Multiple Access with Collision Avoidance, protocol. Carrier sense means that a
node in the network waits before it sends a message if another message is currently being sent on
the bus by another node. Multiple access means that all nodes in a CAN can transmit and receive
traffic. With Collision Avoidance sending nodes must first inform the other nodes of its intention
to transmit. In order to maintain synchronization in the system, bit stuffing is used. CAN uses
twisted pair cables as media.
Scania uses a communication protocol called Higher Layer Protocol, HLP, which admits bit
rates of 250kbit/s. Messages in the Scania heavy truck networks use the extended format with 29
identifier bits, in comparison with the standard message format with 11 identifier bits. [33, 34,
35]
48. 48
5. Implementation and Results
5.1 Creating automated tests for the engine ECU
5.1.1 Introduction
Testing the engine management system is very important. An error in the ECU could, in the
worst case, effect the driving safety. It is very important that new or changed ECU features are
tested continuously. All changes, in soft- or hardware, must be thoroughly tested, under all
possible circumstances. In former times, all tests were made in real trucks, with real components.
Testing directly in trucks is not only very expensive, it is also very slow, and perhaps above all it
is impossible to test everything that must be tested.
Testing the starting motor logic in the engine controller is important. Not only is it central to
verify that the engine starts when the driver switches the ignition key and when all other starting
conditions are fulfilled. It is also of great importance to test that the engine controller does not
activate the starting motor if not all starting conditions are met. Starting the engine when it should
not be started could damage the engine or parts connected to it; or in the worst case affect the
safety of people inside or surrounding the vehicle.
5.1.2 Testing the ECU Functions for the Starting motor
In order to test the starting motor, specifications for it were thoroughly read. The system
description for the Scania ECU contains some information about how the ECU controls the
starter. Another document contains specification about parameters with relevance for the starter,
and yet another document has descriptions on various user functions, describing how the ECU is
expected to act on various messages received, for example to a request for starting the engine.
These documents were carefully read, which could be considered as a form of static testing.
A test case was designed, based on the documents read. The test case was written as a C-file with
steps that were the same for all different tests belonging to the created case. The different tests, or
test scripts, were defined in text files. These test scripts sets different ECU variables, and defines
expected values and possibly fault codes.
49. 49
The tests were written as system tests, where parts of the complete system were tested.
These tests were then run on a real ECU the test rig available at Scania. It should be pointed out
here that it is not the starter itself that has been tested, but the functions in the EMS controlling
the starter. It was tested that the ECU sent out a signal on its digital output, trying to activate the
starter when the different conditions were fulfilled, and that it did not try to activate the starter
when these conditions were not all fulfilled. That means that both negative and positive tests,
mentioned in section 3.1 were made.
5.1.3 The Test Scripts
Except for the test case file, which more or less calls the different steps from the test scripts, a
number of test script files were written. They will be described in turn below. Flowcharts of them
can be found in appendix 1.
5.1.3.1 Test engine started reference speed
The purpose of this test was to make sure that the starter does not activate the starting motor
when the engine has reached the speed when it is considered started, but that it keeps activating
the starter as long as the engine speed is just below the reference speed.
First, as was done in all test scripts, it was tested that the starter was activated under normal
conditions, when the starter conditions were all fulfilled. This was done to make sure that the
system initially worked as expected.
After the first step, the engine was restarted, the ignition key was switched on again, and
the engine speed was set to a speed just below the reference speed. It was ensured that the ECU
was still activating the starter. The next step was to restart the engine once again and then set the
engine speed just above the reference speed when the engine is considered started. It was checked
that the ECU was not activating the starter when the engine was running at this speed.
The last step was to once again restart the engine, with all conditions fulfilled and the engine set
to low idle, around 500 rpm. It was made sure that the ECU did not try to activate the starter
when the engine was running at this speed. This step was done in all tests and was to ensure that
the ECU was still working.
50. 50
5.1.3.2 Start with engine running
The purpose of this test script was to ensure that the engine ECU does not try to activate the
starter if the engine is already running when a demand for starting motor is sent to the ECU. It
should not matter at which speed the engine is running. As long as it is running, the starter should
not be activated. As with all tests, a demand for starting motor was sent to the ECU with all its
activation conditions fulfilled, to make sure it is working.
5.1.3.3 Activation time after reference speed is reached
When the engine reaches a speed above a defined reference speed, where it is considered started,
the ECU shall not try to deactivate the starter immediately, but wait a certain time to make sure
that the engine really is started. The purpose of this test was to ensure that the ECU was still
activating the starter for a specified amount of time, after the engine reached the speed where it is
regarded as started. It was also tested that the ECU stopped activating the starter just after the
specified time.
5.1.3.4 Maximum activation time before engine is started
This test was composed to verify that the ECU stopped activating the engine after a certain
amount of time if the engine does not reach the speed where it is considered started. If the engine
is not started within this time limit, the ECU should also produce a fault code, indicating that
there was a problem starting the engine. The test also verified that this fault code was created.
5.1.3.5 Maximum number of failed starting attempts
The purpose of this test was to verify that a fault code was created after a certain number of failed
attempts of starting the engine. It was verified that this fault code did not appear until the
specified number of attempts were reached.
5.2 Creating a System for Daily Builds
5.2.1 Introduction
This implementation started after it was clarified exactly what was wanted of the system, and a
lot of texts about other daily build systems were read. However, as a test system for automatic
tests already existed at Scania, as well as other parts of the daily build system, the number of
51. 51
implementation methods was limited. The available systems were scrutinized before the actual
implementation began. The following sections describe the various steps of the implementation.
Figure 9 shows the steps in the implemented daily build system.
Figure 10. The Implemented Daily Build System
5.2.2 Checking Out Source Files and Building the ECU Software
Prior to this thesis work, there already existed simple batch scripts for checking out source files
from the version management system, and other scripts that could be used for compiling these
source files. These scripts were used, with a few modifications, to check out and build software
for the different engine types. They were also merged together to reduce the number of steps in
the daily build system, and to reduce the number of script files used.
The modifications that were made were made because of the problems that originated from
the Windows task scheduler. The main problem was that when the daily build machinery was
scheduled to run at times when no user was logged on to the test computer, the paths defined in
the scripts files could not be used. In MS Windows 2000, as well as with other versions of the
operating system, the users can map network drives to various letters, such as E:, F:, G:, and so
52. 52
on. The problem is that when no user is logged on to the system these mappings do not exist. This
means that all references to such network drives had to be replaced by absolute paths.
In addition to getting the latest source files from the version management system, the
scripts also fetched a make script file. This make file was used to compile and link all source files
checked out from the version management system. The make file is a script file with references to
other programs that does the actual building of the software. This file also contains paths to
various programs with these letter mappings. To replace these with absolute paths, a VBScript
file was constructed. This file did nothing else but to replace all known letters to their absolute
paths. The VBScript file was called from the original batch build script file.
Another problem that was raised was that when the version management system tried to
check out the latest version of the just mentioned make file and found that such a file already
existed, it asked the user if it wanted to replace the already existing file. However, when no user
is logged on to the system, no one can answer that question, and no one knows that the question
has been asked. The problem is that the system was not able to continue until that question was
answered. This problem was solved by calling yet another VBScript that renamed the existing
file, with the same name, but with a new extension. If a file with this name, and extension,
already existed, which it most often does, as the program runs every night, the old file is deleted.
5.2.3 Flashing
The software in the engine ECU lies on a flash memory, onto which it can be loaded by means of
a CAN-bus. There exists software for flashing controllers at Scania. This software has been
modified so that it is now able of being controlled from external applications or from a command
line, given the path to the binary software file to load onto the flash memory.
The daily build system uses this to flash a controller with the latest software available. After
getting the latest files available from the version management system, compiling these files, and
creating proper software, the daily build software flashes the controller mounted in the test rig.
This is done directly from the test program, when it is known what sort of controller that is
currently mounted in the rig.
As has been hinted throughout this report, there exists more than just one controller system
for engines at Scania, of which the current S6 is one. In the daily build system created in this
thesis, support for flashing, or loading of software onto the controller’s flash memory, is not
53. 53
available to all these systems. It was expected that it would take disproportionate amounts of time
to create this functionality for all systems. However, support for this is expected to be developed
in a near future, and there should not be any problems integrating this into the current system for
daily builds.
5.2.4 The Start Script
All steps in the daily build system were initiated from a test script. It is this script that is started
from the Windows task scheduler every night. First of all, the script calls the different build
scripts, mentioned earlier. After building program files for all ECU types, the script starts the test
program with certain parameters, which makes the test program check which ECU is mounted in
the rig. This result is then returned to the start script. Depending on which ECU that is mounted
in the rig the start script continues execution in a subroutine. All subroutines work the same way,
by calling the test program with appropriate arguments, depending on the engine type that is to be
tested. After the testing is finished, the mail script is called, which finishes the start script, and the
daily build system.
5.2.5 The Test Software
Most work with this part of the thesis was concerned with modifying the existing software system
for engine ECU tests. Changes were made so that the program could be started with arguments,
specifying things such as when to stop the testing and how many cylinders the engine ECU
should control, and the path to the binary software files to be programmed onto the controller’s
flash memory and then tested.
Some functions had to be modified and others had to be added. There are a lot of functions
concerned with initiating the system, such as specifying paths to description files for the ECU
and establish connection to the test hardware. Many of these functions had to be modified to suit
the daily build system.
It had to be made sure that no functionality was dependent on user interaction, as this
would not be possible in the daily build system. As this software system is rather complex and
extensive, these modifications had to be made in man places in the code, and also very scattered
in different software modules. Another thing that had to be managed was to change certain paths
that are used by the system, as network drives that are mapped with letters cannot be used.
54. 54
5.2.6 Running of Tests
The software system was modified so that it started all available tests for the currently connected
ECU. The system checks which ECU is currently mounted in the test rig. It then searches through
a Microsoft Excel file, containing all existing tests. Each engine type is assigned a column in the
Excel document, and each test script is assigned a row. For each test script, the system checks the
cell in the column corresponding to the current engine type. The cell can contain information that
means that the test is applicable for all engine configurations for this specific engine type, or the
cell contains one or more parameter restrictions,
5.2.7 The daily build report
A report generating system was created. It was integrated with the current test software. It
contains plain html, and is mailed to the concerned testers as the tests are complete. This report
contains information about potentially erroneously created software for the ECU, and test scripts
that were not passed during the preceding nights daily build run. It also contains information
about tests that were not run, because the program was forced to quit, due to time constraints.
In addition to this information, it also includes a link to the more extensive test report that is
created by the test system. This implies that this extensive test report must be accessible for all
testers that receive the mail. Usually, the test program just saves a copy locally, on the test
computer, but modifications were made to the system so that when the test program is started as a
daily build, a copy is saved on a network drive. A sample test report is shown in figure 10.
56. 56
6 Discussions
The two main goals in this thesis were to construct a system for daily build and to create a test
case for the starter functions in the engine ECU. A considerable part of this thesis has also
focused on the field of software development methods and on testing and verification at different
levels.
6.1 The Automated Tests
When it comes to the testing of the controller logic for the starting motor a number of automated
tests were created and tested thoroughly and can now be used for feature versions or “sub-
versions” of the engine controller software. My guess is that these tests can be used for a long
time, as I think that the starting motor logic will work pretty much as it does today for a long
period of time to come. Perhaps some of the test parameters must be updated to work properly in
the feature, but that is easily done.
6.2 The Daily Build System
The future success of the daily build system created during this thesis lies on how much it will be
used, and how many errors that are found in the controller software at an early stage. Another
factor influencing on the future success rate is how well integrated it will be with other, not yet
developed functionalities. However, one of the main goals during the development of the system
was to construct a system that can easily be integrated with other systems, and that can also easily
integrate new functionality. I think that this goal was attained.
The daily build system is, and will be, valuable to the developers of the software for the
engine controller. However, the current system has its limitations, and it is likely that the system
must be supplemented with extern functionality that was impossible to be included at this stage.
As long as it is not possible to set parameters arbitrary in the engine controller software in the
daily build system, all tests cannot be run, and some of the test that can be run may return false
results. Yet another thing that must be solved is support for testing all types of engine controllers.
Different controller systems exist, depending on the engine it is set to control, and all different
controller systems are not supported.
57. 57
The good thing is that it is expected that these things will be solved in a not so distant
future, and if these limitations are solved, it is likely that this system will become valuable for the
development and testing teams at Scania for a long time.
Other limitations that exist with the daily build system are concerned with the desire of
being able to run the system without anyone being logged onto the test computer. For example it
is not possible to know if the system gets caught in a state and being unable to continue
executing. If this happens, the testers are not able to understand what has happened as nothing
will be explained when someone logs on to the computer. A simple solution to at least giving the
testers an idea of in which state the system has got caught in, is to regularly print output to a log
file that is saved on the hard drive. Another thing to use could be some sort of time-outs, where
the system is shut down if certain stages of the execution are not finished within a defined time
frame. None of these solutions were implemented in the system, but could most certainly very
easily be added.
One potential problem with the system is that it is divided into many different parts, with
small script files, which purposes are to solve simple tasks. On of the risks is that it could be hard
for the users to maintain the system, if changes must be made to more than just one module. One
of the main reasons as to why the system is split up into smaller pieces is that the applications
that interact with the daily build system cannot interact with the other applications in the daily
build system. Most of the external applications used in the daily build system are developed at
Scania, and it would have been desirable if these were integrated to a greater extent. However,
rebuilding them and integrating them into a large system would probably take a very long time
and would therefore be very expensive. A large system could however be less adaptable to
changes to a small part of the functionality.
In spite of the limitations mentioned, I consider that the constructed system in fact fulfils its
purpose, as it increases the degree of use of the test equipment, and makes testing easier for
developers and testers. The developers are given a quicker response to new and modified
functions in the controller software as they must not wait for the testers to manually start the
automatic tests, which, when run during normal working hours, limits the access to the test rig
and prevents the testers from running special tests that are required to be run manually.
58. 58
The expectation is that this system actually will shorten the development time for new
versions of the controller software, as the daily build system makes it possible to detect possible
bugs at an early stage, which makes it easier to remove or modify erroneous code.
6.3 Other
It has been very interesting to read about different techniques for software development and
testing, and to observe how these things are done at one of the world’s leading engine developers.
As the main assignment for the department for verification and tools at Scania is concerned with
the process of testing software for the engine, and other parts of the power train; it felt natural to
engross in the subject of software development and testing.
The software development for the engine controller at Scania functions in broad outline like
the Spiral model. It is not clearly stated that this software development method should be, or is,
used, but the work is done with incremental releases which all are thoroughly tested and verified
and that the development of each release in principal follows the waterfall model. As mentioned
before, the waterfall model is not an optimal software development technique, but with each time
period between releases being relatively short, the development works pretty fine.
All newly developed software for the engine controller is first tested at a lower level, as
units with unit testing, and sometimes also with other small units in integration tests by the
developers themselves. It is usually when these steps are passed that the developed software
modules are tested, preferably with the daily build system developed in this thesis, with the whole
system at the department for verification and tools. As mentioned above, I am convinced that the
daily build system created in this thesis will be of great value for the development and test teams
of the engine controller systems.
60. 60
References
[1] Beizer, B., (1990). Software Testing Techniques. 2nd
ed. ISBN 1-850-32880-3. International
Thomson Computer Press.
[2] Integrating Software Parts Wherever Homepage, (1998). The Cost of Poor Quality Software.
http://www.ispw.com/articles/Computer%20Finance.htm. Accessed 2005-09-15.
[3] Lux Group (2003). Project Lifecycles: Waterfall, Rapid Application Development,
and All That. http://luxworldwide.com/about/whitepapers/lux-project-lifecycles.pdf. Accessed
2006-04-17.
[4] Extreme Programming Org homepage, Unit Test Framework
for a Coffee Maker, http://www.extremeprogramming.org/rules/unittests.html. Accessed 2006-
02-13.
[5] Spillner, A., (2002). The W-MODEL – Strengthening the Bond Between Development and
Test. ed. STAReast´2002, Orlando, Florida, USA, 15.-17. May 2002.
[6] Systeme Evolutif Limited, (2005). The W-Model. http://www.bitspi.com/
company/rbtambiguityrevs.html. 2005-09-15.
[7] Als A., Greenidge C., (2003) The Spiral Model. http://scitec.uwichill.edu.bb/cmp/
online/cs22l/spiralmodel.htm. Accessed 2006-02-13.
[8] Chen, P., Agile Programming, http://www.codeproject.com/gen/
design/agileprogramming.asp. Accessed 2006-02-25
[9] Eriksson P., Petersson J., (2006) Implemetation of a System för Automatic Software
Verification. Master’s thesis, Dept. of Computing Science, Umeå University.
61. 61
[10] Wikipedia, Agile software development, http://en.wikipedia.org/wiki/Agile_programming.
Accessed 2006-02-25.
[11] Wells D., Extreme Programming: A gentle introduction.
http://www.extremeprogramming.org/ Accessed 2006-02-25
[12] Microsoft Developer Network Homepage, Unit Testing, http://msdn.microsoft.com/library/
default.asp?url=/library/en-us/vsent7/html/vxconunittesting.asp. Accessed 2006-02-13
[13]. Ahamad, (2005). Unit Testing, http://testingsoftware.blogspot.com/2005/07/unit-
testing.html. Accessed 2006-02-13.
[14] Extreme Programming Org homepage, Unit Tests.
http://www.extremeprogramming.org/example/utframe.html. Accessed 2006-02-13.
[15] McGregor J. D., Component Testing,
http://www.cs.clemson.edu/~johnmc/joop/col3/column3.html. Accessed 2006-02-13
[16] Sysoptima.net homepage.Software Testing - Unit Testing, Integration Testing, Regression
Testing, System Testing and User Testing, http://www.sysoptima.net/software_testing/testing.php
[17]. Microsoft Developer Network Homepage, Integration Testing, http://msdn.microsoft.com/
library/default.asp?url=/library/en-us/vsent7/html/vxconintegrationtesting.asp.
Checked 2006-02-13
[18] Wikipedia, User Acceptance Testing. http://en.wikipedia.org/wiki/User_acceptance_testing.
Accessed 2006-02-13
[19] Wikipedia, Regression Testing, http://en.wikipedia.org/wiki/Regression_testing. Accessed
2006-02-18
62. 62
[20] Ranger P., A Few Words About Regression Testing, http://www.automatedqa.com/
techpapers/regression.asp, Accessed 2006-02-18
[21] Chen Y., Probert R. L., Sims P., Specification-Based Regression Test Management with Risk
Analysis: A New Regression Test Selection Approach. http://www.site.uottawa.ca/
~ychen/Paper_CASCON02.pdf. Accessed 2006-02-17
[22] Kandler, J., (2000). Automated Testing of Embedded Software, [This paper was originally
presented at a STAR conference produced by Software Quality Engineering. For more
information on this conference, visit the current STAR Web site.]
[23] Berner S., Weber R., Keller K. R., (2005) Observations and Lessons Learned From
Automated Testing. Proceedings of the 27th
international conference on Software engineering.
[24] Melzer, I., (1996). http://www.mathematik.uni-ulm.de/~melzer/thesis/node28.html,
Accessed 2006-02-13.
[25] Pan, J., (1999). Software testing. http://www.ece.cmu.edu/~koopman/des_s99/sw_testing.
Accessed 2006-02-13.
[26] McConnell, S. Gauging Software Readiness With Defect Tracking, Best Practices, IEEE
Software, Vol. 14, No. 3, May/June 1997
[27] Wikipedia, Embedded System, http://en.wikipedia.org/wiki/Embedded_system. Accessed
2006-02-20.
[28] Encontre V., Testing Embedded Systems: Do you have the GuTs for it? http://www-
128.ibm.com/developerworks/rational/library/459.html. Accessed 2006-02-20
[29] Wikipedia, Daily Build. http://en.wikipedia.org/wiki/Daily_build. Accessed 2006-02-13.
63. 63
[30] Microsoft Developer Network Homepage, Windows 2000 Task Scheduler.
http://www.microsoft.com/technet/prodtechnol/windows2000serv/maintain/featusability/tsksched
.mspx. Accessed 2006-02-20.
[31] Van der Woude, R., Batch files for DOS, OS/2, Windows 95/98, NT 4, 2000 and XP,
http://www.robvanderwoude.com/ Accessed 2006-02-20.
[32] CAN in Automation homepage. Controller Area Network (CAN), an overview.
http://www.can-cia.org/can. Accessed 2005-08-25.
[33] Karlsson, P., Pettersson, F., (2004). Kartläggning av CAN-nätverk. Master’s thesis, Dept. of
Science and Technology, Linköping Institute of Technology
[34] Robert Bosch GmbH homepage. Controller Area Network (CAN).
http://www.semiconductors.bosch.de/de/20/can/index.asp. Accessed 2005-08-25
[35] Wikipedia. Controller Area Network. http://de.wikipedia.org/wiki/
Controller_Area_Network. Accessed 2005-08-25.
65. 65
ECU activates the starter?
Start with initial
conditions.
Step failedNo
Restart without error.
Engine speed set above
start reference speed.
Yes
Time is less than reference
time. Starter still active?
Step failedNo
Restart without error.
Engine speed set above
start reference speed.
Yes
Starter still active after
reference time?
No
Yes Step failed
Restart engine and set
engine speed to low idle
(500 rpm), engine started.
ECU stops activating
starter?
No Step failed
Yes
Test script passed Test script failed
Test reference time when engine has reached a speed that
indicates that the engine is started
