"Software Testing is the process of executing a program or system with the intent of finding errors.", Myers, 1979. The most important activity in this process is designing the required set of effective test cases. Thus, the problem is narrowed down to determining the exact number of required test cases and increasing their effectiveness. Project FoX is a production ready tool developed in Java, which offers Java developers the opportunity to leverage the proven theories and concepts of formal testing using generalized state automata (X-Machines) as a theoretical model of computation. The formal testing strategy FoX is applying, is proven to generate a complete test set that ensures the correctness of the implementation with respect to the specification. FoX enhances a novel testing process that is fully automated, ranging from complete test set generation, to test preparation and execution. This method can be applied to any Java based software system, regardless of its underlying technologies. Utilizing a formal approach will provide unambiguous test cases which are objective and not subjective to the tester's experience and intuition. The formal testing strategy provides functional testing that tests not only for the desired system behaviour (the system does what it should) but also tests that the system has no undesired behaviour (the system does not do anything it should not do). This short presentation will strive to give the audience an overview of the formal testing methodology and a demonstration of the tool (FoX). It will also showcase a real-life demo of the project FoX, applied to a Java SE application and will discuss how the methodology can be applied to any Java EE or ME application. Anyone with a software engineering background will be able to easily follow the talk and understand the benefits which this process offers to modern day software engineering.

1. PROJECT FOX
A TOOL THAT OFFERS AUTOMATED TESTING USING A FORMAL
APPROACH
Ivo Neskovic
CITY College Thessaloniki, an International Faculty of the University of Sheffield
462

2. AGENDA
> SOFTWARE ENGINEERING: WHAT COULD GO WRONG?
> FORMAL METHODS
> PROJECT FOX
> CASE STUDY: THE BUFFER SYSTEM
> CONCLUSIONS AND FUTURE WORK
> BIBLIOGRAPHY
2

3. THE PROBLEM OF SOFTWARE ENGINEERING
> Faulty systems are a common notion nowadays.
> SE is an engineering discipline, yet lacking the engineering formality.
> Subjective and informal testing.
> Impossible to prove that the system:
– Does what it is supposed to do.
– Does not do what it is not supposed to do.
> Needs structured and precise system designs.
3

4. FORMAL METHODS
> The applied mathematics of computer systems engineering, used to specify and
model the behaviour of a system and mathematically verify that the system
design and implementation satisfy functional and safety properties.
> Specification Languages:
– Abstract State Machines
– Generalized State Machines
– Communicating Sequential Processes
– Specification and Description Language
– Petri Nets
– Temporal Logic of Actions
– B and event-B method
– Z
4

5. FORMAL METHODS AT A TRIAL
> Benefits:
– Specification may be used as a basis for proving the presence or lack of
certain properties in the design and by inference in the developed system.
– Mathematical proof of correctness (Theorem proving).
– Model checking (Proving desired properties in a design).
– Formal Testing.
> Used mainly for safety critical systems such as aerospace engineering.
> Criticism:
– Expensive and time consuming approach (though questionable).
– Lack of tooling support.
5

6. INCORPORATING FORMAL METHODS IN THE
DEVELOPMENT CYCLE
6

7. PROJECT FOX
> Produce the complete set of test cases from a formal specification.
> Execute the tests on the systems implementation.
> Locate errors and non-equivalences and report them to the user.
> Developed in Java for Java.
> Compatible with Java Standard Edition, Enterprise Edition, Mobile Edition.
> Can be extend to work in conjunction with popular Java frameworks.
> Operates on compiled bytecode with the addition of a few specific annotations.
> Utilizes the test drivers of JUnit.
> FoX provides a bridge between regular Java developers and the benefits of
complete positive and negative testing, proven to find all faults.
7

8. USING PROJECT FOX
> Two artefacts necessary:
– Formal specification of the
system.
– The system's implementation.
8

9. BUFFER CASE STUDY – DESCRIPTION
> Simple buffer in a factory.
> Accepts parts, any parts.
> Parts have a name and an ID.
> The buffer has a capacity of 2.
> The buffer can be empty, partially full
or completely full.
> Supports adding and removing items.
> If the capacity is reached, no
additional items can be placed in the
buffer unless an item is removed first.
9

10. BUFFER CASE STUDY – FORMAL
SPECIFICATION
> Modeled as a Generalized State
Machine (stream X-Machine).
> A theoretical model of computing,
pioneered by Samuel Eilenberg in
1974 (X-Machine).
> Separates flow control from
processing.
> Flow control is abstracted to a level
suitable for representation as a finite
state machine.
> Complex data structures are modeled
as an infinite memory.
> Able to model both static (data) and
dynamic (control) parts of a system.
10

11. BUFFER CASE STUDY – FORMAL
SPECIFICATION (cont.)
> Simple buffer in a factory.
< xMachine name = " Buffer " >
> The buffer can be empty, partially full or completely full.
< states >
< state initialState = " true " > empty </ state >
< state > non_empty </ state >
< state > full </ state >
</ states >
11

12. BUFFER CASE STUDY – FORMAL
SPECIFICATION (cont.)
> Accepts parts, any parts.
< input name = " part " ref = " BufferObject " / >
> The buffer has a capacity of 2.
< types >
< builtInType name = " capacity " type = " integer " / >
< builtInType name = " buffer " type = " set: BufferObject " / >
</ types >
< memory >
< memoryBlock ref = " buffer " initialValue = " null " / >
< memoryBlock ref = " capacity " initialValue = " 2 " / >
</ memory >
12

13. BUFFER CASE STUDY – FORMAL
SPECIFICATION (cont.)
> Parts have a name and an ID.
< types >
< complexType name = " ItemType " >
< attributes >
< builtInType name = " type " type = " string " / >
</ attributes >
</ complexType >
< complexType name = " BufferObject " >
< attributes >
< complexType name = " type " ref = " ItemType " / >
< builtInType name = " itemId " type = " integer " / >
</ attributes >
</ complexType >
< /type > 13

14. BUFFER CASE STUDY – FORMAL
SPECIFICATION (cont.)
> Supports adding and removing items.
< functions >
< function name = " add_part " >
< guard >
!buffer. contains ( part ) && buffer . size () + 1 < capacity . value ()
</ guard >
< body > buffer . add ( part ) ; </ body >
< output > Part Added </ output >
</ function >
...
</ functions >
< transitions >
< transition >
< startingState > empty </ startingState >
< appliedFunction > add_part </ appliedFunction >
< endingState > non_empty </ endingState >
</ transition >
...
14
</ transitions>

15. BUFFER CASE STUDY – IMPLEMENTATION
public class ItemType {
private String type;
public ItemType(String type) {
this.type = type;
}
}
public class BufferObject {
private int itemId;
private ItemType type;
public BufferObject(int itemId, ItemType type) {
this.itemId = itemId;
this.type = type;
}
15
}

16. BUFFER CASE STUDY – IMPLEMENTATION
> @Xmachine ­ annotating the class representing the system modeled with the
specification.
> XMachineModel – a class representing the model, containing a number of
useful helper methods.
@XMachine(inputType = "BufferObject",
sampleInputs = {
"integer: 10, ItemType: (string:Box)",
"integer: 17, ItemType: (string:HeavyBox)",
"integer: 25, ItemType: (string:ReallyHeavyBox)",
"integer: 20, ItemType: (string:Dragon)",
"integer: 17, ItemType: (string:Planeswalker)",
"integer: 187, ItemType: (string:Nekrataal)",
"integer: 23, ItemType: (string:Michael Jordan)"
})
16
public class Buffer extends XMachineModel {

17. BUFFER CASE STUDY – IMPLEMENTATION
> @XMMemoryBlock – a field level annotation, associating Java data structures
with their specification equivalents
@XMMemoryBlock(name = "buffer")
private List<BufferObject> buffer;
@XMMemoryBlock(name = "capacity")
private int capacity;
public Buffer() {
super("Buffer");
buffer = new LinkedList<BufferObject>();
capacity = 2;
}
17

18. BUFFER CASE STUDY – IMPLEMENTATION
> @XMFunction – a method level annotation, referencing the modeled functions
implementations.
> reportOutcome( outcome: String) – one of the many helper methods
of the XmachineModel class.
@XMFunction(name = "add_part")
public void addPart(BufferObject part) {
if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
reportOutcome("Part Added");
}
}
18

19. BUFFER CASE STUDY – EXECUTING FOX
19

20. BUFFER CASE STUDY – EXECUTING FOX
(implanted error)
if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
capacity++;
reportOutcome("Part Added");
}
20

21. BUFFER CASE STUDY – GENERATED TEST
CASES
> Tests report the sequence of inputs used for the specific scenario, the sequence
of expected outputs and the actual output.
> Outcome is reported to the user via the usual JUnit red / green notifications.
<tests>
<test testID=”1”>
<input>[ itemId: 187 type: Nekrataal ]</input>
<expectedOutput>[ Part Added ]</expectedOutput>
<output>[ Part Added ]</output>
</test>
<test testID=”2”>
<input>[ itemId: 17 type: Planeswalker, itemId: 20 type: Dragon]</input>
<expectedOutput>
[ Part Added, Part Added – Become Full ]
</expectedOutput>
<output>[ Part Added, Part Added – Become Full ]</output>
</test>
</tests>
21

22. CONCLUSIONS AND FUTURE WORK
> FoX enables developers to leverage the already proven theories for formal testing.
> Provides a fully automated testing process, ranging from complete test set
generation (satisfying some design for test conditions), to test preparation and
execution.
> Operates on any Java based software system, being transparent to it's underlining
technologies.
> Provides complete positive and complete negative testing.
> Nest steps:
– Thorough evaluation.
– An additional tool to make the specification step easier and closer to the
developer, aiming to “hide” the formality as much as possible.
– NetBeans and Eclipse integration.
– A standalone X-Machine IDE providing additional related functionalities.
– Branch out to other languages and frameworks (eg. C# and .NET).
22

23. BIBLIOGRAPHY
> S. Eilenberg, Automate, Languages and Machines, Vol. A. Academic Press, London,
1974.
> M. Holcombe, “X-Machines as a basis for dynamic system specification,” Software
Engineering Journal, vol. 3(2), pp. 69-76, 1988.
> F. Ipate and M. Holcombe, “Specification and Testing using Generalized Machines: a
Presentation and a Case Study,” Softw. Test. Verif. Reliab, vol. 8, pp. 61-81, 1998.
> M. Holcombe and F. Ipate, Correct Systems: Building a Business Process Solution.
Springer, Applied Computing Series, November 1998.
> G. Eleftherakis and A. Cowling, “An Agile Formal Development Methodology,” in 1st
South Eastern European workshop on Formal Methods (SEEFM 03), (Thessaloniki),
pp. 36-47, Nov. 2002. Agile Formal Methods: Practical, Rigorous Methods for a
changing world.
> P. Kefalas, G. Eleftherakis, and E. Kehris, “Communicating X-Machines: a practical
approach for formal and modular specification of large systems,” Information and
Software Technology, vol. 45, pp. 269-280, Apr. 2003.
23

24. Ivo Neskovic http://twitter.com/trumpets
CITY College ivo.neskovic@gmail.com