Software Testing Achievements - Dreams Ecaterina Valic ă http://students.info.uaic.ro/~evalica/

" Program testing can be used to show the presence of bugs, but never to show their absence! " Edsger Dijkstra

" Program testing can be used to show the presence of bugs, but never to show their absence! "

Agenda About Software Testing Roadmap Achievements Dreams Challenges

About Software Testing

Roadmap

Achievements

Dreams

Challenges

Software Testing Process used to help identify the correctness , completeness , security and quality of developed computer software; Observing the execution of a software system to validate whether it behaves as intended and identify potential malfunctions;

Process used to help identify the correctness , completeness ,

security and quality

of developed computer software;

Observing the execution of a software system to validate whether it

behaves as intended and

identify potential malfunctions;

Software Testing Essential activity in software engineering; Used in industry for quality assurance; Estimated can consume fifty percent, or even more, of the development costs;

Essential activity in software engineering;

Used in industry for quality assurance;

Estimated can consume fifty percent, or even more, of the development costs;

Software Testing Different activities : the testing of a small piece of code by the developer (unit testing); the customer validation of a large information system (acceptance testing); the monitoring at run-time of a network-centric service-oriented application; etc.

Different activities :

the testing of a small piece of code by the developer (unit testing);

the customer validation of a large information system (acceptance testing);

the monitoring at run-time of a network-centric service-oriented application;

etc.

Software Testing Different objectives : exposing deviations from user’s requirements; assessing the conformance to a standard specification; evaluating robustness to stressful load conditions or to malicious inputs; measuring given attributes, such as performance or usability; or estimating the operational reliability; etc.

Different objectives :

exposing deviations from user’s requirements;

assessing the conformance to a standard specification;

evaluating robustness to stressful load conditions or to malicious inputs;

measuring given attributes, such as performance or usability;

or estimating the operational reliability;

etc.

Software Testing Testing consists of observing a sample of executions , and giving a verdict over them; Specific aspects that can characterize the sample of observations: Why? How? How much? What? Where? When?

Testing consists of observing a sample of executions , and giving a verdict over them;

Specific aspects that can characterize the sample of observations:

Why?

How?

How much?

What?

Where?

When?

Software Testing Why? ( test objective) why is it that we make the observations? looking for faults? decide if the product can be released? evaluate the usability of the User Interface? How? ( test selection ) which sample do we observe, how do we choose it? selection done ad hoc, random, or in systematic way by applying algorithmic or statistical technique

Why? ( test objective)

why is it that we make the observations?

looking for faults?

decide if the product can be released?

evaluate the usability of the User Interface?

How? ( test selection )

which sample do we observe, how do we choose it?

selection done ad hoc, random, or in systematic way by applying algorithmic or statistical technique

Software Testing How Much? ( test adequacy) how big of a sample? “ classical” approaches: coverage analysis or reliability; What? ( levels of testing ) what is it that we execute? execution as a whole; focusing only on a part of it (unit test, component / subsystem test, integration test);

How Much? ( test adequacy)

how big of a sample?

“ classical” approaches: coverage analysis or reliability;

What? ( levels of testing )

what is it that we execute?

execution as a whole;

focusing only on a part of it (unit test, component / subsystem test, integration test);

Software Testing Where? where do we perform the observation? done in house, in a simulated environment or in the target final context; (embedded systems) When? when is it in the product lifecycle that we perform the observations? the earliest, the most convenient, since the cost of fault removal increases as the lifecycle proceeds;

Where?

where do we perform the observation?

done in house, in a simulated environment or in the target final context; (embedded systems)

When?

when is it in the product lifecycle that we perform the observations?

the earliest, the most convenient, since the cost of fault removal increases as the lifecycle proceeds;

Roadmap A roadmap provides directions to reach a desired destination starting from the “you are here” Contains: the most notable achievements from past research the identified destinations are a set of four ultimate and unachievable goals called dreams the challenges faced by current and future testing research

A roadmap provides directions to reach a desired destination starting from the “you are here”

Contains:

the most notable achievements from past research

the identified destinations are a set of four ultimate and unachievable goals called dreams

the challenges faced by current and future testing research

Achievements Testing was conceived like an art, and was exemplified as the “destructive” process of executing a program with the intent of finding errors, opposed to design which constituted the “constructive” party; Testing process techniques and tools which can be incorporates within the development process; several test process models have been proposed for industrial adoption > “V model”;

Testing was conceived like an art, and was exemplified as the “destructive” process of executing a program with the intent of finding errors, opposed to design which constituted the “constructive” party;

Testing process

techniques and tools which can be incorporates within the development process;

several test process models have been proposed for industrial adoption > “V model”;

Achievements Test criteria the test criteria used for systematic identification of test cases; distinguished between white-box (a.k.a. structural) and black-box (a.k.a. functional), depending on whether or not the source code is exploited in driving the testing; a more refined classification can be laid according to the source from which the test cases are derived; the real challenge becomes the capability to make a justified choice, or rather to understand how they can be most efficiently combined;

Test criteria

the test criteria used for systematic identification of test cases;

distinguished between white-box (a.k.a. structural) and black-box (a.k.a. functional), depending on whether or not the source code is exploited in driving the testing;

a more refined classification can be laid according to the source from which the test cases are derived;

the real challenge becomes the capability to make a justified choice, or rather to understand how they can be most efficiently combined;

Achievements Comparison among test criteria evaluation of the various test criteria, and especially of the factors which make one technique better than another at fault finding; permitted to establish a hierarchy between comparable criteria, and to understand the factors influencing the probability of finding faults, focusing more in particular on comparing partition (i.e., systematic) against random testing;

Comparison among test criteria

evaluation of the various test criteria, and especially of the factors which make one technique better than another at fault finding;

permitted to establish a hierarchy between comparable criteria, and to understand the factors influencing the probability of finding faults, focusing more in particular on comparing partition (i.e., systematic) against random testing;

Achievements Object-oriented testing everything in software testing in also applied to OO code; OO development introduced new risks and difficulties: encapsulation can help hide bugs and makes test harder; inheritance requires extensive retesting of inherited code; polymorphism and dynamic binding call for new coverage models;

Object-oriented testing

everything in software testing in also applied to OO code;

OO development introduced new risks and difficulties:

encapsulation can help hide bugs and makes test harder;

inheritance requires extensive retesting of inherited code;

polymorphism and dynamic binding call for new coverage models;

Achievements Component-based testing Challenges: components must be generic enough for being deployed in different platforms and contexts, therefore the component user needs to retest the component in the assembled system where it is deployed; crucial problem: the lack of information for analysis and testing of externally developed components;

Component-based testing

Challenges:

components must be generic enough for being deployed in different platforms and contexts, therefore the component user needs to retest the component in the assembled system where it is deployed;

crucial problem: the lack of information for analysis and testing of externally developed components;

Achievements Protocol testing verifying the conformance of protocol implementations against their specifications; Reliability testing tries to eliminate those failures which would manifest themselves more frequently: intuitively the tester mimics how the users will employ the system;

Protocol testing

verifying the conformance of protocol implementations against their specifications;

Reliability testing

tries to eliminate those failures which would manifest themselves more frequently: intuitively the tester mimics how the users will employ the system;

Dreams Universal test theory; Test-based modeling; 100% automatic testing; Efficacy-maximized test engineering; the theory is at the basis of the adopted models, models are needed for automation, automation is instrumental to cost-effective test engineering

Universal test theory;

Test-based modeling;

100% automatic testing;

Efficacy-maximized test engineering;

the theory is at the basis of the adopted models,

models are needed for automation,

automation is instrumental to cost-effective test engineering

Dreams Universal test theory dream: to build a sound comprehensive theory which is useful to backup and nourish test technology; framework to which testers can refer to understand the relative strengths and limitations of existing test techniques, and to be guided in selecting the most adequate one; test machinery which ties a statement of the goal for testing with the most effective technique;

Universal test theory

dream: to build a sound comprehensive theory which is useful to backup and nourish test technology;

framework to which testers can refer to understand the relative strengths and limitations of existing test techniques, and to be guided in selecting the most adequate one;

test machinery which ties a statement of the goal for testing with the most effective technique;

Dreams Test-based modeling idea is to use models defined in software construction to drive the testing process; ideal situation: instead of taking a model and see how we can best exploit it for testing, let us consider how we should build the model so that the software can be effectively tested; “ model-based testing” > “test-based modeling”

Test-based modeling

idea is to use models defined in software construction to drive the testing process;

ideal situation:

instead of taking a model and see how we can best exploit it for testing,

let us consider how we should build the model so that the software can be effectively tested;

“ model-based testing” > “test-based modeling”

Dreams 100% automatic testing [1] automation is one of the ways to keep quality analysis and testing in line with the growing quantity and complexity of software (development tools generating code with less effort); can be made: by developing advanced techniques for generating the test inputs; by finding innovative support procedures to automate the testing process;

100% automatic testing [1]

automation is one of the ways to keep quality analysis and testing in line with the growing quantity and complexity of software (development tools generating code with less effort);

can be made:

by developing advanced techniques for generating the test inputs;

by finding innovative support procedures to automate the testing process;

Dreams 100% automatic testing [2] dream: a powerful integrated test environment which by itself, as a piece of software is completed and deployed, can automatically take care of possibly instrumenting it and generating or recovering the needed scaffolding code (drivers, stubs, simulators), generating the most suitable test cases, executing them and finally issuing a test report. promising steps have been made for unit testing;

100% automatic testing [2]

dream: a powerful integrated test environment which by itself, as a piece of software is completed and deployed, can automatically take care of possibly instrumenting it and generating or recovering the needed scaffolding code (drivers, stubs, simulators), generating the most suitable test cases, executing them and finally issuing a test report.

promising steps have been made for unit testing;

Dreams Efficacy-maximized test engineering dream: cost-effectively engineering “practical testing methods, tools and processes for development of high quality software”; theoretical, technical and organization issues should be reconciled into a viable test process yielding the maximum efficiency and effectiveness; main obstacle is the growing complexity of modern systems;

Efficacy-maximized test engineering

dream: cost-effectively engineering “practical testing methods, tools and processes for development of high quality software”;

theoretical, technical and organization issues should be reconciled into a viable test process yielding the maximum efficiency and effectiveness;

main obstacle is the growing complexity of modern systems;

Challenges Challenges arise, such as: how to conciliate model-based derivation of test cases with modern dynamically evolving systems; how to effectively select and use runtime data collected from real usage after deployment; how to qualify and evaluate the effectiveness of testing criteria; how to minimize the amount of retesting after the software is modified; etc.

Challenges arise, such as:

how to conciliate model-based derivation of test cases with modern dynamically evolving systems;

how to effectively select and use runtime data collected from real usage after deployment;

how to qualify and evaluate the effectiveness of testing criteria;

how to minimize the amount of retesting after the software is modified;

etc.

Bibliography Software Testing Research : Achievements, Challenges, Dreams. Antonia Bertolino

Software Testing Research : Achievements, Challenges, Dreams. Antonia Bertolino

Thank you ! Questions?

Questions?