EuroSTAR Software Testing Conference 2008 presentation on The Truth About Model-Based Quality Improvements by Bart Knaack. See more at conferences.eurostarsoftwaretesting.com/past-presentations/

1. © Logica 2008. All rights reserved
The Truth about Model Based Improvements
Bart Knaack

2. Agenda
• Introduction
• The future of testing
• What is a model?
• Different ways of Model Based Quality Improvement
• Conclusions
• What next

3. Bart Knaack
• Bart Knaack, Senior Test Advisor, Logica, The Netherlands
• 15 years experience in IT, of which 12 in testing
• Father of 2 kids (age 6 and 8)
Bart.knaack@logica.com

4. The future of testing
• Crystal ball called Academia
• The past as the mirror of the future
• Trends and threats
But:
• Testing is not quality improvement
• Testing is not LEAN

5. What is a model?
A model is an abstraction or conceptual object used in the creation
of a predictive formula (wikipedia)
A model in science is a physical, mathematical, or logical
representation of a system of entities, phenomena, or processes.
Abstract representation of the reality focussing on a limited set of
aspects
Programs are models too!!

6. Model for this presentation
Behavioral models: Models that somehow define the
(intended) behavior of a system
•UML models
•Finite State Machines
•Algebraic models
•Graphical model
•Textual model

7. Model Based Quality Improvement
No. 725 March 2015 Testing a Virtual Juggling Program

8. Different ways of Model Based Quality Improvement?
• Model Based testing
–Online
–Offline
• Model Metrics and Code Conformance
• Model checking the models
• Applying Model checking techniques.

9. Model Based testing (Online)
• Definition: Directly coupling the test generation model to
the SUT and directly executing the tests.
• Project: IRIS DOMAIN: Insurances
SUT: Actuarial calculation engine.
Torx modelling language for modelling the formulae.
Result: 5 Additional errors were found of which 3 were
significant. Development and test-time for upcoming
changes and fixes was reduced by 50%.

10. Model Based testing (Online)
• Advantages:
–Quick (re)testing of systems
• Disadvantages:
–Model SUT adaptor needs to be set up.
–Model errors need to be taken into account
–Not all systems and tests are suitable for this approach.

11. Model Based testing (Offline)
• Definition: Tests are generated by a Testgenerator and
executed manually on the SUT.
• Project: CCBS Domain: Telecom.
SUT: 5ESS Telephony switch, new feature
Result: 10 features Modeled.1200 Tests designed of which
75% specific on feature interactions. 20% tests failed due to
modelling errors.

12. Model Based testing (Offline)
•Advantages:
–Quick adaption of testcases
–Models can be used for rapid prototyping.
–Quick reponse on Changes
•Disadvantages:
–Most methods do not support the selection of testcases.
–Testexecution still manual

13. Model Metrics and code conformance
• Definition: Using Metrics to determine when models are
good enough.
• Project:Siu Wai Tang Domain: Banking and Finance
SUT: Electronic Payments System.
Result: Improved Insight in elevated problem areas;
redefinition of 3 models based on this insight.

14. Model Metrics
•Advantages:
–Early stage warnings on model quality.
–Multidimensional view on models.
•Disadvantages:
–No hard figures to adhere to.
–Statistical co-relation instead of direct cause-and-effect.

15. Model checking the models
• Definition: Requirements checking by checking the models
that form the requirements.
• Project: Bos project Domain: Industry
SUT: ‘Waterkering’ (lock) protocols.
Result: Error free software (after 20 years of operation).

16. What is Model Checking?
•Statespace search
•Validation of a
negative scenario.
•Structural traversal.

17. Model checking the models
•Advantages:
–Finding CONCEPTUAL errors at an early stage.
–Finding DIFFERENT types of errors
•Disadvantages:
–Model needs to be implemented in a formal manner
–Model checker needs to be available for the used paradigm.

18. Applying Model checking techniques.
• Definition: CODE is used model. Model checking techniques
used for faultfinding
• Project: FeaVer Domain: Telecom
SUT: transaction based system (V5 interface)
5000 lines of code 1000 installments over 10 years.
Result: 35 bugs found (25 race conditions)

19. Applying Model checking techniques.
•Advantages:
–Quick (re)test functionality
–Model errors are REAL errors
–Errors are found that are next to impossible to find using ‘traditional’
methods.
•Disadvantages:
–Not all languages are supported by modelcheckers.
–Stubs and drivers need to be implemented to FEED the system.

20. Conclusions
• Modelling by itself can be quality improving.
• A lot can be gained from modelling and model improvement
initiatives.
• Model Based Testing can help in t ackling the problems in
complex systems.
BUT:
• New skills
• Academic image
• No standards

21. What next ?
• Integrated Toolsupport
• Simplify
• Educate
• Not included: Model Based development.