Presentation from Modelica Conference 2015

1. OPTIMICA TESTING TOOLKIT
Anders Tilly Victor Johnsson Jon Sten
Alexander Perlman Johan Åkesson
2015-09-29 © Modelon 1

2. MOTIVATION
• Why do we need software testing?
 Ensure consistent functionality over time
 Software without tests can not be updated safely
 Safe migration between platforms and tool
versions
 Quality software development requires testing
• Modelica code is just like any other software, it
needs testing
2015-09-29 © Modelon 2

3. BACKGROUND
• Test specification formats
 Experiment annotation
 TestCase
2015-09-29 © Modelon 3

4. BACKGROUND
• Additional requirements on test specification
 Specification of reference variables
 Model structure
• Equation system structure
• Analytic Jacobians
 Tool specific options
 Setup and teardown scripts
 Ability to override reference results for specific
tools
2015-09-29 © Modelon 4

5. BACKGROUND – CSV COMPARE
• Compares and displays deviation between two
result files
• Calculates an tube around the reference curve
• The checked result should inside the tube
2015-09-29 © Modelon 5

6. PREVIOUS WORK
• UnitTesting
 Modelica based library
• MoUnit
 Used in OneModelica
 Support for multiple Modelica and FMI tools
• RegressionTest
 For Dymola testing
2015-09-29 © Modelon 6

7. OPTIMICA TESTING TOOLKIT
• Key features
 Modelica and FMI cross testing platform
 Flexible test authoring
 Automated test execution and reporting
• Architecture
 Core
• Command line tool for running tests
• Uses OPTIMICA compiler toolkit for test retrieval
 GUI
• Tool for creating, updating and running tests
• Reviewing and updating results
2015-09-29 © Modelon 7

8. CORE DESIGN
• Design goals
 Modularized and Plugin based
 Support multiple Modelica and FMI tools
 Defining high-level API for common operations
 Oriented around testing cycle:
 Canonical form between steps
2015-09-29 © Modelon 8
Sieve Execute Report

9. CORE OVERVIEW
2015-09-29 © Modelon 9
.mo.mo
Test
spec.
.m
o
.m
o
Reports
Modelica / FMI
Tools
Sieve Execute Report
.mo.mo
Library

10. RUN CONFIGURATION
• From command line
• From file
2015-09-29 © Modelon 10
.
m
o
.
m
o
Test
spec.
.
m
o
.
m
o
Reports
Modelica / FMI
Tools
Sieve Execute Report
.
m
o
.
m
o
Library
OutputsToolsType
Test
Specification
Library
Resources

11. SEIVES
• Find models to test
• Extracts test settings (start-, stop-time,
tolerance, signals, etc.)
• Translate to canonical form
• Examples of sieves:
 Experiment annotations
 All models
 TestCase annotations
2015-09-29 © Modelon 11

12. SIEVE CORE
1. Based on provided tests, different sieves can be
used
2. The sieve implementation will the provide a list
of tests. If needed the OCT compiler will be used
to extract information from Modelica models
2015-09-29 © Modelon 12
Sieve Core
Sieve
Implementation
OCT Frontend
.mo.mo
1 2

13. SIEVE EXAMPLE
• Experiment annotation Sieve:
1. Forward Modelica packages to OCT. OCT returns tree
representation of packages
2. Recursively for each package:
3. Retrieve sub classes, if it is:
a) Sub-package, then go to 2
b) If class, check for experiment annotation, if found add as test
4. Return list of tests
2015-09-29 © Modelon 13
OCT Frontend
.mo.mo.mo 1
2
3
Sieve Core 4

14. TEST EXECUTION
• Testers can be divided into two main categories:
 Static
• Testing is controlled by OTT
• Each test (model) is tested though a preselected sequence
 Scripted
• Each test is programmatically defined
• OTT provides high level API
2015-09-29 © Modelon 14

15. STATIC TESTING
• Modes:
 Check
 Compile
 Simulate
 Verify
 Testcase
2015-09-29 © Modelon 15
Core
Check Compile Simulate Verify
Compare
Tools
Translation Interfaces
Simulation
Interfaces
Modelica
Tools
FMI
Tools

16. STATIC TESTING EXAMPLE
1. A list of tests is provided by the sieve
2. The OCT compiler is used to compile FMUs
3. The FMUs are simulated using OCT
4. The simulation results are compared using CSV Compare
5. Each step records results which are passed along to the
report step
2015-09-29 © Modelon 16
Core
Check Compile Simulate Verify
CSV
Compare
OCT
51
2 3 4

17. SCRIPTED TESTING
• Each test can be unique, custom sequence of
commands
• Written in Python
• OTT provides high level API to supported Modelica
and FMI tools
• OTT also provides support for generating reports
2015-09-29 © Modelon 17
Core
Compare
Tools
Modelica
Tools
FMI
Tools
.py
Tests
.py
Test

18. REPORTERS
• Each test step produces artefacts
• Artefacts are then processed by the reporters
• Available reporters:
 HTML
 JUnit
 Pickle
2015-09-29 © Modelon 18
.html.html.html
.xml
Core
HTML JUnit Pickle

19. CONTINUOUS INTEGRATION SUPPORT
• CI integration using command line and JUnit XML
results
• Hudson, Jenkins, TeamCity
2015-09-29 © Modelon 19
.m
o
.m
o
Test
spec.
.
m
o
.
m
o
JUnit
Modelica / FMI Tools
Sieve Execute Report
.m
o
.m
o
Library
Job configuration Record JUnit results

20. RESULT
2015-09-29 © Modelon 20

21. GUI
• Test authoring
• Test execution
• Test run configuration
• Test result reviewing
• Still work in progress!
2015-09-29 © Modelon 21

22. GUI
• Developed with usability in mind
 Easy to specify modifications and variables to
compare
 Easy to review and update result differences
 Inheritance between test cases
2015-09-29 © Modelon 22

23. GUI
2015-09-29 © Modelon 23

24. BACKLOG
• New test specification format
 Possibility to specify compare variables
 Tool specific options
 Comparing test model structure
 …
• Run configuration
• Traceability and reproducibility support
• Extend GUI functionality
• Additional Modelica and FMI tools
2015-09-29 © Modelon 24

25. jon.sten@modelon.com
2015-09-29 © Modelon 25