1) An automatic code generator (ACG) produces source code from a model specification but must undergo rigorous testing to be qualified and trusted. 2) Integrated unit testing is proposed to achieve the exhaustiveness of unit testing using only integration tests by instrumenting the ACG with ghost code to monitor test coverage and results during integration testing. 3) The approach generates a large set of integration tests from block specifications to thoroughly cover the many configurations, allowing unit tests to be evaluated without editing intermediate languages.

1. Elie Richa
Gem: Integrated Unit Testing
for a Trusted Code Generator
QGen Developer and QA Coordinator

2. Model-driven Development of
Critical Software
• An Automatic Code Generator (ACG) produces the source code from a
model-based specification
• To avoid re-verifying the generated code each time the ACG must be
"Qualified" (i.e. Trusted) (or “Qualifiable”)
• Qualification requires a rigorous and thorough verification of the ACG
§ Extensive testing of the ACG with high exhaustiveness
• Testing is a major cost in ACG qualification
Model
Automatic Code
Generator!
(ACG)
Source Code
2

3. DO-178C Tool Qualification Levels
Criteria 1: A tool whose output is part of the airborne so5ware and thus could insert an error.
Criteria 2: A tool that automates verifica=on process(es) and thus could fail to detect an error, and
whose output is used to jus0fy the elimina0on or reduc0on of:
• Verifica=on process(es) other than that automated by the tool, or
• Development process(es) that could have an impact on the airborne so5ware.
Criteria 3: A tool that, within the scope of its intended use, could fail to detect an error.
So$ware Levels, failure is …
Level A: Catastrophic
Level B: Hazardous
Level C: Major
Level D: Minor
3

4. TQL-1 Qualification: Requirements-
Based Testing Model for Tools
Tool Opera=onal
Requirements (TORs)
Tool
Requirements (TRs)
Test Cases
Test Procedures
Source Code
implemented by
allocated on Tool
Architecture
implemented by
implemented by
verified by
implemented by
implemented by
4

5. QGen
Architecture of QGen
5
Importer Preprocessor Sequencer
Code
Model
Generator
(CMG)
Op=mizer Expander Printer
Simulink®
Model
Source
Code
internal model representation internal code representation

6. Unit Testing
Input Output
Integration Testing
Input Output
User
Language
Ti−1 TiMi
Source
Code
Ti+1
Unit testing vs. Integration Testing
Integration testing is preferred to unit testing
Objective
Achieve unit testing exhaustiveness using only integration tests
Unit Testing Integration Testing
Test Data Editor/Viewer Internal languages have no editors User language has a good editor
Test Data Complexity Intermediate languages are complex User language is simpler, has higher abstraction
Test Exhaustiveness Achievable thanks to isolation of units Hard to achieve with no visibility on the internals of the tool
Mi+1
6

7. Executable Unit Test Cases
• Write unit test cases as executable queries
§ A query specifying the required input : Test Requirement
§ A query specifying the expected output : Test Oracle
• Example: function Sqrt (Arg : Float) return Float;
7
User
Language
Ti−1 TiMi
Source
Code
Ti+1Mi+1
Unit Test
Requirements
tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test
Oracles
… …
Test Case ID Test Requirement Test Oracle
1 Arg = 0.0 Sqrt'Result = 0.0
2 Arg > 0.0 and then Arg <= 1.0 Sqrt'Result > 0.0 and then Sqrt'Result <= 1.0
3 Arg > 1.0 and then Arg <= 100.0 Sqrt'Result > 1.0 and then Sqrt'Result <= 10.0

8. Instrumentation for Test Case
Evaluation and Logging
• All instrumentation is marked as ghost code
• Compiler produces ghost code only when assertions are enabled
Log which test requirements are satisfied by the input data
Evaluate the test oracles of the covered test cases
Subprogram
Under Test
control flow
instrumentation
instrumentation
8

9. Unit Test Cases
Integration
Tests
tr0,0 tr0,1 tr0,2 to0,0 to0,1 to0,2 tr1,0 tr1,1 to1,0 to1,1 tr2,0 tr2,1 to2,0 to2,1
Test0 SAT – – PASS – – – SAT – PASS – SAT – PASS
Test1 – – SAT – – PASS – SAT – PASS – SAT – PASS
Test2 – – SAT – – FAIL SAT – PASS – – SAT – PASS
Test3 – – – – – – SAT SAT PASS PASS – SAT – FAIL
Test4 SAT – – PASS – – – SAT – PASS – SAT – PASS
Non-covered
Unit Test Case
Non-covered
Unit Test CaseUnit Test Failure Unit Test Failure
Integrated Unit Testing (IUT)
Integration
Test Input
Ti−1 TiMi
Source
Code
Ti+1Mi+1
Unit Test
Requirements
tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test
Oracles
… …
Test Execution
9

10. Achieving coverage of Unit Test Cases
• To achieve coverage, Integration Testing must be thorough
• Example: A Simulink® block type with 4 parameters
§ 3 possible types each (i.e. Scalar, Vector, Matrix) ⇒ 34 = 81 configurations
§ Taking into account primitive types (int8, int16 …) ⇒ ~600000 configurations!
§ But wait, only 14 out of 81 are valid configurations (~7000 out of ~600000)
§ The behavior is generally similar for all configurations, with subtle differences
§ How can we do this accurately for 120+ Simulink® block types?
• Use the BlockLibrary specification language
§ Specify variability parameters: input/output ports, data parameters
§ Specify constraints defining valid combinations of parameter values and types
§ Factorise common specification elements
§ Auto-generate a large test set with a meaningful/reasonable coverage criteria of the
possible configurations
• Achieve Unit Testing using only integration test data (Simulink® models)
10

11. Integrated Unit Testing Framework
• Other features of the Integrated Unit Testing framework include
§ Checking the completeness and disjointness of test cases
§ Storing temporary data specific to a test case to improve expressiveness
§ Selecting/excluding specific groups of test cases in the instrumentation
§ Optimising the instrumentation by only evaluating previously non-covered test cases
§ Specifying test cases in separate files than the tested source code
• Let us know if you’d be interested in Integrated Unit Testing for your code
§ We are considering making it a GNATtest feature
11
Unit Test
Requirements
Unit Test
Oracles
Integration
Test Input
Ti−1 TiMi
Source
Code
Ti+1Mi+1
Unit Test
Requirements
tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test
Oracles
Test Execution
… …

12. Conclusion: Integrated Unit Testing
• Conventional Unit Testing of multi-pass code generators is painful
§ Difficult to create inputs and check outputs of individual passes
• Integration Testing is generally easier
§ Can use normal Models as input; generated Source Code as output
• Integrated Unit Testing: accomplish Unit Testing using Integration Testing
§ Instrument the tested tool with ghost code
§ Monitor internal data during Integration Testing
§ Log evidence of the coverage and passing of Unit Test Cases
• The approach is applicable to other software
§ Let us know if Integrated Unit Testing is interesting in your context
12