Martin heininger - software unit testing autonomous cars verified by aerospace best practices

1. 1© HEICON – Global Engineering GmbH
HEICON Global Engineering GmbH
Kreuzweg 22, 88477 Schwendi
Internet: www.heicon-ulm.de
Blog: http://blog.heicon-ulm.de
Software unit testing:
Autonomous cars verified
by Aerospace Best practices?

2. 2© HEICON – Global Engineering GmbH
HEICON is a specialized engineering company which provides consulting- and development support with a
focus on software-based embedded systems.
The efficient implementation of methods and processes is the area of our engagement.
Founding: 2018 Headquarter: South of Germany (Memmingen) Membership:
Employees: 1 Legal form: GmbH
Revenue Distribution:
HEICON
71% 72%
39%
16%
23%
20%
28%
36%
35%
6%
18%
14% 4%
3%
10% 11%
19%
1%
2% 8%
19% 18%
2% 8% 4% 5% 7%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2013 2014 2015 2016 2017
Other Sectors
Military
Space
Railway
Industrial Automation
Automotive
Aerospace

3. 3© HEICON – Global Engineering GmbH
HEICON
Aero-
space
Auto-
motive
Railway Industry Defence Agri-
culture
HEICON - Starter
HEICON - Consulting
HEICON - Services
HEICON - Training
HEICON - Webinars

4. 4© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Best Practice Aerospace
Unit Tests considerations
Contact

5. 5© HEICON – Global Engineering GmbH
Unit Tests considerations
 ISO 26262 und IEC 61508 require such tests.
 The Aerospace (DO178) does have more than
30 years of experience in doing this kind of
testing.
 Software Unit Tests providing a 100%
structural source code Coverage, find software
bugs early.
 To create the required documentation (Test
Cases, Test Procedures, Test Reports,
Traceability) for SW Unit Tests is costly and
takes a lot of time.
 Therefore efficient ways to create these tests
are essential.

6. 6© HEICON – Global Engineering GmbH
Main.c
(Scheduler)
Function 1
Function 2
Function 3
FBlock1.1.c
Function 1
Function 2
FBlock1.2.c
Function 1
Function 2
Function 3
Function 4
Function 5
Function 6
FBlock2.3.c
Function 1
Function 2
Function 3
Function 4
Function 5
FBlock2.1.c
Function 1
Function 2
Function 3
Function 4
Function 5
Function 6
Function 7
FBlock2.2.c
Function 1
Function 2
SoftwareUnittest
Unit Tests considerations
Software/Software
Integrationstest 1
Software/Software
Integrationstest 2
Main.c
(Scheduler)
Function 1
Function 2
Function 3
FBlock1.1.c
Function 1
Function 2
FBlock1.2.c
Function 1
Function 2
Function 3
Function 4
Function 5
Function 6
FBlock2.3.c
Function 1
Function 2
Function 3
Function 4
Function 5
FBlock2.1.c
Function 1
Function 2
Function 3
Function 4
Function 5
Function 6
Function 7
FBlock2.2.c
Function 1
Function 2
What is a Unit?
Unit Testing as special case of
SW/SW Integration Testing….

7. 7© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Best Practice Aerospace
Unit Tests considerations
Contact

8. 8© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
System
Architecture
High-Level
Requirements
Low-Level
Requirements
Unittest
HW/SW
Integration Test
System Test
1
2
System
Architecture
SW
Architecture
Design
Coding
Requirements – Aerospace:
 Normal Test Cases
 Robustness Test Cases
 100% Structural Coverage
+

9. 9© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Requirements – Automotive:
System design
Software unit
design and
implementation
Software unit
testing
Item integration
and testing
Specification of
Software safety
requirements
Software
architectural
design
Software
integration and
testing
Verification of
software safety
requirements
Item testing
Software testing
Software
testing
Software
testing

10. 10© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Aerospace - DO-178C Applicability by SW Level
Verification of Outputs of Software Requirements Process A B C D
High-level requirements comply with system requirements    
High-level requirements are accurate and consistent    
High-level requirements are compatible with target computer  
High-level requirements are verifiable   
High-level requirements conform to standards   
High-level requirements are traceable to system requirements    
Algorithms are accurate   
Automotive - ISO 26262 –
Part 6 Chapter 6: Specification of software safety requirements
Requirements:

11. 11© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Requirements:
Aerospace - DO-178C Applicability by SW Level
Verification of Outputs of Software Design Process A B C D
Low-level requirements comply with high-level requirements    
Low-level requirements are accurate and consistent    
Low-level requirements are compatible with target computer  
Low-level requirements are verifiable   
Low-level requirements conform to standards   
Low-level requirements are traceable to high-level
requirements
   
Algorithms are accurate   
Automotive - ISO 26262 –
Notation for software unit design
ASIL A ASIL B ASIL C ASIL D
Natural language ++ ++ ++ ++
Informal notations ++ ++ + +
Semi-formal notations + ++ ++ ++
Formal notations + + + +

12. 12© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Structural Coverage:
Aerospace - DO-178C Applicability by SW Level
Verification of Verification Process Results A B C D
Test coverage of software structure (statement coverage)
achieved
 
Test coverage of software structure (decision coverage)
achieved
  
Test coverage of software structure (modified
condition/decision coverage) achieved
  
Automotive - ISO 26262 – Structural coverage
metrics at the software unit level
ASIL A ASIL B ASIL C ASIL D
Statement coverage ++ ++ + +
Branch coverage + ++ ++ ++
MC/DC (Modified Condition/Decision Coverage) + + + ++

13. 13© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Aerospace - DO-178C Applicability by SW Level
Verification of Outputs of Software Coding&Integration A B C D
Source Code complies with low-level requirements   
Source Code complies with software architecture   
Source Code is verifiable  
Source Code conforms to standards   
Source Code is traceable to low-level requirements   
Source Code is accurate and consistent   
Output of software integration process is complete and
correct
  
Automotive - ISO 26262 – Methods for the
verification of software unit design and
implementation
ASIL A ASIL B ASIL C ASIL D
Walk-through ++ + o o
Inspection + ++ ++ ++
Static code analysis + ++ ++ ++
Formal Verification o o + +
Reviews:

14. 14© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Reviews:
Automotive - ISO 26262 – Methods for the
verification of software unit design and
implementation
ASIL A ASIL B ASIL C ASIL D
Control flow analysis + + ++ ++
Data flow analysis + + ++ ++
Aerospace - DO-178C Applicability by SW Level
Verification of Verification Process Results A B C D
Test coverage of software structure (data coupling and
control coupling) achieved
  

15. 15© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Aerospace
1. Two (!) level of Software
Requirements
2. Requirements based Tests on
Source Code Level (Unit Testing)!
3. Normal and Robustness Tests –
that’s it!
4. 100% Structural Test Coverage !
5. DO178C is interpreted very similar
in the aerospace industry.
Automotive:
1. Missing clear separation between functional
requirements and architecture.
2. Unit tests are written against architecture/Detailed
Design and not against functional requirements.
3. There are several Test Categories defined in the
ISO26262:
Requirements based testing, fault Injection test,
interface test, resource usage test, back-to-back
comparison between model and code.
4. Many different interpretation of ISO26262 in the
automotive industry

16. 16© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Effort in Reviews
Automotive
 Failure found in Reviews are
incorporated without proper
documentation of the review
process.
 Content of Reviews often not
properly defined (Checklists).
Aerospace
 The execution of Code-, Requirements-, Design-
Reviews have to be proven (Part of any audit!)
=> traceable documentation of reviews and the
findings is required.
 Checklists have to be provided to assessors in
the planning phases of projects (Early in the
project!)
Examples for different Interpretation of the Standards

17. 17© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Examples for different Interpretation of the Standards
100% Structural Code Coverage
Automotive
 20% and more can be provided by
argumentation.
 Tests to achieve structural
coverage often written against
source code and not design and
requirements.
Aerospace
 A minimum of 95% of coverage have to
be achieved by tests.
 Tests to achieve structural coverage
have to be written against requirements
(To be proven in Audits!).

18. 18© HEICON – Global Engineering GmbH
Comparison Aerospace and Automotive Industry
Best Practice Aerospace
Unit Tests considerations
Contact

19. 19© HEICON – Global Engineering GmbH
Best Practice Aerospace
 Structural Source Code Coverage
 Requirement based Tests
 Functional Software Tests
 Traceability
 Coding guidelines
 Tool Qualification
Ideal to identify Req.- and Test weakness
80% and more structural coverage
achieved
Proper Simulation and Emulation Test
Environments are often sufficient
Define a strategy!
Coding strategy first! Selection of MISRA
rules second!
Very good cost/benefit ratio!

20. 20© HEICON – Global Engineering GmbH
Contact - Publications
Contact:
HEICON – Global Engineering GmbH
Martin Heininger Dipl.-Ing(FH)
Kreuzweg 22
D-88477 Schwendi
Tel.: +49 7353 - 98 17 81
Mobil: +49 176 - 24 73 99 60
martin.heininger@heicon-ulm.de
http://www.heicon-ulm.de
Publications:
Testing power electronics according ISO26262, ATZ 04/15
Monthly: Blog article about Functional Safety Topics: http://blog.heicon-
ulm.de