Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Front Loaded Test Approach Verhaert

1,128 views

Published on

A case study on integrating 'Software in the loop' testing with 'Hardware in the loop' testing by Verhaert on Innovation Day 06.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Front Loaded Test Approach Verhaert

  1. 1. Integrated hard- & software testing for first-time right development FIRST-TIME RIGHT www.mastersininnovation.com DEVELOPMENT ... A DIFFERENT TEST APPROACH Koen Puimege Koen.puimege@verhaert.com Frederik Wouters frederik.wouters@verhaert.com www.verhaert.com Commercially confidence – This presentation contains ideas and information which are proprietary of VERHAERT, Masters in Innovation®*, it is given in confidence. You are authorized to open and view the electronic copy of this document and to print a single copy. Otherwise, the material may not in whole or in part be copied, stored electronically or communicated to third parties without prior agreement of VERHAERT, Masters in Innovation®*. * VERHAERT, Masters in Innovation is a registered trade name of Verhaert New Products & Services NV. www.mastersininnovation.com Slide 1
  2. 2. Integrated hard- & software testing for first-time right development Agenda Introduction www.mastersininnovation.com Software & Hardware in the loop Front loaded test approach of Verhaert Case study: Proba Satellite Slide 2
  3. 3. Integrated hard- & software testing for first-time right development Verhaert mission, strategy and activities Activities: Mission: www.mastersininnovation.com Innovative product development services Space & Security •Instrumentation •Small satellites Strategy: •HALE UAV’s Industry • Act as system developer in key markets •Marine • Offer unique knowledge in industrial •Finance & Retail (ICT) product development •Life Science & Care •Automotive & Transport •Materials & Machines Slide 3
  4. 4. Integrated hard- & software testing for first-time right development Introduction Good development practices www.mastersininnovation.com Focus on early decisions & on specific design alternatives on a sound basis Analysis and specification freeze before initial physical prototyping Feedback on design trade off and optimisations Slide 4
  5. 5. Integrated hard- & software testing for first-time right development Performance Introduction Requirements Testing www.mastersininnovation.com System Functional Specifications Testing Detail Low-level Practices of testing Specifications Testing Requirements based testing D&D Integrated development & test approach Front loaded testing Slide 5
  6. 6. Integrated hard- & software testing for first-time right development Introduction When can a large-scale test methodology be cost www.mastersininnovation.com effective ? Cost driven: Critical development schedule High burden rate i.e. expensive equipment (crashtesting) Usability - Man in the loop Duration Safety Critical Slide 6
  7. 7. Integrated hard- & software testing for first-time right development Introduction Debugging tools don’t determine what www.mastersininnovation.com your bug is, but where it is. Slide 7
  8. 8. Integrated hard- & software testing for first-time right development SIL - Software in the loop Virtual simulation of embedded system on host www.mastersininnovation.com Microprocessor Application Used for debugging and functional verification Alow debugging before HW is available Code validation on simulated hardware Slide 8
  9. 9. Integrated hard- & software testing for first-time right development SIL - Advantages Provide feedback & selection for HW design www.mastersininnovation.com Enable testing/simulation of off-nominal behavior (error conditions, fault recovery) Concurrent HW & SW development No HW resource conflict when developing/testing with >1 person Cross validation of environment (interfaces, payloads) through behavioral models Slide 9
  10. 10. Integrated hard- & software testing for first-time right development SIL - Disadvantages Disadvantage: Accuracy of model www.mastersininnovation.com Need for code validation on real hardware (HIL) No Possibility to test system in real-time Verification SW timing and performance Slide 10
  11. 11. Integrated hard- & software testing for first-time right development HIL - Hardware in the loop Highly realistic simulation of equipment in an operational real time www.mastersininnovation.com virtual environment. Replacing plant interaction by RT simulations. Slide 11
  12. 12. Integrated hard- & software testing for first-time right development HIL - Advantages Testing of extreme conditions upto the limits Performance, Real time behaviour, Environmental www.mastersininnovation.com ... Anomaly testing & Isolate deficiencies Testing with/without subsystems & environment Reducing cost Equipment & manpower Time to market Slide 12
  13. 13. Integrated hard- & software testing for first-time right development HIL & Complex Systems Multi-processor Platform Solar Batteries www.mastersininnovation.com Multi-unit Array Redundant system Reaction Power Payload#1 Many interfaces wheels Control System Magneto- Payload#2 torquers Board Payload AOCS i/f Magneto- Computer Computer Payload#3 meters Star GPS Tracker Payload#4 Slide 13
  14. 14. Integrated hard- & software testing for first-time right development HIL - Disadvantages Difficulty to emulate REAL environment (crash, satellite launch, ...) www.mastersininnovation.com Doesn’t debug your embedded system, No breakpoint or pause-on-command No info of erroneously code No info on bug location Slide 14
  15. 15. Integrated hard- & software testing for first-time right development Front loaded Test Approach www.mastersininnovation.com Requirements based testing Software Validation Facility (SIL) Integrated Test Environment (Extended HIL) Slide 15
  16. 16. Integrated hard- & software testing for first-time right development Requirements Based Testing Identify Requirements related to eSW www.mastersininnovation.com Check verification method (Analysis / Design / Test) Provide list of tests Identify which tests to do in which project phase(s) and on which level Simulation / Emulation Unit test / System test Result is test matrix used for Test progress Traceability «Test plan must be established from the beginning together with the architectural design» Slide 16
  17. 17. Integrated hard- & software testing for first-time right development VALIDATION LEVELS VHDL TEST BENCH AVIONICS TEST BENCH (SCOS) 0 1 2 3 4 5 6 7 EQM VHDL TEST RESULTS (PRIOR TO EQM BURN) (test board functions at system level) parallel execution of selected tests) (functional system performance by (test single functions of a board) www.mastersininnovation.com (verified by the designer) ELECTRICAL TESTS SUBSYSTEM TESTS TEST-ID SUFFIX SYSTEM TESTS SYSTEM TESTS MATURE CORE VHDL MODULE VHDL SYSTEM CORE DESIGN PASS/FAIL PASS/FAIL PASS/FAIL PASS/FAIL PASS/FAIL PASS/FAIL PASS/FAIL PASS/FAIL TEST-ID TEST DEFINITION PROFILE NAME TEST ITEMS 01 TME functional tests registers power-up defaults X PASS X PASS X (*) TTM-50 register write/readback X PASS X PASS X (*) 02 VC7 generation of idle data / VC0 OBET 03 time source packet insertion X PASS X PASS X PASS VC0 HKM data integrity and handshaking X PASS X PASS X PASS 04 VC3 VC4 External PacketWire data 05 initegrity of packetized data X PASS X PASS X PASS VC3 VC4 External PacketWire data 06 initegrity of non-packetized data X PASS X PASS X (*) VC1 Internal PacketWire data integrity of 07 non-packetized-data X PASS X PASS VC2 Internal PacketWire data integrity of 08 non-packetized-data X PASS X PASS VC1 VC2 Internal PacketWire data 09 integrity of non-packetized-data X PASS X PASS X (*) VC1 VC2-Internal PacketWire data 10 integrity of packetized data X PASS X PASS X PASS X PASS Time-source-packet X PASS X PASS X (*) 11 TX ringbuffer empty interrupt X PASS X (*) 12 Core reset via global reset register X PASS X PASS X PASS X 13 VC1 VC2 channel reset X PASS X PASS 14 FECW function X PASS X PASS X PASS 15 Reed Solomon function X PASS X PASS X (*) 16 Convolutional function X PASS X PASS X (*) 17 FECW, RS and CONV enabled X PASS X PASS X (*) 18 Mark-NRZ-M, SPLIT-PHASE, PSEUDO- 19 RANDOM functions X PASS X PASS X (*) VC1 VC2-Internal-PacketWire data 20 integrity of packetized data (PD-flag=0) X PASS X PASS CLCW-bit-16-17 option X PASS X PASS 21 VC bandwidth allocation (dynamic based) X PASS X PASS X FAIL 22 VC bandwidth allocation (priority based) X PASS X PASS X (*) 23 Slide 17
  18. 18. Integrated hard- & software testing for first-time right development SVF - Key data Software in the loop system www.mastersininnovation.com TSIM - Leon Sparc simulation model used from Gaisler Research Behavioral models developed in-house in C eSW Written in C SVF fully transparant to application Binary code runs on TSIM Slide 18
  19. 19. Integrated hard- & software testing for first-time right development Software Validation Facility (SVF) www.mastersininnovation.com DUT Computer Model Environmental Model Stimuli I/F Behavioral model (C) Processor simulation I/F Behavioral model (C) (TSIM) I/F Behavioral model (C) Response eSW (C) Slide 19
  20. 20. Integrated hard- & software testing for first-time right development Software Validation Facility (SVF) TEST APPLICATION BINAIRE www.mastersininnovation.com CODE IL DEVICE DEVICE DEVICE DEVICE DRIVER DRIVER DRIVER DRIVER SOFTWARE SOFTWARE IP IP IP IP PROCESSOR CORE CORE CORE CORE PROCESSOR FPGA Synthesisable model (real) HW HW HW HW Executable code (binary) DRIVER DRIVER DRIVER DRIVER Behavioural model (simulation) MEMORY BOARD BOARD BOARD BOARD No model Required Slide 20
  21. 21. Integrated hard- & software testing for first-time right development Software Validation Facility (SVF) FUNCTIONAL APPLICATION BINARY www.mastersininnovation.com CODE DEVICE DEVICE DEVICE DEVICE DRIVER DRIVER DRIVER DRIVER SOFTWARE SOFTWARE IP IP IP IP PROCESSOR CORE CORE CORE CORE FPGA Synthesisable model (real) HW HW HW HW Executable code (binary) DRIVER DRIVER DRIVER DRIVER Behavioural model (simulation) MEMORY BOARD BOARD BOARD BOARD No model Required Slide 21
  22. 22. Integrated hard- & software testing for first-time right development Software Validation Facility (SVF) FUNCTIONAL/TEST APPLICATION BINARY www.mastersininnovation.com CODE DEVICE DEVICE DEVICE DEVICE DRIVER DRIVER DRIVER DRIVER SOFTWARE SOFTWARE IP IP IP IP PROCESSOR CORE CORE CORE CORE Hardware FPGA Synthesisable model (real) HW HW HW HW Executable code (binary) DRIVER DRIVER DRIVER DRIVER Behavioural model (simulatie) MEMORY BOARD BOARD BOARD BOARD No model Required Slide 22
  23. 23. Integrated hard- & software testing for first-time right development Integrated Test Environment (ITE) DUT computer Test Bench (TB) www.mastersininnovation.com Stimuli Behavioral I/F model (C) Leon Sparc I/F 2x embedded RTEMS Behavioral cPCI PIII I/F model (C) Response RT Linux Debug Behavioral eSW (C) DCA (C) I/F model (C) LAN Test IF Bridge (TIFB) Monitoring & Control PC-104 486 Pentium IV PC RT Linux SuSe Linux Slide 23
  24. 24. Integrated hard- & software testing for first-time right development ITE - Differences with HIL systems? ITE features integrated debugging functionalities www.mastersininnovation.com DUT , HIL & DUT – HIL interaction I.e. Specific test software on DUT Custom interface cards to validate off–nominal behaviour; Baud rate deviation, ... Space specific HW cards Custom HW Custom Drivers for test software Re-use of SVF SW Slide 24
  25. 25. Integrated hard- & software testing for first-time right development ITE – Key data Functional and performance testing www.mastersininnovation.com ITE Components Simulates HW environment of sub-system or whole satellite Same/similar behavioral models can be used through hardware abstraction DUT (Proba Satellite) Debug interface Real hardware (Leon Sparc, Provides access to embedded system via Ethernet I/F, PCB) RTEMS Monitoring & Control Application software User i/f Device Control Application Set of tools (compile, run, debug, test, log, scripting, ...) Slide 25
  26. 26. Integrated hard- & software testing for first-time right development ITE – Techniques and Tools Device Control Application (DCA) www.mastersininnovation.com Debug interface Scripting Monitoring & Control Transparency Independent testing Slide 26
  27. 27. Integrated hard- & software testing for first-time right development ITE - Device Control Application (DCA) Dedicated application www.mastersininnovation.com Separate from application SW Aimed to test and validate all sub-units linked to Board computer (payloads) Implemented according to agreed ICD (Interface Control Document) Enables black-box testing of sub-units from within ITE Supports integrated testing of whole system Inherent cross-validation Sub-unit made by sub-contractor/other team DCA i/f made by system integrator «The DCA facilitates integration testing on system level» Slide 27
  28. 28. Integrated hard- & software testing for first-time right development ITE - Debug Interface Supports debugging and testing of the HW www.mastersininnovation.com Debug interface of Board computer allows deep control of whole system, not only processor Two front-end applications used GDB (open source) Dedicated «An adequate debug interface is imperative to test the eSW application» Slide 28
  29. 29. Integrated hard- & software testing for first-time right development ITE - Scripting Automatic/repeated running of tests www.mastersininnovation.com Management of ITE Top-level Prevents human errors Same tests must be run in different project phases (Engineering/Qualification/Flight Model) Advantage of Tcl scripting language Easy to read/understand Interpreted -> flexible (no compilation) Slide 29
  30. 30. Integrated hard- & software testing for first-time right development ITE - Monitoring & Control Visual representation of system state www.mastersininnovation.com Change system state LAN based No keyboard/display i/f on Board computer Enables remote testing and debugging Good support for TCP/IP in most operating systems Reliable connection Platform independent GUI (Qt - Trolltech) Slide 30
  31. 31. Integrated hard- & software testing for first-time right development ITE - Transparency Layered SW model www.mastersininnovation.com Transparency towards Application HW i/f (Hardware abstraction) POSIX Host platform (x86, Leon Sparc) Operating system POSIX compliant OS + Device drivers Advantages Hardware Re-use of SW Test SW in appropriate environment Early development/testing of SW «Transparency leads to more effective eSW development and testing» Slide 31
  32. 32. Integrated hard- & software testing for first-time right development ITE - Independent Testing eSW Developer www.mastersininnovation.com Focused on application Highly involved, dedicated to implementation Biasing eSW Tester Focused on requirements/functionality Does not know implementation Advantages Bugs are found earlier More bugs are found «The developer should not validate his own code» Slide 32
  33. 33. Integrated hard- & software testing for first-time right development Conclusions Integrated testing, an essential part of eSW development www.mastersininnovation.com Good testplan, testmethodology & enhanced testtools will save your time Simulation enables early testing and provides valuable input for the development PROBA 1 mission successful Launched 22.10.2001 Commissioned in space Planned to operate for 2 years & still operational Slide 33
  34. 34. Integrated hard- & software testing for first-time right development PRoject for On-Board Autonomy Platform for small satellites www.mastersininnovation.com Demonstration of Operational autonomy New technologies (batteries, positioning) Payloads Create wider path for use of small satellites Autonomy (low cost of ownership) Powerful on-board computer (advanced data processing) Advanced Attitude & Orbit Control System (AOCS) Slide 34
  35. 35. Integrated hard- & software testing for first-time right development PROBA – Mission Main task www.mastersininnovation.com PROBA 1: Observation of the Earth PROBA 2: Observation of the Sun Schedule PROBA 1: launched 22.10.2001 PROBA 2: scheduled for mid 2007 PROBA 3: on the roadmap Low earth orbit (LEO) – 600km Slide 35
  36. 36. Integrated hard- & software testing for first-time right development PROBA – Key data Micro satellite www.mastersininnovation.com 120kg, 80cm x 60cm x 60cm 40W to 120W power Modular & Flexible Launcher interface Accommodation of off-the-shelf payloads Software modularity Powerful processors: ERC32,LEON SPARC OS : VxWorks, RTEMS Slide 36
  37. 37. Integrated hard- & software testing for first-time right development www.mastersininnovation.com Slide 37
  38. 38. Integrated hard- & software testing for first-time right development www.mastersininnovation.com Verhaert New Products & Services nv Hogenakkerhoekstraat 21 9150 Kruibeke Belgium Tel +32 (0)3 250 19 00 Fax +32 (0)3 254 10 08 www.verhaert.com info@verhaert.com www.mastersininnovation.com Slide 38

×