Testing Dynamic Behavior in Executable Software Models - Making Cyber-physical Systems Testable
•
2 likes•979 views
This document discusses testing dynamic behavior in executable software models for cyber-physical systems. It presents challenges for model-in-the-loop (MiL) testing due to large input spaces, expensive simulations, and lack of simple oracles. The document proposes using search-based testing to generate critical test cases by formulating it as a multi-objective optimization problem. It demonstrates the approach on an advanced driver assistance system and discusses improving performance with surrogate modeling.
More Related Content
What's hot
What's hot (20)
Similar to Testing Dynamic Behavior in Executable Software Models - Making Cyber-physical Systems Testable
Similar to Testing Dynamic Behavior in Executable Software Models - Making Cyber-physical Systems Testable (20)
More from Lionel Briand
More from Lionel Briand (20)
Recently uploaded
Recently uploaded (20)
Testing Dynamic Behavior in Executable Software Models - Making Cyber-physical Systems Testable
- 1. .lusoftware verification & validation VVS Testing Dynamic Behavior in Executable Software Models! -! Making Cyber-physical Systems Testable! ! Lionel Briand July 18, ISSTA 2016
- 2. Acknowledgements • Shiva Nejati • Reza Matinejad • Raja Ben Abdessalem 2
- 3. Cyber-Physical Systems • Increasingly complex and critical systems • Complex environment • Complex requirements, e.g., temporal, timing, resource usage • Dynamic behavior • Uncertainty, e.g., about the environment • Testing is expensive and difficult, e.g., HW in the loop 3 Cyber Space Physical Sensing Actuation Information Networks Object Domain Real Space
- 4. Dynamic Behavior • Common when dealing with physical entities • Inputs and outputs are variables evolving over time (signals) • Properties to be verified consider change over time, for individual or sets of outputs 4 Time-Continuous Magnitude-Continuous time value
- 5. MiL Testing Hardware-in-the-Loop Stage Model-in-the-Loop Stage Simulink Modeling Generic Functional Model MiL Testing Software-in-the-Loop Stage Code Generation and Integration Software Running on ECU SiL Testing Software Release HiL Testing 5 Both the system and environment are modeled Testing is highly expensive and time consuming
- 6. Simulink Models - Simulation • Simulation Models • heterogeneous 6 Time-Continuous Simulink Model Hardware Model Network Model • continuous behavior • are used for • algorithm design testing • comparing design options
- 7. MiL Test Cases 7 Model Simulation Input Signals Output Signal(s) S3 t S2 t S1 t S3 t S2 t S1 t Test Case 1 Test Case 2
- 8. MiL Testing Challenges • Space of test input signals is extremely large. • Model execution, especially when involving physical modeling, is extremely expensive. • Oracles are not simple Boolean properties – they involve analyzing changes in value over time (e.g., signal patterns) and assess levels of risk. 8
- 9. MiL Testing Challenges (2) • Simulable model of the (physical) environment is required for test automation, but not always available. • Effectiveness of test coverage strategies is questionable, e.g., model coverage. • No equivalence classes on input signal domains, no combinatorial approaches. 9
- 10. We need novel, automated, and cost-effective MiL testing strategies for CPS 10
- 12. Advanced Driver Assistance Systems (ADAS) Decisions are made over time based on sensor data 12 Sensors Software
- 13. Pedestrian Detection System (PeVi) 13 • The PeVi system is a camera-based assistance system providing improved vision
- 14. Challenges • Simulation/testing is performed using physics-based simulation environments • Challenge 1: A large number of simulation scenarios • more than 2000 configuration variables • Challenge 2: Simulations are computationally expensive 14 weather road sensors human vehicles Simulation Scenario
- 15. Approach 15 Generation of Test specifications Static [ranges/values/ resolution] Dynamic [ranges/ resolution] (2) test case specification Specification Documents (Simulation Environment and PeVi System) Domain model Requirements model (1)Development of Requirements and domain models
- 16. Domain Model 16 - intensity: Real SceneLight Dynamic Object 1 - weatherType: Condition Weather - fog - rain - snow - normal «enumeration» Condition Output Trajectory - field of view: Real Camera Sensor RoadSide Object - roadType: RT Road 1 - curved - straight - ramped «enumeration» RT - vc: Real Vehicle - x0: Real - y0: Real - θ: Real - vp: Real Pedestrian - x: Real - y: Real Position 1 * 1 * 1 1 - state: Boolean Collision Parked Cars Trees - simulationTime: Real - timeStep: Real Test Scenario PeVi - state: Boolean Detection 1 1 11 1 1 1 1 «positioned» «uses» 1 1
- 17. Requirements Model 17 <<trace>> <<trace>> Speed Profile Path 1 1 Slot Path Segment 1..** 1 Trajectory Human 1* trajectory Warning Sensors posx1, posx2 posy1, posy2 AWACar/Motor/ Truck/Bus sensor has has awa 1 1 1 * human appears posx1 posx2 posy1 posy2 The PeVi system shall detect any person located in the Acute Warning Area of a vehicle
- 18. Test Generation Overview 18 Simulator + PeVi Environment Settings (Roads, weather, vehicle type, etc.) Fixed during Search Manipulated by Search Human Simulator (initial position, speed, orientation) Car Simulator (speed) PeVi Meta-heuristic Search (multi-objective) Generate scenarios Detection or not? Collision or not?
- 19. Multi-Objective Search • Search algorithm needs objective or fitness functions for guidance • In our case several independent functions can be interesting (heuristics): • Distance between car and pedestrian • Distance between pedestrian and AWA • Time to collision 19 posx1 posx2 posy1 posy2
- 20. Pareto Front 20 Individual A Pareto dominates individual B if A is at least as good as B in every objective and better than B in at least one objective. Dominated by x O1 O2 Pareto front x • A multi-objective optimization algorithm must achieve: • Guide the search towards the global Pareto-Optimal front. • Maintain solution diversity in the Pareto-Optimal front.
- 21. MO Search with NSGA-II 21 Non-Dominated Sorting Selection based on rank and crowding distance Size: 2*N Size: 2*N Size: N • Based on Genetic Algorithm • N: Archive and population size • Non-Dominated sorting: Solutions are ranked according to how far they are from the Pareto front, fitness is based on rank. • Crowding Distance: Individuals in the archive are being spread more evenly across the front (forcing diversity) • Runs simulations for close to N new solutions
- 25. Improving Time Performance • Individual simulations take on average more than 1min • It takes 10 hours to run our search-based test generation ! (≈ 500 simulations) è We use surrogate modeling to improve the search • Goal: Predict fitness based on dynamic variables • Neural networks 25
- 26. Multi-Objective Search with Surrogate Models 26 Non-Dominated Sorting Selection based on rank and crowding distance Size: 2*N Size: 2*N Size: N Original Algorithm - Runs simulations for all! new solutions New Algorithm - Uses prediction values &! prediction errors to run ! simulations only! for the solutions that ! might be selected
- 27. Results – Surrogate Modeling 27 0.00 0.25 0.50 0.75 1.00 Time (min) HV 50 100 15010 (a) Comparing HV values obtained by NSGAII and NSGAII-SM NSGAII (mean) NSGAII-SM (mean)
- 28. Results – Worst Runs 28 Time (min) 0.00 0.25 0.50 0.75 1.00 Time (min) HV 50 100 15010 (c) HV values for worst runs of NSGAII, NSGAII-SM and RS RS NSGAII-SM NSGAII
- 29. Results – Random Search 29 0.00 0.25 0.50 0.75 1.00 Time (min) 50 100 15010 (b) Comparing HV values obtained by RS and NSGAII-SM HV RS (mean) NSGAII-SM (mean) (c) HV values for worst runs of NSGAII,
- 30. Conclusion • A general testing approach for ADAS, and many CP systems • Formulated the generation of critical test cases as a multi-objective search problem using NSGAII algorithm • Improved the search performance with surrogate models based on neural networks • Generated some critical scenarios: no detection in the AWA, collision and no detection • No clear cut oracle – failure to detect may be deemed acceptable risk 30
- 32. • Supercharger bypass flap controller ü Flap position is bounded within [0..1] ü Implemented in MATLAB/Simulink ü 34 (sub-)blocks decomposed into 6 abstraction levels ü The simulation time T=2 seconds Supercharger Bypass Flap Supercharger Bypass Flap Flap position = 0 (open) Flap position = 1 (closed) Simple Example 32
- 33. Initial Desired Value Final Desired Value time time Desired Value Actual Value T/2 T T/2 T Test Input Test Output Plant Model Controller (SUT) Desired value Error Actual value System output+ - MiL Testing of Controllers 33
- 34. Configurable Controllers at MIL Plant Model + + + ⌃ + - e(t) actual(t) desired(t) ⌃ KP e(t) KD de(t) dt KI R e(t) dt P I D output(t) Time-dependent variables Configuration Parameters 34
- 35. Requirements and Test Objectives InitialDesired (ID) Desired ValueI (input) Actual Value (output) FinalDesired (FD) time T/2 T Smoothness Responsiveness Stability 35
- 36. A Search-Based Test Approach Initial Desired (ID) FinalDesired(FD) Worst Case(s)? • Continuous behavior • Controller’s behavior can be complex • Meta-heuristic search in (large) input space: Finding worst case inputs • Possible because of automated oracle (feedback loop) • Different worst cases for different requirements • Worst cases may or may not violate requirements 36
- 37. Initial Solution HeatMap Diagram 1. Exploration List of Critical RegionsDomain Expert Worst-Case Scenarios + Controller- plant model Objective Functions based on Requirements 2. Single-State Search time Desired Value Actual Value 0 1 2 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Initial Desired Final Desired 37
- 38. Results • We found much worse scenarios during MiL testing than our partner had found so far • These scenarios are also run at the HiL level, where testing is much more expensive: MiL results -> test selection for HiL • But further research was needed: • Simulations are expensive • Configuration parameters 38
- 39. Final Solution + Controller Model (Simulink) Worst-Case Scenarios List of Critical PartitionsRegression Tree 1.Exploration with Dimensionality Reduction 2.Search with Surrogate Modeling Objective Functions Domain Expert Visualization of the 8-dimension space using regression trees Dimensionality reduction to identify the significant variables (Elementary Effect Analysis) Surrogate modeling to predict the objective function and speed up the search (Machine learning) 39
- 40. Regression Tree All Points FD>=0.43306 Count Mean Std Dev Count Mean Std Dev FD<0.43306 Count Mean Std Dev ID>=0.64679 Count Mean Std Dev Count Mean Std Dev Cal5>=0.020847 Cal5>0.020847 Count Mean Std Dev Count Mean Std Dev Cal5>=0.014827 Cal5<0.014827 Count Mean Std Dev Count Mean Std Dev 1000 0.007822 0.0049497 ID<0.64679 574 0.0059513 0.0040003 426 0.0103425 0.0049919 373 0.0047594 0.0034346 201 0.0081631 0.0040422 182 0.0134555 0.0052883 244 0.0080206 0.0031751 70 0.0106795 0.0052045 131 0.0068185 0.0023515 40
- 41. Surrogate Modeling Any supervised learning or statistical technique providing fitness predictions with confidence intervals 1. Predict higher fitness with high confidence: Move to new position, no simulation 2. Predict lower fitness with high confidence: Do not move to new position, no simulation 3. Low confidence in prediction: Simulation Surrogate Model Real Function x Fitness 41
- 42. ü Our approach is able to identify more critical violations of the controller requirements that had neither been found by our earlier work nor by manual testing. MiL-Testing different configurations Stability Smoothness Responsiveness MiL-Testing fixed configurations Manual MiL-Testing - -2.2% deviation 24% over/undershoot 20% over/undershoot 5% over/undershoot 170 ms response time 80 ms response time 50 ms response time Results 42
- 43. Open Loop Controllers On Off CtrlSig • Mixed discrete-continuous behavior: Simulink stateflows • No plant model: Much quicker simulation time • No feedback loop -> no automated oracle • The main testing cost is the manual analysis of output signals • Goal: Minimize test suites • Challenge: Test selection • Entirely different approach to testing techniques. We consider this criterion flows have complex internal structures uations, making them less amenable to they have rich time-continuous outputs. e following contributions: blem of testing Stateflows with mixed behaviours. We propose two new test a output stability and output continuity cting test inputs that are likely to pro- puts exhibiting instability and disconti- ively. box coverage and the blackbox output iteria to Stateflows, and evaluate their for continuous behaviours. The former raditional state and transition coverage nd the latter is defined based on the re- ss criterion [?]. eness of our newly proposed and the eria by applying them to three Stateflow two industrial and one public domain. RESULT. r. (c) Engaging state of SCC -- mixed discrete-continuous behaviour Disengaging Engaged [disengageReq]/time := 0 [time>5] [time>5] time + +; OnMoving OnSlipping OnCompleted time + +; ctrlSig := f(time) Engaging time + +; ctrlSig := g(time) time + +; ctrlSig := 1.0 [¬(vehspd = 0) time > 2] [(vehspd = 0) time > 3] [time > 4] Figure 1: Supercharge Clutch Controller (SCC) Stateflow. transient states [?], engaging and disengaging, specifying that mov- ing from the engaged to the disengaged state and vice versa takes six milisec. Since this model is simplified, it does not show han- dling of alterations of the clutch state during the transient states. In addition to adding the time variable, we note that the variable 43
- 44. Selection Strategies Based on Search • Input signal diversity • White-box structural coverage • State Coverage • Transition Coverage • Output signal diversity • Failure-Based selection criteria • Domain specific failure patterns • Output Stability • Output Continuity S3 t S3 t 44
- 45. ! Output Diversity -- Vector-Based 45 Output Time Output Signal 2 Output Signal 1
- 46. 46 Output Diversity -- Feature-Based increasing (n) decreasing (n)constant-value (n, v) signal features derivative second derivative sign-derivative (s, n) extreme-derivatives 1-sided discontinuity discontinuity 1-sided continuity with strict local optimum value instant-value (v) constant (n) discontinuity with strict local optimum increasing C A B
- 47. Failure-based Test Generation 47 Instability Discontinuity 0.0 1.0 2.0 -1.0 -0.5 0.0 0.5 1.0 Time CtrlSigOutput • Search: Maximizing the likelihood of presence of specific failure patterns in output signals • Domain-specific failure patterns elicited from engineers 0.0 1.0 2.0 Time 0.0 0.25 0.50 0.75 1.0 CtrlSigOutput
- 48. Results • The test cases resulting from state/transition coverage algorithms cover the faulty parts of the models • However, they fail to generate output signals that are sufficiently distinct from expectations, hence yielding a low fault revealing rate • Output-based algorithms are much more effective • Existing commercial tools: Not effective at finding faults, not applicable to entire Simulink models 48
- 49. Reflecting 49
- 50. Commonalities • Large input spaces • Combinatorial approaches not applicable • Coverage? • Expensive testing: test execution time, oracle analysis effort • Complex oracle (dynamic behavior) • Testing is driven by risk • Search-based solution: Highest risk scenarios 50
- 51. Differences • Model execution time, e.g., plant or not • Fitness function: Exact or heuristic • Single or multi objectives • Automated oracle or not • Other techniques involved to achieve scalability: regression trees, neural networks, sensitivity analysis, … 51
- 52. Related Work 52
- 53. Constraint Solving • Test data generation via constraint solving is not feasible when: • Continuous mathematical models, e.g., differential equations • Library functions in binary code • Complex operations • Constraints capturing dynamic properties (discretized) tend not to be scalable 53
- 54. Search-Based Testing • Largely focused on unit or function testing, where the goal is to maximize model coverage, check temporal properties (state transitions) … • To address CPS, we need more work on system-level testing, targeting dynamic properties in complex input spaces capturing the behavior of physical entities. 54
- 55. Future: ! A more general methodological and automation framework, targeting more complex and heterogeneous CPS models 55
- 56. Future Work • Shifting the bulk of testing from implemented systems to models of such systems and their environments requires: • Heterogeneous modeling and co-simulation • Modeling dynamic properties and risk • Uncertainty modeling enabling probabilistic test oracles • Executable model at a proper level of precision for testing purposes • Use results to make the best out of available time and resources for testing the implemented system with hardware in the loop • Focus on high risk test scenarios within budget and time constraints 56
- 57. References • R. Ben Abdessalem et al., "Testing Advanced Driver Assistance Systems Using Multi-Objective Search and Neural Networks”, ASE 2016 • R. Matinnejad et al., “Automated Test Suite Generation for Time-continuous Simulink Models“, ICSE 2016 • R. Matinnejad et al., “Effective Test Suites for Mixed Discrete-Continuous Stateflow Controllers”, ESEC/FSE 2015 (Distinguished paper award) • R. Matinnejad et al., “MiL Testing of Highly Configurable Continuous Controllers: Scalable Search Using Surrogate Models”, ASE 2014 (Distinguished paper award) • R. Matinnejad et al., “Search-Based Automated Testing of Continuous Controllers: Framework, Tool Support, and Case Studies”, Information and Software Technology, Elsevier (2014) 57
- 58. .lusoftware verification & validation VVS Testing Dynamic Behavior in Executable Software Models! ! Lionel Briand ** WE HIRE! ** July 18, ISSTA 2016