Download as PDF, PPTX
Uploaded byLionel Briand
Test Case Prioritization for Acceptance Testing of Cyber Physical Systems
1) The document describes a multi-objective search-based approach for minimizing and prioritizing acceptance test cases for cyber physical systems to address challenges like time overhead, uncertainties in execution time, and risks of hardware damage. 2) The approach models acceptance tests, minimizes test cases by removing redundant operations, and prioritizes test cases using Monte Carlo simulation to estimate execution times while optimizing for criticality and risk. 3) An empirical evaluation on an industrial case study of in-orbit satellite testing shows the approach generates test suites that cover more test cases and have lower hardware risk within time budgets compared to manually created test suites.
More Related Content
Similar to Test Case Prioritization for Acceptance Testing of Cyber Physical Systems
Test Case Prioritization for Acceptance Testing of Cyber Physical Systems
- 1. .lusoftware verification &validation VVS Test Case Prioritization for Acceptance Testing of Cyber Physical Systems: A Multi-objective Search-based Approach Seung Yeob Shin1, Shiva Nejati1, Mehrdad Sabetzadeh1, Lionel C. Briand1, and Frank Zimmer2 1 SnT Centre / University of Luxembourg 2 SES Networks ISSTA 2018
- 2. Acceptance Testing of CyberPhysical Systems • Validation • Actual hardware platform • Motivating case study • In-orbit satellite testing 2 Launch effect: e.g., vibration Environment: e.g., space
- 3. Acceptance Test Cases •Manipulating hardware devices • Time-consuming • Uncertainties in the environment • Risks of hardware damage 3 Test instruments Ground station Satellite under test Uplink Downlink
- 4. Time Overhead of Initializationand Teardown • Acceptance test case structure • Redundant initialization and teardown operation calls 4 initialization test scenario teardown Test case A Test case B Test case C Unnecessary teardown à Time overhead S1 S1 S2
- 5. Uncertainties in ExecutionTime • The amount of the actual movement of hardware depends on the environment • Hardware recalibration in response to changes in the environment (e.g., temperature) 5 Recalibrate hardware devices
- 6. Test Time BudgetConstraints • Business (time to market) constraints • Environmental constraints 6 Satellite life-cycle V&V on ground Commercial services 2 months of in-orbit testing SUTNeighboring satellites Risk of interference Launch
- 7. Risks of HardwareDamage Manipulating hardware devices may entail potential damage to the system under test or its environment 7 Satellite communication block diagram Channel 1 Channel 2 Damaged switch Out of service
- 8. Problem Statement How toplan a test suite for acceptance testing of cyber physical systems? to validate that the SUT meets its requirements prior to providing actual services to determine whether the SUT can be sufficiently trusted and put into operation 8 Time Overhead of Initialization and Teardown • Acceptance test case structure • Redundant initialization and teardown operation calls 4 initialization test scenario teardown Test case A Test case B Test case C Unnecessary teardown à Time overhead Sat. 1 Sat. 1 Sat. 2 Uncertainties in Execution Time • The amount of the actual movement of hardware depends on the environment • Hardware recalibration in response to changes in the environment (e.g., temperature) 5 Recalibrate hardware devices Test Time Budget Constraints • Business (time to market) constraints • Environmental constraints 6 Satellite life-cycle V&V on ground Commercial services 2 months of in-orbit testing SUTNeighboring satellites Risk of interference Launch Risks of Hardware Damage Manipulating hardware devices may entail potential damage to the system under test or its environment 7 Satellite communication block diagram Channel 1 Channel 2 Damaged switch Out of service
- 9. A multi-objective approachfor minimizing and prioritizing acceptance test cases in cyber physical systems
- 10. Approach Overview 10 Test Case Minimization Modeling Acceptance Tests TestCase Prioritization Monte-Carlo Simulation
- 11. Modeling Acceptance Tests 11 initialization testscenario teardown Test case Components manipulates Test suite: a sequence of test cases initialization test scenario teardown initialization test scenario teardown Synthesizer Spectrum analyzer Amplifier
- 12. Test Case Minimization 12 initialization testscenario teardown initialization test scenario teardown Power off Power on Power state: on
- 13. Test Case Minimization •Removing redundant initialization and teardown operation calls • This reduces the number of operation calls, hence lowering • execution time of test cases, and • risks of hardware damage incurred by test execution • Our approach uses model checking to ensure that the redundancy removal is behavior-preserving 13
- 14. Monte-Carlo Simulation 14 Historical dataand product spec. Execution time of hardware operations Monte-Carlo Simulation Test suite Estimating distributions of test execution time initialization test scenario teardown initialization test scenario teardown time probability
- 15. Two important prioritizationcriteria: • Test case criticality • Risks of hardware damage Test Case Prioritization 15 Test critical function Test low risks of dmg Test backup function Test critical function Test high risks of dmg Test low risks of dmg time budget time budget
- 16. Multiple Objectives • Rewardearly appearance of test cases that are critical and fast • Reward early appearance of test cases that are low risk 16 time criticality # test cases risk
- 17. Multi-objective Search 17 Search Evaluation Crossover Selection Mutation Initial solutions Asearch algorithm inspired by the process of natural selection
- 18. Minimization Multi-objective Search 18 NSGA II Evaluation Crossover Selection Mutation Initialsolutions Simulation Objective evaluation Test Case Minimization 12 initialization test scenario teardown initialization test scenario teardown Power off Power on Power state: on Multiple Objectives • Reward early appearance of test cases that are critical and fast • Reward early appearance of test cases that are low risk 18 time criticality # test cases risk Monte-Carlo Simulation 14 Historical data and product spec. Execution time of hardware operations Monte-Carlo SimulationTest suite Estimating distributions of test execution time initialization test scenario teardown initialization test scenario teardown time probability
- 19. Empirical Evaluation RQ1 Sanitycheck: How does our test suite generation approach fare against random search? RQ2 Impact of the minimization: How does our test suite generation approach perform with and without the minimization algorithm? RQ3 Usefulness: How do test suites generated by our approach compare with test suites built manually by expert engineers? 19
- 20. Usefulness: Results 20 Objective valueof criticality and execution time Objectivevalueofhardwaredamagerisks -1.04e+08 -1.02e+08 -1e+08 -9.8e+07 -6.4e+07 105000110000115000350000 Test suite by engineers Test suites by our approach Best test suites by our approach
- 21. Usefulness: Results 21 Objective valueof criticality and execution time Objectivevalueofhardwaredamagerisks -1.04e+08 -1.02e+08 -1e+08 -9.8e+07 -6.4e+07 105000110000115000350000 For a given time budget, our approach + 2X # of test cases - 1/3 hardware risk
- 22. Conclusions • Test suiteoptimization for acceptance testing of cyber physical systems • Modeling acceptance tests • Test case prioritization • Test case minimization • Monte-Carlo simulation • Industrial case study • In-orbit satellite testing 22 Usefulness: Results 23 Objective value of criticality and execution time Objectivevalueofhardwaredamagerisks -1.04e+08 -1.02e+08 -1e+08 -9.8e+07 -6.4e+07 105000110000115000350000 For a given time budget, our approach + 2X # of test cases - 1/3 hardware risk Modeling Acceptance Tests 11 initialization test scenario teardown Test case Components manipulates Test suite: a sequence of test cases initialization test scenario teardown initialization test scenario teardown Synthesizer Examples Spectrum analyzer Amplifier Acceptance Testing of Cyber Physical Systems • Validation • Actual hardware platform • Motivating case study • In-orbit satellite testing 2 Launch effect: e.g., vibration Environment: e.g., space Minimization Multi-objective Search 18 NSGA II Evaluation Crossover Selection Mutation Initial solutions Simulation Objective evaluation Test Case Minimization 12 initialization test scenario teardown initialization test scenario teardown Power off Power on Power state: on Multiple Objectives • Reward early appearance of test cases that are critical and fast • Reward early appearance of test cases that are low risk 18 time criti cality # test cases risk
- 23.
- 24. Test Case Prioritization: PriorResearch and Our Contributions • Prior research • Widely studied in the context of regression testing • Fault revealing capability • Many prior techniques require observability into the source code • Code coverages • Our contributions • Characterization of test case prioritization in the context of acceptance testing • A test case minimization algorithm for removing unnecessary operation calls • Direct support for handling uncertainty in test case prioritization 24