Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization
•
0 likes•388 views
This document proposes SAMOTA, a surrogate-assisted many-objective optimization approach for online testing of DNN-enabled systems. SAMOTA uses global and local surrogate models to replace expensive function evaluations. It clusters local data points and builds individual surrogate models for each cluster, rather than one model for all data. An evaluation on a DNN-enabled autonomous driving system shows SAMOTA achieves better test effectiveness and efficiency than alternative approaches, and clustering local data points leads to more effective local searches than using a single local model. SAMOTA is an effective method for online testing of complex DNN systems.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization
Similar to Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization (20)
More from Lionel Briand
More from Lionel Briand (20)
Recently uploaded
Recently uploaded (20)
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization
- 1. Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization Fitash Ul Haq, Donghwan Shin, Lionel Briand Date: May 2022
- 2. 2 Introduction DNN-Enabled System (DADS): • Composed of multiple DNNs capable of various tasks as object tracking, object classification, traffic light detection and traffic sign detection Self-Driving Cars Autonomous Drones
- 3. 3 Introduction Cannot Find Safety Violation Online Testing Can Find Testing DNN-Enabled System (DADS) Offline Testing
- 4. 4 Introduction Challenges for Online Testing: To address the challenges, we propose SAMOTA (Surrogate-Assisted Many-Objective Testing Approach) by leveraging many-objective search and Surrogate Models (SMs) Large Input Space Many Safety Requirements Computationally- intensive Simulation 3rd Party DNNs
- 5. 5 Key Ideas [1, 2] [1] Zhou et al. "Combining global and local surrogate models to accelerate evolutionary optimization." IEEE Transactions on Systems, Man, and Cybernetics, Part C 37, no. 1 (2006): 66-76. [2] Wang et al. "Committee-based active learning for surrogate-assisted particle swarm optimization of expensive problems." IEEE transactions on Cybernetics 47, no. 9 (2017): 2664-2677. Global Search Local Search Local Surrogate Models Global Surrogate Models Shared Database Uses Uses G L for Exploration for Exploitation
- 6. 6 Surrogate Models • Surrogate models are used to replace the computationally expensive function evaluations with much less expensive approximations Polynomial Regression (PR) Radial Basis Function (RBF) Kringing (KR)
- 7. 7 • Ensemble of Surrogate Models Surrogate Model (Contd.) KR Input RBF PR Weight Assignment Ensemble Output Easy uncertainty calculation Low risk of Poorly trained SM Better performance
- 8. 8 Overview: SAMOTA (Surrogate-Assisted Many-Objective Test generation Approach) Execute Simulator Global SMs Many Objective Search Algorithm Most Critical Test Cases Most Uncertain Test Cases Global Search Initialisation Execute Simulator Database Minimal Test Suite 1. Initialization 2. Global search 3. Local search 4. Glocal search 5. Local search 6. … *Note: search will focus on uncovered objectives only (and therefore reduce the number of real simulation executions) Local SM per Cluster Local Search Most Critical Test Cases Single Objective Search Algorithm Clustering Top Points
- 9. 9 Global Search • Uses one unit of “global” SM (single or ensemble) trained on all the data points • Captures global profile of search space (for exploration) • Returns the best predicted test cases and most uncertain test cases • Uncertain test cases maximizes the information gain of the surrogate models, making them more accurate faster. Execute Simulator Global SMs Many Objective Search Algorithm Most Critical Test Cases Most Uncertain Test Cases Global Search Database
- 10. 10 Local Search • Uses a “local” SM constructed by using only the top n% data points • In the literature, the SM generation approach generates one SM for the top n% • It is not optimal Local Search Database Local SM
- 11. 11 Local Search • Uses a “local” SM constructed by using only the top n% data points • In the literature, the SM generation approach generates one SM for the top n% • It is not optimal • We propose using clustering algorithm to cluster the top n% data points and build one SM for each cluster • Captures local profile of promising region in search space (for exploitation) Local SM per Cluster Local Search Most Critical Test Cases Single Objective Search Algorithm Clustering Top Points Database Execute Simulator
- 12. 12 Research Questions What is the best configuration for local search (LS)? How do alternative approaches fare in terms of test effectiveness? How do alternative approaches fare in terms of test efficiency? RQ1 RQ2 RQ3
- 13. 13 Case Study Subject • Pylot (DNN-enabled ADS): • Pylot provides the implementations of state-of-the-art approaches based on pre-trained DNNs • CARLA (Simulator): • Open-source simulator based on the Unreal Engine designed to support training, development, and validation of ADS PYLOT4 CARLA Simulator3 [3] https://carla.readthedocs.io/en/latest/start_introduction/ [4] https://pylot.readthedocs.io/en/latest/
- 14. 14 RQ1: Best Configuration for Local Search SM Generations Local Search SM Types Local Search Effectiveness (LSE) § GA § 2 hours § 20 repetitions One SM for each cluster (using HDBScan) One SM for all top n% RBF PR KR ! - single test case " - single objective # - set of objectives $(!, ") - fitness value of " in ! ()* + = ∑./0 max !4+ $(!, ") |#|
- 15. 15 RQ1: Results • Using our clustering-based approach (cl) for surrogate model generation is significantly better than the existing approach (al) in all cases
- 16. 16 RQ1: Results • Using our clustering-based approach (cl) for surrogate model generation is significantly better than the existing approach (al) in all cases • There is no significant difference overall between different surrogate model types • On average, RFcl performs better than other surrogate model types
- 17. 17 RQ2: Test Effectiveness Online Testing of DADS Test Effectiveness (TE) § 2 hours § 20 repetitions Search Algorithms SAMOTA without Initial database (SE) SAMOTA with Initial database (SI) FITEST (FI) MOSA (MO) Random Search (RS) !" = # %& '(&)*+ ,-%.(*-%/0 # %& '(&)*+ 1)23-4)5)/*0
- 18. 18 RQ2: Results • SAMOTA variants are significantly more effective than other many-objective search algorithms tailored for test suite generation and random search with archive • Furthermore, SAMOTA can achieve acceptable test effectiveness without an initial database
- 19. 19 RQ3: Test Efficiency Online Testing of DADS Test Efficiency § 2 hours § 20 repetitions § 20 minutes interval Search Algorithms SAMOTA without Initial database (SE) SAMOTA with Initial database (SI) FITEST (FI) MOSA (MO) Random Search (RS)
- 20. 20 RQ3: Results • SAMOTA is more efficient than alternative test suite generation approaches as soon as its SMs become sufficiently accurate • An initial database can boost the efficiency of SAMOTA in the initial search phase and allow it to surpass other techniques right from the start.
- 21. 21 Conclusion
- 22. Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization Fitash Ul Haq, Donghwan Shin, Lionel Briand Date: May 2022