SMARLA: Enhancing Safety Monitoring for Reinforcement Learning Agents

SMARLA: Enhancing Safety Monitoring for Reinforcement Learning Agents

• 1.
  1 uOttawa.ca uOttawa.ca | nanda-lab.ca Schoolof Electrical Engineering & Computer Science | Nanda Lab SMARLA: A Safety Monitoring Approach for Reinforcement Learning Agents A. Zolfagharian, M. Abdellatif, L. C. Briand, and Ramesh. S, IEEE Transactions on Software Engineering, vol. , no. 01, pp. 1- 25, PrePrints 5555, doi: 10.1109/TSE.2024.3491496
• 2.
  2 Motivation The Growing Importanceof RL and Real-World Impact: • Reinforcement Learning (RL) has gained significant importance in solving real-world challenges specially in safety-critical domain such as autonomous systems and robotics. • Deep Reinforcement Learning (DRL) is an extension of RL in which the policy is learned by a DNN, enabling agents to process high-dimensional inputs and learn directly from raw data, therefore expanding RL's applicability. Action Reward & State Simulation environment Image: Nvidia Isaak sim
• 3.
  3 Problem • Safety Concerns:Growing use of DRL systems, especially in safety- critical domains, raises significant concerns about their safe and reliable behavior. • Lack of Guarantees: DRL agents, focused on maximizing rewards, may inadvertently violate safety protocols, leading to potentially dangerous situations. • Traditional Testing Limitations: Traditional testing methods struggle to comprehensively evaluate DRL agents due to absence of source code, the probabilistic nature of the environments and vast state spaces. Image: Boston Dynamics and Cruise
• 4.
  4 Solution • Project withGeneral Motors • A Two-Phase Comprehensive Solution Toward RL Safety: • Pre-Deployment Testing: Essential to identify unsafe behaviors in DRL agents before deployment. • Runtime Safety Monitoring: Continuous safety monitoring ensures ongoing protection by predicting and mitigating potential safety violations during operation. STARLA: • A search-based testing approach for DRL agents, with a focus on detecting unsafe episodes. SMARLA: • A runtime safety monitoring approach that predicts and helps prevent potential safety violations during the DRL agent operation.
• 5.
  5 Problem • Testing islimited: Despite significant efforts to test RL agents across various scenarios, ensuring runtime safety remains challenging due to the extremely large agent state space and the sheer number of possible execution scenarios. • Formal Verification is impractical: Analyzing reachable states in DRL has been proven to be an NP-Complete problem. Solution • Runtime safety monitoring of RL agent (Q-Learning) • SMARLA: Safety monitoring approach designed for RL agents, aiming to detect and prevent unsafe behaviors before catastrophic outcomes. • SMARLA targets resource-constrained edge devices, such as drones, and uses a black-box approach.
• 6.
  6 Safety Monitor’s Characteristics KeyCharacteristics of an Ideal Safety Monitoring System: • Accurate Predictions – Ensuring accurate prediction of safety violations and minimizing false alarms. • Early Prediction of Safety Violations – Predicting safety violations as early as possible, allowing for timely corrective or preventive actions. • Lightweight Prediction: inference time, memory usage, training cost Benefits of a Safety Monitoring System: • Provide early warning alerts to prevent accidents or minimize their impact. • Identify situations in real time when corrective action may be necessary.
• 7.
  7 Overview of SMARLA Input: •Agent behavior as states occurring during an episode Output: • At each time step, the output is the probability of a safety violation Agent’s behavior SMARLA (Safety Monitor) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 10 20 30 40 50 60 70 80 90 Probability of safety violation Time step Risk plot Episode 1 Threshold Risk > Threshold Continue execution Initiate safety mechanism No Yes
• 8.
  8 SMARLA Training • Runtests and extract episodes (state-action pairs) • State abstraction reduces the state space • Abstraction level adjusted to achieve optimal accuracy • Only requires access to Q-values (expected cumulative reward) • Black-box, not monitoring the environment • Agnostic to the type of DRL agent’s inputs
• 9.
  9 Approach – Q*State Abstraction • We rely on a Q-irrelevance state abstraction technique. • Q*(s,a) is the predicted overall reward (expected maximum reward) from state s, when the agent selects action a in state s • We consider two states in the same abstraction class when the overall reward for all actions in both states is the same or similar (ceiling values of Q*). • The abstraction level can be changed by parameter 𝑑 Q∗ (s1,a) 𝑑 = Q∗ (s2,a) 𝑑 Q*(s1,a1) Q*(s2,a1) Q*(s2,a2) Q*(s1,a2) ∀𝑎 ∈ 𝐴
• 10.
  10 Features: Presence andabsence of the abstract states SMARLA Training Episodes Abstract state 1 Abstract state 2 …. Abstract state n Label E1 = [(S1,a1), (S2,a2),… , (Sj,aj)] 0 1 0 Safe E2 = [(S’1,a’1), (S’2,a’2),… , (S’i,a’i)] 1 1 1 Unsafe
• 11.
  11 Features: Frequency ofthe abstract states SMARLA Training Episodes Abstract state 1 Abstract state 2 …. Abstract state n Label E1 = [(S1,a1), (S2,a2),… , (Sj,aj)] 0 1 3 Safe E2 = [(S’1,a’1), (S’2,a’2),… , (S’i,a’i)] 1 2 1 Unsafe
• 12.
  12 SMARLA Training Lightweight MLmodel that accurately classifies RL episodes as safe or unsafe, given our focus on resource-constrained edge devices. • Therefore, we exclude DNN models and choose Random Forest as our machine learning model because it: – Can handle many features effectively – Provides short inference time – Is robust to overfitting
• 13.
  13 Safety Monitor inAction
• 14.
  14 Confidence intervals • Randomforest is an ensemble method that combines the predictions of multiple decision trees, each trained on a random subset of the training data. • In order to compute confidence intervals, we use the probability distribution of safety violation generated by decision trees • Then, based on the mean prediction in each time step and standard deviation, confidence intervals is calculated with the following formula. C𝐼 = ത 𝑋 ± 𝑍 × 𝜎 𝑛 • Finally, the confidence interval is: [ ത 𝑋 − 𝑍 × 𝜎 𝑛 , ത 𝑋 + 𝑍 × 𝜎 √𝑛 ] • For 95% confidence, Z = 1.96 • Based on the lower/upper bounds of confidence intervals, we define different triggering criteria to activate the safety mechanism
• 15.
  15 Summary of Results •RQ1: How accurately and early can we predict safety violations during the execution of episodes? – SMARLA demonstrated high accuracy in predicting safety violations (F1-score above 90%) – SMARLA achieved early predictions (halfway through episodes) during the execution of episodes, which may enable proactive prevention and mitigation of potential damages • RQ2: How can the safety monitor determine when to trust the prediction of safety violations? – Compared different decision criteria based on confidence intervals – Using the upper bound of confidence intervals allows for early detection of violations, reducing prediction time steps by 20-45% compared to other methods, though it increases the false positive rate slightly • RQ3: What is the effect of the prediction threshold on the safety monitoring system? – The choice of prediction threshold affects both the timeliness and accuracy of safety violation predictions and should be determined by the criticality of the faults in the environment. • RQ4. What is the effect of the abstraction level on safety monitoring? – It is important to identify the optimal range of abstraction levels, which varies greatly among RL agents.
• 16.
  16 RQ1: How accuratelyand early can we predict safety violations during the execution of episodes? • We monitored the execution of test episodes with SMARLA. Whenever the probability is greater than a threshold (50%) we stop the execution and activate the safety mechanism. • We observe early and accurate predictions of safety violations, which is indicated by precision, recall, and F1-score plateauing from early time steps. • Given the limited information available to the agent in the highway environment, we explored more informative feature representations, such as frequency- based features, which led to significant performance improvements. Mountain-Car Cart-Pole Highway Driving Highway Driving* refers to the highway driving case study with frequency-based features 1000 test episodes, with 59 violations in Highway Driving, 99 violations in Cart-Pole, and 132 in Mountain-Car
• 17.
  17 RQ2. How canthe safety monitor determine when to trust the prediction of safety violations? • We investigate the use of confidence intervals as a mechanism for the safety monitor to determine the appropriate time step to trigger safety mechanisms. • We compare the performance of SMARLA in different time steps relying on the upper bound the lower bound and the probability of safety violation. • Using the upper bound enables earlier detection of safety violations with high accuracy. This provides more time for to mitigate potential damages. Trade-off: Slightly increases false positive rate. Mountain-Car Cart-Pole Highway Driving* Highway Driving* refers to the highway driving case study with frequency-based features
• 18.
  18 RQ3. What isthe effect of the prediction threshold on the safety monitoring system? • We study the impact of varying the prediction threshold values on SMARLA • We evaluate the prediction times and performance of SMARLA using different prediction thresholds – 25%, 50%, 75% • Higher thresholds reduce false positives but delay predictions. Therefore, selecting an optimal threshold is crucial to balance timely and accurate predictions. Mountain-Car Cart-Pole Highway Driving* Highway Driving* refers to the highway driving case study with frequency-based features
• 19.
  19 RQ4. What isthe effect of the abstraction level on the safety monitoring component? (1/2) Accuracy of SMARLA after training with different abstraction levels: This aspect involves evaluating the performance of the Random Forest model once it has been trained on the available training data. We randomly sampled 70% of the dataset to train and 30% to compute the F1-scores of the models using different levels of abstraction. Results suggest an optimal range of abstraction that yields the highest accuracy in predicting safety violations. That range varies greatly among RL agents. Mountain-Car Cart-Pole Highway Driving* Highway Driving* refers to the highway driving case study with frequency-based features
• 20.
  21 Summary • We proposeda black-box safety monitoring approach for reinforcement learning agents. • We rely on a machine learning model to predict safety violations of RL agents early based on (state, action) pairs in episodes. • We employ state abstraction to reduce the state space, enhancing the ML model's learnability. • SMARLA demonstrated high accuracy in predicting safety violations of RL agents. Additionally, our empirical results showed an early prediction of violations, allowing for timely prevention and damage mitigation. • Our method is agnostic to the RL agent's input, improving its applicability across various RL environments and scenarios.
• 21.
  22 uOttawa.ca uOttawa.ca | nanda-lab.ca Schoolof Electrical Engineering & Computer Science | Nanda Lab SMARLA: A Safety Monitoring Approach for Reinforcement Learning Agents A. Zolfagharian, M. Abdellatif, L. C. Briand, and Ramesh. S, IEEE Transactions on Software Engineering, vol. , no. 01, pp. 1- 25, PrePrints 5555, doi: 10.1109/TSE.2024.3491496
• 22.
  23 Confidence intervals ofsafety monitor • Random forest is an ensemble method that combines the predictions of multiple decision trees, each trained on a random subset of the training data. • In order to compute confidence intervals, we use the probability of safety violation generated by decision trees as the population X • Then, based on the mean prediction in each time step and standard deviation, confidence intervals is calculated with the following formula. C𝐼 = ത 𝑋 ± 𝑍 × 𝜎 𝑛 • Finally, the confidence interval is: [ ത 𝑋 − 𝑍 × 𝜎 𝑛 , ത 𝑋 + 𝑍 × 𝜎 √𝑛 ] • For confidence of 95% the Z is 1.96 • Based on the lower/upper bound of confidence intervals we define different triggering criteria to activate safety mechanism
• 23.
  24 Case studies Cart-Pole Mountain-Car Border HighwayDriving
• 24.
  25 Case study: HighwayDriving • To provide a more comprehensive evaluation of our safety monitoring approach we have added an additional case study called Highway driving. • The Highway driving environment is used for simulating realistic driving scenarios to train and test autonomous driving planning algorithms. • In this environment the vehicle is driving on a straight highway with several lanes, and is rewarded for reaching a high speed, staying on the rightmost lanes and avoiding collisions. • This new environment presents a significant challenge due to its partial observability and the dynamic nature of vehicles appearing in different lanes.
• 25.
  26 Case study: HighwayDriving Reward • Collision penalty: a penalty is received when colliding with a vehicle. • Right lane reward: a reward is received when driving on the right-most lanes. • High speed reward : a reward is received when driving at high speed (above 20 MPH) Termination – Episode duration: configurable (50 timesteps in our case) – Collision Safety violation • In this environment we define safety violation as having a collision • We have a continuous state space of, vehicles kinematic information (15 parameters in total) • We have a discrete action space of : 1. Move to left lane 2. Move to right lane 3. Accelerate 4. Brake 5. Idle
• 26.
  27 Safety monitor training– Best results • We train our safety monitor to predict collisions. • For the training data, we generated 4,000 episodes. • The training data includes: • A total of 115,730 concrete states • A failure rate of 5% • An average episode length of 28.9 time steps • A maximum episode length of 30 time steps • 222 safety violations in total (in the training dataset) • Labels: 1 for collision and 0 for no collision • based on the collisions observed in the simulator
• 27.
  28 GM Case study •We also applied SMARLA to a large-scale industrial case study without resource constraints, allowing us to explore the use of LSTM models. • In this case study an RL agent is trained for lane merging task in different scenarios such as: – T-junction lane merge – Highway lane merge – Roundabout or clover-leaf lane merge • The RL agent plans a trajectory and controls the vehicle to safely merge into traffic without causing an accident [4]. [4] Planning for Negotiations in Autonomous Driving using Reinforcement Learning
• 28.
  29 GM Case study State: •States are lists containing: – kinematic information of actors within a certain range of the host vehicle for example velocity, position and acceleration – kinematic information of the host vehicle Reward: Reward is defined in a way that optimizes the efficiency and comfort: • Efficiency – Reaching to the goal state as soon as possible • Comfort – Avoiding sudden and unnecessary movements (i.e., jerk-optimization) Termination: • Episodes may terminate with one the three following conditions: (1) successful, (2) handover, (3) collision Safety violation: • Having a collision
• 29.
  30 Feature Representations Order ofabstract states - Temporal abstraction • Temporal features consider the order in which abstract states occur, allowing for the modeling of sequences and patterns. This is useful in tasks like time-series analysis, natural language processing (e.g., for understanding sentence structure) • Temporal features can result in a significantly higher-dimensional feature space, thereby holding a greater amount of information. This can lead to increased computational complexity. Features: Order of the abstract states Episodes in intervals State 1 State 2 State 3 …. State n Probability of failure [(S1,a1)] Abstract State ID: 1 Abstract State ID: 0 Abstract State ID: 0 Abstract State ID: 0 10% [(S1,a1), (S2,a2)] Abstract State ID: 1 Abstract State ID: 7 Abstract State ID: 0 Abstract State ID: 0 20% [(S1,a1), (S2,a2),(S3,a3)] Abstract State ID: 1 Abstract State ID: 7 Abstract State ID: 5 Abstract State ID: 0 15% * n = Maximum length of episodes
• 30.
  31 Model • Recurrent NeuralNetworks (RNNs) are a class of neural networks that are designed to work with sequences of data. They have loops within them, allowing information to persist. • Long Short-Term Memory (LSTM) are popular variants of RNNs that address the vanishing gradient problem and are capable of capturing long-range dependencies in data. • We considered different types of LSTM models for example, simple classification LSTM, Deep LSTM, Bidirectional LSTM and hybrid CNN/LSTM
• 31.
  32 Other DNN Architectures CNN/LSTMHybrid Deep LSTM Bidirectional LSTM
• 32.
  33 Summary • We alsoimplemented SMARLA on a GM case study and achieved highly accurate predictions of safety violations. • We extended our method by implementing different feature representations and considering LSTM models in the GM case study. • We achieved a maximum F1 score of 98% using LSTM and temporal features. • Although LSTM models provided higher accuracy, SMARLA with Random Forest without temporal features allowed for earlier prediction time steps.