Download as PDF, PPTX
![Defining the Problem
Learning
Improvements
Importance of Making Mistakes
Eligibility Trace
What do we do when...
The environment is not deterministic
The environment is not fully observable
There are way too many states
The states are not discrete
The agent is acting in continuous time
¨Orjan Ekeberg Machine Learning
Defining the Problem
Learning
Improvements
Importance of Making Mistakes
Eligibility Trace
The Exploration–Exploitation dilemma
If an agent strictly follows a greedy policy based on the current
estimate of Q, learning is not guaranteed to converge to Q
Simple solution:
Use a policy which has a certain probability of ”making mistakes”
-greedy
Sometimes (with probability ) make a random action instead
of the one that seems best (greedy)
Softmax
Assign a probability to choose each action depending on how
good they seem
¨Orjan Ekeberg Machine Learning
Defining the Problem
Learning
Improvements
Importance of Making Mistakes
Eligibility Trace
Accelerated learning
Id´ea: TD updates can be used to improve not only the last state’s
value, but also states we have visited earlier.
∀s, a : Q(s, a) ← Q(s, a) + η [rt+1 + γQ(st+1, at+1) − Q(st, at)] · e
e is a remaining trace (eligibility trace) encoding how long ago we
were in s doing a.
Often denoted TD(λ) where λ is the time constant of the trace e
¨Orjan Ekeberg Machine Learning](https://image.slidesharecdn.com/09-rl-2x2-150823145810-lva1-app6891/75/Reinforcement-Learning-7-2048.jpg)
More Related Content
Similar to Reinforcement Learning
![[DSC Europe 25] Jim Sterne - Adopting Generative AI Capabilities Into the Ent...](https://cdn.slidesharecdn.com/ss_thumbnails/sxhpofuorcagxsaulkmt-3-251204082258-7e66bc48-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...](https://cdn.slidesharecdn.com/ss_thumbnails/8brigo2sbu6qur6gxrra-7-251205085715-6ae07d24-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...](https://cdn.slidesharecdn.com/ss_thumbnails/7nw2xxixrxqdxvrb5wca-6-251205085714-ab09a2ac-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dusan Jovicic - AI Story: From on-prem to cloud and back agai...](https://cdn.slidesharecdn.com/ss_thumbnails/8kp49m6uq22ifnbwhfnk-2-251205085715-964d11a6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Max Talanov - Non digital NNs.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/wif8tr3gtua74qvtopke-non-digital-nns-251205090438-26b0eea6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...](https://cdn.slidesharecdn.com/ss_thumbnails/o2gptrmtoyqndgoshwgq-dsc2025-tenstorrent-rajovic-251205090438-814685f5-thumbnail.jpg?width=640&height=640&fit=bounds)
Reinforcement Learning
- 1. Defining the Problem Learning Improvements ReinforcementLearning ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements 1 Defining the Problem Reward Maximization Simplifying Assumptions Bellman’s Equation 2 Learning Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning 3 Improvements Importance of Making Mistakes Eligibility Trace ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation 1 Defining the Problem Reward Maximization Simplifying Assumptions Bellman’s Equation 2 Learning Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning 3 Improvements Importance of Making Mistakes Eligibility Trace ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation Reinforcement Learning Learning of a behavior without explicit information about correct actions A reward gives information about success ¨Orjan Ekeberg Machine Learning
- 2. Defining the Problem Learning Improvements RewardMaximization Simplifying Assumptions Bellman’s Equation Agent–Environment Abstraction: Agent Environment ActionState Reward Policy — Choice of action, depending on current state Learning Objective: Develop a policy which maximizes the reward ... total reward over the agents lifetime! ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation Credit Assignment Problems The reward does not necessarily arrive when you do something good Temporal credit assignment The reward does not say what was good Structural credit assignment ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation Consider a minimal behavior selection situation: What is the best action to make? −1 2 −1 2 −1 2 −1 1 0 0 3 −1 Reward — Immediate consequence of our decision Planning — Taking future reward into account Optimal Behavior — Maximize total reward ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation Can the agent make the right decision without explicitly modeling the future? Yes! If the Sum of Future Rewards for each situation is known. −1 2 −1 1 0 0 3 −1 4 −2 2 −2 −1 2 8 −4 Call this the State Value State Values are subjective Depend on the agents own behavior Must be re-adjusted as the agents behavior improves ¨Orjan Ekeberg Machine Learning
- 3. Defining the Problem Learning Improvements RewardMaximization Simplifying Assumptions Bellman’s Equation Planning Horizon — How long is the future? −1 0 1 2 Reward Time −1 0 1 2 Reward Time Infinite future allows for infinite postponing Discounting — Make early reward more valuable Discount factor (γ) — Time scale of planning ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation Finite Horizon max h t=0 rt Infinite Horizon max ∞ t=0 γt rt Requires discount of future reward (0 < γ < 1) ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation The Reward function controls which task should be solved Game (Chess, Backgammon) Reward only at the end: +1 when winning, −1 when loosing Avoiding mistakes (cycling, balancing, ...) Reward −1 at the end (when failing) Find a short/fast/cheap path to a goal Reward −1 at each step ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation Simplifying Assumptions Discrete time Finite number of actions ai ai ∈ a1, a2, a3, . . . , an Finite number of states si si ∈ s1, s2, s3, . . . , sm Environment is a stationary Markov Decision Process Reward and next state depends only on s, a and chance ¨Orjan Ekeberg Machine Learning
- 4. Defining the Problem Learning Improvements RewardMaximization Simplifying Assumptions Bellman’s Equation Classical model problem: Grid World Each state is represented by a position in a grid The agent acts by moving to other positions G G Trivial labyrinth Reward: −1 at each step until a goal state (G) is reached ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation The values of a state depends on the current policy. 0 0 −1 −2 −3 −1 −2 −3 −2 −2 −3 −2 −1 −3 −2 −1 V with an optimal policy −14 −14 −14 −140 −20 −22 −18 −22 −20 0 −18−22 −22 −20 −20 V with a random policy ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation The Agents Internal Representation Policy The action chosen by the agent for each state π(s) → a Value Function Expected total future reward from s when following policy π V π (s) → ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Reward Maximization Simplifying Assumptions Bellman’s Equation Representation of the Environment Where does an action take us? δ(s, a) → s How much reward do we receive? r(s, a) → The values of different states are interrelated Bellman’s Equation: V π (s) = r(s, π(s)) + γ · V π (δ(s, π(s))) ¨Orjan Ekeberg Machine Learning
- 5. Defining the Problem Learning Improvements Monte-CarloMethod Temporal-Difference Learning to Act Q-Learning 1 Defining the Problem Reward Maximization Simplifying Assumptions Bellman’s Equation 2 Learning Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning 3 Improvements Importance of Making Mistakes Eligibility Trace ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning Normal scenario: r(s, a) and δ(s, a) are not known V π must be estimated by experience Monte-Carlo Method Start at a random s Follow π, store the rewards and st When the goal is reached, update V π(s)-estimation for all visited stated with the future reward we actually received Painfully slow! ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning Temporal Difference Learning −1 2 8 −4 −1 2 8 −4 6 Temporal Difference — the difference between: Experienced value (immediate reward + value of next state) Expected value Measure of unexpected success Two sources: Higher immediate reward than expected Reached better situation than expected ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning Expected Value V π (st) One-step Experienced Value rt+1 + γ · V π (st+1) TD-signal — measure of surprise / disappointment TD = rt+1 + γ · V π (st+1) − V π (st) TD Learning V π (st) ← V π (st) + ηTD ¨Orjan Ekeberg Machine Learning
- 6. Defining the Problem Learning Improvements Monte-CarloMethod Temporal-Difference Learning to Act Q-Learning Learning to Act How do we get a policy from the TD signal? Many possibilities... Actor–Critic Model (Barto, Sutton, Anderson IEEE Trans. Syst. Man & Cybern. 1983) Q-Learning (Watkins, 1989; Watkins & Dayan Machine Learning 1992) ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning Actor–Critic Model TDState Reward Action Actor Critic Actor — Associates states with actions π(·) Critic — Associates states with their value V (·) TD-signal is used as a reinforcement signal to update both! High TD (unexpected success) Increase value of preceeding state Increase tendency to make same action again Low TD (unexpected failure) Decrease value of preceeding state Decrease tendency to make same action again ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning Q-Learning Estimate Q(s, a) instead of V (s) Q(s, a): Expected total reward when doing a from s. π(s) = argmax a Q(s, a) V (s) = max a Q (s, a) The Q-function can also be learned using Temporal-Difference Q(s, a) ← Q(s, a) + η r + γ max a Q(s , a ) − Q(s, a) s is the next state. ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Importance of Making Mistakes Eligibility Trace 1 Defining the Problem Reward Maximization Simplifying Assumptions Bellman’s Equation 2 Learning Monte-Carlo Method Temporal-Difference Learning to Act Q-Learning 3 Improvements Importance of Making Mistakes Eligibility Trace ¨Orjan Ekeberg Machine Learning
- 7. Defining the Problem Learning Improvements Importanceof Making Mistakes Eligibility Trace What do we do when... The environment is not deterministic The environment is not fully observable There are way too many states The states are not discrete The agent is acting in continuous time ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Importance of Making Mistakes Eligibility Trace The Exploration–Exploitation dilemma If an agent strictly follows a greedy policy based on the current estimate of Q, learning is not guaranteed to converge to Q Simple solution: Use a policy which has a certain probability of ”making mistakes” -greedy Sometimes (with probability ) make a random action instead of the one that seems best (greedy) Softmax Assign a probability to choose each action depending on how good they seem ¨Orjan Ekeberg Machine Learning Defining the Problem Learning Improvements Importance of Making Mistakes Eligibility Trace Accelerated learning Id´ea: TD updates can be used to improve not only the last state’s value, but also states we have visited earlier. ∀s, a : Q(s, a) ← Q(s, a) + η [rt+1 + γQ(st+1, at+1) − Q(st, at)] · e e is a remaining trace (eligibility trace) encoding how long ago we were in s doing a. Often denoted TD(λ) where λ is the time constant of the trace e ¨Orjan Ekeberg Machine Learning