The document discusses on-policy control using approximate action values in reinforcement learning, focusing on episodic 1-step and n-step semi-gradient Sarsa methods. It details the application of these methods to the mountain car problem and an access-control queuing example, highlighting the benefits of using n-step updates for faster learning and better performance. Various techniques such as tile coding and differential return are introduced to enhance policy quality and learning efficiency.

1. Chapter 10: On-policy Control with
Approximation
Seungjae Ryan Lee

2. Episodic 1-step semi-gradient Sarsa
● Approximate action values (instead of state values)
● Use Sarsa to define target
● Converges the same ways as TD(0) with same error bound

3. Control with Episodic 1-step semi-gradient Sarsa
● Select action and improve policy using an ε-greedy action w.r.t.

4. Mountain Car Example
● Task: Drive an underpowered car up a steep mountain road
○ Gravity is stronger than car’s engine
○ Must swing back and forth to build enough inertia
● State: position , velocity
● Actions: Forward (+1), Reverse (-1), No-op (0)
● Reward: -1 until the goal is reached

5. Approximation for Mountain Car
● Tile coding used to select binary features (8 tiles)

6. Results of Mountain Car
● Plot the cost-to-go function:
● Initial action values set to 0
○ Very optimistic

7. Results of Mountain Car

8. Episodic n-step Semi-gradient Sarsa
● Use n-step return as the update target

9. Episodic n-step Semi-gradient Sarsa in Practice

10. Episodic n-step Semi-gradient Sarsa Results
● Faster learning
● Better asymptotic performance

11. Episodic n-step Semi-gradient Sarsa Results
● Best performance for intermediate values of n-step

12. Average Reward Setting
● Quality of policy defined by the average reward following policy
● Continuing tasks without discounting

13. Differential Return and Value Functions
Differential Return: differences between rewards and average reward
Differential Value Functions: Expected differential returns

14. Bellman Equations
● Remove all
● Replace rewards with difference of rewards

15. Differential semi-gradient Sarsa
● Same update rule with differential TD error
● Original TD error:
● Differential TD error:

16. Differential semi-gradient Sarsa

17. Access-Control Queuing Example
● Agent can grant access to 10 servers
○ Agent can accept or reject customers
● Customers arrive at a single queue
○ Customers have 4 different priorities, randomly distributed
○ Pay a reward of 1, 2, 4, or 8 when granted access to a server
● A busy server is freed with some probability

18. Access-Control Queuing Results
● Tabular solution with differential semi-gradient Sarsa

19. n-step Semi-gradient Sarsa
● Use n-step return
○

20. Thank you!
Original content from
● Reinforcement Learning: An Introduction by Sutton and Barto
You can find more content in
● github.com/seungjaeryanlee
● www.endtoend.ai