The document outlines reinforcement learning concepts and implementation strategies, focusing on deep Q-networks and epsilon-greedy policies. It presents various challenges in the OpenAI Gym, including environments like 'mountaincar-v0' and 'cartpole-v0', detailing their observation and action spaces, reward structures, and objectives. Additionally, it discusses the importance of exploration vs. exploitation in learning, exemplifying it through learning algorithms and state-action value updates.

1. Model: Deep Q Network
Approximator: Convolutional
Neural Network
Policy: Epsilon Greedy
Player 2Player 1
Model: Original
Deterministic Pong
Approximator: Hard-
Coded
Policy: Greedy

2. Intro to
Reinforcement
Learning

3. Reinforcement Learning VS Other Types of ML

4. Classification / Regression
Reinforcement Learning VS Other Types of ML

5. Classification / Regression Clustering / DR
Reinforcement Learning VS Other Types of ML

6. Classification / Regression Clustering / DR Optimal Decision Sequences
Reinforcement Learning VS Other Types of ML

7. Trial and Error
Optimal Decision Sequences
Specific Traits of RL

8. Trial and Error
Maps States to Actions
Optimal Decision Sequences
Specific Traits of RL

9. Trial and Error
Optimizes policies
Maps States to Actions
Optimal Decision Sequences
Specific Traits of RL

10. Trial and Error
Maximizes Rewards
Optimizes policies
Maps States to Actions
Optimal Decision Sequences
Specific Traits of RL

11. Markov Decision
Process
P =0.3
P = 1.0
P =0.7
P = 1.0 P = 0.3
P = 0.4
P = 0.3
P = 1.0
Trial and Error
Maximizes Rewards
Optimizes policies
Maps States to Actions
Specific Traits of RL

12. Provides States
Provides Feedback
How RL Works?
?
Learns to Maximize Rewards

13. Exploitation VS Exploration
Best Known
Strategy
“Greedy”
Highest Known
Value
Can’t Learn
New Skills
Random
Decisions
Not “Greedy”
Most
Information
Gained
Can’t Use
Learned Skill

14. Exploitation VS Exploration
Primary
Technique
Exploration policy:
• How Random,
How Long
Examples:
• E-Greedy,

15. Exploitation VS Exploration
Primary
Measures
Regret:
• Reward vs Max-
Reward
Time/Cost:
• Training Costs
Primary
Technique
Exploration policy:
• How Random,
How Long
Examples:
• E-Greedy,

16. Why Start with Gym?

17. ?
#1 It Takes Care of Most Necessities

18. Writing Simulations and Games
Defining Reward Systems
We Can Focus on the Learner
?
Left = Q(s,a1)
Right = Q(s,a2)
Jump = Q(s,a3)
Creating Learning Agents

19. #2
Gym
Environments
Classical Control
Robotics
MuJoCo
Box2D
Atari
Toy Text

20. RL in Python
GPU Access
Easy Start with Keras-RL
#3
Python Ecosystem

21. Getting started

22. A Simple Example

23. A Simple Example
gym.make()
• Environment

24. A Simple Example
gym.make()
• Environment
env.reset()
• Initial State • Resets Episodes

25. A Simple Example
gym.make()
• Environment
env.render()
• Video Display or Text Map
env.reset()
• Initial State • Resets Episodes

26. A Simple Example
gym.make()
• Environment
env.render()
• Video Display or Text Map
env.action_space.sample()
• Sample of Action
env.reset()
• Initial State • Resets Episodes

27. A Simple Example
gym.make()
• Environment
env.render()
• Video Display or Text Map
env.action_space.sample()
• Sample of Actions
env.step()
• State
• Reward
• Done
• Info
env.reset()
• Initial State • Resets Episodes

28. Challenge 1

29. MountainC
ar-v0
Premise:
Under-Powered Car
Must Go Up a Steep
Hill.

30. MountainC
ar-v0
• Position • Velocity
Observation Space: Box(2)
Position
Velocity

31. MountainC
ar-v0
• Position • Velocity
Observation Space: Box(2)
• Push Left
• Push Right
• No Push
Action Space: Discrete(3)
Left Right
None

32. MountainC
ar-v0
• Position • Velocity
Observation Space: Box(2)
• Push Left
• Push Right
• No Push
• -1 Per Step
Action Space: Discrete(3)
Rewards

33. MountainC
ar-v0
• Position • Velocity
Observation Space: Box(2)
• Push Left
• Push Right
• No Push
• -1 Per Step
• 200 Steps • Reach Goal
Game Ends
Action Space: Discrete(3)
Rewards

34. MountainC
ar-v0
• Position • Velocity
Observation Space: Box(2)
• Push Left
• Push Right
• No Push
• -1 Per Step
• 200 Steps • Reach Goal
Game Ends
Goal: Get up the Hill with Minimum
Regret
Action Space: Discrete(3)
Rewards

35. Import Packages
Make Environment

36. Set Parameters
Set Epsilon Decay Schedule

37. Discretization and Q-Table

38. Q-Table
Action 1 … Action n
State 1 Value
(s1, a1)
… Value
(s1, an)
State 2 … … …
… … … …
… … … …
State n Value
(sn, a1)
… Value
(sn, an)
Q-Value
Current State
Action

39. Decision and Action

40. Decision and Action

41. Decision and Action

42. Update Q-Table

43. Update Q-Table

44. Update Q-Table

45. Episode 0 Episode 400 Episode 600
Episode 1200 Episode 1800

46. Results
Amended From
Original Version

47. Results
Exploration
Exploitation
Amended From
Original Version

48. Challenge 2

49. CartPole-
v0
Premise:
Overcome Complex
Dynamics to Hold the
Pole Vertical.

50. CartPole-
v0
• Cart Position
• Cart Velocity
• Pole Angle
• Pole Velocity
Observation Space: Box(4)

51. CartPole-
v0
• Cart Position
• Cart Velocity
• Pole Angle
• Pole Velocity
Observation Space: Box(4)
Action Space: Discrete(2)
• Push Left
• Push Right
Left Right

52. CartPole-
v0
• Cart Position
• Cart Velocity
• Pole Angle
• Pole Velocity
Observation Space: Box(4)
Action Space: Discrete(2)
• Push Left
• Push Right
Rewards
• 1 Point Per Step

53. CartPole-
v0
• Cart Position
• Cart Velocity
• Pole Angle
• Pole Velocity
Observation Space: Box(4)
Action Space: Discrete(2)
• Push Left
• Push Right
Rewards
• 1 Point Per Step
• 200 Steps
• Pole Angle ±12°
• Cart Position ±2.4
Game Ends
Game
Over
±12°
Cart Position ±2.4
Game
Over

54. CartPole-
v0
• Cart Position
• Cart Velocity
• Pole Angle
• Pole Velocity
Observation Space: Box(4)
Action Space: Discrete(2)
• Push Left
• Push Right
Rewards
• 1 Point Per Step
• 200 Steps
• Pole Angle ±12°
• Cart Position ±2.4
Game Ends
Solved When: Avg Reward >= 195
(over 100 consecutive episodes)

55. Import Packages
Create CartPole Environment

56. Function Approximator
Input Layer
Hidden Layer
Output Layer
States
Value
Action 1
Value
Action 2

57. Function Approximator
Input Layer
Hidden Layer
Output Layer
States
Value
Action 1
Value
Action 2

58. Function Approximator
Input Layer
Hidden Layer
Output Layer
States
Value
Action 1
Value
Action 2

59. Function Approximator
Input Layer
Hidden Layer
Output Layer
States
Value
Action 1
Value
Action 2

60. Policy, Agent, and Fitting

61. Policy, Agent, and Fitting

62. Policy, Agent, and Fitting
(On-Policy)
(Off-Policy)

63. Policy, Agent, and Fitting

64. CartPole-v0 Test
Learner
Agent: SARSA
Function Approximator:
Deep Neural Net
Policy: Epsilon-Greedy
Test
Episodes: 100
Average: 200
Solved: Yes (avg reward >=
195)

65. Questions