Recommended
More Related Content
What's hot
What's hot (20)
Similar to Getting started with reinforcement learning in open ai gym
Similar to Getting started with reinforcement learning in open ai gym (20)
Recently uploaded
Recently uploaded (20)
Getting started with reinforcement learning in open ai gym
- 1. Model: Deep Q Network Approximator: Convolutional Neural Network Policy: Epsilon Greedy Player 2Player 1 Model: Original Deterministic Pong Approximator: Hard- Coded Policy: Greedy
- 3. Reinforcement Learning VS Other Types of ML
- 4. Classification / Regression Reinforcement Learning VS Other Types of ML
- 5. Classification / Regression Clustering / DR Reinforcement Learning VS Other Types of ML
- 6. Classification / Regression Clustering / DR Optimal Decision Sequences Reinforcement Learning VS Other Types of ML
- 7. Trial and Error Optimal Decision Sequences Specific Traits of RL
- 8. Trial and Error Maps States to Actions Optimal Decision Sequences Specific Traits of RL
- 9. Trial and Error Optimizes policies Maps States to Actions Optimal Decision Sequences Specific Traits of RL
- 10. Trial and Error Maximizes Rewards Optimizes policies Maps States to Actions Optimal Decision Sequences Specific Traits of RL
- 11. Markov Decision Process P =0.3 P = 1.0 P =0.7 P = 1.0 P = 0.3 P = 0.4 P = 0.3 P = 1.0 Trial and Error Maximizes Rewards Optimizes policies Maps States to Actions Specific Traits of RL
- 12. Provides States Provides Feedback How RL Works? ? Learns to Maximize Rewards
- 13. Exploitation VS Exploration Best Known Strategy “Greedy” Highest Known Value Can’t Learn New Skills Random Decisions Not “Greedy” Most Information Gained Can’t Use Learned Skill
- 14. Exploitation VS Exploration Primary Technique Exploration policy: • How Random, How Long Examples: • E-Greedy,
- 15. Exploitation VS Exploration Primary Measures Regret: • Reward vs Max- Reward Time/Cost: • Training Costs Primary Technique Exploration policy: • How Random, How Long Examples: • E-Greedy,
- 16. Why Start with Gym?
- 17. ? #1 It Takes Care of Most Necessities
- 18. Writing Simulations and Games Defining Reward Systems We Can Focus on the Learner ? Left = Q(s,a1) Right = Q(s,a2) Jump = Q(s,a3) Creating Learning Agents
- 20. RL in Python GPU Access Easy Start with Keras-RL #3 Python Ecosystem
- 21. Getting started
- 22. A Simple Example
- 23. A Simple Example gym.make() • Environment
- 24. A Simple Example gym.make() • Environment env.reset() • Initial State • Resets Episodes
- 25. A Simple Example gym.make() • Environment env.render() • Video Display or Text Map env.reset() • Initial State • Resets Episodes
- 26. A Simple Example gym.make() • Environment env.render() • Video Display or Text Map env.action_space.sample() • Sample of Action env.reset() • Initial State • Resets Episodes
- 27. A Simple Example gym.make() • Environment env.render() • Video Display or Text Map env.action_space.sample() • Sample of Actions env.step() • State • Reward • Done • Info env.reset() • Initial State • Resets Episodes
- 28. Challenge 1
- 29. MountainC ar-v0 Premise: Under-Powered Car Must Go Up a Steep Hill.
- 30. MountainC ar-v0 • Position • Velocity Observation Space: Box(2) Position Velocity
- 31. MountainC ar-v0 • Position • Velocity Observation Space: Box(2) • Push Left • Push Right • No Push Action Space: Discrete(3) Left Right None
- 32. MountainC ar-v0 • Position • Velocity Observation Space: Box(2) • Push Left • Push Right • No Push • -1 Per Step Action Space: Discrete(3) Rewards
- 33. MountainC ar-v0 • Position • Velocity Observation Space: Box(2) • Push Left • Push Right • No Push • -1 Per Step • 200 Steps • Reach Goal Game Ends Action Space: Discrete(3) Rewards
- 34. MountainC ar-v0 • Position • Velocity Observation Space: Box(2) • Push Left • Push Right • No Push • -1 Per Step • 200 Steps • Reach Goal Game Ends Goal: Get up the Hill with Minimum Regret Action Space: Discrete(3) Rewards
- 36. Set Parameters Set Epsilon Decay Schedule
- 38. Q-Table Action 1 … Action n State 1 Value (s1, a1) … Value (s1, an) State 2 … … … … … … … … … … … State n Value (sn, a1) … Value (sn, an) Q-Value Current State Action
- 42. Update Q-Table
- 43. Update Q-Table
- 44. Update Q-Table
- 45. Episode 0 Episode 400 Episode 600 Episode 1200 Episode 1800
- 48. Challenge 2
- 49. CartPole- v0 Premise: Overcome Complex Dynamics to Hold the Pole Vertical.
- 50. CartPole- v0 • Cart Position • Cart Velocity • Pole Angle • Pole Velocity Observation Space: Box(4)
- 51. CartPole- v0 • Cart Position • Cart Velocity • Pole Angle • Pole Velocity Observation Space: Box(4) Action Space: Discrete(2) • Push Left • Push Right Left Right
- 52. CartPole- v0 • Cart Position • Cart Velocity • Pole Angle • Pole Velocity Observation Space: Box(4) Action Space: Discrete(2) • Push Left • Push Right Rewards • 1 Point Per Step
- 53. CartPole- v0 • Cart Position • Cart Velocity • Pole Angle • Pole Velocity Observation Space: Box(4) Action Space: Discrete(2) • Push Left • Push Right Rewards • 1 Point Per Step • 200 Steps • Pole Angle ±12° • Cart Position ±2.4 Game Ends Game Over ±12° Cart Position ±2.4 Game Over
- 54. CartPole- v0 • Cart Position • Cart Velocity • Pole Angle • Pole Velocity Observation Space: Box(4) Action Space: Discrete(2) • Push Left • Push Right Rewards • 1 Point Per Step • 200 Steps • Pole Angle ±12° • Cart Position ±2.4 Game Ends Solved When: Avg Reward >= 195 (over 100 consecutive episodes)
- 55. Import Packages Create CartPole Environment
- 56. Function Approximator Input Layer Hidden Layer Output Layer States Value Action 1 Value Action 2
- 57. Function Approximator Input Layer Hidden Layer Output Layer States Value Action 1 Value Action 2
- 58. Function Approximator Input Layer Hidden Layer Output Layer States Value Action 1 Value Action 2
- 59. Function Approximator Input Layer Hidden Layer Output Layer States Value Action 1 Value Action 2
- 60. Policy, Agent, and Fitting
- 61. Policy, Agent, and Fitting
- 62. Policy, Agent, and Fitting (On-Policy) (Off-Policy)
- 63. Policy, Agent, and Fitting
- 64. CartPole-v0 Test Learner Agent: SARSA Function Approximator: Deep Neural Net Policy: Epsilon-Greedy Test Episodes: 100 Average: 200 Solved: Yes (avg reward >= 195)
- 65. Questions