The document provides a comprehensive overview of reinforcement learning, covering key concepts such as exploration versus exploitation, Markov decision processes, and various learning algorithms including Q-learning and policy gradient methods. It includes lecture slides and tutorials from prominent courses, highlighting techniques like deep reinforcement learning and asynchronous methods. The document serves as a foundational resource for understanding both traditional and advanced topics within reinforcement learning.

1. Reinforcement Learning

2. Overview

3. Introduction to Reinforcement
Learning
Chapter 1 – Reinforcement Learning: An Introduction
Imitation Learning Lecture Slides from CMU Deep
Reinforcement Learning Course

4. What is Reinforcement Learning?

5. Exploration versus Exploitation

6. Reinforcement Learning Systems

7. Policy

8. Reward Signal

9. Value Function (1)

10. Value Function (2)

11. Model-free versus Model-based

12. On-policy versus Off-policy

13. Credit Assignment Problem

14. Reward Design

15. What is Deep Reinforcement Learning?

16. Finite Markov Decision Processes
Chapter 3 – Reinforcement Learning: An Introduction

17. Markov Decision Process (MDP)

18. Time Discounting

19. Agent-Environment Interaction (1)

20. Agent-Environment Interaction (2)

21. Action Selection

22. MDP Dynamics

23. State Transition Probabilities

24. Expected Rewards

25. State-Value Function (1)

26. State-Value Function (2)

27. Action-Value Function

28. Bellman Equation (1)

29. Bellman Equation (2)

30. Optimality

31. Temporal-Difference Learning
Chapter 6 – Reinforcement Learning: An Introduction
Playing Atari with Deep Reinforcement Learning
Asynchronous Methods for Deep Reinforcement Learning
David Silver’s Tutorial on Deep Reinforcement Learning

32. What is TD learning?

33. Value-based Reinforcement Learning

34. Update Rule for TD(0)

35. Update Rule Intuition

36. Tabular TD(0) Algorithm

37. SARSA – On-policy TD Control

38. SARSA Update Rule

39. SARSA Algorithm

40. Q-learning – Off-policy TD Control

41. One-step Q-learning Algorithm

42. Epsilon-greedy Policy

43. Deep Q-Networks (DQN)

44. Q-Networks

45. Experience Replay

46. State representation

47. Q-Network Training

48. Loss Function Gradient Derivation

49. DQN Algorithm

50. Comments

51. Policy Gradient Methods
Chapter 13 – Reinforcement Learning: An Introduction
Policy Gradient Lecture Slides from David Silver’s
Reinforcement Learning Course
David Silver’s Tutorial on Deep Reinforcement Learning

52. What are Policy Gradient Methods?

53. Policy-based Reinforcement Learning

54. Notation

55. Policy Approximation

56. Types of Policy Gradient Method

57. Finite Difference Policy Gradient

58. REINFORCE: Monte Carlo Policy Gradient

59. REINFORCE Properties

60. REINFORCE Algorithm

61. Actor-Critic Methods

62. One-step Actor-Critic Update Rules

63. One-step Actor-Critic Algorithm

64. Asynchronous Reinforcement
Learning
Asynchronous Methods for Deep Reinforcement Learning

65. What is Asynchronous Reinforcement Learning?

66. Parallelism (1)

67. Parallelism (2)

68. No Experience Replay

69. Asynchronous Algorithms

70. Asynchronous one-step Q-learning

71. Exploration

72. Asynchronous one-step Q-learning Algorithm

73. Asynchronous one-step SARSA

74. n-step Q-learning

75. n-step Returns

76. Asynchronous n-step Q-learning Algorithm

77. A3C

78. Advantage Definition

79. A3C Algorithm

80. Summary