The document discusses reinforcement learning concepts including Markov chains and Monte Carlo methods. It defines reinforcement learning as an area where an agent learns from interactions with an environment by receiving rewards or punishments for actions. Markov chains and Monte Carlo methods are probabilistic models used in reinforcement learning to model state transitions and estimate values based on sampled returns or rewards. The document provides examples to illustrate reinforcement learning, Markov chains, and how Monte Carlo methods estimate values from episodes without knowing the environment's dynamics.

1. Reinforcement Learning:
Markov Chain and Monte Carlo
MSC-IT Part-1
By - Ajay Chaurasiya
1

2. What is Reinforcement Learning?
 “Teach by experience”
 For each step, an agent will:
Execute an action
Observe a new state
Receive a reward
 Agent takes an action in an environment to maximize a reward
MSC-IT Part-1 By - Ajay Chaurasiya
2

3. Main points in Reinforcement learning
 Input
 Output
 Training
 The model keeps continues to learn.
 The best solution is decided based on the maximum reward.
MSC-IT Part-1 By - Ajay Chaurasiya
3

4.  Example: The problem is as follows: We have an agent and a reward, with
many hurdles in between. The agent is supposed to find the best possible
path to reach the reward.
The above image shows robot, diamond and fire. The goal of the robot is to get the
reward that is the diamond and avoid the hurdles that is fire.
The robot learns by trying all the possible paths and then choosing the path which gives
him the reward with the least hurdles. Each right step will give the
robot a reward and each wrong step will subtract the reward of the robot. The total
reward will be calculated when it reaches the final reward that is the diamond.
MSC-IT Part-1 By - Ajay Chaurasiya
4

5. Markov Chain Learning
 A Markov chain is a probabilistic model.
 It describing a sequence of possible events in which the probability of each
event depends only on the state attained in the previous event. In
continuous-time.
 it is also known as Markov process.
 The Markov property states that the future depends only on the present and
not on the past.
 Moving from one state to another is called transition and its probability is
called a transition probability.
MSC-IT Part-1 By - Ajay Chaurasiya
5

6.  Example: A robot car wants to travel far, quickly
1. Three states: Cool, Warm, Overheated
2. Two actions: Slow. Fast
3. Going faster gets double reward
Note: In Markov Decision, the probability of going
from one state to another state will always be one.
MSC-IT Part-1 By - Ajay Chaurasiya
6

7. MDP can be represented by 5 important elements.
 State(S)
 Actions(A)
 Transition Probability(Pᵃₛ₁ₛ₂)
 Reward Probability(Rᵃₛ₁ₛ₂)
 discount factor (γ)
MSC-IT Part-1 By - Ajay Chaurasiya
7

8. Continue..
 It is based on the action our agent performs, it receives a reward. A reward is nothing
but a numerical value, say, +1 for good action and -1 for a bad action.
 The total amount of rewards the agent receives from the environment is called
returns. We can formulate the total amount of reward as
R(t) = r(t+1)+r(t+2)+r(t+3)+r(t+4)……………+r(Τ)
 Since we don’t have any final state for a continuous task, we can define our
return for continuous tasks as R(t) = r(t+1)+r(t+2)+r(t+3)…+r(Τ) which will sum up to ∞
 That’s why we introduce the notion of a discount factor. We can redefine our return
with a discount factor, as follows R(t) = r(t+1)+γr(t+2)+γ² r(t+3)+ ……………
 The value of the discount factor lies within 0 to 1
Rewards
Discount factor
MSC-IT Part-1 By - Ajay Chaurasiya
8

9. Monte Carlo Learning
 The Monte Carlo method for reinforcement learning learns directly from
episodes of experience without any prior knowledge of MDP transitions. Here,
the random component is the return or reward.
 Below are key characteristics of Monte Carlo (MC) method:
1. There is no model (agent does not know state MDP transitions)
2. Agent learns from sampled experience
3. learn state value vπ(s) under policy π by experiencing average return from
all sampled episodes (value = average return)
4. only after a complete episode, values are updated
5. There is no bootstrapping
6. Only can be used in episodic problems
MSC-IT Part-1 By - Ajay Chaurasiya
9

10. Continue..
 In Monte Carlo Method instead of expected return we use empirical return
that agent has sampled based following the policy.
MSC-IT Part-1 By - Ajay Chaurasiya
10

11.  Example: Gems collection
 Agent follows policy and complete an episode, along the way in each
step it collects rewards in the form of gem. To get state value agent
sum-up all the gems collected after each episode starting from that
state.
MSC-IT Part-1 By - Ajay Chaurasiya
11

12. Continue..
 Return(Sample 01) = 2 + 1 + 2 + 2 + 1 + 5 = 13 gems
 Return(Sample 02) = 2 + 3 + 1 + 3 + 1 + 5 = 15 gems
 Return(Sample 03) = 2 + 3 + 1 + 3 + 1 + 5 = 15 gems
 Observed mean return (based on 3 samples) = (13 + 15 + 15)/3 = 14.33 gems
 Thus state value as per Monte Carlo Method, v π(S 05) is 14.33 gems based on
3 samples following policy π.
MSC-IT Part-1 By - Ajay Chaurasiya
12

13.  (even if agent comes-back to the same state multiple time in the episode,
only first visit will be counted). Detailed step as below:
 To evaluate state s, first we set number of visit, N(s) = 0, Total return TR(s) =
0 (these values are updated across episodes)
 The first time-step t that state s is visited in an episode, increment counter
N(s) = N(s) + 1
 Increment total return TR(s) = TR(s) + Gt
 Value is estimated by mean return V(s) = TR(s)/N(s)
 By law of large numbers, V(s) -> vπ(s) (this is called true value under policy π)
as N(s) approaches infinity
MSC-IT Part-1 By - Ajay Chaurasiya
13

14. Thank You !
MSC-IT Part-1 By - Ajay Chaurasiya
14