SlideShare a Scribd company logo
1 
Reinforcement Learning 
By: 
Chandra Prakash 
IIITM Gwalior
22 
Outline 
Introduction 
Element of reinforcement learning 
Reinforcement Learning Problem 
Problem solving methods for RL 
2
33 
Introduction 
Machine learning: Definition 
 Machine learning is a scientific discipline that is 
concerned with the design and development of 
algorithms that allow computers to learn based on 
data, such as from sensor data or databases. 
A major focus of machine learning research is to 
automatically learn to recognize complex patterns and 
make intelligent decisions based on data . 
3
4 
Machine learning Type: 
With respect to the feedback type to learner: 
 Supervised learning : Task Driven (Classification) 
 Unsupervised learning : Data Driven (Clustering) 
 Reinforcement learning — 
 Close to human learning. 
 Algorithm learns a policy of how to act in a given 
environment. 
 Every action has some impact in the environment, 
and the environment provides rewards that guides 
the learning algorithm.
5 
Supervised Learning Vs 
Reinforcement Learning 
Supervised Learning 
 Step: 1 
Teacher: Does picture 1 show a car or a flower? 
Learner: A flower. 
Teacher: No, it’s a car. 
Step: 2 
Teacher: Does picture 2 show a car or a flower? 
Learner: A car. 
Teacher: Yes, it’s a car. 
Step: 3 ....
6 
Supervised Learning Vs 
Reinforcement Learning (Cont...) 
Reinforcement Learning 
Step: 1 
World: You are in state 9. Choose action A or C. 
Learner: Action A. 
World: Your reward is 100. 
Step: 2 
World: You are in state 32. Choose action B or E. 
Learner: Action B. 
World: Your reward is 50. 
Step: 3 ....
77 
Introduction (Cont..) 
 Meaning of Reinforcement: Occurrence of an 
event, in the proper relation to a response, that tends to 
increase the probability that the response will occur 
again in the same situation. 
Reinforcement learning is the problem faced by an 
agent that learns behavior through trial-and-error 
interactions with a dynamic environment. 
 Reinforcement Learning is learning how to act in 
order to maximize a numerical reward. 
7
88 
Introduction (Cont..) 
 Reinforcement learning is not a type of neural 
network, nor is it an alternative to neural networks. 
Rather, it is an orthogonal approach for Learning 
Machine. 
 Reinforcement learning emphasizes learning feedback 
that evaluates the learner's performance without 
providing standards of correctness in the form 
of behavioral targets. 
Example: Bicycle learning 
8
99 
Element of reinforcement learning 
Agent 
State Reward Action 
Environment 
Policy 
 Agent: Intelligent programs 
 Environment: External condition 
 Policy: 
Defines the agent’s behavior at a given time 
A mapping from states to actions 
Lookup tables or simple function 
9
1100 
Element of reinforcement learning 
 Reward function : 
 Defines the goal in an RL problem 
 Policy is altered to achieve this goal 
 Value function: 
 Reward function indicates what is good in an immediate 
sense while a value function specifies what is good in 
the long run. 
 Value of a state is the total amount of reward an agent can expect 
to accumulate over the future, starting form that state. 
 Model of the environment : 
 Predict mimic behavior of environment, 
 Used for planning & if Know current state and action 
then predict the resultant next state and next reward. 
10
1111 
Agent- Environment Interface 
11
1122 
Steps for Reinforcement Learning 
1. The agent observes an input state 
2. An action is determined by a decision making 
function (policy) 
3. The action is performed 
4. The agent receives a scalar reward or reinforcement 
from the environment 
5. Information about the reward given for that state / 
action pair is recorded 
12
1133 
Silent Features of Reinforcement 
Learning : 
 Set of problems rather than a set of techniques 
 Without specifying how the task is to be achieved. 
 “RL as a tool” point of view: 
 RL is training by rewards and punishments. 
 Train tool for the computer learning. 
 The learning agent’s point of view: 
 RL is learning from trial and error with the world. 
 Eg. how much reward I much get if I get this. 
13
1144 
Reinforcement Learning (Cont..) 
 Reinforcement Learning uses Evaluative Feedback 
 Purely Evaluative Feedback 
 Indicates how good the action taken is. 
 Not tell if it is the best or the worst action possible. 
 Basis of methods for function optimization. 
 Purely Instructive Feedback 
 Indicates the correct action to take, independently of 
the action actually taken. 
 Eg: Supervised Learning 
 Associative and Non Associative: 
14
15 
Associative & Non Associative Tasks 
 Associative : 
 Situation Dependent 
 Mapping from situation to the actions that are best in that 
situation 
 Non Associative: 
 Situation independent 
 No need for associating different action with different 
situations. 
 Learner either tries to find a single best action when the task 
is stationary, or tries to track the best action as it changes 
over time when the task is non stationary.
1166 
Reinforcement Learning (Cont..) 
Exploration and exploitation 
Greedy action: Action chosen with greatest estimated 
value. 
Greedy action: a case of Exploitation. 
Non greedy action: a case of Exploration, as it 
enables us to improve estimate the non-greedy 
action's value. 
 N-armed bandit Problem: 
We have to choose from n different options or actions. 
We will choose the one with maximum reward. 
16
Bandits Problem 
One-Bandit 
“arms” 
Pull arms sequentially so as to maximize the total 
expected reward 
•It is non associative and evaluative
1188 
Action Selection Policies 
Greediest action - action with the highest estimated 
reward. 
 Ɛ -greedy 
 Most of the time the greediest action is chosen 
 Every once in a while with a small probability Ɛ, an 
action is selected at random. The action is selected 
uniformly, independent of the action-value estimates. 
 Ɛ -soft - The best action is selected with probability (1 
–Ɛ) and the rest of the time a random action is chosen 
uniformly. 
18
1199 
Ɛ-Greedy Action Selection Method : 
Let the a* is the greedy action at time t and Qt(a) is the 
value of action a at time. 
Greedy Action Selection: 
 Ɛ –greedy 
19 
Î 
* = argmax 
with probability 1 t a 
{ 
-Î 
random action with probability 
at= 
at = at 
a 
Qt (a)
20 
Action Selection Policies (Cont…) 
 Softmax – 
 Drawback of Ɛ -greedy & Ɛ -soft: Select random 
actions uniformly. 
 Softmax remedies this by: 
 Assigning a weight with each actions, according 
to their action-value estimate. 
 A random action is selected with regards to the 
weight associated with each action 
 The worst actions are unlikely to be chosen. 
 This is a good approach to take where the worst 
actions are very unfavorable.
21 
Softmax Action Selection( Cont…) 
 Problem with Ɛ-greedy: Neglects action values 
 Softmax idea: grade action probs. by estimated values. 
 Gibbs, or Boltzmann action selection, or exponential 
weights: 
t is the “computational temperature” 
At t  0 the Softmax action selection method become 
the same as greedy action selection.
22 
Incremental Implementation 
 Sample average: 
 Incremental computation: 
 Common update rule form: 
NewEstimate = OldEstimate + StepSize[Target – 
OldEstimate] 
 The expression [ Target - Old Estimate] is an error in the 
estimate. 
 It is reduce by taking a step toward the target. 
 In proceeding the (t+1)st reward for action a the step-size 
parameter will be 1(t+1).
Some terms in Reinforcement Learning 
 The Agent Learns a Policy: 
2233 
 Policy at step t, : a mapping from states to action 
probabilities will be: 
 Agents changes their policy with Experience. 
 Objective: get as much reward as possible over a 
long run. 
 Goals and Rewards 
 A goal should specify what we want to achieve, not 
how we want to achieve it. 
23
24 
Some terms in RL (Cont…) 
 Returns 
 Rewards in long term 
 Episodes: Subsequence of interaction between agent-environment 
e.g., plays of a game, trips through a 
maze. 
 Discount return 
 The geometrically discounted model of return: 
 Used to: 
 To determine the present value of the future 
rewards 
 Give more weight to earlier rewards
2255 
The Markov Property 
 Environment's response at (t+1) depends only on State 
(s) & Action (a) at t, 
Markov Decision Processes 
 Finite MDP 
 Transition Probabilities: 
 Expected Reward 
Transition Graph: Summarize dynamics of finite MDP. 
25
Value function 
 States-action pairs function that estimate how good it is 
for the agent to be in a given state 
 Type of value function 
2266 
 State-Value function 
 Action-Value function 
26
2277 
Backup Diagrams 
 Basis of the update or backup operations 
 backup diagrams to provide graphical summaries of 
the algorithms. (State value & Action value function) 
Bellman Equation for a Policy Π 
27
28 
Model-free and Model-based Learning 
 Model-based learning 
 Learn from the model instead of interacting with the world 
 Can visit arbitrary parts of the model 
 Also called indirect methods 
 E.g. Dynamic Programming 
 Model-free learning 
 Sample Reward and transition function by interacting with 
the world 
 Also called direct methods
29 
Problem Solving Methods for RL 
1) Dynamic programming 
2) Monte Carlo methods 
3) Temporal-difference learning.
3300 
1.Dynamic programming 
 Classical solution method 
 Require a complete and accurate model of the 
environment. 
 Popular method for Dynamic programming 
 Policy Evaluation : Iterative computation of the value 
function for a given policy (prediction Problem) 
 Policy Improvement: Computation of improved policy 
for a given value function. 
30 
( ) { ( )} t t 1 t V s E r gV s p ¬ + +
Dynamic Programming 
T 
st 
rt+1 
T 
T 
st+1 
T 
T T 
T 
T T 
T 
31 
( ) { ( )} t t 1 t V s E r gV s p ¬ + +
32
33
34 
 NewEstimate = OldEstimate + 
StepSize[Target – 
OldEstimate] 
NewEstimate = OldEstimate + StepSize[Target – OldEstimate]
Computation of improved policy for a given value function 
35
36 
 Policy Evaluation and Policy improvement 
together leads 
 Policy Iteration 
 Value Iteration:
3377 
Policy Iteration : Alternates policy evaluation and policy 
improvement steps 
37
 Combines evaluation and improvement in one update 
38
3399 
Value Iteration: 
Combines evaluation and improvement in one update
40 
Generalized Policy Iteration (GPI) 
Consist of two iteration process, 
 Policy Evaluation :Making the value function 
consistent with the current policy 
 Policy Improvement: Making the policy greedy 
with respect to the current value function
4411 
2. Monte Carlo Methods 
 Features of Monte Carlo Methods 
 No need of Complete knowledge of environment 
 Based on averaging sample returns observed after visit 
to that state. 
 Experience is divided into Episodes 
 Only after completing an episode, value estimates and 
policies are changed. 
 Don't require a model 
 Not suited for step-by-step incremental computation 
41
4422 
MC and DP Methods 
 Compute same value function 
 Same step as in DP 
 Policy evaluation 
 Computation of VΠ and QΠ for a fixed arbitrary 
policy (Π). 
 Policy Improvement 
 Generalized Policy Iteration 
42
4433 
To find value of a State 
 Estimate by experience, average the returns observed 
after visit to that state. 
 More the return, more is the average converge to 
expected value 
43
First-visit Monte Carlo Policy evaluation 
4444 
44
4455
In MCES, all returns for each state-action pair are accumulated 
and averaged, irrespective of what Policy was in force when 
they were observed 
4466 
46
4477
4488 
Monte Carlo Method (Cont…) 
 All actions should be selected infinitely for 
optimization 
 MC exploring starts can not converge to any 
suboptimal policy. 
 Agents have 2 approaches 
 On-policy 
 Off-policy 
48
49
5500 
On-policy MC algorithm 
50
51 
Off Policy MC algorithm
Monte Carlo and Dynamic Programming 
5522 
 MC has several advantage over DP: 
 Can learn from interaction with environment 
 No need of full models 
 No need to learn about ALL states 
 No bootstrapping 
52
5533 
3. Temporal Difference (TD) methods 
 Learn from experience, like MC 
 Can learn directly from interaction with environment 
 No need for full models 
 Estimate values based on estimated values of next states, 
like DP 
 Bootstrapping (like DP) 
 Issue to watch for: maintaining sufficient exploration 
53
for a given policy p, compute the state-value function 
target: the actual return after time t 
5544 
TD Prediction 
Policy Evaluation (the prediction problem): 
Simple every - visit Monte Carlomethod : 
V(s ) V(s )+α[R V(s )] t t t t ¬ - 
V p 
( ) : 
The simplest TD method, TD 0 
V(s ) V(s )+α[r + V(s ) V(s )] 
¬ g 
- t t t+ 1 t+ 1 
t target: an estimate of the return 
54
Simple Monte Carlo 
T T T 
T 
T T 
T st 
T T 
T T T 
55
Simplest TD Method 
st+1 
rt+1 
st 
T T T T T 
T T 
T T T T T T 
56 
V(st )¬V(st) +a rt+1 +g V(st+1 ) - V(st [ )]
Dynamic Programming 
T 
st 
rt+1 
T 
T 
st+1 
T 
T T 
T 
T T 
T 
57 
( ) { ( )} t t 1 t V s E r gV s p ¬ + +
5588 
TD methods bootstrap and sample 
 Bootstrapping: update involves an estimate 
 MC does not bootstrap 
 DP bootstraps 
 TD bootstraps 
 Sampling: update does not involve an expected 
value 
 MC samples 
 DP does not sample 
 TD samples 
58
After every transition from a nonterminal state , do this ( ) ( ) [ ( ) ( )] 
If is terminal, then 0. 
1 1 
5599 
Learning An Action-Value Function 
Estimate Qp for the current behavior policy p . 
59 
Q s ,a ¬Q s ,a +α r + gQ s ,a Q s ,a 
t t t t t+ 1 t+ 1 
t+1 t t 
s Q(s ,a )= 
s 
t+ t+ t+1 
t 
-
Q-Learning: Off-Policy TD Control 
60 
One - step Q- learning : 
Q st , at ( )¬Q st , at ( )+a rt +1 +g max 
[ Q(s, )- +aQ ( s, a)] 
t1t t a
6611 
Sarsa: On-Policy TD Control 
S A R S A: State Action Reward State Action 
61 
Turn this into a control method by always updating the policy 
to be greedy with respect to the current estimate:
6622 
Advantages of TD Learning 
 TD methods do not require a model of the 
environment, only experience 
 TD, but not MC, methods can be fully incremental 
 You can learn before knowing the final outcome 
 Less memory 
 Less peak computation 
 You can learn without the final outcome 
 From incomplete sequences 
 Both MC and TD converge 
62
63 
References: RL 
 Univ. of Alberta 
 http://www.cs.ualberta.ca/~sutton/book/ebook/node 
1.html 
 Sutton and barto,”Reinforcement Learning an 
introduction.” 
 Univ. of South Wales 
 http://www.cse.unsw.edu.au/~cs9417ml/RL1/tdlear 
ning.html
6644 
Thank You 
64

More Related Content

What's hot

Types of Machine Learning
Types of Machine LearningTypes of Machine Learning
Types of Machine Learning
Samra Shahzadi
 
Reinforcement Learning Q-Learning
Reinforcement Learning   Q-Learning Reinforcement Learning   Q-Learning
Reinforcement Learning Q-Learning
Melaku Eneayehu
 
I. AO* SEARCH ALGORITHM
I. AO* SEARCH ALGORITHMI. AO* SEARCH ALGORITHM
I. AO* SEARCH ALGORITHM
vikas dhakane
 
Decision Tree Learning
Decision Tree LearningDecision Tree Learning
Decision Tree Learning
Milind Gokhale
 
Lstm
LstmLstm
Heuristic Search Techniques {Artificial Intelligence}
Heuristic Search Techniques {Artificial Intelligence}Heuristic Search Techniques {Artificial Intelligence}
Heuristic Search Techniques {Artificial Intelligence}
FellowBuddy.com
 
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
vikas dhakane
 
K Nearest Neighbors
K Nearest NeighborsK Nearest Neighbors
Reinforcement Learning : A Beginners Tutorial
Reinforcement Learning : A Beginners TutorialReinforcement Learning : A Beginners Tutorial
Reinforcement Learning : A Beginners Tutorial
Omar Enayet
 
Problem solving agents
Problem solving agentsProblem solving agents
Problem solving agents
Megha Sharma
 
Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep Learning
Oswald Campesato
 
Intelligent Agent PPT ON SLIDESHARE IN ARTIFICIAL INTELLIGENCE
Intelligent Agent PPT ON SLIDESHARE IN ARTIFICIAL INTELLIGENCEIntelligent Agent PPT ON SLIDESHARE IN ARTIFICIAL INTELLIGENCE
Intelligent Agent PPT ON SLIDESHARE IN ARTIFICIAL INTELLIGENCE
Khushboo Pal
 
Supervised vs Unsupervised vs Reinforcement Learning | Edureka
Supervised vs Unsupervised vs Reinforcement Learning | EdurekaSupervised vs Unsupervised vs Reinforcement Learning | Edureka
Supervised vs Unsupervised vs Reinforcement Learning | Edureka
Edureka!
 
Local search algorithm
Local search algorithmLocal search algorithm
Local search algorithm
Megha Sharma
 
Artificial Intelligence -- Search Algorithms
Artificial Intelligence-- Search Algorithms Artificial Intelligence-- Search Algorithms
Artificial Intelligence -- Search Algorithms
Syed Ahmed
 
Artificial Intelligence
Artificial IntelligenceArtificial Intelligence
Artificial Intelligence
Vinod Kumar Meghwar
 
BackTracking Algorithm: Technique and Examples
BackTracking Algorithm: Technique and ExamplesBackTracking Algorithm: Technique and Examples
BackTracking Algorithm: Technique and Examples
Fahim Ferdous
 
Markov decision process
Markov decision processMarkov decision process
Markov decision process
Hamed Abdi
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer Perceptron
Mostafa G. M. Mostafa
 
Back propagation
Back propagationBack propagation
Back propagation
Nagarajan
 

What's hot (20)

Types of Machine Learning
Types of Machine LearningTypes of Machine Learning
Types of Machine Learning
 
Reinforcement Learning Q-Learning
Reinforcement Learning   Q-Learning Reinforcement Learning   Q-Learning
Reinforcement Learning Q-Learning
 
I. AO* SEARCH ALGORITHM
I. AO* SEARCH ALGORITHMI. AO* SEARCH ALGORITHM
I. AO* SEARCH ALGORITHM
 
Decision Tree Learning
Decision Tree LearningDecision Tree Learning
Decision Tree Learning
 
Lstm
LstmLstm
Lstm
 
Heuristic Search Techniques {Artificial Intelligence}
Heuristic Search Techniques {Artificial Intelligence}Heuristic Search Techniques {Artificial Intelligence}
Heuristic Search Techniques {Artificial Intelligence}
 
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
 
K Nearest Neighbors
K Nearest NeighborsK Nearest Neighbors
K Nearest Neighbors
 
Reinforcement Learning : A Beginners Tutorial
Reinforcement Learning : A Beginners TutorialReinforcement Learning : A Beginners Tutorial
Reinforcement Learning : A Beginners Tutorial
 
Problem solving agents
Problem solving agentsProblem solving agents
Problem solving agents
 
Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep Learning
 
Intelligent Agent PPT ON SLIDESHARE IN ARTIFICIAL INTELLIGENCE
Intelligent Agent PPT ON SLIDESHARE IN ARTIFICIAL INTELLIGENCEIntelligent Agent PPT ON SLIDESHARE IN ARTIFICIAL INTELLIGENCE
Intelligent Agent PPT ON SLIDESHARE IN ARTIFICIAL INTELLIGENCE
 
Supervised vs Unsupervised vs Reinforcement Learning | Edureka
Supervised vs Unsupervised vs Reinforcement Learning | EdurekaSupervised vs Unsupervised vs Reinforcement Learning | Edureka
Supervised vs Unsupervised vs Reinforcement Learning | Edureka
 
Local search algorithm
Local search algorithmLocal search algorithm
Local search algorithm
 
Artificial Intelligence -- Search Algorithms
Artificial Intelligence-- Search Algorithms Artificial Intelligence-- Search Algorithms
Artificial Intelligence -- Search Algorithms
 
Artificial Intelligence
Artificial IntelligenceArtificial Intelligence
Artificial Intelligence
 
BackTracking Algorithm: Technique and Examples
BackTracking Algorithm: Technique and ExamplesBackTracking Algorithm: Technique and Examples
BackTracking Algorithm: Technique and Examples
 
Markov decision process
Markov decision processMarkov decision process
Markov decision process
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer Perceptron
 
Back propagation
Back propagationBack propagation
Back propagation
 

Viewers also liked

Lecture 9 10 .mobile ad-hoc routing protocols
Lecture 9 10 .mobile ad-hoc routing protocolsLecture 9 10 .mobile ad-hoc routing protocols
Lecture 9 10 .mobile ad-hoc routing protocols
Chandra Meena
 
Lecture 11 14. Adhoc routing protocols cont..
Lecture 11 14. Adhoc  routing protocols cont..Lecture 11 14. Adhoc  routing protocols cont..
Lecture 11 14. Adhoc routing protocols cont..
Chandra Meena
 
Lecture 2 evolution of mobile cellular
Lecture 2  evolution of mobile cellular Lecture 2  evolution of mobile cellular
Lecture 2 evolution of mobile cellular
Chandra Meena
 
Lecture 7 8 ad hoc wireless media access protocols
Lecture 7 8 ad hoc wireless media access protocolsLecture 7 8 ad hoc wireless media access protocols
Lecture 7 8 ad hoc wireless media access protocols
Chandra Meena
 
Lecture 3 4. prnet
Lecture 3 4. prnetLecture 3 4. prnet
Lecture 3 4. prnet
Chandra Meena
 
Lecture 1 mobile and adhoc network- introduction
Lecture 1  mobile and adhoc network- introductionLecture 1  mobile and adhoc network- introduction
Lecture 1 mobile and adhoc network- introduction
Chandra Meena
 
Lecture 19 22. transport protocol for ad-hoc
Lecture 19 22. transport protocol for ad-hoc Lecture 19 22. transport protocol for ad-hoc
Lecture 19 22. transport protocol for ad-hoc
Chandra Meena
 
Lecture 23 27. quality of services in ad hoc wireless networks
Lecture 23 27. quality of services in ad hoc wireless networksLecture 23 27. quality of services in ad hoc wireless networks
Lecture 23 27. quality of services in ad hoc wireless networks
Chandra Meena
 
Lecture 5 6 .ad hoc network
Lecture 5 6 .ad hoc networkLecture 5 6 .ad hoc network
Lecture 5 6 .ad hoc network
Chandra Meena
 

Viewers also liked (9)

Lecture 9 10 .mobile ad-hoc routing protocols
Lecture 9 10 .mobile ad-hoc routing protocolsLecture 9 10 .mobile ad-hoc routing protocols
Lecture 9 10 .mobile ad-hoc routing protocols
 
Lecture 11 14. Adhoc routing protocols cont..
Lecture 11 14. Adhoc  routing protocols cont..Lecture 11 14. Adhoc  routing protocols cont..
Lecture 11 14. Adhoc routing protocols cont..
 
Lecture 2 evolution of mobile cellular
Lecture 2  evolution of mobile cellular Lecture 2  evolution of mobile cellular
Lecture 2 evolution of mobile cellular
 
Lecture 7 8 ad hoc wireless media access protocols
Lecture 7 8 ad hoc wireless media access protocolsLecture 7 8 ad hoc wireless media access protocols
Lecture 7 8 ad hoc wireless media access protocols
 
Lecture 3 4. prnet
Lecture 3 4. prnetLecture 3 4. prnet
Lecture 3 4. prnet
 
Lecture 1 mobile and adhoc network- introduction
Lecture 1  mobile and adhoc network- introductionLecture 1  mobile and adhoc network- introduction
Lecture 1 mobile and adhoc network- introduction
 
Lecture 19 22. transport protocol for ad-hoc
Lecture 19 22. transport protocol for ad-hoc Lecture 19 22. transport protocol for ad-hoc
Lecture 19 22. transport protocol for ad-hoc
 
Lecture 23 27. quality of services in ad hoc wireless networks
Lecture 23 27. quality of services in ad hoc wireless networksLecture 23 27. quality of services in ad hoc wireless networks
Lecture 23 27. quality of services in ad hoc wireless networks
 
Lecture 5 6 .ad hoc network
Lecture 5 6 .ad hoc networkLecture 5 6 .ad hoc network
Lecture 5 6 .ad hoc network
 

Similar to Reinforcement learning

reinforcement-learning-141009013546-conversion-gate02.pdf
reinforcement-learning-141009013546-conversion-gate02.pdfreinforcement-learning-141009013546-conversion-gate02.pdf
reinforcement-learning-141009013546-conversion-gate02.pdf
VaishnavGhadge1
 
reinforcement-learning-141009013546-conversion-gate02.pptx
reinforcement-learning-141009013546-conversion-gate02.pptxreinforcement-learning-141009013546-conversion-gate02.pptx
reinforcement-learning-141009013546-conversion-gate02.pptx
MohibKhan79
 
An efficient use of temporal difference technique in Computer Game Learning
An efficient use of temporal difference technique in Computer Game LearningAn efficient use of temporal difference technique in Computer Game Learning
An efficient use of temporal difference technique in Computer Game Learning
Prabhu Kumar
 
CS3013 -MACHINE LEARNING.pptx
CS3013 -MACHINE LEARNING.pptxCS3013 -MACHINE LEARNING.pptx
CS3013 -MACHINE LEARNING.pptx
logesswarisrinivasan
 
Head First Reinforcement Learning
Head First Reinforcement LearningHead First Reinforcement Learning
Head First Reinforcement Learning
azzeddine chenine
 
RL.ppt
RL.pptRL.ppt
RL.ppt
AzharJamil15
 
Reinforcement Learning
Reinforcement LearningReinforcement Learning
Reinforcement Learning
SVijaylakshmi
 
Reinforcement learning
Reinforcement learningReinforcement learning
Reinforcement learning
Ding Li
 
Introduction to Deep Reinforcement Learning
Introduction to Deep Reinforcement LearningIntroduction to Deep Reinforcement Learning
Introduction to Deep Reinforcement Learning
IDEAS - Int'l Data Engineering and Science Association
 
Reinforcement learning
Reinforcement  learningReinforcement  learning
Reinforcement learning
SKS
 
Reinforcement Learning
Reinforcement LearningReinforcement Learning
Reinforcement Learning
Salem-Kabbani
 
Deep reinforcement learning from scratch
Deep reinforcement learning from scratchDeep reinforcement learning from scratch
Deep reinforcement learning from scratch
Jie-Han Chen
 
acai01-updated.ppt
acai01-updated.pptacai01-updated.ppt
acai01-updated.ppt
butest
 
REINFORCEMENT LEARNING
REINFORCEMENT LEARNINGREINFORCEMENT LEARNING
REINFORCEMENT LEARNING
pradiprahul
 
Reinforcement Learning Guide For Beginners
Reinforcement Learning Guide For BeginnersReinforcement Learning Guide For Beginners
Reinforcement Learning Guide For Beginners
gokulprasath06
 
Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017
Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017
Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017
MLconf
 
Reinforcement learning
Reinforcement learningReinforcement learning
Reinforcement learning
Elias Hasnat
 
reiniforcement learning.ppt
reiniforcement learning.pptreiniforcement learning.ppt
reiniforcement learning.ppt
charusharma165
 
How to formulate reinforcement learning in illustrative ways
How to formulate reinforcement learning in illustrative waysHow to formulate reinforcement learning in illustrative ways
How to formulate reinforcement learning in illustrative ways
YasutoTamura1
 
Reinforcement Learning.ppt
Reinforcement Learning.pptReinforcement Learning.ppt
Reinforcement Learning.ppt
POOJASHREEC1
 

Similar to Reinforcement learning (20)

reinforcement-learning-141009013546-conversion-gate02.pdf
reinforcement-learning-141009013546-conversion-gate02.pdfreinforcement-learning-141009013546-conversion-gate02.pdf
reinforcement-learning-141009013546-conversion-gate02.pdf
 
reinforcement-learning-141009013546-conversion-gate02.pptx
reinforcement-learning-141009013546-conversion-gate02.pptxreinforcement-learning-141009013546-conversion-gate02.pptx
reinforcement-learning-141009013546-conversion-gate02.pptx
 
An efficient use of temporal difference technique in Computer Game Learning
An efficient use of temporal difference technique in Computer Game LearningAn efficient use of temporal difference technique in Computer Game Learning
An efficient use of temporal difference technique in Computer Game Learning
 
CS3013 -MACHINE LEARNING.pptx
CS3013 -MACHINE LEARNING.pptxCS3013 -MACHINE LEARNING.pptx
CS3013 -MACHINE LEARNING.pptx
 
Head First Reinforcement Learning
Head First Reinforcement LearningHead First Reinforcement Learning
Head First Reinforcement Learning
 
RL.ppt
RL.pptRL.ppt
RL.ppt
 
Reinforcement Learning
Reinforcement LearningReinforcement Learning
Reinforcement Learning
 
Reinforcement learning
Reinforcement learningReinforcement learning
Reinforcement learning
 
Introduction to Deep Reinforcement Learning
Introduction to Deep Reinforcement LearningIntroduction to Deep Reinforcement Learning
Introduction to Deep Reinforcement Learning
 
Reinforcement learning
Reinforcement  learningReinforcement  learning
Reinforcement learning
 
Reinforcement Learning
Reinforcement LearningReinforcement Learning
Reinforcement Learning
 
Deep reinforcement learning from scratch
Deep reinforcement learning from scratchDeep reinforcement learning from scratch
Deep reinforcement learning from scratch
 
acai01-updated.ppt
acai01-updated.pptacai01-updated.ppt
acai01-updated.ppt
 
REINFORCEMENT LEARNING
REINFORCEMENT LEARNINGREINFORCEMENT LEARNING
REINFORCEMENT LEARNING
 
Reinforcement Learning Guide For Beginners
Reinforcement Learning Guide For BeginnersReinforcement Learning Guide For Beginners
Reinforcement Learning Guide For Beginners
 
Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017
Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017
Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017
 
Reinforcement learning
Reinforcement learningReinforcement learning
Reinforcement learning
 
reiniforcement learning.ppt
reiniforcement learning.pptreiniforcement learning.ppt
reiniforcement learning.ppt
 
How to formulate reinforcement learning in illustrative ways
How to formulate reinforcement learning in illustrative waysHow to formulate reinforcement learning in illustrative ways
How to formulate reinforcement learning in illustrative ways
 
Reinforcement Learning.ppt
Reinforcement Learning.pptReinforcement Learning.ppt
Reinforcement Learning.ppt
 

Recently uploaded

IS Code SP 23: Handbook on concrete mixes
IS Code SP 23: Handbook  on concrete mixesIS Code SP 23: Handbook  on concrete mixes
IS Code SP 23: Handbook on concrete mixes
Mani Krishna Sarkar
 
Introduction to IP address concept - Computer Networking
Introduction to IP address concept - Computer NetworkingIntroduction to IP address concept - Computer Networking
Introduction to IP address concept - Computer Networking
Md.Shohel Rana ( M.Sc in CSE Khulna University of Engineering & Technology (KUET))
 
CONFINED SPACE ENTRY TRAINING FOR OIL INDUSTRY ppt
CONFINED SPACE ENTRY TRAINING FOR OIL INDUSTRY pptCONFINED SPACE ENTRY TRAINING FOR OIL INDUSTRY ppt
CONFINED SPACE ENTRY TRAINING FOR OIL INDUSTRY ppt
ASHOK KUMAR SINGH
 
Conservation of Natural Resources Biodiversity.pptx
Conservation of Natural Resources Biodiversity.pptxConservation of Natural Resources Biodiversity.pptx
Conservation of Natural Resources Biodiversity.pptx
AdarshaMR1
 
Design and Application of Side Channel Spillways
Design and Application of Side Channel SpillwaysDesign and Application of Side Channel Spillways
Design and Application of Side Channel Spillways
ahmed42488
 
Chlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptxChlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptx
yadavsuyash008
 
OSHA LOTO training, LOTO, lock out tag out
OSHA LOTO training, LOTO, lock out tag outOSHA LOTO training, LOTO, lock out tag out
OSHA LOTO training, LOTO, lock out tag out
Ateeb19
 
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
YanKing2
 
Benefits of Studying Artificial Intelligence - KRCE.pptx
Benefits of Studying Artificial Intelligence - KRCE.pptxBenefits of Studying Artificial Intelligence - KRCE.pptx
Benefits of Studying Artificial Intelligence - KRCE.pptx
krceseo
 
readers writers Problem in operating system
readers writers Problem in operating systemreaders writers Problem in operating system
readers writers Problem in operating system
VADAPALLYPRAVEENKUMA1
 
Jet Propulsion and its working principle.pdf
Jet Propulsion and its working principle.pdfJet Propulsion and its working principle.pdf
Jet Propulsion and its working principle.pdf
KIET Group of Institutions
 
Monitoring and reporting of transparent forest data and information under the...
Monitoring and reporting of transparent forest data and information under the...Monitoring and reporting of transparent forest data and information under the...
Monitoring and reporting of transparent forest data and information under the...
Pilar Valbuena Perez
 
Conservation of Taksar through Economic Regeneration
Conservation of Taksar through Economic RegenerationConservation of Taksar through Economic Regeneration
Conservation of Taksar through Economic Regeneration
PriyankaKarn3
 
Unblocking The Main Thread - Solving ANRs and Frozen Frames
Unblocking The Main Thread - Solving ANRs and Frozen FramesUnblocking The Main Thread - Solving ANRs and Frozen Frames
Unblocking The Main Thread - Solving ANRs and Frozen Frames
Sinan KOZAK
 
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-IDUNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
GOWSIKRAJA PALANISAMY
 
Lecture 3 Biomass energy...............ppt
Lecture 3 Biomass energy...............pptLecture 3 Biomass energy...............ppt
Lecture 3 Biomass energy...............ppt
RujanTimsina1
 
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptxPresentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Er. Kushal Ghimire
 
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
Jim Mimlitz, P.E.
 
Rotary Intersection in traffic engineering.pptx
Rotary Intersection in traffic engineering.pptxRotary Intersection in traffic engineering.pptx
Rotary Intersection in traffic engineering.pptx
surekha1287
 
RECENT DEVELOPMENTS IN RING SPINNING.pptx
RECENT DEVELOPMENTS IN RING SPINNING.pptxRECENT DEVELOPMENTS IN RING SPINNING.pptx
RECENT DEVELOPMENTS IN RING SPINNING.pptx
peacesoul123
 

Recently uploaded (20)

IS Code SP 23: Handbook on concrete mixes
IS Code SP 23: Handbook  on concrete mixesIS Code SP 23: Handbook  on concrete mixes
IS Code SP 23: Handbook on concrete mixes
 
Introduction to IP address concept - Computer Networking
Introduction to IP address concept - Computer NetworkingIntroduction to IP address concept - Computer Networking
Introduction to IP address concept - Computer Networking
 
CONFINED SPACE ENTRY TRAINING FOR OIL INDUSTRY ppt
CONFINED SPACE ENTRY TRAINING FOR OIL INDUSTRY pptCONFINED SPACE ENTRY TRAINING FOR OIL INDUSTRY ppt
CONFINED SPACE ENTRY TRAINING FOR OIL INDUSTRY ppt
 
Conservation of Natural Resources Biodiversity.pptx
Conservation of Natural Resources Biodiversity.pptxConservation of Natural Resources Biodiversity.pptx
Conservation of Natural Resources Biodiversity.pptx
 
Design and Application of Side Channel Spillways
Design and Application of Side Channel SpillwaysDesign and Application of Side Channel Spillways
Design and Application of Side Channel Spillways
 
Chlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptxChlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptx
 
OSHA LOTO training, LOTO, lock out tag out
OSHA LOTO training, LOTO, lock out tag outOSHA LOTO training, LOTO, lock out tag out
OSHA LOTO training, LOTO, lock out tag out
 
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
 
Benefits of Studying Artificial Intelligence - KRCE.pptx
Benefits of Studying Artificial Intelligence - KRCE.pptxBenefits of Studying Artificial Intelligence - KRCE.pptx
Benefits of Studying Artificial Intelligence - KRCE.pptx
 
readers writers Problem in operating system
readers writers Problem in operating systemreaders writers Problem in operating system
readers writers Problem in operating system
 
Jet Propulsion and its working principle.pdf
Jet Propulsion and its working principle.pdfJet Propulsion and its working principle.pdf
Jet Propulsion and its working principle.pdf
 
Monitoring and reporting of transparent forest data and information under the...
Monitoring and reporting of transparent forest data and information under the...Monitoring and reporting of transparent forest data and information under the...
Monitoring and reporting of transparent forest data and information under the...
 
Conservation of Taksar through Economic Regeneration
Conservation of Taksar through Economic RegenerationConservation of Taksar through Economic Regeneration
Conservation of Taksar through Economic Regeneration
 
Unblocking The Main Thread - Solving ANRs and Frozen Frames
Unblocking The Main Thread - Solving ANRs and Frozen FramesUnblocking The Main Thread - Solving ANRs and Frozen Frames
Unblocking The Main Thread - Solving ANRs and Frozen Frames
 
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-IDUNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
 
Lecture 3 Biomass energy...............ppt
Lecture 3 Biomass energy...............pptLecture 3 Biomass energy...............ppt
Lecture 3 Biomass energy...............ppt
 
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptxPresentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
 
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
SCADAmetrics Instrumentation for Sensus Water Meters - Core and Main Training...
 
Rotary Intersection in traffic engineering.pptx
Rotary Intersection in traffic engineering.pptxRotary Intersection in traffic engineering.pptx
Rotary Intersection in traffic engineering.pptx
 
RECENT DEVELOPMENTS IN RING SPINNING.pptx
RECENT DEVELOPMENTS IN RING SPINNING.pptxRECENT DEVELOPMENTS IN RING SPINNING.pptx
RECENT DEVELOPMENTS IN RING SPINNING.pptx
 

Reinforcement learning

  • 1. 1 Reinforcement Learning By: Chandra Prakash IIITM Gwalior
  • 2. 22 Outline Introduction Element of reinforcement learning Reinforcement Learning Problem Problem solving methods for RL 2
  • 3. 33 Introduction Machine learning: Definition  Machine learning is a scientific discipline that is concerned with the design and development of algorithms that allow computers to learn based on data, such as from sensor data or databases. A major focus of machine learning research is to automatically learn to recognize complex patterns and make intelligent decisions based on data . 3
  • 4. 4 Machine learning Type: With respect to the feedback type to learner:  Supervised learning : Task Driven (Classification)  Unsupervised learning : Data Driven (Clustering)  Reinforcement learning —  Close to human learning.  Algorithm learns a policy of how to act in a given environment.  Every action has some impact in the environment, and the environment provides rewards that guides the learning algorithm.
  • 5. 5 Supervised Learning Vs Reinforcement Learning Supervised Learning  Step: 1 Teacher: Does picture 1 show a car or a flower? Learner: A flower. Teacher: No, it’s a car. Step: 2 Teacher: Does picture 2 show a car or a flower? Learner: A car. Teacher: Yes, it’s a car. Step: 3 ....
  • 6. 6 Supervised Learning Vs Reinforcement Learning (Cont...) Reinforcement Learning Step: 1 World: You are in state 9. Choose action A or C. Learner: Action A. World: Your reward is 100. Step: 2 World: You are in state 32. Choose action B or E. Learner: Action B. World: Your reward is 50. Step: 3 ....
  • 7. 77 Introduction (Cont..)  Meaning of Reinforcement: Occurrence of an event, in the proper relation to a response, that tends to increase the probability that the response will occur again in the same situation. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment.  Reinforcement Learning is learning how to act in order to maximize a numerical reward. 7
  • 8. 88 Introduction (Cont..)  Reinforcement learning is not a type of neural network, nor is it an alternative to neural networks. Rather, it is an orthogonal approach for Learning Machine.  Reinforcement learning emphasizes learning feedback that evaluates the learner's performance without providing standards of correctness in the form of behavioral targets. Example: Bicycle learning 8
  • 9. 99 Element of reinforcement learning Agent State Reward Action Environment Policy  Agent: Intelligent programs  Environment: External condition  Policy: Defines the agent’s behavior at a given time A mapping from states to actions Lookup tables or simple function 9
  • 10. 1100 Element of reinforcement learning  Reward function :  Defines the goal in an RL problem  Policy is altered to achieve this goal  Value function:  Reward function indicates what is good in an immediate sense while a value function specifies what is good in the long run.  Value of a state is the total amount of reward an agent can expect to accumulate over the future, starting form that state.  Model of the environment :  Predict mimic behavior of environment,  Used for planning & if Know current state and action then predict the resultant next state and next reward. 10
  • 11. 1111 Agent- Environment Interface 11
  • 12. 1122 Steps for Reinforcement Learning 1. The agent observes an input state 2. An action is determined by a decision making function (policy) 3. The action is performed 4. The agent receives a scalar reward or reinforcement from the environment 5. Information about the reward given for that state / action pair is recorded 12
  • 13. 1133 Silent Features of Reinforcement Learning :  Set of problems rather than a set of techniques  Without specifying how the task is to be achieved.  “RL as a tool” point of view:  RL is training by rewards and punishments.  Train tool for the computer learning.  The learning agent’s point of view:  RL is learning from trial and error with the world.  Eg. how much reward I much get if I get this. 13
  • 14. 1144 Reinforcement Learning (Cont..)  Reinforcement Learning uses Evaluative Feedback  Purely Evaluative Feedback  Indicates how good the action taken is.  Not tell if it is the best or the worst action possible.  Basis of methods for function optimization.  Purely Instructive Feedback  Indicates the correct action to take, independently of the action actually taken.  Eg: Supervised Learning  Associative and Non Associative: 14
  • 15. 15 Associative & Non Associative Tasks  Associative :  Situation Dependent  Mapping from situation to the actions that are best in that situation  Non Associative:  Situation independent  No need for associating different action with different situations.  Learner either tries to find a single best action when the task is stationary, or tries to track the best action as it changes over time when the task is non stationary.
  • 16. 1166 Reinforcement Learning (Cont..) Exploration and exploitation Greedy action: Action chosen with greatest estimated value. Greedy action: a case of Exploitation. Non greedy action: a case of Exploration, as it enables us to improve estimate the non-greedy action's value.  N-armed bandit Problem: We have to choose from n different options or actions. We will choose the one with maximum reward. 16
  • 17. Bandits Problem One-Bandit “arms” Pull arms sequentially so as to maximize the total expected reward •It is non associative and evaluative
  • 18. 1188 Action Selection Policies Greediest action - action with the highest estimated reward.  Ɛ -greedy  Most of the time the greediest action is chosen  Every once in a while with a small probability Ɛ, an action is selected at random. The action is selected uniformly, independent of the action-value estimates.  Ɛ -soft - The best action is selected with probability (1 –Ɛ) and the rest of the time a random action is chosen uniformly. 18
  • 19. 1199 Ɛ-Greedy Action Selection Method : Let the a* is the greedy action at time t and Qt(a) is the value of action a at time. Greedy Action Selection:  Ɛ –greedy 19 Î * = argmax with probability 1 t a { -Î random action with probability at= at = at a Qt (a)
  • 20. 20 Action Selection Policies (Cont…)  Softmax –  Drawback of Ɛ -greedy & Ɛ -soft: Select random actions uniformly.  Softmax remedies this by:  Assigning a weight with each actions, according to their action-value estimate.  A random action is selected with regards to the weight associated with each action  The worst actions are unlikely to be chosen.  This is a good approach to take where the worst actions are very unfavorable.
  • 21. 21 Softmax Action Selection( Cont…)  Problem with Ɛ-greedy: Neglects action values  Softmax idea: grade action probs. by estimated values.  Gibbs, or Boltzmann action selection, or exponential weights: t is the “computational temperature” At t  0 the Softmax action selection method become the same as greedy action selection.
  • 22. 22 Incremental Implementation  Sample average:  Incremental computation:  Common update rule form: NewEstimate = OldEstimate + StepSize[Target – OldEstimate]  The expression [ Target - Old Estimate] is an error in the estimate.  It is reduce by taking a step toward the target.  In proceeding the (t+1)st reward for action a the step-size parameter will be 1(t+1).
  • 23. Some terms in Reinforcement Learning  The Agent Learns a Policy: 2233  Policy at step t, : a mapping from states to action probabilities will be:  Agents changes their policy with Experience.  Objective: get as much reward as possible over a long run.  Goals and Rewards  A goal should specify what we want to achieve, not how we want to achieve it. 23
  • 24. 24 Some terms in RL (Cont…)  Returns  Rewards in long term  Episodes: Subsequence of interaction between agent-environment e.g., plays of a game, trips through a maze.  Discount return  The geometrically discounted model of return:  Used to:  To determine the present value of the future rewards  Give more weight to earlier rewards
  • 25. 2255 The Markov Property  Environment's response at (t+1) depends only on State (s) & Action (a) at t, Markov Decision Processes  Finite MDP  Transition Probabilities:  Expected Reward Transition Graph: Summarize dynamics of finite MDP. 25
  • 26. Value function  States-action pairs function that estimate how good it is for the agent to be in a given state  Type of value function 2266  State-Value function  Action-Value function 26
  • 27. 2277 Backup Diagrams  Basis of the update or backup operations  backup diagrams to provide graphical summaries of the algorithms. (State value & Action value function) Bellman Equation for a Policy Π 27
  • 28. 28 Model-free and Model-based Learning  Model-based learning  Learn from the model instead of interacting with the world  Can visit arbitrary parts of the model  Also called indirect methods  E.g. Dynamic Programming  Model-free learning  Sample Reward and transition function by interacting with the world  Also called direct methods
  • 29. 29 Problem Solving Methods for RL 1) Dynamic programming 2) Monte Carlo methods 3) Temporal-difference learning.
  • 30. 3300 1.Dynamic programming  Classical solution method  Require a complete and accurate model of the environment.  Popular method for Dynamic programming  Policy Evaluation : Iterative computation of the value function for a given policy (prediction Problem)  Policy Improvement: Computation of improved policy for a given value function. 30 ( ) { ( )} t t 1 t V s E r gV s p ¬ + +
  • 31. Dynamic Programming T st rt+1 T T st+1 T T T T T T T 31 ( ) { ( )} t t 1 t V s E r gV s p ¬ + +
  • 32. 32
  • 33. 33
  • 34. 34  NewEstimate = OldEstimate + StepSize[Target – OldEstimate] NewEstimate = OldEstimate + StepSize[Target – OldEstimate]
  • 35. Computation of improved policy for a given value function 35
  • 36. 36  Policy Evaluation and Policy improvement together leads  Policy Iteration  Value Iteration:
  • 37. 3377 Policy Iteration : Alternates policy evaluation and policy improvement steps 37
  • 38.  Combines evaluation and improvement in one update 38
  • 39. 3399 Value Iteration: Combines evaluation and improvement in one update
  • 40. 40 Generalized Policy Iteration (GPI) Consist of two iteration process,  Policy Evaluation :Making the value function consistent with the current policy  Policy Improvement: Making the policy greedy with respect to the current value function
  • 41. 4411 2. Monte Carlo Methods  Features of Monte Carlo Methods  No need of Complete knowledge of environment  Based on averaging sample returns observed after visit to that state.  Experience is divided into Episodes  Only after completing an episode, value estimates and policies are changed.  Don't require a model  Not suited for step-by-step incremental computation 41
  • 42. 4422 MC and DP Methods  Compute same value function  Same step as in DP  Policy evaluation  Computation of VΠ and QΠ for a fixed arbitrary policy (Π).  Policy Improvement  Generalized Policy Iteration 42
  • 43. 4433 To find value of a State  Estimate by experience, average the returns observed after visit to that state.  More the return, more is the average converge to expected value 43
  • 44. First-visit Monte Carlo Policy evaluation 4444 44
  • 45. 4455
  • 46. In MCES, all returns for each state-action pair are accumulated and averaged, irrespective of what Policy was in force when they were observed 4466 46
  • 47. 4477
  • 48. 4488 Monte Carlo Method (Cont…)  All actions should be selected infinitely for optimization  MC exploring starts can not converge to any suboptimal policy.  Agents have 2 approaches  On-policy  Off-policy 48
  • 49. 49
  • 50. 5500 On-policy MC algorithm 50
  • 51. 51 Off Policy MC algorithm
  • 52. Monte Carlo and Dynamic Programming 5522  MC has several advantage over DP:  Can learn from interaction with environment  No need of full models  No need to learn about ALL states  No bootstrapping 52
  • 53. 5533 3. Temporal Difference (TD) methods  Learn from experience, like MC  Can learn directly from interaction with environment  No need for full models  Estimate values based on estimated values of next states, like DP  Bootstrapping (like DP)  Issue to watch for: maintaining sufficient exploration 53
  • 54. for a given policy p, compute the state-value function target: the actual return after time t 5544 TD Prediction Policy Evaluation (the prediction problem): Simple every - visit Monte Carlomethod : V(s ) V(s )+α[R V(s )] t t t t ¬ - V p ( ) : The simplest TD method, TD 0 V(s ) V(s )+α[r + V(s ) V(s )] ¬ g - t t t+ 1 t+ 1 t target: an estimate of the return 54
  • 55. Simple Monte Carlo T T T T T T T st T T T T T 55
  • 56. Simplest TD Method st+1 rt+1 st T T T T T T T T T T T T T 56 V(st )¬V(st) +a rt+1 +g V(st+1 ) - V(st [ )]
  • 57. Dynamic Programming T st rt+1 T T st+1 T T T T T T T 57 ( ) { ( )} t t 1 t V s E r gV s p ¬ + +
  • 58. 5588 TD methods bootstrap and sample  Bootstrapping: update involves an estimate  MC does not bootstrap  DP bootstraps  TD bootstraps  Sampling: update does not involve an expected value  MC samples  DP does not sample  TD samples 58
  • 59. After every transition from a nonterminal state , do this ( ) ( ) [ ( ) ( )] If is terminal, then 0. 1 1 5599 Learning An Action-Value Function Estimate Qp for the current behavior policy p . 59 Q s ,a ¬Q s ,a +α r + gQ s ,a Q s ,a t t t t t+ 1 t+ 1 t+1 t t s Q(s ,a )= s t+ t+ t+1 t -
  • 60. Q-Learning: Off-Policy TD Control 60 One - step Q- learning : Q st , at ( )¬Q st , at ( )+a rt +1 +g max [ Q(s, )- +aQ ( s, a)] t1t t a
  • 61. 6611 Sarsa: On-Policy TD Control S A R S A: State Action Reward State Action 61 Turn this into a control method by always updating the policy to be greedy with respect to the current estimate:
  • 62. 6622 Advantages of TD Learning  TD methods do not require a model of the environment, only experience  TD, but not MC, methods can be fully incremental  You can learn before knowing the final outcome  Less memory  Less peak computation  You can learn without the final outcome  From incomplete sequences  Both MC and TD converge 62
  • 63. 63 References: RL  Univ. of Alberta  http://www.cs.ualberta.ca/~sutton/book/ebook/node 1.html  Sutton and barto,”Reinforcement Learning an introduction.”  Univ. of South Wales  http://www.cse.unsw.edu.au/~cs9417ml/RL1/tdlear ning.html