View stunning SlideShares in full-screen with the new iOS app!Introducing SlideShare for AndroidExplore all your favorite topics in the SlideShare appGet the SlideShare app to Save for Later — even offline
View stunning SlideShares in full-screen with the new Android app!View stunning SlideShares in full-screen with the new iOS app!
1.
Optimal Nudging
A new approach to solving SMDPs
Reinaldo Uribe M
Universidad de los Andes — Oita University
Colorado State University
Nov. 11, 2013
2.
Snakes & Ladders
Player advances the
number of steps indicated
by a die.
Landing on a snake’s
mouth sends the player
back to the tail.
Landing on a ladder’s
bottom moves the player
forward to the top.
Goal: reaching state 100.
3.
Snakes & Ladders
Player advances the
number of steps indicated
by a die.
Boring!
(No skill required, only luck.)
Landing on a snake’s
mouth sends the player
back to the tail.
Landing on a ladder’s
bottom moves the player
forward to the top.
Goal: reaching state 100.
4.
Variation: Decision Snakes and Ladders
Sets of “win” and
“loss” terminal states.
Actions: either
“advance” or “go
back,” to be decided
before throwing the die.
5.
Reinforcement Learning: Finding an optimal policy.
“Natural” Rewards: ±1
on “win”/“lose”, 0
othw.
Optimal policy
maximizes total
expected reward.
Dynamic programming
quickly ﬁnds the
optimal policy.
Probability of winning:
pw = 0.97222 . . .
6.
We know a lot!
Markov Decision Process: States, Actions, Transition
Probabilities, Rewards.
7.
We know a lot!
Markov Decision Process: States, Actions, Transition
Probabilities, Rewards.
Policies and policy value.
8.
We know a lot!
Markov Decision Process: States, Actions, Transition
Probabilities, Rewards.
Policies and policy value.
Max winning probability = max earnings.
9.
We know a lot!
Markov Decision Process: States, Actions, Transition
Probabilities, Rewards.
Policies and policy value.
Max winning probability = max earnings.
Taking an action costs (in units diﬀerent from rewards.)
10.
We know a lot!
Markov Decision Process: States, Actions, Transition
Probabilities, Rewards.
Policies and policy value.
Max winning probability = max earnings.
Taking an action costs (in units diﬀerent from rewards.)
Diﬀerent actions may have diﬀerent costs.
11.
We know a lot!
Markov Decision Process: States, Actions, Transition
Probabilities, Rewards.
Policies and policy value.
Max winning probability = max earnings.
Taking an action costs (in units diﬀerent from rewards.)
Diﬀerent actions may have diﬀerent costs.
Semi-Markov model with average rewards.
13.
Better than optimal?
(Optimal policy) with
average reward
ρ = 0.08701
14.
Better than optimal?
(Optimal policy) with
average reward
ρ = 0.08701
pw = 0.48673 (was
0.97222 — 50.06%)
d = 11.17627 (was
84.58333 — 13.21%)
15.
Better than optimal?
(Optimal policy) with
average reward
ρ = 0.08701
pw = 0.48673 (was
0.97222 — 50.06%)
d = 11.17627 (was
84.58333 — 13.21%)
This policy maximizes
pw
d
16.
So, how are average-reward optimal policies found?
Algorithm 1 Generic SMDP solver
Initialize
repeat forever
Act
Do RL to ﬁnd value of current π
Update ρ.
Usually 1-step Q-learning
Average-adjusted Q-learning:
Qt+1 (st , at ) ← (1 − γt ) Qt (st , at ) + γt rt+1 − ρt ct+1 + max Qt (st+1 , a)
a
17.
Generic Learning Algorithm
Table of algorithms. ARRL
Algorithm
Gain update
t
r(si , π i (si ))
AAC
Jalali and Ferguson 1989
R–Learning
ρt+1 ←
ρ
t+1
Tadepalli and Ok 1998
SSP Q-Learning
t+1
← (1 − α)ρt +
α rt+1 + max Qt (st+1 , a) − max Qt (st , a)
Schwartz 1993
H–Learning
i=0
a
ρt+1 ← ρt + αt min Qt (ˆ, a)
s
a
Abounadi et al. 2001
t
r(si , π i (si ))
HAR
Ghavamzadeh and Mahadevan 2007
a
ρt+1 ← (1−αt )ρt +αt rt+1 − H t (st ) + H t (st+1 )
αt
αt+1 ←
αt + 1
ρt+1 ←
i=0
t+1
18.
Generic Learning Algorithm
Table of algorithms. SMDPRL
Algorithm
Gain update
SMART
t
r(si , π i (si ))
Das et al. 1999
ρt+1 ←
i=0
t
MAX-Q
Ghavamzadeh and Mahadevan 2001
c(si , π i (si ))
i=0
19.
Nudging
Algorithm 2 Nudged Learning
Initialize (π, ρ, Q)
repeat
Set reward scheme to (r − ρc).
Solve by any RL method.
Update ρ
until Qπ (sI ) = 0
20.
Nudging
Algorithm 3 Nudged Learning
Initialize (π, ρ, Q)
repeat
Set reward scheme to (r − ρc).
Solve by any RL method.
Update ρ
until Qπ (sI ) = 0
Note: ‘by any RL method’ refers to a well-studied problem for
which better algorithms (both practical and with theoretical
guarantees) exist.
ρ can (and will) be updated optimally.
22.
The w − l space.
Value and Cost
(Policy π has expected average reward v π and expected average
cost c π . Let D be a bound on the absolute value of v π )
wπ =
D + vπ
,
2c π
lπ =
D
5D
−0.
1
l
0
−D
D
l
D − vπ
.
2c π
2
0.5D
4
8
D
w
D
w
D
29.
Sample task: two states, continuous actions
Policy Space (Actions)
1
a2
0
0
a1
1
30.
Sample task: two states, continuous actions
Policy Values and Costs
Policy value
4
Policy cost
4
31.
Sample task: two states, continuous actions
Policy Manifold in w − l
l
D/2
w
D/2
32.
And the rest...
Neat geometry, linear problems in w − l.
Easily exploited using straightforward algebra / calculus.
Updating average reward between iterations can be optimized.
Becomes ﬁnding the (or rather an) intersection between two
conics.
Which can be solved in O(1) time.
33.
And the rest...
Neat geometry, linear problems in w − l.
Easily exploited using straightforward algebra / calculus.
Updating average reward between iterations can be optimized.
Becomes ﬁnding the (or rather an) intersection between two
conics.
Which can be solved in O(1) time.
Worst case, uncertainty reduces in half.
Typically much better than that.
Little extra complexity added to already PAC methods.
34.
Thank you.
r-uribe@uniandes.edu.co
Untitled by Li Wei, School of Design, Oita University, 2009.
Views
Actions
Embeds 0
Report content