This chapter shows how to use knowledge about the wlorld to make decisions even when the outcomes of an action are uncertain and the rewards for acting might not be reaped until many actions have passed. The main points are as follows: e Sequential decision problems in uncertain envirsinments,also called Markov decision processes, or MDPs, are defined by a transition model specifying the probabilistic outcomes of actions and a reward function specifying the reward in each state. o The utility of a state sequence is the sum of all the rewards over the sequence, possibly discounted over time. The solution of an MDP is a policy that associates a decision with every state that the agent might reach. An optimal policy maximizes the utility of the state sequences encountered when it is execut~ed. e The utility of a state is the expected utility of the state sequences encountered when an optimal policy is executed, starting in that state. The value iteration algorithm for solving MDPs works by iteratively solving the equations relating the utilities of each state to that of its neighbors. Policy iteration alternates between calculating the utilities of states under the current policy and improving the current policy with respect to the current utilities. * Partially observable MDPs, or POMDPs, are much more difficult to solve than are MDPs. They can be solved by conversion to an MDP in the continuous space of belief states. Optimal behavior in POMDPs includes information gathering to reduce uncertainty and therefore make better decisions in the fiuture. A decision-theoretic agent can be constructed for POMDP environments. The agent uses a dynamic decision network to represent the transition and observation models, to update its belief state, and to project forward possible action sequences. Game theory describes rational behavior for agents in situations where multiple agents interact simultaneously. Solutions of games are Nash equilibria-strategy profiles in which no agent has an incentive to deviate from the specified strategy. Mechanism design can be used to set the rules by which agents will interact, in order to maximize some global utility through the operation of individually rational agents. Sometimes, mechanisms exist that achieve this goal without requiring each agent to consider the choices made by other agents. We shall return to the world of MDPs and POMDP in Chapter 21, when we study reinforcement learning methods that allow an agent to improve its behavior from experience in sequential, uncertain environments.

1. Making Complex Decisions
Department of Computer Science & Engineering
Hamdard University Bangladesh
1

2. Sequential Decisions
• Agent’s utility depends on a sequence of decisions
• Also known as accessible, deterministic domains
 Utilities,
 Uncertainty,
 Sensing
 Generalize search
 Planning problems.
2

3. Simple Robot Navigation Problem
• In each state, the possible actions are U, D, R, and L
3

4. Probabilistic Transition Model
• In each state, the possible actions are U, D, R, and L
• The effect of U is as follows (transition model):
• With probability 0.8 the robot moves up one square
4

5. Probabilistic Transition Model
• In each state, the possible actions are U, D, R, and L
• The effect of U is as follows (transition model):
• With probability 0.8 the robot moves up one square
•With probability 0.1 the robot moves right one square
•With probability 0.1 the robot moves left one square.
5

6. Markov Property
 The transition properties depend only on
the current state, not on previous history
(how that state was reached)
6

7. Markov Decision Process (MDP)
• Defined as a tuple: <S, A, M, R>
– S: State
– A: Action
– M: Transition function
– R: Reward
• Choose a sequence of actions
- Utility based on a sequence of decisions
7

8. Generalization
 Inputs:
• Initial state s0
• Action model
• Reward R(si) collected in each state si
 A state is terminal if it has no successor
 Starting at s0, the agent keeps executing actions until it
reaches a terminal state
 Its goal is to maximize the expected sum of rewards collected
(additive rewards)
 Additive rewards: U(s0, s1, s2, …) = R(s0) + R(s1) + R(s2)+ …
 Discounted rewards
U(s0, s1, s2, …) = R(s0) + R(s1) +  2R(s2) + … (0    1)
8

9. Example MDP
• Machine can be in one of three states: good, deteriorating, broken
• Can take two actions: maintain, ignore
9

10. o To know what state have reached (accessible)
o Calculate best action for each state
- Always know what to do next!
o Mapping from states to actions is called a policy
Policies
10

11. Policies
• No time period is different from the others
• Optimal thing to do in state s should not depend on time period
– … because of infinite horizon
– With finite horizon, don’t want to maintain machine in last period
• A policy is a function π from states to actions
• Example policy: π(good shape) = ignore, π(deteriorating) = ignore,
π(broken) = maintain 11

12. Evaluating a policy
• Key observation:
MDP + policy = Markov process with rewards
• To evaluate Markov process with rewards: system of linear
equations
• Gives algorithm for finding optimal policy: try every
possible policy, evaluate
– Terribly inefficient
12

13. Bellman equation
• Suppose state s, and play optimally from there on
• This leads to expected value v*(s)
• Bellman equation:
v*(s) = maxa R(s, a) + δΣsꞌ P(s, a, sꞌ) v*(sꞌ)
• Given v*, finding optimal policy is easy
13

14. o Calculate utility of each state U (state)
o Use state utilities to select optimal action
Value iteration
14

15. Value iteration algorithm for finding
optimal policy
 Start with arbitrary utility values
 Update to make them locally consistent with bellman eqn.
 Repeat until “no change”
15

16. Policy iteration algorithm for finding
optimal policy
• Easy to compute values given a policy
– No max operator
• Policies may not be highly sensitive to exact utility values
⇒ May be less work to iterate through policies than utilities
16

17. π ← an arbitrary initial policy
repeat until no change in π
compute utilities given π (value determination)
Update π
as if utilities were correct (i.e., local MEU)
Policy Iteration Algorithm
17

18. Thanks To All
18