W04

1. Exploit vs. Explore

2. Multi-Armed Bandit problem
Image: medium.com/10x-curiosity
Exploit vs Explore

3. MORE ABOUT
ENVIRONMENTS

4. Types of environments
Fully observable: agent knows completely about its current state
Partially observable: agent knows just a part of its current state
Unobservable: agent don’t knows anything its current state.

5. Types of environments
Single agent vs. multiagent

6. Episodic vs. sequential
Types of environments

7. Other types:
 Deterministic vs. stochastic
 Discrete vs. continuous
 Static vs. dynamic
 …
Learn more from Section 2.3 of
(Russell, 2016)
Types of environments

8. State representations
Source: (Russell, 2016)

9. MORE ABOUT
AGENTS

10. PEAS description of agents
Source: (Russell, 2016)

11. Source: (Russell, 2016)

12. Basic kinds of agents

13. Simple reflex agents
Source: (Russell, 2016)
Agents that select actions based ONLY on the current percept, ignoring the previous percepts.
For example,

14. Simple reflex agents can NOT work well in cases, e.g., partially
observable environments.

15. Model-based reflex agents
Agents that try to "guess" the unobserved part of the environment by using some model of the
environment.

16. Goal-based agents
The mentioned agents are reflex ones: they act just according to the state of the environment.
Goal-based agents select action regarding both state and its goal.
For example, when seeing brake lights of the car in front of it:
+ A reflex agent
+ A goal-based agent, with the goal "not hitting other cars",
The goal-based agent is more flexible.

17. Utility-based agents
Goals just provide a binary distinction between “DONE” and “NOT DONE” states.
Utilities consider HOW WELL it is done.
For example,

18. Learning agents
State-of-the-art systems.
A learning agent includes 4 components:
+ Performance element:
+ Critic:
+ Learning element:
+ Problem generator:

19. Example: A taxi agent
+ It drives using the
+ The critic observes the world and gives feedback to the learning element. For example,
+ See this feedback, the learning element creates a rule to
and gives it to the performance element to use.
+ The problem generator might identify some behaviors in need of improvement and suggest
experiments, such as

20. SOLVING PROBLEMS BY
SEARCHING
Main reference:
Chapters 3 of Russell, S., & Norvig, P. (2016). Artificial intelligence: a modern approach.

21. Focus on problem-solving agents
 A kind of goal-based agent
 Uses atomic representation for environment state
Assumptions of the environment:
 Oversable:
 Discrete:
 Deterministic:

22. Concepts
Goal:
Solution:
Optimal solution:
Search:
State space:
Level of abstraction:

23. Problem description using graph
Node:
Edge:
Path:

24. BẾN TRE

25. Components of a problem
1. Initial state: a state that the agent starts in.
For example,
2. Possible actions: actions that are applicable to the agent at a given state.
Notation: ACTIONS(state s).
For example,
3. Transition model: result of an action in a state.
Notation: RESULT(state s, action a).
For example,
Note:
Successor:

26. 4. Goal test: determines whether a state is a goal state.
For example,
5. Cost: a performance measure (lower is better).
Step cost:
Path cost:

27. Example problem 1
Source: (Russell, 2016)

28. 1. Initial state:
2. Possible actions:
3. Transition model:
4. Goal test:
5. Cost:

29. Example problem 2
Source: (Russell, 2016)

30. 1. Initial state:
2. Possible actions:
3. Transition model:
4. Goal test:
5. Cost:

31. Exercise: Route-finding problem
Fomulate the problem of finding routes to go from one city/province to another.
Cities/provinces are shown on the map given in the next slide.
FYI: Route-finding algorithms are used in package routing on the Internet, airline travel-planning systems and so on.
1. Initial state:
2. Possible actions:
3. Transition model:
4. Goal test:
5. Cost:

32. BẾN TRE

33. Other real-world problems
Touring problems
The traveling salesperson problem (TSP)
A VLSI layout problem
Robot navigation
Automatic assembly sequencing
See section 3.2.2 in (Russell, 2016) for more details.