This document discusses multi-objective reinforcement learning and introduces Deep OLS Learning, which combines multi-objective learning with deep Q-networks. It presents Deep OLS Learning with Partial Reuse and Full Reuse to handle multi-objective Markov decision processes by finding a convergence set of policies that optimize multiple conflicting objectives, such as maximizing server performance while minimizing power consumption. The approach is evaluated on multi-objective versions of mountain car and deep sea treasure problems.

1. Multi Objective DRL
https://arxiv.org/pdf/1610.02707.pdf
Dealing Multi Criteria

2. Before start
Single-objective Markov Decision Process
optimal action-value function ( Q function )
Deep Q Network
Minimizing loss

3. Before start
Multi-Objective MDP
rl : maximization of sum of received scalar signal
Linear case
http://roijers.info/motutorial.html

4. Before start
Multi-Objective MDP
Multiple policy

5. Before start
Optimistic Linear Support (OLS)
Linear Support

6. Introduction
DQN, AlphaGo, Robot Control
=> Focus on Single-objective
In real world -> multiple conflicting objectives
Maximize
performance of server
Minimize
Power consumption
=> No single optimal policy
=> Convergence set with (multiple policies)

7. Method
Deep OLS learning : multi objective learning + DQN
-> Deep OLS Learning with Partial (DOL-PR), Full Reuse (DOL-FR)
> Deep OLS Learning
Just Do deep Q-Learning on each step

8. Method
Deep OLS learning : multi objective learning + DQN
-> Deep OLS Learning with Partial (DOL-PR), Full Reuse (DOL-FR)
> Deep OLS Learning
If max value
https://github.com/hossam-mossalam/multi-
objective-deep-rl
DOL
DOL-FR
DOL-PR

9. Experiment
multi-objective mountain car problem
single-objective
variant is −1 for all
time steps and 0
when the goal is
reached
fuel consumption for
each time step, which
is proportional to the
force exerted by the
car

10. Experiment
Deep Sea Treasure
a treasure value was
received
a time penalty of −1
for each time-step