This document discusses using random network distillation to provide intrinsic rewards for reinforcement learning agents. It summarizes a previous study that used prediction error as an intrinsic reward across diverse environments. The document then proposes using random network distillation, where a randomly initialized fixed network predicts the outputs of a randomly initialized training network, to provide intrinsic rewards. This allows scaling to high-dimensional observations and solving problems with stochastic dynamics. Experiments are proposed to test combining intrinsic rewards from random network distillation with extrinsic rewards on diverse environments.

1. Exploration By Random
Network Distillation
권휘

2. Intrinsic vs Extrinsic Reward
reward extrinsic reward
(environment’s reward
intrinsic reward
(exploration bonus)

3. Intrinsic Reward
From P.-Y. Oudeyer., et al., Intrinsic Motivation Systems for Autonomous Mental Development, IEEE TRANSACTIONS ON EVOLUTIONARY
COMPUTATION, 2007.
Check also P.-Y. Oudeyer and F. Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 2009.

4. Intrinsic Reward
Tabular case:
visitation count

5. Intrinsic Reward
Non-tabular case: pseudo-counts, prediction
error(forward dynamics, inverse dynamics, …)

6. Previous study (‘18 Aug)
From https://blog.openai.com/reinforcement-learning-with-prediction-based-rewards/
(Forward dynamics)
Pixels
Random Features (RF)
Variational Autoencoder (VAE)
Inverse Dynamics Features (IDF)
compact, sufficient, stationary

7. Previous study (‘18 Aug)
- Diverse environments(48 Atari, Mario, 2 Roboschool, 2 player Pong,
2 Unity Mazes)
- Large-Scale(~2048 parallel envs)
- Curiosity(Intrinsic reward) only
- Infinite horizon
- Stabilization techniques
- Limitation: stochastic dynamics(Noisy-TV)
Check Yuri Burda et al., Large-Scale Study of Curiosity-Driven Learning, arXiv:1808.04355v1, 2018

8. Environment

9. Environment
From https://blog.openai.com/reinforcement-learning-with-prediction-based-rewards/

10. Random Network Distillation (‘18 Oct)
From https://blog.openai.com/reinforcement-learning-with-prediction-based-rewards/
Random initialized fixed network
Random initialized training network

11. Contribution
1. Works well with high-dimensional observations
2. Easily scale up to large numbers of parallel environments
3. Solve stochastic dynamic problem
4. Intrinsic + Extrinsic reward
5. Get high score in Montezuma’s Revenge (pass 1st level)

12. Sources of Prediction Errors
1. Amount of training data
2. Stochasticity
3. Model misspecification
4. Learning dynamics

13. Relation to Uncertainty Quantification

14. Relation to Uncertainty Quantification
From Osbald et al., Randomized Prior Functions for Deep Reinforcement Learning, arXiv:1806.03335v2, 2018.

15. Intrinsic + Extrinsic returns
1. Intrinsic: Non-episodic, separate value function( )
2. Extrinsic: Episodic, separate value function( )
3. Reward and Observation Normalization

16. Experiments
Condition:
- 30K rollouts of length 128 per environments with 128 parallel environments.
- Discount factor(0.99 vs 0.999)
- RNN vs CNN
- # of parallel envs
Metrics:
- Mean episodic return
- the number of rooms the agent finds over the training run.

17. Experiments

18. Experiments

19. Pseudo-code
PPO
Get observation
initial statistics
왜?