1) The document summarizes a research paper on recurrent environment simulators that use deep learning models to predict the next observation in an environment given the previous state and action. 2) The model was tested on Atari games to predict the next video frame based on the previous frame and the agent's action. 3) The results showed that the model was able to achieve state-of-the-art performance on the Atari games by learning to accurately predict future frames both when using predicted frames and observed frames from the environment.

1. 第1回 最新のML，CV，NLP
関連論文読み会
2017/06/18
@aimpast

2. Introduce myself
名前: 蓑手智紀 （みのて ともき）
経歴: 都立高専 → 豊橋技術科学大学 学部卒 → ABEJA入社（2017年4月）
専攻: Computer Science
研究の一環で深層強化学習でレースゲームをプレイするエージェントを作ったりしました。
2

3. Recurrent Environment Simulators
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra & Shakir Mohamed
DeepMind, London, UK

4. Reinforcement Learning
Which is the best action in current condition?
4
Hall
Hall Player Hall Goal
Hall Wall Wall Wall
Wall Wall
Wall Wall Wall Wall Wall

5. Reinforcement Learning
5
Agent
Environment
state reward action
st rt at
• The agent interacts the environment at every time-step

6. • The agent observes current state
Reinforcement Learning
6
Agent
Environment
state
st
action
at
st
• And it selects action at

7. Reinforcement Learning
7
Agent
Environment
state reward action
atst+1 rt+1
• The action is passed to environment and modifies its state
• The agent receives next state and rewardst+1 rt+1

8. • The goal of the agent is to interact with environment in a way
that maximizes future reward.
• the future discounted future return at time t is given by
• The optical strategy is to maximize the expectation.
Reinforcement Learning
8
V⇡ = rt + rt+1 + 2
rt+2 =
1X
k=0
k
rt+k
Q⇡
(s, a) = E⇡ {V⇡|st = s, at = a}
: discount factor < 1.0

9. Abstract
• This research topic is a part of Reinforcement Learning.
• They introduce the model can predict next observation from previous
state and action and predicted/observed state.
• Models that can simulate how environments change in response to
actions can be used by agents to plan and act efficiently.
9

10. Introduction
10
Oh et al., ’Action-Conditional Video Prediction using Deep Network in Atari Games’
: observed frame (pixel)
: predicted frame (pixel)

11. Introduction
11

12. Introduction
12
: State
: state transition function : encoding function
: predicted frame (pixel)
: observed frame (pixel)
: decoding function
: action
: return or
Action-Dependent State Transition

13. Introduction
13
: State
: state transition function : encoding function
: predicted frame (pixel)
: observed frame (pixel)
: decoding function
: action
: return or
Prediction-Dependent State Transition

14. LSTM Block
14
tanh
+
σ
+
+
σ +
tanh
σ+
output
recurrent
recurrent
recurrent
recurrent
recurrent
input
input
input
input
f
i
o
output gate
input gate
forget gate
at-1
at-1
at-1
at-1

15. Recurrent Environment Simulators
The selection of the predicted frame or real frame
15
: Prediction-dependent transition
: Observation-dependent transition

16. Experiment
• 0% PDT
• 33% PDT
• 0%-20%-33%PDT
• Only observation-dependent transitions in the first 10,000 parameter updates
• Prediction-dependent:Observation-dependent= 8:2 for the the subsequent
100,000 parameters updates
• PDT:ODT=1:2 after the subsequence 100,000 parameters updates
• 46% PDT Alt.: Alternate between ODT and ODT from a time-step to the next.
• 46% PDT
• 67% PDT
• 0%-100% PDT:
• Only observation-dependent transitions in the first 1000 parameter updates
• Only prediction-dependent transitions in the subsequent parameter updates.
• 100% PDT
16

17. Train
The model Trained to minimize the mean squared error (MSE)
error between the observed time-series and predicted time-
series.
17

18. Results
18

19. Results
19

20. Results
20

21. Results
21

22. Discussion
We showed state-of-the-art results on Atari, and demonstrated the feasibility of
live human play in all three task families. The system is able to capture
complex and long-term interactions, and displays a sense of spatial and
temporal coherence that has, to our knowledge, not been demonstrated on
high-dimensional time-series data such as these.
Complex environments have compositional structure, such as independently
moving objects and other phenomena that only rarely interact. In order for our
simulators to better capture this compositional structure, we may need to
develop specialised functional forms and memory stores that are better suited
to dealing with independent representations and their interlinked interactions
and relationships.
22