The paper introduces Value Iteration Networks, which utilize Convolutional Neural Networks (CNNs) to interpret the value iteration algorithm in reinforcement learning. It highlights the effectiveness of CNNs for structured prediction and path planning tasks, proposing a fully differentiable neural network with an embedded planning module. The authors conclude that this innovative approach has significant potential for solving inverse reinforcement learning challenges.

1. Value Iteration
Networks
Aviv Tamar, Sergey Levine, and Pieter Abbeel
Presenter: Sungjoon Choi
arXiv:1602.02867v1 [cs.AI] 9 Feb 2016

2. This paper can be used for

3. Convolutional Networks
Today, we will see a very clever interpretation of CNN !
CNN is not just used for efficient feature extractor but this
paper finds an analogy between operations in CNN and
value iteration algorithm in reinforcement learning.

4. Convolutional Networks
When it comes to an image processing, CNN is used
in almost Everywhere!

5. Structured Prediction?
Structured prediction is an umbrella term for supervised
machine learning techniques that involve predicting
structured objects, rather than scalar discrete or real values.

6. Path Planning?

7. Why not just End to End?

8. Is it Deep Q Learning?
No, it is different.
DQN only models the Q-function with CNN.

9. Reinforcement Learning
We only get the reward at certain points.
What makes RL different from other methods?
But we have to make decision every time.

10. RL: Value Iteration
So, we introduce the notion of value.
And of course, ways to find the value function.

11. Value Iteration via CNN?
This papers says
“ We introduce the value iteration network: a fully
differentiable neural network with a panning module
embedded within.”

12. Value Iteration via CNN?

13. Value Iteration Block

14. Value Iteration Block
The depth of the Q layer need not to be the same as the
number of actions.

15. Value Iteration Network
VI Block

16. Value Iteration Network
Or just a feature extraction stage. (I guess)

17. Hierarchical VI Network

18. Grid-World Experiment

19. Grid-World Experiment
Input: Sequence of states (locations)
Output: Sequence of actions (controls)

20. Grid-World Experiment
Value Iteration Network vs. Direct Policy Learning

21. Mars Rover Navigation

22. Conclusion
Very clever idea of using CNN as a building
block for solving inverse reinforcement
learning problem!
Make things differentiable and use deep
networks, deep learning tools will take care
of the rest.
Still conceptual level, but potentials are
limitless