Slides prepared by Miriam Bellver for the Computer Vision Reading Group at the Universitat Politecnica de Catalunya (UPC). Based on the original paper: Caicedo, Juan C., and Svetlana Lazebnik. "Active object localization with deep reinforcement learning." In Proceedings of the IEEE International Conference on Computer Vision, pp. 2488-2496. 2015.

1. Active Object Localization with Deep
Reinforcement Learning
Juan C. Caicedo & Svetlana Lazebnik (ICCV 2015)
Slides by Miriam Bellver, from the Computer Vision Reading Group. (16/02/2016)
https://imatge.upc.edu/web/teaching/computer-vision-reading-group
[Paper] [Reddit] [Slides by Jiren Jin]

2. Introduction
Goal: Localizing Objects in scenes
Efficient Strategy
Visual attention model
Active detection model: Uses an ‘agent’ to identify the correct locations
Class specific

3. Introduction
“The agent learns to deform a bounding box using simple transformation
actions, with the goal of determining the most specific location of target objects
following a top-down reasoning”
The agent is trained using Deep reinforcement learning

4. Model
Top-down search strategy
whole scene

5. Object Localization as a Dynamic Decision Process
Markov Decision Process (MDP)
Set of states S
Set of actions A
Reward function R

6. Object Localization as a Dynamic Decision Process
Set of actions A
Transformation actions

7. Object Localization as a Dynamic Decision Process
Set of actions A
Terminates the sequence of the current search
Marks the region, inhibition-of-return (IoR)

8. Object Localization as a Dynamic Decision Process
Set of states S
(o,h)
o = feature vector from pre-trained CNN fc6 : 4096 dim
h = history of taken actions binary vector dim 90

9. Object Localization as a Dynamic Decision Process
Reward Function R
ground-truthbounding box

10. Object Localization as a Dynamic Decision Process
Reward Function R for trigger action
The Reward function considers the number of steps as a cost
3
minimum
IoU:
0.6

11. Localization Policy with Reinforcement Learning
Policy function
If the current state is S, which should be the next action A?
Reinforcement Learning using a Q-learning

12. Localization Policy with Reinforcement Learning
The action-value function is estimated using a neural network that:
● has as many output units as actions
● the algorithm incorporates a replay-memory to collect experiences
● category-specific Q-network
Policy of the agent: selection action A with maximum estimated value of the
learnt action-value function.

13. Localization Policy with Reinforcement Learning

14. Localization Policy with Reinforcement Learning
● RL is in between supervised learning and unsupervised learning.
● RL is based on the interaction of an agent who executes an action and its environment who
gives to the agent positive or negative feedback. (reward)
● The agent’s aim is to optimize his actions to receive the best feedback possible

15. Localization Policy with Reinforcement Learning
Training Localization Agents
● Q-network parameters initialized at random.
● Policy used during training:
● 15 epochs, and parameters updated using stochastic gradient descent
and backpropagation.
exploration exploitation
random actions
to gather
experiences
selected actions
according policy
learnt, and learns
from the results

16. Localization Policy with Reinforcement Learning
Testing a Localization Agent
● The agent runs for max. 200 steps
● When trigger is used, the search for other object continues
● After 40 steps without triggering ---> object not found

17. Experiments and Results
Datasets for training and testing : PASCAL VOC
Two modes of evaluation:
1) All attended Regions (AAR)
2) Terminal regions (TR)

18. Experiments and Results

19. Experiments and Results
Gain

20. Experiments and Results

21. Experiments and Results

22. Experiments and Results

23. Conclusions
System localizes objects using an attention-action strategy
Reinforcement learning demonstrated to be efficient strategy to learn a
localization policy.
The system can localize a single instance of an object processing between 11
and 25 regions only, so it is a very efficient strategy
Runtime detail: If we run 200 steps per image, 1.54s is average time/image

24. The EndThank you!