This document discusses closing the loop between biological and artificial vision by using deep convolutional neural networks (DCNNs) to model human vision. Specifically, it discusses: 1) Research showing DCNNs develop representations that become more similar to human brain regions involved in processing faces and places as the networks become more specialized for those categories. 2) Methods for using optimized images to visualize feature differences between DCNN layers and brain regions. 3) Opportunities for closing the loop by using DCNNs to efficiently test models of human vision, identify features detected in different brain regions, and find differences between brain regions.

1. Closing the loop between biological and
artificial vision
Dr Kate Storrs, MRC Cognition & Brain Sciences
Unit, Cambridge
@katestorrs

2. @pintofscience #pint17www.xkcd.com

3. @pintofscience
why vision is
“virtually impossible”

4. @pintofscience #pint17
why vision is
“virtually impossible”

5. 1021 (Alhazen)
- 1604 (Kepler)
visual neuroscience tl;dr
1900
(Cajal) 1950s
1990s
2000s

6. deep conv nets: inspired by the visual system

7. convergent solutions?
Mahendran & Vedaldi (2015)

8. Brain data
1
2
...
62
...
activation pattern to
each image in a region
. . .
. . .
. . .
1
2
62
...
activation pattern to
each image in a layer
. . .
. . .
. . .
1
2
62
correlate
VGG
DCNN data
1
2
...
62
Khaligh-Razavi & Kriegeskorte et al. (2014)
are deep nets solving vision like the brain does?
Representational
dissimilarity matrix

9. specialised models for specialised areas
“face
region”
(FFA)
“place
region”
(PPA)

10. 0.1
0.2
0.3
network depth
5
10
15
representationalsimilarity
(Kendalltau-aaveragedacross24participants)
0
0.1
0.2
0.3
network depth
5
10
15
face region (FFA)
place region (PPA)
specialised nets for specialised areas
network layer
network layer
best performance possible
best performance possible
similaritytohumanbrain
Storrs et al. (unpublished)

11. 0.1
0.2
0.3
network depth
5
10
15
representationalsimilarity
(Kendalltau-aaveragedacross24participants)
0
0.1
0.2
0.3
network depth
5
10
15
face region (FFA)
place region (PPA)
network layer
network layer
best performance possible
best performance possible
similaritytohumanbrain
Storrs et al. (unpublished)
random network
random network
specialised nets for specialised areas

12. representationalsimilarity
(Kendalltau-aaveragedacross24participants)
0
0.1
0.2
0.3
network depth
5
10
15
0.1
0.2
0.3
network depth
5
10
15
face region (FFA)
place region (PPA)
network layer
network layer
best performance possible
best performance possible
similaritytohumanbrain
Storrs et al. (unpublished)
random network
object-trained
random network
object-trained
specialised nets for specialised areas

13. representationalsimilarity
(Kendalltau-aaveragedacross24participants)
0
0.1
0.2
0.3
network depth
5
10
15
0.1
0.2
0.3
network depth
5
10
15
face region (FFA)
place region (PPA)
network layer
network layer
best performance possible
best performance possible
similaritytohumanbrain
Storrs et al. (unpublished)
random network
object-trained
face-trained
random network
object-trained
specialised nets for specialised areas

14. 0.1
0.2
0.3
network depth
5
10
15
representationalsimilarity
(Kendalltau-aaveragedacross24participants)
0
0.1
0.2
0.3
network depth
5
10
15
face region (FFA)
place region (PPA)
network layer
network layer
best performance possible
best performance possible
similaritytohumanbrain
Storrs et al. (unpublished)
random network
object-trained
face-trained
random network
object-trained
face-trained
specialised nets for specialised areas

15. representationalsimilarity
(Kendalltau-aaveragedacross24participants)
0
0.1
0.2
0.3
network depth
5
10
15
0.1
0.2
0.3
network depth
5
10
15
face region (FFA)
place region (PPA)
network layer
network layer
best performance possible
best performance possible
similaritytohumanbrain
Storrs et al. (unpublished)
random network
object-trained
face-trained
place-trained
random network
object-trained
face-trained
specialised nets for specialised areas

16. 0.1
0.2
0.3
network depth
5
10
15
representationalsimilarity
(Kendalltau-aaveragedacross24participants)
0
0.1
0.2
0.3
network depth
5
10
15
face region (FFA)
place region (PPA)
network layer
network layer
best performance possible
best performance possible
similaritytohumanbrain
Storrs et al. (unpublished)
random network
object-trained
face-trained
place-trained
random network
object-trained
face-trained
place-trained
specialised nets for specialised areas

17. input image
Optimisation:
Update pixels to ‘caricature’ this
response pattern.
https://github.com/google/deepdream
optimised
image
visualising feature differences

18. faces-VGG places-VGG
random seed image

19. faces-VGG places-VGG
layer 4

20. faces-VGG places-VGG
layer 7

21. faces-VGG places-VGG
layer 13

22. Matchtobrain
A B C
B
C
A
candidate models A B C
process candidate stimuli
maximise model disagreement
get brain data
evaluate
closing the loop
…to efficiently
test many
models.

23. closing the loop
…to find
the features
detected in
a brain
region.
show stimuli
measure brain activity
measure network activations
predict brain activity from features
optimise

24. closing the loop
…to find
differences
between
brain
regions.
stimuli

25. stimuli
evaluate similarity of stimuli
find informative stimuli
optimise
get brain data
FFA
OFA
OFA
FFA
Close in OFA; far in FFA
closing the loop
…to find
differences
between
brain
regions.

26. closing the loop

27. Deep nets: our first fully explicit models of human vision!
• fully interrogate-able
• + easy optimisation
• = new opportunities!

28. @pintofscience #pint17
Collaboration with artist
Claudia Flandoli

29. @pintofscience #pint17
Collaboration with artist
Claudia Flandoli

30. 1

31. 1

32. 1

33. 2

34. 2

35. 2

36. 3

37. 3

38. 3

39. 4

40. 4

41. 4

42. Thanks!
@katestorrs
Niko Kriegeskorte (PI)
Alex Walther, Hannes Mehrer (collaborators)