This PhD thesis by Gonzalo Sanguinetti from Universidad de la República in Montevideo, Uruguay examines the relationship between phenomenology, image statistics, and neuroscience in models of visual perception. The thesis contains analyses of natural image statistics, the visual cortex as a fiber bundle, and the modeling of lateral connectivity in the cortex using integral curves. Histograms of oriented wavelet responses from a large image database are presented and compared to psychophysical data on association fields in human vision.

1. Invariant models of vision between
phenomenology, image statistics and neurosciences
Gonzalo Sanguinetti
Universidad de la Rep´blica, Montevideo, Uruguay
u
Thesis Directors:
Prof. Giovanna Citti
Prof. Alessandro Sarti
March 28th, 2011
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 1 / 34

2. Outline
1 Background
2 Natural Image Statistics
Computation of the histograms
Relation with the Cortical Model
Stochastic Model
3 Scale
The symplectic model
Image Statistics
4 Ladders
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 2 / 34

3. Association Fields
Psychophysical experiment
[Field, Hayes,Hess, 1993]
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 3 / 34

4. Association Fields
Psychophysical experiment
[Field, Hayes,Hess, 1993]
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 3 / 34

5. The visual pathway and the 3 main cortical structures
L
R
L
R I
L
II
R
III
L
R IV
V
VI
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 4 / 34

6. The visual Cortex is a Fiber Bundle: R2 ×S 1
V1 Simple cells
[DeAngelis et al., 1995]
Fitting with a DoG wavelet
To each retinal point (x, y ) is associated a copy of the set S 1 .
Each point g = (x, y , θ) ∈ R2 ×S 1 represents a set of cells with the same OP
θ and RP centered at (x, y )
The space is identified with the SE (2) if it is considered with the appropriate
group operation [Citti and Sarti, 2006].
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 5 / 34

7. Lifting of Curves into the Left Invariant Structure
Non-maximal suppression
Left-invariant basis:
X1 = (cos θ, sin θ, 0)
X2 = (0, 0, 1)
X3 = (− sin θ, cos θ, 0)
associated to the diff. operators
(Xi (f ) =< X , f >):
Sub-Riemannian structure
X1 = cos θ∂x + sin θ∂y
Only the vector fields X1 and X2 are
X2 = ∂θ
considered, a subset of the tangent space.
X3 = − sin θ∂x + cos θ∂y
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 6 / 34

8. Integral curves and horizontal connections
The lateral connectivity is modeled as
integral curves with constant coefficients:
y
γ(t) = X1 (t) + k X2 (t)
˙
The solution is (for x0 = y0 = θ0 = 0)
sin(kt)

 x= k
1−cos(kt)
 y= k Θ
θ = kt
x
[Field, Hayes, Hess, 1993]
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 7 / 34

9. Outline
1 Background
2 Natural Image Statistics
Computation of the histograms
Relation with the Cortical Model
Stochastic Model
3 Scale
The symplectic model
Image Statistics
4 Ladders
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 8 / 34

10. Natural Image database
Image Database: 4000 images, 1536×1024 pixels
From http://hlab.phys.rug.nl/imlib/index.html
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 9 / 34

11. Processing of the images
Bank of oriented wavelets (steerable).
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 10 / 34

12. Processing of the images
Bank of oriented wavelets (steerable).
(x0 , y0 , θ0 )
(x1 , y1 , θ1 )
(xi , yi , θi )
(xN , yN , θN )
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 10 / 34

13. Cross-correlation assuming translation invariance
Construction of a 4D histogram
H(∆x, ∆y , θ0 , θ1 )
x1 ,y1 ,Θ1
y
xo ,yo ,Θo x
|∆x|, |∆y | < 32 px, S 1 discretized in 32 different values,
then H is large 65×65×32×32
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 11 / 34

14. Co-occurrence Histogram
H(∆x, ∆y , 0, 0) (two horizontal edges)
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 12 / 34

15. Co-occurrence Histogram
H(∆x, ∆y , 0, 0) (two horizontal edges)
1.5 107
1.0 107
5.0 106
x
30 20 10 10 20 30
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 12 / 34

16. 4D histogram
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 13 / 34

17. 4D histogram
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 13 / 34

18. 4D histogram
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 13 / 34

19. 3D histogram
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 14 / 34

20. 3D histogram
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 15 / 34

21. Comparison with the Association Fields
H(x, y , θm (x, y )) = maxθ∈S 1 H(x, y , θ) Mean error from co-circularity condition:
V (x, y ) = (cos(θm (x, y )), sin(θm (x, y )))
Eθ = n
1
x,y
θm (x, y ) − 2 arctan x
y
2
≈ 0.2rad ≈ 8◦
3Π
8
Π
4
Π
8
Θm x,y
0
Π
8
Π
4
3Π
8
3Π Π Π Π Π 3Π
4 8
0 8 4
8 8
2atan y x
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 16 / 34

22. Probabilistic framework
Mumford’s direction process
Langevin equation (SDE):
s
x(s) = 0 cos θ(t)dt + x(0)
s
y (s) = 0 sin θ(t)dt + y (0)
θ(s) = σW (s)
W (s) is a Brownian motion
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 17 / 34

23. Time dependent Fokker-Planck equation
Fokker-Planck equation associated to Mumford’s stochastic process:
σ2
∂t p = − cos(θ)∂x p − sin(θ)∂y p + ∂θθ p
2
where p(x, y , θ, t) is the transition probability from an initial state:
 
x ≤ x(t) ≤ x + ∆x x(0) = 0
p(x, y , θ, t)∆x∆y ∆θ = P  y ≤ y (t) ≤ y + ∆y y (0) = 0 
θ ≤ θ(t) ≤ θ + ∆θ θ(0) = 0
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 18 / 34

24. Time dependent Fokker-Planck equation
Fokker-Planck equation associated to Mumford’s stochastic process:
σ2
∂t p = − cos(θ)∂x p − sin(θ)∂y p + ∂θθ p
2
where p(x, y , θ, t) is the transition probability from an initial state:
 
x ≤ x(t) ≤ x + ∆x x(0) = 0
p(x, y , θ, t)∆x∆y ∆θ = P  y ≤ y (t) ≤ y + ∆y y (0) = 0 
θ ≤ θ(t) ≤ θ + ∆θ θ(0) = 0
It can be written in terms of left invariant vector fields of SE(2):
σ2
∂t p = −X1 p + X22 p, X22 = X2 (X2 ) = ∂θθ
2
advection in the direction X1 = cos θ∂x + sin θ∂y and;
diffusion in the direction of X2 = ∂θ .
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 18 / 34

25. Time independent Fokker-Planck equation
forward - backward
We propose to use the fundamental solution of the time independent
equation, i.e. the solution of
σ2 1
−X1 p(x, y , θ) + X22 p(x, y , θ) = δ(x, y , θ)
2 2
plus the fundamental solution of its backward equation:
σ2 1
X1 p(x, y , θ) +
X22 p(x, y , θ) = δ(x, y , θ)
2 2
Only one free parameter: σ, the variance of the underlying stochastic process.
The solution was numerically computed using COMSOL Multiphysics, a
commercial FEM solver.
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 19 / 34

26. Fitting with the statistics.
σ = 1.7px minimizes the mean square error E . At the minimum, E ≈ 2%
Fokker-Planck fundamental solution. Histogram of co-occurrences.
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 20 / 34

27. Comparison with the probabilistic model
Level curves of the Fokker-Planck fundamental solution
Fokker−Plank fundamental solution level sets: θ = 0 x 10
−4 Fokker−Plank fundamental solution level sets: θ = 11.25 x 10
−4 Fokker−Plank fundamental solution level sets: θ = 22.5 x 10
−4
30 30 5 30
8
2.5
4.5
20 7 20 20
4
6 2
10 10 3.5 10
5
3
∆y
∆y
∆y
0 0 0 1.5
4 2.5
−10 −10 2 −10
3
1
1.5
2
−20 −20 −20
1
1 0.5
−30 −30 0.5 −30
−30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30
∆x ∆x ∆x
Fokker−Plank fundamental solution level sets: θ = 45 x 10
−5 Fokker−Plank fundamental solution level sets: θ = 56.25 x 10
−5 Fokker−Plank fundamental solution level sets: θ = 67.5 x 10
−5
9 6.5
30 30 30
12
11 6
8
20 20 20
10
5.5
7
10 10 10
9
5
8
6
∆y
∆y
∆y
0 0 0
4.5
7
−10 −10 5 −10
6 4
5
−20 −20 4 −20 3.5
4
3
−30 −30 3 −30
3
−30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30
∆x ∆x ∆x
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 21 / 34

28. Comparison with the probabilistic model
Level curves of the histogram
Co−occurrences histogram level sets: θ = 0 x 10
−4 Co−occurrences histogram level sets: θ = 11.25 x 10
−4 Co−occurrences histogram level sets: θ = 22.5 x 10
−4
30 30 5 30 2.5
8
4.5
20 7 20 20
4 2
6
10 10 3.5 10
5 3
1.5
∆y
∆y
∆y
0 0 0
2.5
4
−10 −10 2 −10
1
3
1.5
−20 −20 −20
2 1
0.5
−30 −30 0.5 −30
1
−30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30
∆x ∆x ∆x
Co−occurrences histogram level sets: θ = 45 x 10
−5 Co−occurrences histogram level sets: θ = 56.25 x 10
−5 Co−occurrences histogram level sets: θ = 67.5 x 10
−5
30 30 30 6.5
12
8
11 6
20 20 20
10
7 5.5
10 10 10
9
5
6
8
∆y
∆y
∆y
0 0 0
4.5
7
5
−10 −10 −10 4
6
5
4 3.5
−20 −20 −20
4
3
−30 −30 3 −30
3
−30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30
∆x ∆x ∆x
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 21 / 34

29. Outline
1 Background
2 Natural Image Statistics
Computation of the histograms
Relation with the Cortical Model
Stochastic Model
3 Scale
The symplectic model
Image Statistics
4 Ladders
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 22 / 34

30. Extension to the affine group [Sarti, Citti, Petitot, 2008]
Invariant under translations, rotations and scaling transformations
2
+η 2 )
ϕ0 (ξ, η) = e −(ξ cos(2η)
ϕx,y ,θ,σ (ξ, η) = ϕ0 A−1 (ξ, η)
Ax,y ,θ,σ (ξ, η) =
x cos θ − sin θ ξ
+ eσ
y sin θ cos θ η
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 23 / 34

31. Lifting into R2 × S 1 × R+
Geometric interpretation
The scale parameter σ may be interpreted as the distance to boundary.
Θ
1
eΣ
2
x,y
Oθm ,σm (x, y ) = max Oθ,σ (x, y )
(θ,σ)
The function O represents the output of the cells, i.e. the cortical activity.
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 24 / 34

32. Space differential structure and connectivity
Symplectic structure, 2 sub-Riemannian metrics.
y y
Left Invariant Vector Fields:
Θ Σ
Xσ,1 = e σ (cos θ∂x + sin θ∂y )
Xσ,2 = ∂θ
Xσ,3 = e σ (− sin θ∂x + cos θ∂y )
Xσ,4 = ∂σ x x
2 types of Integral curves:
γ(t) = Xσ,1 (γ(t)) + kXσ,2 (γ(t))
˙
γ(0) = (x0 , y0 , θ0 , σ0 ) y
γ(t) = Xσ,3 (γ(t)) + kXσ,4 (γ(t))
˙
γ(0) = (x0 , y0 , θ0 , σ0 )
x
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 25 / 34

33. Image Statistics
Same methodology
Even symmetric filters, implemented with the steerable architecture:
Each detected edge is a 4d point (x, y , θ, σ)
Histogram of co-occurrences (translation invariance assumption) is 6D
H(∆x, ∆y , θc , θp , σc , σp )
|∆x|, |∆y | ≤ 16px, 8 different orientation, 10 different scales.
Dimensions of H, 33 × 33 × 8 × 8 × 10 × 10
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 26 / 34

34. Results
Visualization of 5 dimensions
σc = 3px fixed at the lowest scale
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 27 / 34

35. Plane spanned by {X3,σ , X4,σ }
θc , θp are fixed
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 28 / 34

36. The model of the connectivity.
Integral curves of X3,σ + αX4,σ :
y
Pure advection in the directions:
Xσ,3 − Xσ,4 and Xσ,3 + Xσ,4 .
Σ
x
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 29 / 34

37. Test on the synthetic cartoon image database
100 images randomly generated
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 30 / 34

38. Outline
1 Background
2 Natural Image Statistics
Computation of the histograms
Relation with the Cortical Model
Stochastic Model
3 Scale
The symplectic model
Image Statistics
4 Ladders
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 31 / 34

39. Snakes vs Ladders
Psychophysical experiment
[May-Hess, 2008]
“increasing the separation between the
elements had a disruptive effect on the
detection of snakes but had no effect on
ladders, so that as separation increased,
performance on the two types
converged”
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 32 / 34

40. Reinterpretation of the results
s s
Α Α
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 33 / 34

41. Reinterpretation of the results
s s
Α Α
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 33 / 34

42. Natural Image Statistics
Snake vs Ladder
1.0
0.9
0.8
0.7
0.6
0.5
1.09 1.54 2.18 3.08 0.4
1.09 1.54 2.18 3.08
1.0 0.30
0.25
0.8
0.20
0.6
0.15
0.4
0.10
0.2
0.05
0.0 0.00
40 20 0 20 40 10 15 20 25 30 35 40 45
Gonzalo Sanguinetti (Udelar) Phd thesis March 28th, 2011 34 / 34