Introduction to harmonic analysis on groups, links with spatial correlation.

Introduction to harmonic analysis on groups,
links with spatial correlation.
Valentin De Bortoli
March 16, 2018
ENS Paris Saclay, Université Paris Descartes
1 / 36

Table of contents
1. Motivations and background
Two introductory problems
Correlation and geometry (3D)
Correlation and geometry (2D)
Representation theory toolbox
2. Spherical CNN
Theoretical grounds
Experiments and interests
3. The Special Euclidean motion group
Unitary representations
Embedding and consequences
4. Conclusion
2 / 36

Signal processing on the sphere
*
Problem
S2 is not a group. How to convolve two spherical signals?
3 / 36

Two convolutions?
Let (X, Y ) ∈ L2(S2)
2
R → X ∗ Y (R) :=
S2
X(u)Y (R−1
u) du ∈ SO3(R) = S2
.
Thus:
• S2 ∗ S2 ∈ SO3(R),
• SO3(R) ∗ SO3(R) ∈ SO3(R).
Question
How to embed these convolutions into a mutual framework?
4 / 36

A detection problem
Where are the squares? → ﬁltering → thresholding output image.
5 / 36

Introducing rotations
Previous strategy → FAILURE!
6 / 36

A solution
Rotating ﬁlter → Multiple ﬁltering → thresholding output stack.
7 / 36

Group action and images
“Different group actions yield different spatial correlations.”
• R2 (translations)
• SO2(R) (or SO3(R)) (rotations)
• SE2(R) (direct isometries)
• E2(R) (isometries)
• Aff2(R) (affine transformations)
• GP2(R) & GP1(C) (homographies)
8 / 36

Our goal
So, what is our goal?
1. Introduce a “convolution” with respect to other groups than
R2, i.e
f ∗
G
h ≈
x∈X
f (x)h(g−1
x) dx.
2. Introduce a “Fourier transform” w.r.t other groups,
3. Design discrete counterparts for numerical implementation
(G-FFT?), see Maslen and Rockmore (1997).
⇒ We need a group theoretical point of view of usual signal
processing tools Sugiura (1990), Chirikjian and Kyatkin (2000).
9 / 36

Group actions and representations
Group action
Let X be a set and G a group.
• G acts on X if there exists a group-morphism
π : (G, ∗) → (Bij(X), ◦). π is a group action.
• Suppose G topological and X := H a Hilbert space. If π takes
its values in Isom(H) and (g, u) → π(g)(u) is continuous then
(π, H) is a representation of G.
A ﬁrst example: R2 acts on itself via the mapping (x, y) → t(x,y).
10 / 36

Two representations
Suppose G is a Haussdorf locally compact topological group (thus
it admits a Haar measure).
Left regular representation
We deﬁne the left regular representation RG as
∀g ∈ G, ∀f ∈ L2
(G), LG (g)(f )(·) = f (g−1
·).
Space representation
Suppose (π, R2) is a representation of G, we extend this action to
L2(R2) via the representation π.
∀g ∈ G, ∀f ∈ L2
(R2
), π(g)(f )(·) = f π(g−1
)(·) .
11 / 36

Convolution(s)
G-convolution
We deﬁne the G-convolution ∗
G
: L2
(G)
2
→ L1
(G) as
∀(f , h) ∈ L2
(G)2
, ∀g ∈ G, f ∗
G
h (g) := f , LG (g)(h) G
=
g∈G
f (g )h(g−1
g ) dg .
(G, π)-convolution
We deﬁne the (G, π)-convolution ∗
G,π
: L2
(R2
)
2
→ CG
as
∀(f , h) ∈ L2
(R2
)2
, ∀g ∈ G, f ∗
G,π
h (g) := f , π(g)(h) R2 .
=
x∈R2
f (x)h(π(g)−1
x) dx
Question: is there a G-Fourier transform such that f ∗ h = ˆh∗
ˆg?
12 / 36

Unitary and irreducible representations
Irreducible representation
Let (G, π) be a representation which acts on H. π is said to be
reducible if
∃V subspace of H, ∀g ∈ G, π(g)(V ) ⊂ V .
A representation which is not reducible is said to be irreducible.
Unitary representation
A representation (G, π) on H is said to be unitary if
∀g ∈ G, π(g) ∈ U(H),
where U(H) is the set of all unitary operators on H.
13 / 36

Examples for some groups
• irreducible unitary representations of Z, k → eikξ
ξ∈T
,
• irreducible unitary representations of R, x → eixξ
ξ∈R
,
• irreducible unitary representations of Z
nZ, k → e
2iπkl
n
l∈ 0,n−1
,
see Peyré (2004) for example.
Thus, irreducible unitary representation of G ≈ Fourier atoms of G.
Objective
Compute irreducible unitary representations of other groups. Theoretical
results:
• compact groups, Sugiura (1990), Hewitt and Ross (2012),
• locally compact abelian groups Rudin (2017), Folland (2016).
14 / 36

Link between convolutions
Suppose (π, R2
) is a representation of G.
Wavelet Grossmann et al. (1985)
Suppose that π, L2
(R2
) is an irreducible representation of G. Let
ψ ∈ L2
(R2
) a π-admissible vector, i.e g → π(g)(ψ), ψ ∈ L2
(G). Then
W : L2
(R2
) → Im(W) ⊂ L2
(G) deﬁned as
W[f ] =
1
Cψ
π(·)(ψ), f
intertwines π and LG in the sense that
∀g ∈ G, ∀u ∈ L2
(R2
), LG (g)(W[u]) = W[π(g)(u)].
The unitary W operator is called a wavelet transform.
Convolution(s)
Let W be a wavelet transform.
∀(u, v) ∈ L2
(R2
)
2
, u ∗
G,π
v = W[u] ∗
G
W[v].
15 / 36

The origins: Grossmann & Morlet
In Grossmann et al. (1985)
16 / 36

Summary on wavelets
f ∈ L2(R2) π(g)(f ) ∈ L2(R2)
W [f ] ∈ L2(G) W [π(g)(f )] ∈ L2(G)
L2(G)
L2(R2)
space representation
17 / 36

Summary on wavelets
f ∈ L2(R2) π(g)(f ) ∈ L2(R2)
W [f ] ∈ L2(G) W [π(g)(f )] ∈ L2(G)
L2(G)
L2(R2)
wavelet transform
17 / 36

Summary on wavelets
f ∈ L2(R2) π(g)(f ) ∈ L2(R2)
W [f ] ∈ L2(G) LG (g)(W [f ]) ∈ L2(G)
L2(G)
L2(R2)
regular representation
17 / 36

A take-home message
Recall (π, R2) is a unitary representation of G and (π, L2(R2)) a
unitary representation of G.
• A Fourier transform extension on L2(G) can be deﬁned via the
irreducible unitary representations of G, see Sugiura (1990).
• If π is irreducible and an admissible vector exists then there
exists a wavelet transform Grossmann et al. (1985).
• But then LG is irreducible! Thus there exists no “simple”
Fourier transform.
18 / 36

Different situations, different needs...
u & v signals
u ∈ L2(R2),
v ∈ L2(R2), ex:
images...
u ∈ L2(G),
v ∈ L2(G), ex:
first layer of scat-
tering transform...
u ∗
G,π
v ∈ CG
operations specific
to group G...
u ∗
G
v ∈ L1(G)
19 / 36

Different situations, different needs...
u & v signals
u ∈ L2(R2),
v ∈ L2(R2), ex:
images...
u ∈ L2(G),
v ∈ L2(G), ex:
first layer of scat-
tering transform...
u ∗
G,π
v ∈ L2(G)
operations specific
to group G...
u ∗
G
v ∈ L1(G)
19 / 36

Spherical harmonics
Deﬁnition, see Stein and Weiss (2016)
(Y m
l )l∈N,m∈ −l,l ∈ L2(S2) with expression1
∀l ∈ N, m ∈ −l, l , Y m
l (θ, ϕ) ∝ Pl,m(cos(θ))eimϕ
.
with (Pl,m)l∈N,m∈ −m,m the associated Legendre polynomials.
We have that:
Properties
1. (Y m
l )l∈N,m∈ −l,l is an orthonormal basis of L2(S2),
2. ∀l ∈ N, m ∈ −l, l , −∆Y m
l = l(l + 1)Y m
l .
1
this expression can be found algebraically using representation theory on
SU2(C), see Sugiura (1990).
20 / 36

Some spherical harmonics
Figure 1: The real parts of the spherical harmonics (Y m
l )l∈N,m∈ 1,m . As
l increases the number of oscillations increases as well (links with the
behaviour of the usual Fourier atoms).
21 / 36

Irreducible unitary representations
D-Wigner matrix, see Sugiura (1990)
The irreducible unitary representation of SO3(R) are given by
Ul
: SO3(R) → Wl .
For every l ∈ N, g ∈ SO3(R) we denote by
(Dl
m,n(g))(m,n)∈ −l,l ∈ U2l+1(C) the D-Wigner matrices, where Wl is
a space of dimension 2l + 1.2
Properties
We have that:
1. Dl
m,n l∈N,(m,n)∈ −l,l 2 is an orthonormal basis of L2
(SO3(R)),
2. the irreducible unitary representations of SO3(R) are ﬁnite
dimensional, this is true for every compact group by the
Peter-Weyl theorem.
2
There exists explicit expressions for the D-Wigner matrices.
22 / 36

Fundamental properties
Fourier transform
Let f ∈ L1
(S2
) we deﬁne the Fourier transform3
∀l ∈ N, ˆf (l) =
x∈S2
f (x)Yl dx ∈ R2l+1
.
Let f ∈ L1
(SO3(R)) we deﬁne the Fourier transform4
∀l ∈ N, ˆf (l) =
x∈SO3(R)
f (x)Dl
dx ∈ M2l+1(R).
Convolution to product
Let (f , h) ∈ L2
(SO3(R)). We have
f ∗
SO3(R)
h = ˆh∗ ˆf .
Let (f , h) ∈ L2
(S2
). We have
f ∗
SO3(R),π
h = ˆh∗ ˆf .
23 / 36

A S2
and a SO3(R)-FFT
Fourier atoms, spherical harmonics and Wigner-D matrices
• Dl
m,n(α, β, γ) ∝ dl
m,n(β)e−imα
e−inγ
,
• Y l
m(θ, ϕ) ∝ Pl,m(cos(θ))eimϕ
.5
Operations
Cost
S2
SO3(R)
Fourier transform N2
θ log(Nθ)2
N3
θ log(Nθ)3
Inverse Fourier transform N2
θ log(Nθ)2
N3
θ log(Nθ)3
Convolution N2
θ log(Nθ)2
+ N3
θ + N3
θ log(Nθ)3
N3
θ log(Nθ)3
+ N3+
θ
Table 1: Computational cost of Fourier operations in S2
and SO3(R). With
Nθ the maximal number of angles for each variable. Discretization discussed in
Kostelec and Rockmore (2008).6
5
dl
m,n and Pl
m satisfy order 2 recursion in l thus can be computed eﬃciently
using the Fast Polynomial Transform see Potts et al. (1998) (thanks Thibaud!).24 / 36

SO3(R)-Convolutional Neural Networks
25 / 36

3 × 3 R2 ∗ R2 ﬁltering
3 × 3 R2 ∗ R2 ﬁltering
25 / 36

S2 ∗SO3(R),π S2 ﬁltering
SO3(R) ∗SO3(R) SO3(R) ﬁltering
25 / 36

Spherical MNIST dataset
Figure 2: Two MNIST digits projected onto the sphere using
stereographic projection. Mapping back to the plane results in non-linear
distortions. Courtesy of Cohen et al. (2018)
Group
Training
Testing NR/NR R/R NR/R
R2
0.98 0.23 0.11
SO3(R) 0.96 0.95 0.94
Table 2: Accuracy of R2
and SO3(R) CNN. Both are two layers neural
networks with number of parameters ≈ 60k. “R” means that each digit
has been randomly rotated. 26 / 36

The Special Euclidean motion
group

Many authors have used the Special Euclidean motion group:
• Mallat (2012), Sifre and Mallat (2013), Sifre and Mallat
(2014), Oyallon and Mallat (2015) → Group Scattering
transform (roto-translation = SE2(R)),
• Duits et al. (2007), Sanguinetti et al. (2015), Duits et al.
(2016) → “Wavelet” on the Special Euclidean motion group +
tracking in SE2(R) (Mathematica implementation can be
found here http://www.lieanalysis.nl/)),
• Chirikjian and Kyatkin (2000), Kyatkin and Chirikjian (2000)
→ algorithms for convolution on motion groups + application
to medical image analysis.
27 / 36

Unitary representations
Principal series representations of SE2(R), Sugiura (1990)
Let a ∈ C. We deﬁne the principal series representations
Ua : G → BL(L2(T)) as
∀(r, z) ∈ SE2(R), ∀F ∈ L2
(T), Ua
(r, z)(F)(s) = ei z,sa
F(r−1
s),
where BL(L2(T)) is the set of bounded linear operators on L2(T).
For each a ∈ C, Ua is a irreducible unitary representation.
inﬁnite dimensional representation...
28 / 36

Fourier transform & Hilbert-Schmidt operators
Fourier transform
Let f ∈ L1
(SE2(R)). We define the Fourier transform, a function defined on
R∗
+ taking its values on BL(L2
(SE2(R)))
∀a ∈ R∗
+, ˆf (a) =
SE2(R)
f (g)Ua
(g) dg.
Hilbert-Schmidt operator
We define the space of integral Hilbert-Schmidt operators
B2(T, T) = {φ, ∃kφ ∈ L2
(T × T), ∀f ∈ L2
(T), φ(f )(y) =
T
kφ(x, y)f (x) dx},
φ → kφ is an isometry between B2(T, T) and L2
(T, T).
⇒ We have that ∀a ∈ R∗
+, ˆf (a) ∈ B2(T, T).
29 / 36

On the way to discretization... (1)
To compute ˆf (a) for all a ∈ R∗
+:
• for a fixed a ∈ R∗
+ we ought to compute a kernel kf ,a (explicit
expression),
• this operation can be done in N log(N) where N is the
cardinality of a discretization of SE2(R).
Questions
• is there an inverse Fourier transform? If so, can we compute it
efficiently?
• is there a formula for the Fourier transform of a convolution?
If so, can we compute it efficiently?
30 / 36

Inverse Fourier transform
Let f ∈ S(SE2(R)) then for almost every g ∈ SE2(R)
∀g ∈ SE2(R), f (g) =
+∞
0
Tr(Ua
(g)ˆf (a))a da.
⇒ computational cost is N log(N).
Fourier transform and convolution
Let (f , g) ∈ L1(SE2(R)). ∀a ∈ R∗
+, f ∗ h(a) = ˆh(a)∗ ˆf (a).
31 / 36

Operations Cost
Fourier transform N log(N)
Inverse Fourier transform N log(N)
Convolution N log(N) + N1+
Table 3: Computational cost of Fourier operations in SE2(R). N1+
can
be replaced by Nx NθNm−2
θ = NNm−2
θ where m ≥ 2 is such that nm
is the
complexity of a matrix multiplication in Mn(R).
Take-home message
We can build fast Fourier operations on SE2(R). However, the
computational cost for convolution is higher than in the abelian
case.
32 / 36

Where are the wavelets?
The left regular representation on SE2(R) is reducible.7
Question
Can we deﬁne a wavelet extension?
1. set W[f ](r, z) = f (z), → corresponds to set δ0 on L2
(R2
) as an
admissible vector.
2. see Duits et al. (2007) for an extension of wavelets (called
orientation scores) via Reproducing Kernel Hilbert Spaces.
7
This does not imply that no wavelet can be constructed. However it is shown
in Führ (2002) that if there exists an admissible vector on G, with G
unimodular, then G is discrete.
33 / 36

Figure 3: In Bekkers et al. (2018) the authors lift medical images in
SE2(R) then ﬁnd a template in L(SE2(R)).
34 / 36

Working scheme
f ∈ L2(R2) π(g)(f ) ∈ L2(R2)
W [f ] ∈ L2(SE2(R))K W [π(g)(f )] ∈ L2(SE2(R))K
L2(SE2(R))
L2(R2)
space representation
35 / 36

Working scheme
f ∈ L2(R2) π(g)(f ) ∈ L2(R2)
W [f ] ∈ L2(SE2(R))K W [π(g)(f )] ∈ L2(SE2(R))K
L2(SE2(R))
L2(R2)
“wavelet” transform
35 / 36

Working scheme
f ∈ L2(R2) π(g)(f ) ∈ L2(R2)
W [f ] ∈ L2(SE2(R))K LSE2 (R)(g)(W [f ]) ∈ L2(SE2(R))K
L2(SE2(R))
L2(R2)
regular representation
35 / 36

Conclusion & perspectives
• diﬀerent signal geometries ⇒ diﬀerent group actions ⇒ it may
be useful to include this geometry directly in correlation tools,
• representation theory yields a very elegant framework to build
the mathematical foundations of group harmonic analysis,
• it is hard and not clear how to go from the continuous case
to the discrete implementation.
Some lines of work:
• try to derive the same results for other interesting groups for
image processing,
• embed the SE2(R) convolution in CNN.
36 / 36

References
Erik Johannes Bekkers, Marco Loog, Bart M ter Haar Romeny, and Remco Duits. Template matching
via densities on the roto-translation group. IEEE transactions on pattern analysis and machine
intelligence, 40(2):452–466, 2018.
Gregory S Chirikjian and Alexander B Kyatkin. Engineering applications of noncommutative harmonic
analysis: with emphasis on rotation and motion groups. CRC press, 2000.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint
arXiv:1801.10130, 2018.
Remco Duits, Michael Felsberg, Gösta Granlund, and Bart ter Haar Romeny. Image analysis and
reconstruction using a wavelet transform constructed from a reducible representation of the
euclidean motion group. International Journal of Computer Vision, 72(1):79–102, 2007.
Remco Duits, Stephan PL Meesters, Jean-Marie Mirebeau, and Jorg M Portegies. Optimal paths for
variants of the 2d and 3d reeds-shepp car with applications in image analysis. arXiv preprint
arXiv:1612.06137, 2016.
Gerald B Folland. A course in abstract harmonic analysis, volume 29. CRC press, 2016.
Hartmut Führ. Admissible vectors for the regular representation. Proceedings of the American
Mathematical Society, 130(10):2959–2970, 2002.
Alex Grossmann, Jean Morlet, and T Paul. Transforms associated to square integrable group
representations. i. general results. Journal of Mathematical Physics, 26(10):2473–2479, 1985.
Edwin Hewitt and Kenneth A Ross. Abstract Harmonic Analysis: Volume I Structure of Topological
Groups Integration Theory Group Representations, volume 115. Springer Science & Business Media,
2012.
37 / 36

Peter J Kostelec and Daniel N Rockmore. Ffts on the rotation group. Journal of Fourier analysis and
applications, 14(2):145–179, 2008.
Alexander B Kyatkin and Gregory S Chirikjian. Algorithms for fast convolutions on motion groups.
Applied and Computational Harmonic Analysis, 9(2):220–241, 2000.
Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics, 65
(10):1331–1398, 2012.
David K Maslen and Daniel N Rockmore. Generalized ffts—a survey of some recent results. In Groups
and Computation II, volume 28, pages 183–287. American Mathematical Soc., 1997.
Edouard Oyallon and Stéphane Mallat. Deep roto-translation scattering for object classification. In
CVPR, volume 3, page 6, 2015.
Gabriel Peyré. L’algèbre discrète de la transformée de Fourier. Ellipses, 2004.
Daniel Potts, Gabriele Steidl, and Manfred Tasche. Fast algorithms for discrete polynomial transforms.
Mathematics of Computation of the American Mathematical Society, 67(224):1577–1590, 1998.
Walter Rudin. Fourier analysis on groups. Courier Dover Publications, 2017.
Gonzalo Sanguinetti, Erik Bekkers, Remco Duits, Michiel HJ Janssen, Alexey Mashtakov, and
Jean-Marie Mirebeau. Sub-riemannian fast marching in se (2). In Iberoamerican Congress on
Pattern Recognition, pages 366–374. Springer, 2015.
Laurent Sifre and PS Mallat. Rigid-motion scattering for image classification. PhD thesis, Citeseer,
2014.
Laurent Sifre and Stéphane Mallat. Rotation, scaling and deformation invariant scattering for texture
discrimination. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on,
pages 1233–1240. IEEE, 2013.
Elias M Stein and Guido Weiss. Introduction to Fourier analysis on Euclidean spaces (PMS-32),
volume 32. Princeton university press, 2016.
Mitsuo Sugiura. Unitary representations and harmonic analysis: an introduction, volume 44. Elsevier,
1990.
38 / 36

Introduction to harmonic analysis on groups, links with spatial correlation.

More Related Content

What's hot

Similar to Introduction to harmonic analysis on groups, links with spatial correlation.

Recently uploaded

Introduction to harmonic analysis on groups, links with spatial correlation.