This document discusses macrocanonical models for texture synthesis. It begins by introducing the goal of texture synthesis and providing a brief history. It then describes the parametric question of combining randomness and structure in images. Specifically, it discusses maximizing entropy under geometric constraints. The document goes on to discuss links to statistical physics, defining microcanonical and macrocanonical models. It focuses on studying the macrocanonical model, describing how to find optimal parameters through gradient descent and how to sample from the model using Langevin dynamics. The document provides examples of texture synthesis and compares results to other methods.

1. Macrocanonical Models for Texture
Synthesis
Valentin De Bortoli
With: Agnès Desolneux, Bruno Galerne, Arthur Leclaire
May 16, 2019
1 / 19

2. Models for texture synthesis

3. 2 / 19

4. Goal: sample images x ∼ Π(x) which look like an input texture x0
but are not verbatim copies of x0.
A brief history of exemplar-based texture synthesis...
1975 1980 1985 1990 1995 2000 2005 2010 2015
T
im
e-series
(M
cCorm
ick
and
Jayaram
am
urthy)
FractionalB
row
nian
m
otion
(Fournier
et
al.)
Spot
noise
(van
W
ijk)
Co-occurence
m
atrices
(G
agalow
icz
and
M
a)
Steerable
pyram
id
(H
eeger
and
B
ergen)
M
axim
um
entropy
(Zhu
et
al.)
G
radient
descent
(Portilla
and
Sim
oncelli)
G
eom
etricalwavelets
(Peyré)
First
tim
e
neuralnetworks
(G
atys
et
al.)
N
euralnew
tork
+
spectrum
(Liu
et
al.)
N
-gram
m
odels
(G
arber
and
Sawchuk)
Pixel-w
ise
updates
(Efros
and
Leung)
B
lock-w
ise
updates
(Efros
and
Freem
an)
Texture
optim
ization
(K
watra
et
al.)O
ptim
altransport
in
patch
space
(G
alerne
et
al.)
– non-parametric synthesis (patch-based)
– parametric synthesis (feature-based)
3 / 19

5. Goal: sample images x ∼ Π(x) which look like an input texture x0
but are not verbatim copies of x0.
The parametric question: How to combine randomness and
structure in an image model?
A possible (vague) answer: Maximize the entropy (H) under
geometrical constraints (f ).
4 / 19

6. Links with statistical physics (Bruna and Mallat (2018))...
Microcanonical model
The probability distribution function Π ∈ P is a microcanonical
model associated with the exemplar texture x0 ∈ Rd , statistics
f : Rd → Rp if
H(Π) = max {H(Π), Π ∈ P, f (X) = f (x0), X ∼ Π} .
Macrocanonical model
The probability distribution function Π ∈ P is a macrocanonical
model associated with the exemplar texture x0 ∈ Rd , statistics
f : Rd → Rp if
H(Π) = max {H(Π), Π ∈ P, Π(f ) = f (x0)} .
Notations: EΠ [f (X)] := Π(f ).
5 / 19

7. Links with statistical physics (Bruna and Mallat (2018))...
Microcanonical model
The probability distribution function Π ∈ P is a microcanonical
model associated with the exemplar texture x0 ∈ Rd , statistics
f : Rd → Rp if
H(Π) = max {H(Π), Π ∈ P, f (X) = f (x0), X ∼ Π} .
Macrocanonical model
The probability distribution function Π ∈ P is a macrocanonical
model associated with the exemplar texture x0 ∈ Rd , statistics
f : Rd → Rp if
H(Π) = min {KL(Π, µ), Π ∈ P, Π(f ) = f (x0)} .
Notations: EΠ [f (X)] := Π(f ), µ = reference probability measure
5 / 19

8. Microcanonical model
• Introduced in 1987
(Gagalowicz and Ma)
• White noise + Gradient
descent
• First theoretical results in
2018 (Bruna and Mallat)
• “Optimal distribution” can be
low dimensional
• “Optimal distribution” is
usually unknown
Macrocanonical model
• Introduced in 1997 (Zhu
et al.) (FRAME)
• Markov Chain(s) + Gradient
descent
• First theoretical results in
2018 (D., Desolneux, Galerne
and Leclaire)
• “Optimal distribution” has
dimension of the image space
• “Optimal distribution” is
known up to parameters
6 / 19

9. Study of the macrocanonical
model

10. f : Rd → Rp with
• d - number of pixels,
• p - number of parameters.
Gibbs measure: dΠθ
dµ (x) ∝ exp [− θ, f (x) − f (x0) ]
Gibbs measures are macrocanonical models
Under mild assumptions there exists ˜θ ∈ Rd such that Π˜θ is a
macrocanonical model associated with the exemplar texture
x0 ∈ Rd and statistics f .
Two questions remain:
1. how to find the optimal parameters ˜θ ?
2. how to sample from the model, i.e. sample from a Gibbs
measure Πθ?
7 / 19

11. Finding the optimal parameters...
The optimal parameters ˜θ minimizes the log-partition function
L(θ) = log
Rd
exp(− θ, f (x) − f (x0) )dµ(x) .
Properties of the log-partition function
• θL(θ) = −(Πθ(f ) − f (x0)) ,
• 2
θL(θ) = CovΠθ
(f ) ⇒ convexity!
θn+1 = ProjK [θn + δn+1Πθn (f − f (x0))] ,
Boiling down to...
Computing Πθ(f ) ⇒ Computing L(θ) ⇒ Gradient descent ⇒ Finding ˜θ
8 / 19

12. Sampling from Πθ...
Usually it is not possible to sample from Π(dx) ∝ exp[−U(x)]dx, but...
Geometric ergodicity, Meyn and Tweedie (2009)
Let R be a Markov kernel over Rd
× B(Rd
) which satisfies a
minorization condition and a Foster-Lyapunov drift then
• R admits an invariant probability ˜Π,
• for any xinit ∈ Rd
and n ∈ N, δxinit
Rn
− ˜Π TV Cρn
with
ρ ∈ (0, 1) and C 0.
for some R, ˜Π − Π TV ε.
Langevin chain - (Roberts et al. (1996), Durmus and Moulines (2017)...)
Xn+1 = Xn − γn+1 x U(Xn) + 2γn+1Zn ,
with Zn ∼ N(0, Id), i.i.d. and γn stepsizes. → sample ≈ C exp(−U(x))
9 / 19

13. We denote V (x, θ) = θ, f (x) − f (x0) + r(x) (assuming that
dµ
dLeb (x) ∝ exp(−r(x))).
Finding optimal parameters
˜θ is the minimum of the
log-partition function which is a
convex problem.
Gradient descent dynamics
θn+1 = θn + δn+1Πθn ( θV (·, θn))
Sampling from a Gibbs measure
The potential x → V (x, θ) is
usually non-convex but has
curvature at infinity.
Langevin dynamics
Xn+1 = Xn−γn+1 x V (Xn, θ)+ 2γn+1Zn+1
⇒ Combining dynamics
Xn
k+1 = Xn
k − γn x V (Xn
k , θn) + 2γnZn
k+1 , with Xn
0 = Xn−1
mn−1
,
θn+1 = ProjK θn + δn+1m−1
n
mn
k=1
θV (Xn
k , θn) ,
where Zn
k are i.i.d d-dimensional standard Gaussian r.v.
10 / 19

14. Our main result: D., Durmus, Pereyra, Fernandez Vidal (2018).
Convergence of the parameters
If f is smooth and f V with V : Rd → [1, +∞) a Lyapunov
function for the Langevin dynamics and
+∞
k=1
δk = +∞ ,
+∞
k=1
δk
mkγk
< +∞ ,
+∞
k=1
δkγ
1/2
k < +∞ .
then (θn)n∈N converges almost surely and in L1 to the optimal
parameters.
A few remarks:
• f does not need to be convex (neural network features )
• explicit constants (strongly convex ⇒ linear in the dimension)
and speed n
k=1 δk
• other result → mn can be set to 1
• this algorithm works in a more general statistical context...
11 / 19

15. Some textures

16. L = family of layers
C = channels of layer
T ,c = feature at layer and channel c
Figure 1: Neural network features. Structure of the neural network
VGG-19.
Choice of features: mean of each channel for selected layers,
p ≈ 103, i.e. f (x) = (
nc,
i=1 T ,c(x)i /nc, ) ∈L,c∈C .
12 / 19

17. (a) (b) (c)
Figure 2: Texture synthesis. (a) input texture, (b) is the initialization
of the algorithm and (c) the output.
13 / 19

18. Original DeepFrame Our result Original DeepFrame Our result
Lu et al. (2016) Lu et al. (2016)
14 / 19

19. Original (512 × 512) Initialization (Gaussian) After 5000 iterations
15 / 19

20. (a) (b) (c)
Figure 3: Texture synthesis. (a) input texture, (b) is the initialization
of the algorithm and (c) the output.
16 / 19

21. (a) (b)
(c) (d)
Figure 4: Comparison. The input image is shown in (a), the output of Gatys et al.
(2015) in (b), the output of Lu et al. (2016) (DeepFRAME) in (c), the output of
Jetchev et al. (2016) (GAN) in (d) and our result in (e). We lack spatial organization.
17 / 19

22. Exemplar-based texture synthesis
Patch-based Feature-based
Microcanonical Macrocanonical
The two approaches yield comparable visual results.
Problem: Efficient Markov chain sampling is still too slow in high
dimension...
How to improve the mixing time of our Markov chains?
The Big Question: what are the best “perceptual” geometrical
constraints f ?
18 / 19

23. References
J. Bruna and S. Mallat. Multiscale sparse microcanonical models. ArXiv e-prints, January 2018.
Alain Durmus and Éric Moulines. Nonasymptotic convergence analysis for the unadjusted Langevin
algorithm. Ann. Appl. Probab., 27(3):1551–1587, 2017. ISSN 1050-5164. doi:
10.1214/16-AAP1238. URL https://doi.org/10.1214/16-AAP1238.
Alexei A. Efros and William T. Freeman. Image quilting for texture synthesis and transfer. In
Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques,
SIGGRAPH 2001, Los Angeles, California, USA, August 12-17, 2001, pages 341–346, 2001.
Alexei A. Efros and Thomas K. Leung. Texture synthesis by non-parametric sampling. In ICCV, 1999.
Alain Fournier, Donald S. Fussell, and Loren C. Carpenter. Computer rendering of stochastic models.
Commun. ACM, 25(6):371–384, 1982. doi: 10.1145/358523.358553. URL
https://doi.org/10.1145/358523.358553.
André Gagalowicz and Song De Ma. Model driven synthesis of natural textures for 3-d scenes.
Computers & Graphics, 1986.
B. Galerne, A. Leclaire, and J. Rabin. A texture synthesis model based on semi-discrete optimal
transport in patch space. SIIMS, 2018.
David D Garber and Alexander A Sawchuk. Computational models for texture analysis and synthesis. In
Techniques and Applications of Image Understanding, volume 281, pages 254–274. International
Society for Optics and Photonics, 1981.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Texture synthesis using convolutional neural
networks. In NIPS, 2015.
David J. Heeger and James R. Bergen. Pyramid-based texture analysis/synthesis. In ICIP, 1995.
19 / 19

24. Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf. Texture synthesis with spatial generative
adversarial networks. CoRR, 2016.
Vivek Kwatra, Arno Schödl, Irfan A. Essa, Greg Turk, and Aaron F. Bobick. Graphcut textures: image
and video synthesis using graph cuts. ACM Trans. Graph., 22(3):277–286, 2003. doi:
10.1145/882262.882264. URL http://doi.acm.org/10.1145/882262.882264.
Gang Liu, Yann Gousseau, and Gui-Song Xia. Texture synthesis through convolutional neural networks
and spectrum constraints. In ICPR, 2016.
Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Learning FRAME models using CNN filters. In AAAI,
2016.
Bruce H. McCormick and Sadali N. Jayaramamurthy. Time series model for texture synthesis.
International Journal of Parallel Programming, 3(4):329–343, 1974. doi: 10.1007/BF00978978.
URL https://doi.org/10.1007/BF00978978.
Sean Meyn and Richard L. Tweedie. Markov chains and stochastic stability. Cambridge University Press,
Cambridge, second edition, 2009. ISBN 978-0-521-73182-9. doi: 10.1017/CBO9780511626630.
URL https://doi.org/10.1017/CBO9780511626630. With a prologue by Peter W. Glynn.
Gabriel Peyré. Texture synthesis with grouplets. IEEE Trans. Pattern Anal. Mach. Intell., 32(4):
733–746, 2010. doi: 10.1109/TPAMI.2009.54. URL https://doi.org/10.1109/TPAMI.2009.54.
Javier Portilla and Eero P. Simoncelli. A parametric texture model based on joint statistics of complex
wavelet coefficients. IJCV, 2000.
Gareth O Roberts, Richard L Tweedie, et al. Exponential convergence of langevin distributions and their
discrete approximations. Bernoulli, 2(4):341–363, 1996.
Jarke J. van Wijk. Spot noise texture synthesis for data visualization. In Proceedings of the 18th Annual
Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1991, Providence, RI,
USA, April 27-30, 1991, 1991. doi: 10.1145/122718.122751. URL
http://doi.acm.org/10.1145/122718.122751.
Song Chun Zhu, Ying Nian Wu, and David Mumford. Filters, random fields and maximum entropy
(FRAME): towards a unified theory for texture modeling. IJCV, 1998.
20 / 19

25. Original Scattering Our result Original Scattering Our result
21 / 19

26. Original Galerne et al. (2018) Gatys et al. (2015)
Portilla and Simoncelli (2000) Our result Jetchev et al. (2016)
22 / 19