Slides of the presentation at the conference "Optimal Transport to Orsay", Orsay, France, June 18-22, 2012.

1. Optimal Transport
in Imaging Sciences
Gabriel Peyré
www.numerical-tours.com

2. Statistical Image Models
Colors distribution: ) each pixel
Source image (X point in R3
Source image after color transfer
Style image (Y )
J. Rabin Wasserstein Regularization

3. Statistical Image Models
Colors distribution: ) each pixel
Source image (X point in R3
Optimal transport framework Sliced Wasserstein projection Applications
Application to Color Transfer
Optimal transport framework Sliced Wasserstein projection Applications
Application to Color Transfer Sliced Wasserstein projection color style
Source image after of X to transfer
image color statistics Y
Style image (Y )
J. Rabin Wasserstein Regularization
Source image (X ) Sliced Wasserstein projection of X to style
image color statistics Y
Input image
Source image (X ) Modified color transfer
Source image after
image
Style image (Y )
J. Rabin Wasserstein Regularization

4. Statistical Image Models
Colors distribution: ) each pixel
Source image (X point in R3
Optimal transport framework Sliced Wasserstein projection Applications
Application to Color Transfer
Optimal transport framework Sliced Wasserstein projection Applications
Application to Color Transfer Sliced Wasserstein projection color style
Source image after of X to transfer
image color statistics Y
Style image (Y )
J. Rabin Wasserstein Regularization
Source image (X ) Sliced Wasserstein projection of X to style
image color statistics Y
Input image
Source image (X ) Modified color transfer
Source image after
image
Style image (Y )
Texture synthesis
Other applications: J. Rabin Wasserstein Regularization
Texture segmentation

5. Overview
• Wasserstein Distance
• Sliced Wasserstein Distance
• Color Transfer
• Wasserstein Barycenter
• Gaussian Texture Models

6. Discrete Distributions
N 1
Discrete measure: µ = pi Xi Xi Rd pi = 1
i=0 i

7. Discrete Distributions
N 1
Discrete measure: µ = pi Xi Xi Rd pi = 1
i=0 i
Point cloud
Constant weights: pi = 1/N .
Xi
Quotient space:
RN d / N

8. Discrete Distributions
N 1
Discrete measure: µ = pi Xi Xi Rd pi = 1
i=0 i
Point cloud Histogram
Constant weights: pi = 1/N . Fixed positions Xi (e.g. grid)
Xi
Quotient space: A ne space:
RN d / N {(pi )i i pi = 1}

9. From Images to Statistics
Source image (X )
Discretized image f RN d
fi Rd = R3
N = #pixels, d = #colors.
Source image
Style image (Y )
J. Rabin Wasserstein Regularizatio

10. From Images to Statistics Source image (X )
Discretized image f RN d
fi Rd = R3
N = #pixels, d = #colors.
Source image
Disclamers: images are not distributions. image (Y )
Style
J. Rabin Wasserstein Regularizatio
Needs an estimator: f µf
Modify f by controlling µf .

11. From Images to Statistics Source image (X )
Discretized image f RN d
fi Rd = R3
N = #pixels, d = #colors.
Source image
Disclamers: images are not distributions. image (Y )
Style
J. Rabin Wasserstein Regularizatio
Needs an estimator: f µf
Modify f by controlling µf .
Xi
Point cloud discretization: µf = fi
i

12. From Images to Statistics Source image (X )
Discretized image f RN d
fi Rd = R3
N = #pixels, d = #colors.
Source image
Disclamers: images are not distributions. image (Y )
Style
J. Rabin Wasserstein Regularizatio
Needs an estimator: f µf
Modify f by controlling µf .
Xi
Point cloud discretization: µf = fi
i
Histogram discretization: µf = pi Xi
i
1
Parzen windows: pi = (xi fj )
Zf j

13. From Images to Statistics Source image (X )
Discretized image f RN d
fi Rd = R3
N = #pixels, d = #colors.
Source image
Disclamers: images are not distributions. image (Y )
Style
J. Rabin Wasserstein Regularizatio
Needs an estimator: f µf
Modify f by controlling µf .
Xi
Point cloud discretization: µf = fi
i
Today’s focus
Histogram discretization: µf = pi Xi
i
1
Parzen windows: pi = (xi fj )
Zf j

14. Optimal Transport Distances
Grayscale: 1-D
Vector X RN d µX = Xi
(image, coe cients, . . . ) i
Colors: 3-D
B G
R

15. Optimal Transport Distances
Grayscale: 1-D
Vector X RN d µX = Xi
(image, coe cients, . . . ) i
Colors: 3-D
B G
Optimal assignment:
⇥
⇥ argmin ||Xi Y (i) ||p R
N i

16. Optimal Transport Distances
Grayscale: 1-D
Vector X RN d µX = Xi
(image, coe cients, . . . ) i
Colors: 3-D
B G
Optimal assignment:
⇥
⇥ argmin ||Xi Y (i) ||p R
N i
Wasserstein distance: Wp (µX , µY )p = ||Xi Y (i) ||p
i
Metric on the space of distributions.

17. Optimal Transport Distances
Grayscale: 1-D
Vector X RN d µX = Xi
(image, coe cients, . . . ) i
Colors: 3-D
B G
Optimal assignment:
⇥
⇥ argmin ||Xi Y (i) ||p R
N i
Wasserstein distance: Wp (µX , µY )p = ||Xi Y (i) ||p
i
Metric on the space of distributions.
Projection on statistical constraints: C = {f µf = µY }
ProjC (f ) = Y

18. Computing Transport Distances
Explicit solution for 1D distribution (e.g. grayscale images):
Xi
Yi
sorting the values, O(N log(N )) operations.

19. Computing Transport Distances
Explicit solution for 1D distribution (e.g. grayscale images):
Xi
Yi
sorting the values, O(N log(N )) operations.
Higher dimensions: combinatorial optimization methods
Hungarian algorithm, auctions algorithm, etc.
O(N 5/2 log(N )) operations.
intractable for imaging problems.

20. Computing Transport Distances
Explicit solution for 1D distribution (e.g. grayscale images):
Xi
Yi
sorting the values, O(N log(N )) operations.
Higher dimensions: combinatorial optimization methods
Hungarian algorithm, auctions algorithm, etc.
O(N 5/2 log(N )) operations.
intractable for imaging problems.
Arbitrary distributions: µ= pi Xi ⇥= qi Yi
i i
Wp (µ, ) solution of a linear program.
p

21. Overview
• Wasserstein Distance
• Sliced Wasserstein Distance
• Color Transfer
• Wasserstein Barycenter
• Gaussian Texture Models

22. Approximate Sliced Distance
Key idea: replace transport in Rd by series of 1D transport.
Xi
[Rabin, Peyr´, Delon & Bernot 2010]
e
Projected point cloud: X = { Xi , ⇥}i .
Xi ,

23. Approximate Sliced Distance
Key idea: replace transport in Rd by series of 1D transport.
Xi
[Rabin, Peyr´, Delon & Bernot 2010]
e
Projected point cloud: X = { Xi , ⇥}i .
Xi ,
Sliced Wasserstein distance: (p = 2)
SW (µX , µY )2 = W (µX , µY )2 d
|| ||=1

24. Approximate Sliced Distance
Key idea: replace transport in Rd by series of 1D transport.
Xi
[Rabin, Peyr´, Delon & Bernot 2010]
e
Projected point cloud: X = { Xi , ⇥}i .
Xi ,
Sliced Wasserstein distance: (p = 2)
SW (µX , µY )2 = W (µX , µY )2 d
|| ||=1
Theorem: E(X) = SW (µX , µY )2 is of class C 1 and
E(X) = Xi Y⇥ (i) , d .
where N are 1-D optimal assignents of X and Y .

25. Approximate Sliced Distance
Key idea: replace transport in Rd by series of 1D transport.
Xi
[Rabin, Peyr´, Delon & Bernot 2010]
e
Projected point cloud: X = { Xi , ⇥}i .
Xi ,
Sliced Wasserstein distance: (p = 2)
SW (µX , µY )2 = W (µX , µY )2 d
|| ||=1
Theorem: E(X) = SW (µX , µY )2 is of class C 1 and
E(X) = Xi Y⇥ (i) , d .
where N are 1-D optimal assignents of X and Y .
Possible to use SW in variational imaging problems.
Fast numerical scheme : use a few random .

26. Sliced Assignment
Theorem: X is a local minima of E(X) = SW (µX , µY )2
N, X = Y

27. Sliced Assignment
Theorem: X is a local minima of E(X) = SW (µX , µY )2
N, X = Y
Stochastic gradient descent of E(X):
Step 1: choose at random. E (X) = W (X , Y )2
Step 2: X( +1)
= X( )
E (X ( ) )

28. Sliced Assignment
Theorem: X is a local minima of E(X) = SW (µX , µY )2
N, X = Y
Stochastic gradient descent of E(X):
Step 1: choose at random. E (X) = W (X , Y )2
Step 2: X( +1)
= X( )
E (X ( ) )
X( )
converges to C = {X µX = µY }.

29. Sliced Assignment
Theorem: X is a local minima of E(X) = SW (µX , µY )2
N, X = Y
Stochastic gradient descent of E(X):
Step 1: choose at random. E (X) = W (X , Y )2
Step 2: X( +1)
= X( )
E (X ( ) )
X( )
converges to C = {X µX = µY }. Final assignment
Numerical observation: X ( )
ProjC (X (0) )

30. Overview
• Wasserstein Distance
• Sliced Wasserstein Distance
• Color Transfer
• Wasserstein Barycenter
• Gaussian Texture Models

31. pplication to Color Transfer
Color Histogram Equalization
1
Input color images: fi RN 3 . projectioniof= to style
Sliced Wasserstein
⇥ X
N x
fi (x)
image color statistics Y
Optimal transport framework Sliced Wasserstein projection Applications
Application to Color Transfer
Source image (X )
f1 f0 Sliced Wasserstein projec
image color statistics Y
f0
Source image after color transfer
µ1 image (Y )
Style Source image (X )
µ0
J. Rabin Wasserstein Regularization

32. pplication to Color Transfer
Color Histogram Equalization
1
Input color images: fi RN 3 . projectioniof= to style
Sliced Wasserstein
⇥ X
N x
fi (x)
image color statistics Y
Optimal assignement: min ||f0 f1 ⇥ ||
N
Optimal transport framework Sliced Wasserstein projection Applications
Application to Color Transfer
Source image (X )
f1 f0 Sliced Wasserstein projec
image color statistics Y
f0
Source image after color transfer
µ1 image (Y )
Style Source image (X )
µ0
J. Rabin Wasserstein Regularization

33. pplication to Color Transfer
Color Histogram Equalization
1
Input color images: fi RN 3 . projectioniof= to style
Sliced Wasserstein
⇥ X
N x
fi (x)
image color statistics Y
Optimal assignement: min ||f0 f1 ⇥ ||
N
Transport: T : f0 (x) R3 f1 ( (i)) R3
Optimal transport framework Sliced Wasserstein projection Applications
Application to Color Transfer
Source image (X )
f1 f0 Sliced Wasserstein projec
image color statistics Y
f0
Source image after color transfer
µ1 image (Y )
Style Source image (X )
µ0
T
J. Rabin Wasserstein Regularization

34. pplication to Color Transfer
Color Histogram Equalization
1
Input color images: fi RN 3 . projectioniof= to style
Sliced Wasserstein
⇥ X
N x
fi (x)
image color statistics Y
Optimal assignement: min ||f0 f1 ⇥ ||
N
Optimal transport framework Sliced Wasserstein projection Applications
Transport: T : f0 (x) R3Application to Color Transfer R3
f1 ( (i))
Optimal transport framework Sliced Wasserstein projection Applications
Application to Color Transfer
˜
Equalization: ) f0 = T (f0 ) ˜
f0 = f1 Sliced Wasserstein projection of X to sty
Source image (X image color statistics Y
f1 f0 T (f )
0
Sliced Wasserstein projec
image color statistics Y
Source image (X )
T
f0
Source image after color transfer
µ1 image (Y )
Style Source image (X )
µ0 Source image after color transfer
µ1
Style image (Y )
T J. Rabin Wasserstein Regularization
J. Rabin Wasserstein Regularization

35. Sliced Wasserstein Transfert
Solving min ||f0 f1 ⇥ || is computationally untractable.
N
Approximate Wasserstein projection:
˜
f0 solves min E(f ) = SW (µf , µf1 )
f
and is close to f0

36. Sliced Wasserstein Transfert
Solving min ||f0 f1 ⇥ || is computationally untractable.
N
Approximate Wasserstein projection:
˜
f0 solves min E(f ) = SW (µf , µf1 )
f
and is close to f0
(Stochastic) gradient descent:
f (0) = f0
f( +1)
= f( )
E(f ( ) )
f( ) ˜
f0
At convergence: µf0 = µf1
˜

37. pplication to Color Transfer
Color Transfert
Sliced Wasserstein projection of X to style
image color statistics Y
Sliced Wasserstein projection of X to styl
sty
image color statistics Y
Source image (X )
Input image f0 )
Source image (X
˜
Source image after image f0
Transfered color transfer
Style image (Y ) Source image after color transfer
Target image )f1
Style image (Y

38. Color Exchange
Source image (X (X )
Source image )
Source image (Xf
Input image )
Source image (X )
0 TargetXimage ⇥ f1 Y
X ⇥⇥ Y
X
Y
X ⇥ Y
Style image (Y )
Style image (Y )
Transfered image f0
Style image (Y )
˜ Transfered X ˜
Y ⇥ image f1

39. Overview
• Wasserstein Distance
• Sliced Wasserstein Distance
• Color Transfer
• Wasserstein Barycenter
• Gaussian Texture Models

40. Wasserstein Barycenter
µ2
Barycenter of {(µi , )}L :
i i=1 i =1
)
L i
2,µ
µ argmin W2 (µi , µ)2
2 (µ
i µ
W
µ
i=1 µ1
)
W2
W2 (µ 1, µ
(µ 3
,µ
)
µ3

41. Wasserstein Barycenter
µ2
Barycenter of {(µi , )}L :
i i=1 i =1
)
L i
2,µ
µ argmin W2 (µi , µ)2
2 (µ
i µ
W
µ
i=1 µ1
)
W2
(µ 1, µ
If µi = Xi , then µ = W2
(µ 3
X
,µ
X =
)
i Xi µ3
i
Generalizes Euclidean barycenter.

42. Wasserstein Barycenter
µ2
Barycenter of {(µi , )}L :
i i=1 i =1
)
L i
2,µ
µ argmin W2 (µi , µ)2
2 (µ
i µ
W
µ
i=1 µ1
)
W2
(µ 1, µ
If µi = Xi , then µ = W2
(µ 3
X
,µ
X =
)
i Xi µ3
i
Generalizes Euclidean barycenter.
Theorem: [Agueh, Carlier, 2010]
if µ0 does not vanish on small sets,
µ exists and is unique.

43. Special Case: 2 Distributions
Case L = 2:
µt argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2
µ
µ2
µ1 µ
t µt is the geodesic path.

44. Special Case: 2 Distributions
Case L = 2:
µt argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2
µ
µ2
µ1 µ
t µt is the geodesic path.
N
Discrete point clouds: µ= k=1 Xi (k) ,
N
Assignment: ⇥
⇥ argmin ||X1 (k) X2 ( (k))||2
N
k=1
N
µ = k=1 X (k) ,
where X = (1 t)X1 (k) + tX2 ( (k))

45. Linear Programming Resolution
Discrete setting: i = 1, . . . , L, µi = k Xi (k)
µ2
µ1
µ µ3

46. Linear Programming Resolution
Discrete setting: i = 1, . . . , L, µi = k Xi (k)
L-way interaction cost:
C(k1 , . . . , kL ) = i,j i j ||Xi (ki ) Xj (kj )||2
µ2
µ1
µ µ3

47. Linear Programming Resolution
Discrete setting: i = 1, . . . , L, µi = k Xi (k)
L-way interaction cost:
C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1
i,j
L-way coupling matrices: P PL
=1
=1
µ2
µ1
µ µ3

48. Linear Programming Resolution
Discrete setting: i = 1, . . . , L, µi = k Xi (k)
L-way interaction cost:
C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1
i,j
L-way coupling matrices: P PL
Linear program: =1
min Pk1 ,...,kL C(k1 , . . . , kL ) =1
P PL
k1 ,...,kL
µ2
µ1
µ µ3

49. Linear Programming Resolution
Discrete setting: i = 1, . . . , L, µi = k Xi (k)
L-way interaction cost:
C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1
i,j
L-way coupling matrices: P PL
Linear program: =1
min Pk1 ,...,kL C(k1 , . . . , kL ) =1
P PL
k1 ,...,kL
Barycenter: µ2
µ = Pk1 ,...,kL µ1
X (k1 ,...,kL )
k1 ,...,kL
L µ µ3
X (k1 , . . . , kL ) = i=1 i Xi (ki )

50. Sliced Wasserstein Barycenter
Sliced-barycenter: µX that solves
min i SW (µX , µXi )2
X
i

51. Sliced Wasserstein Barycenter
Sliced-barycenter: µX that solves
min i SW (µX , µXi )2
X
i
Gradient descent: Ei (X) = SW (µX , µXi )2
L
X( +1)
= X( )
⇥ i Ei (X ( ) )
i=1

52. Sliced Wasserstein Barycenter
Sliced-barycenter: µX that solves
min i SW (µX , µXi )2
X
i
Gradient descent: Ei (X) = SW (µX , µXi )2
L
X( +1)
= X( )
⇥ i Ei (X ( ) )
i=1
Advantages:
µX is a sum of N Diracs.
Smooth optimization problem.
Disadvantage:
Non-convex problem
local minima.

53. Sliced Wasserstein Barycenter
Sliced-barycenter: µX that solves
min i SW (µX , µXi )2
X
i
Gradient descent: Ei (X) = SW (µX , µXi )2
L
X( +1)
= X( )
⇥ i Ei (X ( ) ) µ1
i=1
Advantages:
µX is a sum of N Diracs.
Smooth optimization problem.
Disadvantage:
µ2 µ3
Non-convex problem
local minima.

54. . . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics;
Step 1: compute Sliced-Wasserstein Barycenter color statistics;
Step 1: 1: compute Sliced-Wasserstein Barycenter color statistics;
Application: Color Harmonization
. . . Step compute Sliced-Wasserstein projection ofofof each image onto the
Step 2: compute Sliced-Wasserstein projection each image onto the
Step 2: 2: compute Sliced-Wasserstein projection each image onto the
Barycenter;
Barycenter;
Barycenter;
Raw image
sequence
Raw image sequence
Raw image sequence
Raw image sequence
J. RabinRabinGREYC, University Caen Caen Approximate Wasserstein Metric forfor for Texture Synthesis and Mixing
J. RabinGREYC, University of of Caen
J. – – – GREYC, University of Approximate Wasserstein Metric Texture Synthesis and Mixing
Approximate Wasserstein Metric Texture Synthesis and Mixing

55. . . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics;
Step 1: compute Sliced-Wasserstein Barycenter color statistics;
Step 1: 1: compute Sliced-Wasserstein Barycenter color statistics;
Application: Color Harmonization
. . . Step compute Sliced-Wasserstein projection ofofof each image onto the
Step 2: compute Sliced-Wasserstein projection each image onto the
Step 2: 2: compute Sliced-Wasserstein projection each image onto the
Barycenter;
Barycenter;
Barycenter;
Raw image
sequence
Compute
Wasserstein
Raw image sequence
Raw image sequence
Raw image sequence
barycenter
J. RabinRabinGREYC, University Caen Caen Approximate Wasserstein Metric forfor for Texture Synthesis and Mixing
J. RabinGREYC, University of of Caen
J. – – – GREYC, University of Approximate Wasserstein Metric Texture Synthesis and Mixing
Approximate Wasserstein Metric Texture Synthesis and Mixing

56. . . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics;
Step 1: compute Sliced-Wasserstein Barycenter color statistics;
Step 1: 1: compute Sliced-Wasserstein Barycenter color statistics;
Application: Color Harmonization
. . . Step compute Sliced-Wasserstein projection ofofof each image onto the
Step 2: compute Sliced-Wasserstein projection each image onto the
Step 2: 2: compute Sliced-Wasserstein projection each image onto the
Barycenter;
Barycenter;
Barycenter;
Raw image
stein BarycenterSliced Wasserstein Barycenter Experimental results Applications Conclusion
erstein BarycenterSliced Wasserstein Barycenter Experimental results Applications Conclusion
in Barycenter Sliced Wasserstein Barycenter Experimental results Applications Conclusion
sequence
lor transfer
or transfer
olor transfer
Color harmonization of several images
Color harmonization of of several images
Color harmonization several images
Step 1: compute Sliced-Wasserstein Barycenter of color statistics;
. . . Step 1: compute Sliced-Wasserstein Barycenter color statistics;
Step 1: compute Sliced-Wasserstein Barycenter of of color statistics;
Step 2: compute Sliced-Wasserstein projection of each image onto the
. . . Step 2: compute Sliced-Wasserstein projection each image onto thethe
Step 2: compute Sliced-Wasserstein projection of of each image onto Compute
Barycenter;
Barycenter;
Barycenter;
Wasserstein
Raw image sequence
Raw image sequence
Raw image sequence
barycenter
J. RabinRabinGREYC, University Caen Caen Approximate Wasserstein Metric forfor for Texture Synthesis and Mixing
J. RabinGREYC, University of of Caen
J. – – – GREYC, University of Approximate Wasserstein Metric Texture Synthesis and Mixing
Approximate Wasserstein Metric Texture Synthesis and Mixing
Project on
the barycenter

57. Overview
• Wasserstein Distance
• Sliced Wasserstein Distance
• Color Transfer
• Wasserstein Barycenter
• Gaussian Texture Models

58. Texture Synthesis
“random”
Problem: given f0 , generate f
perceptually “similar”
Exemplar f0

59. Texture Synthesis
“random”
Problem: given f0 , generate f
perceptually “similar”
Probability
analysis
distribution
µ = µ(p)
Exemplar f0

60. Texture Synthesis
“random”
Problem: given f0 , generate f
perceptually “similar”
Probability
analysis synthesis
distribution
µ = µ(p)
Exemplar f0
Outputs f µ(p)

61. Texture Synthesis
“random”
Problem: given f0 , generate f
perceptually “similar”
Probability
analysis synthesis
distribution
µ = µ(p)
Exemplar f0
Outputs f µ(p)
Gaussian models: µ = N (m, ), parameters p = (m, ).

62. Gaussian Texture Model
Input exemplar: f0 RN d
N2
N2
d = 1 (grayscale), d = 3 (color)
N3
N1
Images N1 Videos

63. Gaussian Texture Model
Input exemplar: f0 RN d
N2
N2
d = 1 (grayscale), d = 3 (color)
N3
Gaussian model:
X µ = N (m, ) N1
Images N1 Videos
m RN d
, RN d Nd

64. Gaussian Texture Model
Input exemplar: f0 RN d
N2
N2
d = 1 (grayscale), d = 3 (color)
N3
Gaussian model:
X µ = N (m, ) N1
Images N1 Videos
m RN d
, RN d Nd
Texture analysis: from f0 RN d , learn (m, ).
highly under-determined problem.

65. Gaussian Texture Model
Input exemplar: f0 RN d
N2
N2
d = 1 (grayscale), d = 3 (color)
N3
Gaussian model:
X µ = N (m, ) N1
Images N1 Videos
m RN d
, RN d Nd
Texture analysis: from f0 RN d , learn (m, ).
highly under-determined problem.
Texture synthesis:
given (m, ), draw a realization f = X( ).
Factorize = AA (e.g. Cholesky).
Compute f = m + Aw where w drawn from N (0, Id).

66. Spot Noise Model [Galerne et al.]
Stationarity hypothesis: (periodic BC) X(· + ) X

67. Spot Noise Model [Galerne et al.]
Stationarity hypothesis: (periodic BC) X(· + ) X
Block-diagonal Fourier covariance:
ˆ
y = f computed as y ( ) = ˆ ( )f ( )
ˆ
2ix1 ⇥1 2ix2 ⇥2
ˆ
where f ( ) = f (x)e N1 + N2
x

68. Spot Noise Model [Galerne et al.]
Stationarity hypothesis: (periodic BC) X(· + ) X
Block-diagonal Fourier covariance:
ˆ
y = f computed as y ( ) = ˆ ( )f ( )
ˆ
2ix1 ⇥1 2ix2 ⇥2
ˆ
where f ( ) = f (x)e N1 + N2
x
Maximum likelihood estimate (MLE) of m from f0 :
1
i, mi = f0 (x) Rd
N x

69. Spot Noise Model [Galerne et al.]
Stationarity hypothesis: (periodic BC) X(· + ) X
Block-diagonal Fourier covariance:
ˆ
y = f computed as y ( ) = ˆ ( )f ( )
ˆ
2ix1 ⇥1 2ix2 ⇥2
ˆ
where f ( ) = f (x)e N1 + N2
x
Maximum likelihood estimate (MLE) of m from f0 :
1
i, mi = f0 (x) Rd
N x
1
MLE of : i,j = f0 (i + x) f0 (j + x) Rd d
N x

70. Spot Noise Model [Galerne et al.]
Stationarity hypothesis: (periodic BC) X(· + ) X
Block-diagonal Fourier covariance:
ˆ
y = f computed as y ( ) = ˆ ( )f ( )
ˆ
2ix1 ⇥1 2ix2 ⇥2
ˆ
where f ( ) = f (x)e N1 + N2
x
Maximum likelihood estimate (MLE) of m from f0 :
1
i, mi = f0 (x) Rd
N x
1
MLE of : i,j = f0 (i + x) f0 (j + x) Rd d
N x
ˆ ˆ
= 0, ˆ ( ) = f0 ( )f0 ( ) Cd d
is a spot noise = 0, ˆ ( ) is rank-1.

71. Example of Synthesis
Synthesizing f = X( ), X N (m, ):
= 0, ˆ ˆ
f ( ) = f0 ( )w( )
ˆ w N (N 1
,N 1/2
IdN )
Cd C
Convolve each channel with the same white noise.
Input f0 RN 3 Realizations f

72. Gaussian Optimal Transport
Input distributions (µ0 , µ1 ) with µi = N (mi , i ).
Ellipses: Ei = x Rd (mi x) i
1
(mi x) c
E0 E1

73. Gaussian Optimal Transport
Input distributions (µ0 , µ1 ) with µi = N (mi , i ).
Ellipses: Ei = x Rd (mi x) i
1
(mi x) c
E0 T E1
Theorem: If ker( 0) Im( 1) = {0},
T (x) = Sx + m1 m0 where
1/2 1/2 1/2 1/2 1/2
S= 1
+
0,1 1 0,1 =( 1 0 1 )
W2 (µ0 , µ1 )2 = tr ( 0+ 1 2 0,1 ) + ||m0 m1 ||2 ,

74. Wasserstein Geodesics
Geodesics between (µ0 , µ1 ): t [0, 1] µt
Variational caracterization: (W2 is a geodesic distance)
µt = argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2
µ

75. Wasserstein Geodesics
Geodesics between (µ0 , µ1 ): t [0, 1] µt
Variational caracterization: (W2 is a geodesic distance)
µt = argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2
µ µ1
µ0 µ
Optimal transport caracterization:
µt = ((1 t)Id + tT ) µ0

76. Wasserstein Geodesics
Geodesics between (µ0 , µ1 ): t [0, 1] µt
Variational caracterization: (W2 is a geodesic distance)
µt = argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2
µ µ1
µ0 µ
Optimal transport caracterization:
µt = ((1 t)Id + tT ) µ0
µ0
Gaussian case: µt = Tt µ0 = N (mt , t)
mt = (1 t)m0 + tm1
t = [(1 t)Id + tT ] 0 [(1 t)Id + tT ]
the set of Gaussians is geodesically convex. µ1

77. Geodesic of Spot Noises
Theorem: Let for i = 0, 1, µi = µ(f [i] ) be spot noises,
ˆ ˆ
i.e. ˆ i ( ) = f [i] ( )f [i] ( ) . Then t [0, 1], µt = µ(f [t] )
f [t] = (1 t)f [0] + tg [1]
ˆ
f [1] ( ) ˆ
f [0] ( )
ˆ
g [1] ( ) = f [1] ( )
ˆ
ˆ
|f [1] ( ) ˆ
f [0] ( )|

78. Geodesic of Spot Noises
Theorem: Let for i = 0, 1, µi = µ(f [i] ) be spot noises,
ˆ ˆ
i.e. ˆ i ( ) = f [i] ( )f [i] ( ) . Then t [0, 1], µt = µ(f [t] )
f [t] = (1 t)f [0] + tg [1]
ˆ
f [1] ( ) ˆ
f [0] ( )
ˆ
g [1] ( ) = f [1] ( )
ˆ
ˆ
|f [1] ( ) ˆ
f [0] ( )|
f [0] t f [1]
0 1

79. OT Barycenters
µ2
µ1
µ3

80. OT Barycenters
µ2
µ1
µ3

81. OT Barycenters
µ2
µ1
µ3

82. 2-D Gaussian Barycenters
Euclidean Optimal transport Rao

83. Spot Noise Barycenters
Input

84. Spot Noise Barycenters
Input

85. Results following the geodesic path: 3
Abstract.- This paper tackles static and dynamic texture mixing by combining the statistical properties of an input set of images or videos. We focus on spot noise
images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Dynamic Textures Mixing
textures that follow a stationary and Gaussian model which can be learned from the given exemplars. From here, we define, using optimal transport, the distance
f [0] f [1] f [2]
between texture models, derive the geodesic path, and define the barycenter between several texture 2 models. These derivations are useful because they allow the
4
user to navigate inside the set of texture A T L A interpolating a new texture model at each element 1 the set. From these new interpolated models, new textures
M models, B C O D E of
5
can be synthesized of arbitrary size in space and time. Numerical results obtained from a library of exemplars show the ability of our method to generate new
complex realistic static and dynamic textures. 6
7
[See web page]
0
Results following the geodesic path: <%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start
3
images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
f [0] f [1]2 f [2]
MATLABCODE 4
1
0 1 5 2 3 4 5 6
6
Exemplars 0 7
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
f [0] f [1] f [0] [2]
f [1] [2]
f f
0 1 2 3 4 5 6 7
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>
f [0] [1] [2]
f 0 f 1 2 3 4 5 6
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
f [0] [1]
f
[2]
f
0 1 2 3 4 5 6
0 1 2 3 4 5 6 7
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>
f [0] [1] [2]
f f
0 1 2 3 4 5 6 7

86. Conclusion Source image (X )
Image modeling with statistical constraints
Colorization, synthesis, mixing, . . .
Source ima
Style image (Y )
J. Rabin Wasserstein Regula

87. Conclusion Source image (X )
Image modeling with statistical constraints
Colorization, synthesis, mixing, . . .
Wasserstein distance approach Source ima
Fast sliced approximation. Style image (Y )
J. Rabin Wasserstein Regula

88. Conclusion Source image (X )
Image modeling with statistical constraints
Colorization, synthesis, mixing, . . .
14 Anonymous
Wasserstein distance approach Source ima
Fast sliced approximation. Style image (Y )
J. Rabin Wasserstein Regula
Extension to a wide range
of imaging problems.
Color transfert, segmentation, . . .
P (⇥⇥ ) P (⇥ ) P (⇥ c ) P (⇥⇥ ) in
P (⇥ ) out
P (⇥
) c