Optimal Transport in Imaging Sciences

Optimal Transport
in Imaging Sciences
Gabriel Peyré

www.numerical-tours.com

Statistical Image Models
Colors distribution: ) each pixel
Source image (X point in R3

Source image after color transfer

Style image (Y )

J. Rabin Wasserstein Regularization


Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer


Application to Color Transfer Sliced Wasserstein projection color style
Source image after of X to transfer
image color statistics Y

Style image (Y )

Source image (X ) Sliced Wasserstein projection of X to style

Input image
Source image (X ) Modiﬁed color transfer
Source image after
image
Style image (Y )






Application to Color Transfer Sliced Wasserstein projection color style
Source image after of X to transfer

Style image (Y )

Source image (X ) Sliced Wasserstein projection of X to style

Input image
Source image (X ) Modiﬁed color transfer
Source image after
image
Style image (Y )
Texture synthesis
Other applications: J. Rabin Wasserstein Regularization

Texture segmentation

Overview

• Wasserstein Distance

• Sliced Wasserstein Distance

• Color Transfer

• Wasserstein Barycenter

• Gaussian Texture Models

Discrete Distributions
N 1
Discrete measure: µ = pi Xi Xi Rd pi = 1
i=0 i

N 1
i=0 i

Point cloud
Constant weights: pi = 1/N .

Xi

Quotient space:
RN d / N

N 1
i=0 i

Point cloud Histogram
Constant weights: pi = 1/N . Fixed positions Xi (e.g. grid)

Xi

Quotient space: A ne space:
RN d / N {(pi )i i pi = 1}

From Images to Statistics
Source image (X )

Discretized image f RN d
fi Rd = R3
N = #pixels, d = #colors.
Source image

Style image (Y )

J. Rabin Wasserstein Regularizatio

From Images to Statistics Source image (X )

fi Rd = R3
Source image

Disclamers: images are not distributions. image (Y )
Style


Needs an estimator: f µf
Modify f by controlling µf .


fi Rd = R3
Source image

Style


Xi

Point cloud discretization: µf = fi
i


fi Rd = R3
Source image

Style


Xi

i

Histogram discretization: µf = pi Xi
i
1
Parzen windows: pi = (xi fj )
Zf j


fi Rd = R3
Source image

Style


Xi

i
Today’s focus
Histogram discretization: µf = pi Xi
i
1
Parzen windows: pi = (xi fj )
Zf j

Optimal Transport Distances
Grayscale: 1-D
Vector X RN d µX = Xi
(image, coe cients, . . . ) i
Colors: 3-D
B G

R

Grayscale: 1-D
Colors: 3-D
B G
Optimal assignment:
⇥
⇥ argmin ||Xi Y (i) ||p R
N i

Grayscale: 1-D
Colors: 3-D
B G
Optimal assignment:
⇥
N i

Wasserstein distance: Wp (µX , µY )p = ||Xi Y (i) ||p
i
Metric on the space of distributions.

Grayscale: 1-D
Colors: 3-D
B G
Optimal assignment:
⇥
N i

Wasserstein distance: Wp (µX , µY )p = ||Xi Y (i) ||p
i
Metric on the space of distributions.

Projection on statistical constraints: C = {f µf = µY }
ProjC (f ) = Y

Computing Transport Distances

Explicit solution for 1D distribution (e.g. grayscale images):
Xi
Yi
sorting the values, O(N log(N )) operations.


Xi
Yi

Higher dimensions: combinatorial optimization methods
Hungarian algorithm, auctions algorithm, etc.
O(N 5/2 log(N )) operations.
intractable for imaging problems.


Xi
Yi

Higher dimensions: combinatorial optimization methods
Hungarian algorithm, auctions algorithm, etc.
O(N 5/2 log(N )) operations.
intractable for imaging problems.

Arbitrary distributions: µ= pi Xi ⇥= qi Yi
i i
Wp (µ, ) solution of a linear program.
p

Approximate Sliced Distance
Key idea: replace transport in Rd by series of 1D transport.
Xi
[Rabin, Peyr´, Delon & Bernot 2010]
e
Projected point cloud: X = { Xi , ⇥}i .
Xi ,

Xi
e
Xi ,
Sliced Wasserstein distance: (p = 2)

SW (µX , µY )2 = W (µX , µY )2 d
|| ||=1

Xi
e
Xi ,

SW (µX , µY )2 = W (µX , µY )2 d
|| ||=1

Theorem: E(X) = SW (µX , µY )2 is of class C 1 and
E(X) = Xi Y⇥ (i) , d .
where N are 1-D optimal assignents of X and Y .

Xi
e
Xi ,

SW (µX , µY )2 = W (µX , µY )2 d
|| ||=1

Theorem: E(X) = SW (µX , µY )2 is of class C 1 and
E(X) = Xi Y⇥ (i) , d .
where N are 1-D optimal assignents of X and Y .

Possible to use SW in variational imaging problems.
Fast numerical scheme : use a few random .

Sliced Assignment

Theorem: X is a local minima of E(X) = SW (µX , µY )2
N, X = Y

Sliced Assignment

N, X = Y

Stochastic gradient descent of E(X):
Step 1: choose at random. E (X) = W (X , Y )2
Step 2: X( +1)
= X( )
E (X ( ) )

Sliced Assignment

N, X = Y

Step 2: X( +1)
= X( )
E (X ( ) )
X( )
converges to C = {X µX = µY }.

Sliced Assignment

N, X = Y

Step 2: X( +1)
= X( )
E (X ( ) )
X( )
converges to C = {X µX = µY }. Final assignment

Numerical observation: X ( )
ProjC (X (0) )

pplication to Color Transfer
Color Histogram Equalization
1
Input color images: fi RN 3 . projectioniof= to style
Sliced Wasserstein
⇥ X
N x
fi (x)


Source image (X )

f1 f0 Sliced Wasserstein projec

f0

µ1 image (Y )
Style Source image (X )
µ0

1
Sliced Wasserstein
⇥ X
N x
fi (x)
Optimal assignement: min ||f0 f1 ⇥ ||
N


Source image (X )


f0

µ1 image (Y )
µ0

1
Sliced Wasserstein
⇥ X
N x
fi (x)
N

Transport: T : f0 (x) R3 f1 ( (i)) R3

Source image (X )


f0

µ1 image (Y )
µ0
T

1
Sliced Wasserstein
⇥ X
N x
fi (x)
N

Transport: T : f0 (x) R3Application to Color Transfer R3
f1 ( (i))

˜
Equalization: ) f0 = T (f0 ) ˜
f0 = f1 Sliced Wasserstein projection of X to sty
Source image (X image color statistics Y

f1 f0 T (f )
0
Sliced Wasserstein projec
Source image (X )
T
f0

µ1 image (Y )
µ0 Source image after color transfer
µ1
Style image (Y )

T J. Rabin Wasserstein Regularization

Sliced Wasserstein Transfert
Solving min ||f0 f1 ⇥ || is computationally untractable.
N

Approximate Wasserstein projection:
˜
f0 solves min E(f ) = SW (µf , µf1 )
f
and is close to f0

Sliced Wasserstein Transfert
Solving min ||f0 f1 ⇥ || is computationally untractable.
N

Approximate Wasserstein projection:
˜
f0 solves min E(f ) = SW (µf , µf1 )
f
and is close to f0

(Stochastic) gradient descent:
f (0) = f0
f( +1)
= f( )
E(f ( ) )
f( ) ˜
f0
At convergence: µf0 = µf1
˜

Color Transfert
Sliced Wasserstein projection of X to style
Sliced Wasserstein projection of X to styl
sty

Source image (X )
Input image f0 )
Source image (X

˜
Source image after image f0
Transfered color transfer
Style image (Y ) Source image after color transfer
Target image )f1
Style image (Y

Color Exchange

Source image (X (X )
Source image )
Source image (Xf
Input image )
Source image (X )
0 TargetXimage ⇥ f1 Y
X ⇥⇥ Y
X
Y
X ⇥ Y

Style image (Y )
Style image (Y )
Transfered image f0
Style image (Y )
˜ Transfered X ˜
Y ⇥ image f1

Wasserstein Barycenter
µ2
Barycenter of {(µi , )}L :
i i=1 i =1

)
L i

2,µ
µ argmin W2 (µi , µ)2

2 (µ
i µ

W
µ
i=1 µ1
)

W2
W2 (µ 1, µ

(µ 3
,µ
)
µ3

µ2
i i=1 i =1

)
L i

2,µ

2 (µ
i µ

W
µ
i=1 µ1
)

W2
(µ 1, µ
If µi = Xi , then µ = W2

(µ 3
X

,µ
X =

)
i Xi µ3
i
Generalizes Euclidean barycenter.

µ2
i i=1 i =1

)
L i

2,µ

2 (µ
i µ

W
µ
i=1 µ1
)

W2
(µ 1, µ
If µi = Xi , then µ = W2

(µ 3
X

,µ
X =

)
i Xi µ3
i
Generalizes Euclidean barycenter.

Theorem: [Agueh, Carlier, 2010]
if µ0 does not vanish on small sets,
µ exists and is unique.

Special Case: 2 Distributions
Case L = 2:
µt argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2
µ
µ2
µ1 µ
t µt is the geodesic path.

Special Case: 2 Distributions
Case L = 2:
µt argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2
µ
µ2
µ1 µ
t µt is the geodesic path.

N
Discrete point clouds: µ= k=1 Xi (k) ,
N
Assignment: ⇥
⇥ argmin ||X1 (k) X2 ( (k))||2
N
k=1
N
µ = k=1 X (k) ,

where X = (1 t)X1 (k) + tX2 ( (k))

Linear Programming Resolution
Discrete setting: i = 1, . . . , L, µi = k Xi (k)

µ2
µ1

µ µ3


L-way interaction cost:
C(k1 , . . . , kL ) = i,j i j ||Xi (ki ) Xj (kj )||2

µ2
µ1

µ µ3


C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1
i,j

L-way coupling matrices: P PL

=1

=1

µ2
µ1

µ µ3


C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1
i,j


Linear program: =1

min Pk1 ,...,kL C(k1 , . . . , kL ) =1
P PL
k1 ,...,kL
µ2
µ1

µ µ3


C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1
i,j


Linear program: =1

min Pk1 ,...,kL C(k1 , . . . , kL ) =1
P PL
k1 ,...,kL

Barycenter: µ2

µ = Pk1 ,...,kL µ1
X (k1 ,...,kL )
k1 ,...,kL
L µ µ3
X (k1 , . . . , kL ) = i=1 i Xi (ki )

Sliced Wasserstein Barycenter
Sliced-barycenter: µX that solves
min i SW (µX , µXi )2
X
i

X
i
Gradient descent: Ei (X) = SW (µX , µXi )2
L
X( +1)
= X( )
⇥ i Ei (X ( ) )
i=1

X
i
L
X( +1)
= X( )
⇥ i Ei (X ( ) )
i=1
Advantages:
µX is a sum of N Diracs.
Smooth optimization problem.
Disadvantage:
Non-convex problem
local minima.

X
i
L
X( +1)
= X( )
⇥ i Ei (X ( ) ) µ1
i=1
Advantages:
µX is a sum of N Diracs.
Smooth optimization problem.
Disadvantage:
µ2 µ3
Non-convex problem
local minima.

. . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics;
Step 1: compute Sliced-Wasserstein Barycenter color statistics;
Step 1: 1: compute Sliced-Wasserstein Barycenter color statistics;
Application: Color Harmonization
. . . Step compute Sliced-Wasserstein projection ofofof each image onto the
Step 2: compute Sliced-Wasserstein projection each image onto the
Step 2: 2: compute Sliced-Wasserstein projection each image onto the
Barycenter;
Barycenter;
Barycenter;

Raw image
sequence

Raw image sequence
Raw image sequence
Raw image sequence

J. RabinRabinGREYC, University Caen Caen Approximate Wasserstein Metric forfor for Texture Synthesis and Mixing
J. RabinGREYC, University of of Caen
J. – – – GREYC, University of Approximate Wasserstein Metric Texture Synthesis and Mixing
Approximate Wasserstein Metric Texture Synthesis and Mixing

Barycenter;
Barycenter;
Barycenter;

Raw image
sequence

Compute
Wasserstein
Raw image sequence
Raw image sequence
Raw image sequence
barycenter

Barycenter;
Barycenter;
Barycenter;

Raw image
stein BarycenterSliced Wasserstein Barycenter Experimental results Applications Conclusion
erstein BarycenterSliced Wasserstein Barycenter Experimental results Applications Conclusion
in Barycenter Sliced Wasserstein Barycenter Experimental results Applications Conclusion
sequence
lor transfer
or transfer
olor transfer

Color harmonization of several images
Color harmonization of of several images
Color harmonization several images
Step 1: compute Sliced-Wasserstein Barycenter of color statistics;
. . . Step 1: compute Sliced-Wasserstein Barycenter color statistics;
Step 1: compute Sliced-Wasserstein Barycenter of of color statistics;
Step 2: compute Sliced-Wasserstein projection of each image onto the
. . . Step 2: compute Sliced-Wasserstein projection each image onto thethe
Step 2: compute Sliced-Wasserstein projection of of each image onto Compute
Barycenter;
Barycenter;
Barycenter;
Wasserstein
Raw image sequence
Raw image sequence
Raw image sequence
barycenter

Project on
the barycenter

Texture Synthesis
“random”
Problem: given f0 , generate f
perceptually “similar”

Exemplar f0

Texture Synthesis
“random”

Probability
analysis
distribution
µ = µ(p)

Exemplar f0

Texture Synthesis
“random”

Probability
analysis synthesis
distribution
µ = µ(p)

Exemplar f0

Outputs f µ(p)

Texture Synthesis
“random”

Probability
analysis synthesis
distribution
µ = µ(p)

Exemplar f0

Outputs f µ(p)

Gaussian models: µ = N (m, ), parameters p = (m, ).

Gaussian Texture Model
Input exemplar: f0 RN d
N2
N2
d = 1 (grayscale), d = 3 (color)
N3

N1
Images N1 Videos

N2
N2
N3
Gaussian model:
X µ = N (m, ) N1
Images N1 Videos
m RN d
, RN d Nd

N2
N2
N3
Gaussian model:
X µ = N (m, ) N1
Images N1 Videos
m RN d
, RN d Nd

Texture analysis: from f0 RN d , learn (m, ).
highly under-determined problem.

N2
N2
N3
Gaussian model:
X µ = N (m, ) N1
Images N1 Videos
m RN d
, RN d Nd

Texture analysis: from f0 RN d , learn (m, ).
highly under-determined problem.
Texture synthesis:
given (m, ), draw a realization f = X( ).
Factorize = AA (e.g. Cholesky).
Compute f = m + Aw where w drawn from N (0, Id).

Spot Noise Model [Galerne et al.]
Stationarity hypothesis: (periodic BC) X(· + ) X

Block-diagonal Fourier covariance:
ˆ
y = f computed as y ( ) = ˆ ( )f ( )
ˆ
2ix1 ⇥1 2ix2 ⇥2
ˆ
where f ( ) = f (x)e N1 + N2

x

ˆ
ˆ
2ix1 ⇥1 2ix2 ⇥2
ˆ
where f ( ) = f (x)e N1 + N2

x
Maximum likelihood estimate (MLE) of m from f0 :
1
i, mi = f0 (x) Rd
N x

ˆ
ˆ
2ix1 ⇥1 2ix2 ⇥2
ˆ
where f ( ) = f (x)e N1 + N2

x
1
i, mi = f0 (x) Rd
N x
1
MLE of : i,j = f0 (i + x) f0 (j + x) Rd d
N x

ˆ
ˆ
2ix1 ⇥1 2ix2 ⇥2
ˆ
where f ( ) = f (x)e N1 + N2

x
1
i, mi = f0 (x) Rd
N x
1
MLE of : i,j = f0 (i + x) f0 (j + x) Rd d
N x
ˆ ˆ
= 0, ˆ ( ) = f0 ( )f0 ( ) Cd d
is a spot noise = 0, ˆ ( ) is rank-1.

Example of Synthesis
Synthesizing f = X( ), X N (m, ):

= 0, ˆ ˆ
f ( ) = f0 ( )w( )
ˆ w N (N 1
,N 1/2
IdN )

Cd C
Convolve each channel with the same white noise.

Input f0 RN 3 Realizations f

Gaussian Optimal Transport
Input distributions (µ0 , µ1 ) with µi = N (mi , i ).

Ellipses: Ei = x Rd (mi x) i
1
(mi x) c

E0 E1

Gaussian Optimal Transport
Input distributions (µ0 , µ1 ) with µi = N (mi , i ).

Ellipses: Ei = x Rd (mi x) i
1
(mi x) c

E0 T E1

Theorem: If ker( 0) Im( 1) = {0},
T (x) = Sx + m1 m0 where
1/2 1/2 1/2 1/2 1/2
S= 1
+
0,1 1 0,1 =( 1 0 1 )

W2 (µ0 , µ1 )2 = tr ( 0+ 1 2 0,1 ) + ||m0 m1 ||2 ,

Wasserstein Geodesics
Geodesics between (µ0 , µ1 ): t [0, 1] µt
Variational caracterization: (W2 is a geodesic distance)
µt = argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2
µ

µ µ1
µ0 µ
Optimal transport caracterization:
µt = ((1 t)Id + tT ) µ0

µ µ1
µ0 µ
Optimal transport caracterization:
µt = ((1 t)Id + tT ) µ0
µ0
Gaussian case: µt = Tt µ0 = N (mt , t)

mt = (1 t)m0 + tm1
t = [(1 t)Id + tT ] 0 [(1 t)Id + tT ]

the set of Gaussians is geodesically convex. µ1

Geodesic of Spot Noises
Theorem: Let for i = 0, 1, µi = µ(f [i] ) be spot noises,
ˆ ˆ
i.e. ˆ i ( ) = f [i] ( )f [i] ( ) . Then t [0, 1], µt = µ(f [t] )
f [t] = (1 t)f [0] + tg [1]
ˆ
f [1] ( ) ˆ
f [0] ( )
ˆ
g [1] ( ) = f [1] ( )
ˆ
ˆ
|f [1] ( ) ˆ
f [0] ( )|

Geodesic of Spot Noises
Theorem: Let for i = 0, 1, µi = µ(f [i] ) be spot noises,
ˆ ˆ
i.e. ˆ i ( ) = f [i] ( )f [i] ( ) . Then t [0, 1], µt = µ(f [t] )
f [t] = (1 t)f [0] + tg [1]
ˆ
f [1] ( ) ˆ
f [0] ( )
ˆ
g [1] ( ) = f [1] ( )
ˆ
ˆ
|f [1] ( ) ˆ
f [0] ( )|

f [0] t f [1]
0 1

OT Barycenters

µ2

µ1
µ3

2-D Gaussian Barycenters

Euclidean Optimal transport Rao

Results following the geodesic path: 3
Abstract.- This paper tackles static and dynamic texture mixing by combining the statistical properties of an input set of images or videos. We focus on spot noise
images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Dynamic Textures Mixing
textures that follow a stationary and Gaussian model which can be learned from the given exemplars. From here, we define, using optimal transport, the distance
f [0] f [1] f [2]
between texture models, derive the geodesic path, and define the barycenter between several texture 2 models. These derivations are useful because they allow the
4
user to navigate inside the set of texture A T L A interpolating a new texture model at each element 1 the set. From these new interpolated models, new textures
M models, B C O D E of
5
can be synthesized of arbitrary size in space and time. Numerical results obtained from a library of exemplars show the ability of our method to generate new
complex realistic static and dynamic textures. 6
7

[See web page]
0

Results following the geodesic path: <%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start
3
images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
f [0] f [1]2 f [2]
MATLABCODE 4
1
0 1 5 2 3 4 5 6
6
Exemplars 0 7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
f [0] f [1] f [0] [2]
f [1] [2]
f f
0 1 2 3 4 5 6 7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>
f [0] [1] [2]
f 0 f 1 2 3 4 5 6

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
f [0] [1]
f
[2]
f
0 1 2 3 4 5 6
0 1 2 3 4 5 6 7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>
f [0] [1] [2]
f f
0 1 2 3 4 5 6 7

Conclusion Source image (X )

Image modeling with statistical constraints
Colorization, synthesis, mixing, . . .

Source ima

Style image (Y )

J. Rabin Wasserstein Regula



Wasserstein distance approach Source ima

Fast sliced approximation. Style image (Y )




14 Anonymous
Wasserstein distance approach Source ima

Fast sliced approximation. Style image (Y )


Extension to a wide range
of imaging problems.
Color transfert, segmentation, . . .
P (⇥⇥ ) P (⇥ ) P (⇥ c ) P (⇥⇥ ) in
P (⇥ ) out
P (⇥
) c

Optimal Transport in Imaging Sciences

More Related Content

What's hot

Similar to Optimal Transport in Imaging Sciences

More from Gabriel Peyré

Recently uploaded

Optimal Transport in Imaging Sciences