3. Video Color Transfer
• Per-frame color transfer
• Computationally intensive
• Times of day hallucination for a 3-min video
• 180 sec x 25 frame/sec x 50 sec/frame = 5 hours
• Lack of temporal consistency
• Use local affine models
4. Data-driven Hallucination
of Different Times of Day
from a Single Outdoor Photo
• Synthesize an image at a different time of
day from an input image
• Exploit a database of time-lapse videos
seen as time passes
5. R
G
B
(1) Search the
video that look
like the input
image I
(2) Find a frame at the time of the input and
another frame at the target time
(3) Warp the frame to get the warped match
frame M and the warped target frame T
(4) Model the color transforms using
local affine model learned from M and T
(5) Apply the transform
to input I and get the
hallucinated image O
6. Local Affine Model
• Models {𝐴 𝑘} describe transforms between T and M
• Wish that 𝐼 can be transformed to 𝑂 using the same 𝐴 𝑘
• Add a regularization term using a global affine model G
• A Least-squares Optimization
Ο= argmin
𝑂,{𝐴 𝑘}
𝑘
𝑣 𝑘 𝑂 − 𝐴 𝑘 𝑣 𝑘(𝐼) 2
+ ϵ
𝑘
𝑣 𝑘 𝑇 − 𝐴 𝑘 𝑣 𝑘( 𝑀)
2
+ 𝛾
𝑘
𝐴 𝑘 − 𝐺 𝐹
2
7. • For each local image block, compute an affine model 𝐴 𝑘
• Learn the color transformation between input and output
• The output should have the same structure as the input
• Simpler at a local scale
• Preserve the details
Local Affine Model
input output
𝐴1 𝐴1𝐴 𝑘
8. Local Affine Model
• Ο= argmin
𝑂,{𝐴 𝑘}
𝑘 𝑣 𝑘 𝑇 − 𝐴 𝑘 𝑣 𝑘( 𝑀) 𝐹
2
affine model linear model
• Overlap W-1
• Overlap W/2
• linearly interpolate pixel values weighted by the distance to the
center of the block
9. SLIC Super-pixels
• Partition an image into multiple segments
• Pixels with the same label share certain
characteristics
• A spatially localized version of k-means
clustering
10. Simple Linear Iterative Clustering (SLIC)
• Each pixel is associated to a feature vector
• Initialize k-mean with center of each grid tile
• Use the Lloyd algorithm to refine k-means centers and
clusters iteratively
• Each pixel can be assigned to the 2x2 centers to grid tiles
adjacent to the pixel
(a) Standard k-means (b)SLIC
11. • regionSize: nominal size of the regions (superpixels)
• regularizer: trade-off between clustering appearance and
spatial regularization
14. Transform Recipes for Efficient
Cloud Photo Enhancement
• Limited computing power and battery life of
mobile devices
• Cloud image processing applications which
preserve the overall content of an image
• Use least time and energy cost of
uploading and downloading
15. (1) Generate a compressed input I of the input image I(2) Upload this image I along with the histogram of I(3) Upsample I and applies histogram transfer to compute a proxy input 𝐼(4) Generate a proxy output 𝑂 = f( 𝐼)(5) Compute a compact recipe r using 𝐼 and 𝑂,r 𝐼 ≈ 𝑓( 𝐼)(6) Download the recipe(7) Apply it on the original input I
• Process input I with a filter f to produce output O = f(I)
• Each recipe is specific to a given input-filter pair
16. Image Decomposition
• Multi-scale decomposition
• Work in {𝑌, 𝐶 𝑏, 𝐶𝑟} color space
• Coarsely model the chrominance transformation and
sophisticatedly model the luminance transformation
• Split 𝐼 and 𝑂 into n + 1 levels 𝐿[𝐼] and 𝐿[𝑂]
• First n levels represent the details at increasingly coarser scales
• Last level is the low frequency residual which affects a large area
and affect significantly in final reconstruction
• Combined high-frequency data
H I = 𝑙=0
𝑛−1
𝐿𝑙[𝐼]
17. Layer n :
the low frequency residual
Layer 0~n-1 :
the details at increasingly coarser scales
Combined high-frequency data+
Laplacian stack
18. Compute Recipes (1)
• The low frequency residual part of the transformation
𝑅 𝑐 𝑝 =
𝐿 𝑛 𝑂𝑐 𝑝 + 1
𝐿 𝑛 𝐼𝑐 𝑝 + 1
• Chrominance Transformations
𝑝𝜖ℬ
𝐻 𝑂𝑐𝑐 𝑝 − 𝐴 𝑐 ℬ 𝐻 𝐼 𝑝 − 𝑏 𝑐(ℬ) 2
affine function
19. Compute Recipes (2)
• Luminance Transformations
• Affine function - brightness and contrast
• Multiplicative factor to each stack level - multiscale effects
• Multiplicative factor to non-linearity terms
• Segment Function
• 𝑦𝑖 = min
ℬ
𝐻 𝐼 𝑌 +
𝑖
𝑘
(max
ℬ
𝐻 𝐼 𝑌 − min
ℬ
𝐻 𝐼 𝑌 ) , 𝑖 ∈ {1, … , 𝑘 − 1}
• 𝑠𝑖 ∙ = max ∙ −𝑦𝑖 , 0
21. Lasso Regression
• Include a penalty term to constrain the size of the
coefficients
• min
𝛽0,𝛽
(
1
2𝑁 𝑖=1
𝑁
(𝑦𝑖 − 𝛽0 − 𝑥𝑖
𝑇
𝛽)2+𝜆𝑃𝛼(𝛽))
• 𝑃𝛼 𝛽 =
1−𝛼
2
𝛽 2
2
+ 𝛼 𝛽 1 = 𝑗=1
𝑝
(
1−𝛼
2
𝛽𝑗
2
+ 𝛼 𝛽𝑗 )
• The penalty term Pα(β) interpolates between the L1 norm
of β and the squared L2 norm of β
• As λ increases, the number of nonzero components
of β decreases
22. Reconstructing
• Perform the same decomposition
• Apply the corresponding recipe coefficients to each term
• L 𝑛 𝑂𝑐 𝑝 = 𝑅 𝑐 𝑝 𝐿 𝑛[𝐼𝑐](𝑝) + 1 − 1
• 𝐻ℬ O 𝑐𝑐 p = 𝐴 𝑐 ℬ 𝐻 𝐼 𝑝 + 𝑏 𝑐(ℬ)
• 𝐻ℬ O 𝑌 p = 𝐴 𝑌 ℬ 𝐻 𝐼 𝑝 + 𝑏 𝑌 ℬ +
𝑙=0
𝑛−1
𝑚𝑙 ℬ 𝐿𝑙 𝐼 𝑌 𝑝 + 𝑖=1
𝑘−1
𝑞𝑖(ℬ)𝑠𝑖(𝐻[𝐼 𝑌](𝑝))
• O = up L 𝑛 + H O 𝑐𝑐 + H[O 𝑌]
• Up-sample the low residual term
• Linearly interpolate other terms
28. • H I = 𝑙=0
𝑛
𝐿𝑙[𝐼]
• Remove the low frequency residual
• Add a layer in laplacian stack and the high frequency term
Modified Laplacian Stack Method (1)
Layer 0~n-1
Layer n
Combined high-frequency data+
36. Video color transfer
• Video color transfer using local affine models
• Find approximate nearest-neighbor matches of a video to
a set of reference patches in the first frame
• Patch match
• Ring intersection approximate nearest neighbor search
• Compute local affine models between the original first
frame and the enhanced first frame in the video
• Apply the transforms of the approximate nearest-neighbor
matches to patches in the video
𝑨 𝟏 𝑨 𝟐 𝑨 𝟑 𝑨 𝟏 𝑨 𝟐 𝑨 𝟑
37. Recipe Coefficients
• Use other regression method to stabilize the local affine
model coefficients
lasso regressionpseudo inverse
RR GR BR 1R
RG GG BG 1G
RB GB BB 1B
38. Reference
[1] Transform Recipes for Efficient Cloud Photo Enhancement
Michaël Gharbi, YiChang Shih, Gaurav Chaurasia,
Jonathan Ragan-Kelley, Sylvain Paris, Frédo Durand
SIGGRAPH ASIA 2015
[2] Data-driven Hallucination for Different Times of Day from a Single
Outdoor Photo
YiChang Shih, Sylvain Paris, Frédo Durand, William T. Freeman
SIGGRAPH ASIA 2013
[3] SLIC Superpixels Compared to State-of-the-art Superpixel Methods
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi,
Pascal Fua, and Sabine Susstrunk