Sparsity Based Super Resolution Using Color Channel Constraints

Background Color SR Color Dictionary Learning Experimental Results Conclusion
Sparsity Based Super Resolution
Using Color Channel Constraints
Hojjat Mousavi, Vishal Monga
School of Electrical Engineering and Computer Science
The Pennsylvania State University
September 27, 2016
iPAL Color Super Resolution 1/28

Super Resolution - Problem Deﬁnition
• Multi-frame SR: Traditional Super-resolution problem is the process of
combining multiple low resolution images to form a higher resolution one
• Resulting image should represent reality better than all the input images.
• Single-image SR: given a single low-resolution input, reconstruct a
high-resolution version of the input.
• Advantage: more widely applicable than multi-frame approaches.
• Challenge: single-image super-resolution is an extremely ill-posed
problem.

Sparsity-based Super resolution - Basic idea 1,2
• Construct two coupled dictionaries based on image patches in
luminance (Y) channel
1 Low resolution dictionary: DDDl (High frequency features)
2 High resolution dictionary: DDDh (Actual high resolution patches)
• Atoms of each dictionary correspond to each other and are LR-HR
counterparts of each other extracted from the same locations
1Wright et al. CVPR 2008
2Wright et al. TIP 2009

Sparsity-based Super resolution
• SR for an unseen low resolution image:
1 Extract overlapping patches (overlapped tiling) (yyyl)
2 For each patch ﬁnd the low resolution representation using DDDl
xxx∗
= argmin
xxx
1
2
||yyyl −DDDlxxx||2
2 +λ||xxx||1
Find the sparse linear representation of low resolution patch based on LR
dictionary
3 Find the high resolution representation using DDDh and the same xxx∗.
yyyh = DDDhxxx∗
4 construct the high resolution image from high-res patches.

Color Super Resolution
• YCbCr space. Apply Bicubic interpolation on Cb and Cr channels.
• Human eye is more sensitive to luminance than chrominance
• Some images have varying amount of luminance and chrominance
geometry
• Chrominance channels also contain useful information
• Super-resolution only on luminance channel may not get the best results
• Luminance edge (in Y) → present in R, G and B channels
• Jointly account for cross channel information in an adaptive manner

• How to capture edge similarities?
• Extract edge information in RGB channels
• Find patches that should have have high edge correlation based on amount
of color information in each patch
• Encourage edge similarity in selected patches of high resolution image
• Similarity (Correlation) between edges in different channels. (Based on HR
image): Example: SSSryyyhr
−SSSgyyyhg 2 Or correlation (SSSryyyhr
)T(SSSgyyyhg
)
where SSSr,SSSg,SSSb are highpass edge detector ﬁlters
• Color constraints: Edge differences across color channels are minimized
for selected patches3,4,5
SSSryyyhr −SSSgyyyhg 2 < εrg
SSSgyyyhg −SSSbyyyhb 2 < εgb
SSSbyyyhb
−SSSryyyhr 2 < εbr
3Srinivas et al. CIC 2010
4Farsiu et al. TIP 2006
5Menon et al. TIP 2009

• High resolution representation of patches is incorporated in the cost
function by the following assumption of conventional SR methods.
yyyhr = DDDhrxxxr, yyyhg = DDDhgxxxg, yyyhb
= DDDhb
xxxb
• Incorporating RGB channel information and exploiting our multi-task
framework result in the following optimization problem:
argminxxxc
c={r,g,b}
1
2
yyylc
−DDDlc
xxxc
2
2 +λ xxxc 1
+τ SSSrDDDhr
xxxr −SSSgDDDhg
xxxg
2
2 + SSSgDDDhg
xxxg −SSSbDDDhb
xxxb
2
2 + SSSbDDDhb
xxxb −SSSrDDDhr
xxxr
2
2 .
• Note: Without color channel constraints → Three independent
optimization problems
• With additional color constraints → One optimization problem with
quadratic constraints on pairs of channels
• τ is very crucial and we pick it in an adaptive manner

• Introducing DDD,DDDl,xxx,yyy, we can simply the cost function to
xxx = argminxxx xxxTDDDxxx−yyyTDDDlxxx+λ xxx 1. → FISTA6
where
DDD =



1
2 DDDT
lr
DDDlr +2τDDDT
hr
SSST
r SSSrDDDhr −2τDDDT
hr
SSST
r SSSgDDDhg 000
000 1
2 DDDT
lg
DDDlg +2τDDDT
hg
SSST
g SSSgDDDhg −2τDDDT
hg
SSST
g SSSbDDDhb
−2τDDDT
hb
SSST
b
SSSrDDDhr 000 1
2 DDDT
lb
DDDlb
+2τDDDT
hb
SSST
b
SSSbDDDhb



xxx =


xxxr
xxxg
xxxb

, yyy =


yyylr
yyylg
yyylb

, DDDl =


DDDlr 000 000
000 DDDlg 000
000 000 DDDlb


• Note that matrix DDD can capture cross channel constraints by adding a
term to the appropriate locations
• SSSr,SSSg,SSSb are gradient operators in RGB channels.
6Beck et al. SIAM Journal of Imaging Sciences, 2009

Joint Dictionary Learning for Color Channels
• Correlation between color channels can be even better captured if the
individual color channel dictionaries are also designed to facilitate the
same.
• Given a set of N sampled training image patch pairs {YYYh,YYYl}.
YYYh = {yyy1
h,yyy2
h,...,yyyN
h }: set of HR patches sampled from training
YYYl = {yyy1
l ,yyy2
l ,...,yyyN
l }: set of corresponding LR patches.
• A new joint learning for multi channel dictionary learning is proposed:
arg min
DDDh,DDDl,xxxi
1
N
N
i=1
γ
2
yyyi
l −DDDlxxxi 2
2 +
1−γ
2
yyyi
h −DDDhxxxi 2
2
+τ SSSrDDDhrxxxi
r −SSSgDDDhgxxxi
g
2
2
+ SSSgDDDhgxxxi
g −SSSbDDDhb
xxxi
b
2
2
+ SSSbDDDhb
xxxi
b −SSSrDDDhrxxxi
r
2
2 +λ xxxi
1
st. DDDh(:,k) 2
2 ≤ 1, DDDl(:,k) 2
2 ≤ 1, k = 1,2,...,K

Joint Dictionary Learning for Color Channels
L2 =
1
N
N
i=1
γ
2
yyyi
l −DDDlxxxi 2
2 +
1−γ
2
yyyi
h −DDDhxxxi 2
2 +λ xxxi
1
+2τxxxiT
DDDT
hSSST
(III −PPPT
s )SSSDDDhxxxiT
=
γ
2N
YYYl −DDDlXXX 2
F +
1−γ
2N
YYYh −DDDhXXX 2
F +
λ
N
XXX 1
+
2τ
N
Tr XXXT
DDDT
hSSST
(III −PPPT
s )SSSDDDhXXXT
.
where XXX = [xxx1 xxx2 ... xxxN].
Alternatively solve for XXX, DDDl and DDDh
xxx =


xxxr
xxxg
xxxb

, yyy =


yyylr
yyylg
yyylb

, DDDl =


DDDlr 000 000
000 DDDlg 000
000 000 DDDlb

, DDDh =


DDDhr 000 000
000 DDDhg 000
000 000 DDDhb


SSS =


SSSr 000 000
000 SSSg 000
000 000 SSSb

, PPPs =


000 000 IIIp2×p2
IIIp2×p2 000 000
000 IIIp2×p2 000



Solution for XXX
• DDDl and DDDh ﬁxed.
• Optimize over XXX whose columns can be obtained independently.
• For each column of XXX (i = 1...N) we can rewrite the cost function as:
xxxi
= argmin
xxx
γ
2
yyyi
l −DDDlxxx 2
F +
1−γ
2
yyyi
h −DDDhxxx 2
F +λ xxx 1
+2τxxxT
DDDT
hSSST
(III −PPPT
s )SSSDDDhxxxT
= argmin
xxx
xxxT
[
γ
2
DDDT
l DDDl +
1−γ
2
DDDT
hDDDh
+2τDDDT
hSSST
(III −PPPT
s )SSSDDDh]xxx
− γyyyiT
l DDDl +(1−γ)yyyiT
h DDDh xxx+λ xxx 1
= argmin
xxx
xxxT
AAAxxx−bbbT
xxx +λ xxx 1
AAA = γ
2DDDT
l DDDl + 1−γ
2 DDDT
hDDDh +2τDDDT
hSSST(III −PPPT
s )SSSDDDh
bbbiT
= γyyyiT
l DDDl +(1−γ)yyyiT
h DDDh.
• Can be solved using FISTA7
7Beck et al. SIAM Journal of Imaging Sciences, 2009

Solution for DDDl
• Fix XXX and DDDh, the cost function reduces to:
DDDl = argmin
DDDl
YYYl −DDDlXXX 2
F
s.t. DDDl(:,k) 2
2 ≤ 1, k = 1,2,...,K
DDDl is block diagonal .
• Split into three separate dictionary learning procedures as follows where
c ∈ {r,g,b}.
DDDlc = argmin
DDDlc
YYYlc −DDDlcXXXc
2
F
s.t. DDDlc (:,k) 2
2 ≤ 1, k = 1,2,...,K
where XXXc = [xxx1
c xxx2
c ... xxxN
c ], YYYlc = [yyy1
c yyy2
c ... yyyN
c ] and c takes the subscripts from
{r,g,b} indicating a speciﬁc color channel.
• Each of the above dictionaries are learnt by the ODL method8.
8Mairal et al. ICML, 2009.

Solution for DDDh
• Finally when XXX and DDDl are ﬁxed, L2 reduces to:
DDDh = argmin
DDDh
1
N
N
i=1
1−γ
2
yyyi
h −DDDhxxxi 2
2
+2τxxxiT
DDDT
hSSST
(III −PPPT
s )SSSDDDhxxxiT
s.t DDDh(:,k) 2
2 ≤ 1, k = 1,2,...,K.
• Not very straight forward to solve → ADMM9.
• Deﬁne the function g(DDDh,ZZZ) as follows:
g(DDDh,ZZZ) =
1
N
N
i=1
1−γ
2
yyyi
h −DDDhxxxi 2
2 +2τxxxiT
DDDT
hSSST
(III −PPPT
s )SSSZZZxxxiT
• Solve the equivalent bi-convex problem:
DDDh = argmin
DDDh,ZZZ
g(DDDh,ZZZ)
s.t DDDh −ZZZ = 000,
DDDh(:,k) 2
2 ≤ 1, k = 1,2,...,K.
9Boyd et al. Foundations and Trends in Machine Learning, 2011

Solution for DDDh
Iterative steps of ADMM until a convergence is achieved are as follows:
1 Find DDDt+1
h :
DDDt+1
h = argmin
DDDh
1
N
N
i=1
1−γ
2
yyyi
h −DDDhxxxi 2
2
+2τxxxiT
DDDT
hSSST
(III −PPPT
s )SSSZZZt
xxxiT
+
ρ
2
DDDh −ZZZt
+UUUt 2
F
s.t. DDDh(:,k) 2
2 ≤ 1, k = 1,...,K.
2 Find ZZZt+1:
ZZZt+1
= argmin
ZZZ
2τ
N
N
i=1
xxxiT
DDDt+1T
h SSST
(III −PPPT
s )SSSZZZt
xxxiT
+
ρ
2
DDDt+1
h −ZZZ +UUUt 2
F
3 Find UUUt+1: UUUt+1
= UUUt
+DDDt+1
h −ZZZt+1
Solutions to steps 1 and 2 of the ADMM procedure are not straightforward
and details are in the paper.

• More analytical Results on how to solve optimization problems at each
stage
• Extensive experimental validations in addition to high quality images
• Implementation and MATLAB toolbox
All Available online at:
http://signal.ee.psu.edu/MCcSR.html

State of the art methods to compare with
Single image super resolution methods that incorporate sparsity methods:
1 Sparsity Constrained super resolution (ScSR)10
2 Single Image Scale-up using Sparse Representation 11
3 Adjusted Anchored Neighborhood Regression for Fast Example-Based
Super-Resolution (ANR+)12
4 Global Regression for Fast Super-Resolution (GR)13
5 Neighbor Embedding with Locally Linear Embedding (NE+LLE) 14
6 Neighbor Embedding with NonNegative Least Squares (NE+NNLS) 15
7 Single Image SR using sparse regression and natural image prior16:
Using sparse kernel ridge regression and natural image priors.
8 Image and Video Upscaling from Local Self-Examples17
10Yang, IEEE TIP, 2012
11Zeyde et al, Springer, Curves and Surfaces, 2012
12Timofte et al. ACCV 2014
13Timofte et al. ICCV 2013
14Chang et al. CVPR 2004
15Bevilazqua et al. BMVC 2012
16Kim et al. IEEE Tran on PAMI, 2010
17Freeman et al, ACM Transactions on Graphics, 2011

Experimental Set Up
• Dictionary size: 512 atoms - 100,000 image patches are sampled
• Scaling factor: 2x, 3x, 4x
• Patch size: 5×5, 7×7, 9×9 pixels.
• Quantitative measures: PSNR, SSIM, S-CIELAB18
18Zhang et al., in Proc. IEEE COMPCON Symp. Dig., 1997.

(a) PSNR and SSIM - scale 2:
Top: Original, Bicubic (30.46, 0.840), Zeyde et al. (31.97, 0.887),
Middle: GR (31.70, 0.879), ANR (32.09, 0.889), NENNLS (31.87, 0.884)
Bottom: NELLE (32.03, 0.889), MCcSR (32.23, 0.899), ScSR (32.14,
0.893) .
(b) SCIELAB error map - scale 2:
Top: Original, Bicubic (1.898e4), Zeyde et al. (1.127e4),
Middle: GR (1.198e4), ANR (1.077e4), NENNLS (1.159e4)
Bottom: NELLE (1.099e4), MCcSR (9.770e3) , ScSR (1.014e4).

(c) PSNR and SSIM - scale 3:
0.740) .
(d) SCIELAB error map - scale 3:

(e) PSNR and SSIM - scale 4:
0.608) .
(f) SCIELAB error map - scale 4:

Results - Scale 3
Table: PSNR results of different methods for various images with scaling factor of 3.
Images
PSNR (dB)
Bicub Zeyde GR ANR NENNLS NELLE MCcSR ScSR
baby 38.42 39.51 39.38 39.56 39.22 39.49 39.51 39.40
butterﬂy 28.73 30.60 29.73 30.57 30.29 30.42 30.59 30.64
bird 36.37 37.90 37.44 37.92 37.68 37.90 38.02 37.59
face 35.96 36.44 36.40 36.50 36.39 36.47 36.48 36.37
foreman 35.76 37.67 36.84 37.71 37.37 37.69 37.74 37.64
coastguard 31.31 31.91 31.78 31.84 31.77 31.83 31.95 31.83
ﬂowers 30.92 31.84 31.62 31.88 31.68 31.80 32.07 31.87
head 36.02 36.47 36.42 36.52 36.40 36.50 36.51 36.42
lenna 35.26 36.23 35.99 36.29 36.11 36.24 36.33 36.14
man 31.78 32.68 32.44 32.71 32.50 32.65 32.75 32.68
pepper 35.25 36.27 35.77 36.13 35.99 36.12 36.30 36.20
average 33.08 34.06 33.76 34.07 33.88 34.03 34.14 34.00

Results - Scale 3
Table: SSIM results of different methods for various images with scaling factor of 3.
Images
SSIM
baby 0.88 0.90 0.90 0.90 0.89 0.90 0.90 0.89
butterﬂy 0.79 0.85 0.80 0.84 0.84 0.84 0.85 0.85
bird 0.90 0.92 0.91 0.92 0.92 0.92 0.93 0.91
face 0.72 0.74 0.74 0.74 0.74 0.74 0.75 0.74
foreman 0.89 0.91 0.90 0.91 0.90 0.91 0.91 0.90
coastguard 0.57 0.62 0.63 0.62 0.61 0.62 0.63 0.62
ﬂowers 0.77 0.80 0.79 0.80 0.79 0.80 0.81 0.80
head 0.72 0.74 0.74 0.75 0.74 0.74 0.75 0.74
lenna 0.78 0.80 0.80 0.80 0.80 0.80 0.81 0.80
man 0.72 0.76 0.76 0.77 0.76 0.76 0.76 0.76
pepper 0.78 0.80 0.79 0.80 0.79 0.79 0.80 0.79
average 0.745 0.776 0.769 0.778 0.771 0.775 0.785 0.774

Results - Scale 3
Table: S-CIELAB error results of different methods for various images with scaling
factor of 3.
Images
S-CIELAB
baby 2.07E+04 1.36E+04 1.40E+04 1.32E+04 1.47E+04 1.34E+04 1.34E+04 1.50E+04
butterﬂy 2.28E+04 1.55E+04 1.84E+04 1.55E+04 1.60E+04 1.60E+04 1.54E+04 1.49E+04
bird 1.07E+04 7.36E+03 8.02E+03 7.21E+03 7.73E+03 7.30E+03 6.50E+03 7.81E+03
face 3.79E+03 2.71E+03 2.73E+03 2.57E+03 2.73E+03 2.61E+03 2.47E+03 2.70E+03
foreman 8.46E+03 3.90E+03 4.79E+03 3.48E+03 4.01E+03 3.62E+03 3.72E+03 3.89E+03
coastguard 1.96E+04 1.71E+04 1.70E+04 1.70E+04 1.76E+04 1.71E+04 1.69E+04 1.70E+04
ﬂowers 4.47E+04 3.75E+04 3.89E+04 3.69E+04 3.84E+04 3.74E+04 3.29E+04 3.70E+04
head 3.79E+03 2.69E+03 2.74E+03 2.54E+03 2.79E+03 2.61E+03 2.42E+03 2.65E+03
lenna 2.44E+04 1.74E+04 1.85E+04 1.67E+04 1.79E+04 1.69E+04 1.58E+04 1.72E+04
man 3.80E+04 2.91E+04 3.03E+04 2.84E+04 3.02E+04 2.89E+04 2.88E+04 2.95E+04
pepper 2.48E+04 1.91E+04 2.15E+04 1.96E+04 2.02E+04 1.95E+04 1.73E+04 1.91E+04
average 2.79E+04 2.27E+04 2.36E+04 2.24E+04 2.33E+04 2.26E+04 2.14E+04 2.28E+04

Effect of RGB constraints
Figure: Visual Images as well as S-CIELAB error maps are shown for a scaling factor
of 3. From left to right for each row images correspond to: Original image, applying
SR separately on RGB channels (36.26, 0.83, 1.57e4), ScSR (36.13, 0.83, 1.67e4)
and MCcSR (36.67, 0.85, 1.43e4). Numbers in parenthesis are PSNR, SSIM and
SCIELAB error measures.

Effect of RGB constraints
Figure: Visual Images as well
as S-CIELAB error maps are
shown for a scaling factor of 3.
From left to right for each row
Images correspond to: Original
Image, applying SR separately
on RGB channels, ScSR,
MCcSR
Table: Quantitative measures to show effectiveness of color constraints in SR for a scaling factor of 3.
Images
PSNR SSIM S-CIELAB
Separate RGB ScSR MCcSR Separate RGB ScSR MCcSR Separate RGB ScSR MCcSR
comic 28.37 28.25 28.51 0.74 0.74 0.75 2.80e4 3.00e4 2.70e4
baboon 26.95 26.95 27.11 0.53 0.52 0.54 9.93e4 1.01e5 9.57e4
pepper 36.14 35.85 36.30 0.79 0.77 0.81 1.93e4 2.27e5 1.73e4
bird 37.71 37.59 38.02 0.92 0.912 0.93 7.28e3 8.54e3 6.50e3

Performance Under Noise
Figure: Super Resolution performance under difference noise
standard deviations: 4,6,8,12 (from top to bottom ) with different methods:
Original, bicubic, MCcSR, ScSR (from left to right)
Table: Average performance under different
noise levels.
Measure Method σ = 0 σ = 4 σ = 6 σ = 8 σ = 12
PSNR
Bicubic 33.08 32.99 32.75 32.50 31.88
ScSR 34.00 33.95 33.92 33.90 33.86
MCcSR 34.14 34.11 34.09 34.09 34.07
SSIM
Bicubic 0.745 0.731 0.698 0.672 0.619
ScSR 0.774 0.772 0.766 0.761 0.752
MCcSR 0.785 0.783 0.780 0.775 0.768
SCIELAB
Bicubic 2.79E4 2.92E4 4.40E4 5.25E4 6.31E4
ScSR 2.28E4 2.31E4 2.36E4 2.39E4 2.43E4
MCcSR 2.14E4 2.16E4 2.20E4 2.21E4 2.23E4

Dictionary size
• Dictionaries of size 16,32,64,128,256 and 512 are trained.
Figure: Effect of dictionary size on PSNR, SSIM and S-CIELAB error

Conclusion and Future Work
• Sparsity → powerful prior for the ill-posed problem of single image super
resolution
• Cross channel information and color constraints → Regularizing the
optimization problem for boosting SR performance
• Under different scaling factors, different noise levels, different dictionary
sizes the proposed MCcSR method outperforms the state of the art.
• Expedite the sparse coding problem using neural networks
• Introduce other objective measurements rather than MSE for quality
assessment or in the objective function
• Apply other cross channel constraints or color constraint that can
improve super resolution performance

Thank you!

Backup Slides

Dictionary Atoms
• Low resolution dictionary atoms - Red channel only- 9 by 9 pixels
patches - Extracted from features
Figure: Low resolution dictionary atoms -
Red channel only- 9 by 9 pixels patches -
Extracted from features
Figure: High resolution dictionary atoms -
RGB - 9 by 9 pixels patches

Color Adaptive Patch Processing
• Not all image patches have the same color information (chrominance)
• Parameter τ can be used to control high frequency correlation among
color channels.
• Calculate the variations in color information → Adaptively control τ.
β =
1
2s
HHH1yyyCb + HHH1yyyCr
HHH1yyyY
+
HHH2yyyCb + HHH2yyyCr
HHH2yyyY
where s is normalization parameter, HHH1 and HHH2 are high-pass Scharr
operators.

(a) PSNR and SSIM - scale 2:
Bottom: NELLE (28.66, 0.688), MCcSR (28.78, 0.705, ScSR (28.69, 0.692)
.
(b) SCIELAB error map - scale 2:

(c) PSNR and SSIM - scale 3:
0.524) .
(d) SCIELAB error map - scale 3:

(e) PSNR and SSIM - scale 4:
0.415) .
(f) SCIELAB error map - scale 4:

Sparsity Based Super Resolution Using Color Channel Constraints

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Sparsity Based Super Resolution Using Color Channel Constraints

Similar to Sparsity Based Super Resolution Using Color Channel Constraints (20)

Recently uploaded

Recently uploaded (20)

Sparsity Based Super Resolution Using Color Channel Constraints