Oral presentation at DICTA 2018 conference

1. C A N O N C O N F I D E N T I A L
Asymmetric recursive Gaussian filtering for
space-variant artificial bokeh
Tuan Pham
Canon Information Systems Research Australia
Oral Session 5 15:45-16:00 on Thursday 13 December 2018

2. Overview
2
1. Recursive Gaussian filter
2. Space-variant recursive filter
3. Artificial bokeh application

3. FIR filter versus IIR (recursive) filter
3
Finite Impulse Response (FIR) filter
Output is a weighted average of input samples: O(N), where N is filter size
Xn-2 Xn-1 Xn Xn+1 Xn+2
ynyn-1yn-2 yn+1 yn+2
Impulse
input
Finite
impulse
response
0
0
w1
w2
w3
w4
w5
w1
w2
w3 w4 w5
Infinite Impulse Response (IIR) filter
Output is a weighted average of input and previous output samples
O(1) computation regardless of filter size
Xn-2 Xn-1 Xn
yn-2 yn-1 yn yn+1 yn+2
xn+1 xn+2
Infinite
impulse
response 0
Causal
direction
Xn-2 Xn-1Xn
yn-2 yn-1ynyn+1 yn+2
xn+1 xn+2
Anti-causal
direction
0
∑wi = 1
for a
normalised
filter

4. Recursive Gaussian filter
4
Second-order IIR filter [1] uses up to 2 previous taps
Anti-causal
filter
x + y
Causal
filter y+
y̶
𝑦 𝑛
+
= 𝑛0 𝑥 𝑛 + 𝑛1 𝑥 𝑛−1 − 𝑑1 𝑦 𝑛−1
+
− 𝑑2 𝑦 𝑛−2
+
𝑦 𝑛
−
= 𝑚1 𝑥 𝑛+1 + 𝑚2 𝑥 𝑛+2 − 𝑑1 𝑦 𝑛+1
−
− 𝑑2 𝑦 𝑛+2
−
𝑦 𝑛 = 𝑦 𝑛
+ + 𝑦 𝑛
−
Parallel IIR implementation of
Gaussian filter
4th-order filter is more accurate
𝑦 𝑛
+
= 𝑛0 𝑥 𝑛 + 𝑛1 𝑥 𝑛−1 + 𝑛2 𝑥 𝑛−2 + 𝑛3 𝑥 𝑛−3
− 𝑑1 𝑦 𝑛−1
+
− 𝑑2 𝑦 𝑛−2
+
− 𝑑3 𝑦 𝑛−3
+
− 𝑑4 𝑦 𝑛−4
+
𝑦 𝑛
−
= 𝑚1 𝑥 𝑛+1 + 𝑚2 𝑥 𝑛+2 + 𝑚3 𝑥 𝑛+3 + 𝑚4 𝑥 𝑛+4
− 𝑑1 𝑦 𝑛+1
−
− 𝑑2 𝑦 𝑛+2
−
− 𝑑3 𝑦 𝑛+3
−
− 𝑑4 𝑦 𝑛+4
−
𝑦 𝑛 = 𝑦 𝑛
+ + 𝑦 𝑛
−
-40 -20 0 20 40
10
-6
10
-5
10
-4
10
-3
10
-2
10
-1
t
h(t)
Gaussian =10
2nd
order, RMSE=2×10-3
4nd
order, RMSE=4×10-5
-40 -20 0 20 40
-0.01
0
0.01
0.02
0.03
0.04
0.05
t
h(t)
Gaussian
2nd
order
4nd
order
Log
space
Forth-order IIR filter [2]: use up to 4 previous taps
→ more accurate
where the filter coefficients ni, mi, and di (i=1...4) are
functions of the Gaussian smoothing scale σ
[2] Farnebäck and Westin, Improving Deriche-style recursive Gaussian filters, JMIV, 2006.
[1] Deriche, Fast algorithms for low-level vision, PAMI, 1990.

5. Space-variant IIR filter: the straightforward extension [3]
5
Vary the filter coefficients ni, mi, and di according to the varying σ
This isotropic filter works OK for smoothly varying σ
not OK for discontinuous blur: colour bleeds across blur discontinuities
foveation blur: 0 ≤ σ ≤ 20 discontinuous blur σ ∈ {0, 10}
[3] Tan, et al., Performance of three recursive algorithms for fast space-variant Gaussian filtering, 2003.
y+
Σ
0
1
1
0
0
1 Σ
σ
x σ=0 σ=1 σ=1
leakage
causal filters
y̶
Σ
0
1
1
0
0
1
σ=0 σ=0 σ=1
σ
x
anti-causal filters
causal pass: bleeding anti-causal pass: no bleeding
input
output
Colour bleeds to adjacent pixels if the blur difference is large enough

6. Our solution: constraining the rate of change of directional σ’s
6
xy-separable filtering, each with 2 directions -> 4 directional sigmas:
𝜎𝑥
+
, 𝜎𝑥
−
, 𝜎𝑦
+
, 𝜎𝑦
−
, which can be different at a pixel
Σ
0
1
1
0
0
1
σ ̶
x
y
Σ
anti-causal filterscausal filters
0
1
1
0
1
σ+
0
0
1
σ 0
σ+
=0
σ̶
=0
σ ̶
=1 σ+
= 1 σ̶
=.5σ+
=.1 σ+
=0
σ̶
=0
x
+
x
-
y
+
y
-
0
10
5
To minimise leakage, the rate of increase of 𝜎+
and the rate of decrease of 𝜎− is tapered to at
most 1/3 per pixel.
input
output

7. 7
Bokeh = Japanese for visually pleasing out-of-focus blur
Application: artificial bokeh
Depth map
Large Depth-of-Field input

8. Deriche's 2-tap symmetric IIR Our 2-tap asymmetric IIR
Farneback's 4-tap symmetric IIR Our 4-tap asymmetric IIR
8
Application: artificial bokeh
Bokeh = Japanese for visually pleasing out-of-focus blur
Depth map

9. Summary
9
IIR filter is more efficient than FIR filter for large filter size
Naïve extension of existing IIR filters to space-varying Gaussian
filters produces artefacts at sudden blur discontinuities
We proposed asymmetric IIR filters that minimise intensity
leakage across blur discontinuities
Our asymmetric IIR filters produce good defocus blur for
scenes with depth discontinuities (e.g. portrait shot)

10. 10
Thank you
tuan.pham@cisra.canon.com.au

11. Details: filter normalisation
11
0
0
0
0
0
0
0
Anti-causal
filter σ̶
x + y
Causal
filter σ+
y+
y̶
+
+
w+
δ
w̶
δ
×
×
2σ+
σ+
+ σ̶
2σ̶
σ+
+ σ̶
The asymmetric left & right blur weights may not add up to 1
Need to add a small delta response to normalise the total filter

12. Application: speeded-up anisotropic filtering
12
20 iterations of Perona-Malik
anisotropic diffusion with λ=0.25,
𝑔 𝛻𝐼 = exp (− 𝛻𝐼 /5)
Our edge-stopping blur with
𝜎 = 16 × exp − 𝛻𝐼 in the inset