study Domain Transform for Edge-Aware Image and Video Processing
2011/12/20 Domain Transform DOMAIN TRANSFORM FOR EDGE-AWARE IMAGE AND VIDEO PROCESSING Eduardo S. L. Gastal and Manuel M. Oliveira Instituto de Inform´atica – UFRGS SIGGRAPH 2011
2011/12/20 Domain TransformEdge-aware filter• e.g. Bilateral filter • Weighting F of spacial distance and (color) range distance q0 Spacial distance is fixed. Range distance and kernel value p Fast has to be calculated Bottleneck !How to speed up? How about L1 distance ? ||PQ||1 = |xp-xq| + |yp-yq| , ||PQ||2= √((xp-xq)2+(yp-yq)2) How about replacing 2D filter with 1D vertical and horizontal filter ?
2011/12/20 Domain TransformAbstract• 1D filtering on transformed multiple-dimension domain e.g. a case of 2D filter on a color image (XY, RGB) This paper provides high quality edge-preserving smoothing after • (x, RGB) ct(x). Apply 1D horizontal filter on ct(x) • (y, RGB) ct(y). Apply 1D vertical filter on ct(y)• Real-time at arbitrary scale• Many applications of this real-time edge preserving filter • Depth-of-field effects, stylization, recoloring, colorization, detail enhancement, and tone mapping.
2011/12/20 Domain TransformContribution• Real-time high-quality edge aware filtering of image/video based on a dimensionality reduction • Smoothing on ct(X,RGB) and ct(Y, RGB) ≡ edge preserving on 2D color image 1D• Perform anisotropic edge-preserving on curves of 2D image manifold using 1D linear filters • Isotropic kernel on ct(x) ≡ anisotropic kernel on x• 2D edge-preserving filter as a sequence of 1D linear filter• The first real-time edge-preserving smoothing • Any scale kernel • Process time independent of the filter parameters • Can control kernel shape • Handle color image correctly
2011/12/20 Domain Transform It’s have been well known that … 2D Smoothing ≡ 1D Edge-preservingHowever, replace 1D Bilateral with 2D Gaussian …. High Cost !
2011/12/20 Domain Transform 2D Smoothing ≡ 1D Edge-preserving Bilateral grid Gaussian KD-tree [Chen et al. SIG07] [Adams et al. SIG09] higher dimensional functions wi w down-sample Gaussian convolution division slicinglimited for small kernels, coarse problem not real-time
2011/12/20 Domain TransformAnisotropic Diffusion Filter (AD)• c(p, t) is large when p is not a part of an edge c(p, t) is small when p is a part of an edge Unknown iterations
2011/12/20 Domain TransformEdge-avoid Wavelets (EAW)• [Raanan Fatta SIG2009] Edge-avoiding scaling func. inversely related to the pixel difference. translation-invariant trans. Edge-avoiding trans. Constrain the size of the smoothing kernel to 2^n
2011/12/20 Filter Attributes I σrRelationship to σs, σr and I σs x no longer edge-preserving as input unbounded smoothing as input
2011/12/20 FILTERING IN THE TRANSFORMED DOMAIN Normalized Convolution (NC) Interpolated Convolution (IC) Recursive Filtering (RF)
2011/12/20 Filtering in Ωw• Sample uniformly in Ωw is unnecessary • It’s easy to determine who is neighbor in Ωw I a b c p Ω I a b ct(p)-ct(a) < r a is b’s neighbor ct(c)-ct(p) > r c is not b’s neighbor c Ωw ct(p)
2011/12/20Normalized Convolution (NC)[Knutsson and Westin 93]moving-average [Dougherty 94] r
2011/12/20Interpolated Convolution (IC)[Piroddi and Petrou 04] continuous, not discrete !
2011/12/20 NC vs. ICNormalized convolution (NC) Interpolated convolution (IC) NC, IC
2011/12/20Recursive Filtering (RF) [Smith 07] a ∈ [0, 1] is a feedback coefficient An infinite impulse response (IIR) with exponential decay d ↗ ⇨ ad 0 .. stop propagation and preserve edge
2011/12/20 FILTERING 2D SIGNALS Replacing 2D filter with multiple iterations of 1D vertical and horizontal filter
2011/12/20 Filtering 2D signalsFiltering 2D Signals• Is it possible to apply domain transform on (x,y) ? • ct(p) is the accumulated L1 distance along (p, I(p)) • Only exists in surface with zero Gaussian curvature [O’Neill 06] •• Implementation • Several iterations • Horizontal pass along each image row • Vertical pass along each image column
2011/12/20 Filtering 2D signalsHow to Determine #Iterations ?• 1st iteration 2nd iteration stripes Halve σH at each iteration. Stop while σH is small enough
2011/12/20 Impulse Response Case : the left impulse without strong edge; the right impulse with strong edge #iteration = 3NC:Normalized convolution; IC:Interpolated convolution; RF:Recursive convolution; BF:Bilateral ; AD:Anisotropic Diffusion; WLS:Weighted Least Squares The NC and IC filters have Gaussian-like response, similar to AD and BF. Strong edges : Ω IC ~ AD. NC has a higher response near strong edges : pixels near strong edge edges have less neighbors in the same population, and will weight their contribution strongly. Impulse : RF ~ WLS Ωw
2011/12/20Performance Filtering on CPU Filtering on GPU• 2.8 GHz Quad Core PC • NVIDIA GeForce GTX280• 8GB memory • CUDA• NC, RF by C++• IC in MATLAB • 1M RGB * #3 = 0.007 sec. • Domain tx : 0.7 mx• 1M RGB * #3 = 0.16~0.06 sec. • Each iteration : 2 ms• 10 M RGB * #3 = 1.6 ~ 0.6 sec.• Linearly with image size • Independent on σs or σr• Faster than EAW, PLBF, CTBF
2011/12/20Comparison• Bilateral Grid [Chen et al. 07] • Faster than domain tx. • Luminance only • Due to down-sampling process, not possible for small spatial and range kernel• NVIDIA GeForce GTX280 This paper PLBF WLS 1M RGB * #3 = 0.5 M RGB = 0.1 1 M gray = 1sec 0.007 sec sec
2011/12/20Detail Manipulation• EPF decomposes image into level-of-detail : J0, J1, .. Jk • I = J0 , Di = Ji – Ji+1 input I using D0 by IC EAW by [Fattal 09] (σs=20, σr=.08) [Farbman et al. 08]
2011/12/20Tone Mapping• Edge-aware tone mapping avoids haloing and artifacts due to compression• The compressed luminance channel Lc in HDR [Farbman et al. 08] I = J0 , Di = Ji – Ji+1 μ is the mean of B local min B = 0 local max B = 112 msec on log-luminance byRF ! J1 : σs= 20, σr=0.33 J2 : σs= 50, σr=0.67 J3 : σs=100, σr=1.34 This paper (RF) WLS
2011/12/20Stylization 1/2• Abstract low-contrast region and preserving high-contrast features NC filtering edge + this edge-aware filtered image
2011/12/20Stylization 2/2 Kp : #points in the moving windows On the edge : #point ↘ dark On the smooth area : #point ↗ light• Assign each output pixel a scaled version of the value of the normalization factor Kp in NC 0.11 Kp
2011/12/20Joint Filtering• Alpha can be combined with other map to create some localized or selective stylization alpha map
2011/12/20Joint Filtering depth-of-field effect• Smooth image A based on the edges information of image B simulate depth-of-field (DoF) B is an alpha matte
2011/12/20Colorization• Propagate user color S by blurring them using the edge information User color scribbles using this RF [Levin et al. 04] (σs=100, σr=.03)
2011/12/20Recoloring• Soft segmentation – region Ri defined by color Ci
2011/12/20Conclusions and Future Work• High-quality edge-preserving filtering of image and videos in real time • First real-time edge-preserving filter on color image at arbitrary scale • 1D smoothing filter on transformed domain ≡ Edge-preserving filter • High quality 2D edge-preserving filtering by iterating 1D-filtering • independent of the filter parameters(σs, σr) (due to moving average)• Future work • Edge-preserving applications• Limitation • Not rotationally invariant