study Domain Transform for Edge-Aware Image and Video Processing

2,838 views

Published on

Published in: Technology, Business
  • Be the first to comment

study Domain Transform for Edge-Aware Image and Video Processing

  1. 1. 2011/12/20 Domain Transform DOMAIN TRANSFORM FOR EDGE-AWARE IMAGE AND VIDEO PROCESSING Eduardo S. L. Gastal and Manuel M. Oliveira Instituto de Inform´atica – UFRGS SIGGRAPH 2011
  2. 2. 2011/12/20 Domain TransformEdge-aware filter• e.g. Bilateral filter • Weighting F of spacial distance and (color) range distance q0 Spacial distance is fixed. Range distance and kernel value p Fast  has to be calculated  Bottleneck !How to speed up?  How about L1 distance ? ||PQ||1 = |xp-xq| + |yp-yq| , ||PQ||2= √((xp-xq)2+(yp-yq)2)  How about replacing 2D filter with 1D vertical and horizontal filter ?
  3. 3. 2011/12/20 Domain TransformAbstract• 1D filtering on transformed multiple-dimension domain e.g. a case of 2D filter on a color image (XY, RGB) This paper provides high quality edge-preserving smoothing after • (x, RGB)  ct(x). Apply 1D horizontal filter on ct(x) • (y, RGB)  ct(y). Apply 1D vertical filter on ct(y)• Real-time at arbitrary scale• Many applications of this real-time edge preserving filter • Depth-of-field effects, stylization, recoloring, colorization, detail enhancement, and tone mapping.
  4. 4. 2011/12/20 Domain TransformContribution• Real-time high-quality edge aware filtering of image/video based on a dimensionality reduction • Smoothing on ct(X,RGB) and ct(Y, RGB) ≡ edge preserving on 2D color image 1D• Perform anisotropic edge-preserving on curves of 2D image manifold using 1D linear filters • Isotropic kernel on ct(x) ≡ anisotropic kernel on x• 2D edge-preserving filter as a sequence of 1D linear filter• The first real-time edge-preserving smoothing • Any scale kernel • Process time independent of the filter parameters • Can control kernel shape • Handle color image correctly
  5. 5. 2011/12/20 Domain Transform RELATED WORK
  6. 6. 2011/12/20 Domain Transform It’s have been well known that … 2D Smoothing ≡ 1D Edge-preservingHowever, replace 1D Bilateral with 2D Gaussian …. High Cost !
  7. 7. 2011/12/20 Domain Transform 2D Smoothing ≡ 1D Edge-preserving Bilateral grid Gaussian KD-tree [Chen et al. SIG07] [Adams et al. SIG09] higher dimensional functions wi w down-sample Gaussian convolution division slicinglimited for small kernels, coarse problem not real-time 
  8. 8. 2011/12/20 Domain TransformAnisotropic Diffusion Filter (AD)•  c(p, t) is large when p is not a part of an edge  c(p, t) is small when p is a part of an edge Unknown iterations 
  9. 9. 2011/12/20 Domain TransformEdge-avoid Wavelets (EAW)• [Raanan Fatta SIG2009] Edge-avoiding scaling func. inversely related to the pixel difference. translation-invariant trans. Edge-avoiding trans. Constrain the size of the smoothing kernel to 2^n 
  10. 10. 2011/12/20 Domain Transform DOMAIN TRANSFORM
  11. 11. 2011/12/20 Domain Transform Domain transform ct(u) Ωw is ct(u)Smoothing on transformed domain Ωw ≡ Edge-preserving on Ω
  12. 12. 2011/12/20 Domain Transform I’ |I’|
  13. 13. 2011/12/20 Domain Transform |I’| ct(u) is accumulation of each L1 distance of (x, I(x))
  14. 14. 2011/12/20 Domain TransformDomain Transform ct(u) ct(u) : (u,I(u)) ct(u) is accumulation of each L1 1 : 1 distance of (u, I(u)) ct(u) : (u,I(u)) 1 : 1
  15. 15. 2011/12/20 Domain TransformDomain Transform ct(u) ct(u) : (u,I(u)) ct(u) is accumulation of each L1 1 : 1 distance of (u, I(u)) ct(u) : (u,I(u)) 1 : 1
  16. 16. 2011/12/20 Domain Transform pq Ωw = ct(u)
  17. 17. 2011/12/20 Domain Transform q p Ωw = ct(u) edge edge edge edge
  18. 18. 2011/12/20 Smoothing Ωw ≡ Edge-preserving Ω Smoothing Ωw ≡ Edge-preserving Ω Ωw is ct(u) WHY ?
  19. 19. 2011/12/20 Smoothing Ωw ≡ Edge-preserving Ω Smoothing Ω can not preserve edge …
  20. 20. 2011/12/20 Smoothing Ωw ≡ Edge-preserving Ω Smoothing Ωw ≡ Smoothing neighbors in L1 Smoothing on Ω sampling along the curve sample Ωw uniformly
  21. 21. 2011/12/20 Filter AttributesKernel size of Bilateral filter – σs, σr• I p σr kernel in Ωw x large σs What do σs and σr mean in Ωw ?
  22. 22. 2011/12/20 Filter AttributesKernel size of Bilateral filter – σs, σr• I p σr kernel in Ωw x small σs What do σs and σr mean in Ωw ?
  23. 23. 2011/12/20 Domain TransformKernel size of Bilateral filter – σs, σr• I non-scaled kernel in Ωw p q σr scaled kernel in Ωw x small σs
  24. 24. 2011/12/20 Filter AttributesKernel size of Bilateral filter – σs, σr• I non-scaled kernel in Ωw p σr scaled kernel in Ωw x large σs
  25. 25. 2011/12/20 Filter AttributesScaling Factors – σs, σr•
  26. 26. 2011/12/20 Filter Attributes I σrRelationship to σs, σr and I σs x no longer edge-preserving as input unbounded smoothing as input
  27. 27. 2011/12/20 FILTERING IN THE TRANSFORMED DOMAIN Normalized Convolution (NC) Interpolated Convolution (IC) Recursive Filtering (RF)
  28. 28. 2011/12/20 Filtering in Ωw• Sample uniformly in Ωw is unnecessary • It’s easy to determine who is neighbor in Ωw I a b c p Ω I a b ct(p)-ct(a) < r  a is b’s neighbor ct(c)-ct(p) > r  c is not b’s neighbor c Ωw ct(p)
  29. 29. 2011/12/20Normalized Convolution (NC)[Knutsson and Westin 93]moving-average [Dougherty 94] r
  30. 30. 2011/12/20Normalized Convolution (NC)[Knutsson and Westin 93]moving-average [Dougherty 94] ΣI Ω
  31. 31. 2011/12/20Interpolated Convolution (IC)[Piroddi and Petrou 04] continuous, not discrete !
  32. 32. 2011/12/20 NC vs. ICNormalized convolution (NC) Interpolated convolution (IC) NC, IC
  33. 33. 2011/12/20Recursive Filtering (RF) [Smith 07] a ∈ [0, 1] is a feedback coefficient An infinite impulse response (IIR) with exponential decay d ↗ ⇨ ad  0 .. stop propagation and preserve edge
  34. 34. 2011/12/20 FILTERING 2D SIGNALS Replacing 2D filter with multiple iterations of 1D vertical and horizontal filter
  35. 35. 2011/12/20 Filtering 2D signalsFiltering 2D Signals• Is it possible to apply domain transform on (x,y) ? • ct(p) is the accumulated L1 distance along (p, I(p)) • Only exists in surface with zero Gaussian curvature [O’Neill 06] •• Implementation • Several iterations • Horizontal pass along each image row • Vertical pass along each image column
  36. 36. 2011/12/20 Filtering 2D signalsHow to Determine #Iterations ?• 1st iteration 2nd iteration stripes Halve σH at each iteration. Stop while σH is small enough
  37. 37. 2011/12/20 Filtering 2D signals σH = σs = 40, σr = 0.77σH = σs = 50, σr = 0.5 2 itr.
  38. 38. 2011/12/20 COMPARISON TO OTHER APPROACHES
  39. 39. 2011/12/20 Impulse Response Case : the left impulse without strong edge; the right impulse with strong edge #iteration = 3NC:Normalized convolution; IC:Interpolated convolution; RF:Recursive convolution; BF:Bilateral ; AD:Anisotropic Diffusion; WLS:Weighted Least Squares  The NC and IC filters have Gaussian-like response, similar to AD and BF.  Strong edges : Ω  IC ~ AD.  NC has a higher response near strong edges : pixels near strong edge edges have less neighbors in the same population, and will weight their contribution strongly.  Impulse : RF ~ WLS Ωw
  40. 40. 2011/12/20Smooth Quality
  41. 41. 2011/12/20Performance Filtering on CPU Filtering on GPU• 2.8 GHz Quad Core PC • NVIDIA GeForce GTX280• 8GB memory • CUDA• NC, RF by C++• IC in MATLAB • 1M RGB * #3 = 0.007 sec. • Domain tx : 0.7 mx• 1M RGB * #3 = 0.16~0.06 sec. • Each iteration : 2 ms• 10 M RGB * #3 = 1.6 ~ 0.6 sec.• Linearly with image size • Independent on σs or σr• Faster than EAW, PLBF, CTBF
  42. 42. 2011/12/20Comparison• Bilateral Grid [Chen et al. 07] • Faster than domain tx. • Luminance only • Due to down-sampling process, not possible for small spatial and range kernel• NVIDIA GeForce GTX280 This paper PLBF WLS 1M RGB * #3 = 0.5 M RGB = 0.1 1 M gray = 1sec 0.007 sec sec
  43. 43. 2011/12/20 REA-TIME APPLICATIONS OF EDGE-PRESERVING FILTER (EPF)  Detail Manipulation  Tone Mapping  Stylization  Joint Filtering  Colorization  Recoloring
  44. 44. 2011/12/20Detail Manipulation• EPF decomposes image into level-of-detail : J0, J1, .. Jk • I = J0 , Di = Ji – Ji+1 input I using D0 by IC EAW by [Fattal 09] (σs=20, σr=.08) [Farbman et al. 08]
  45. 45. 2011/12/20Tone Mapping• Edge-aware tone mapping avoids haloing and artifacts due to compression• The compressed luminance channel Lc in HDR [Farbman et al. 08] I = J0 , Di = Ji – Ji+1 μ is the mean of B local min  B = 0 local max  B = 112 msec on log-luminance byRF ! J1 : σs= 20, σr=0.33 J2 : σs= 50, σr=0.67 J3 : σs=100, σr=1.34 This paper (RF) WLS
  46. 46. 2011/12/20Stylization 1/2• Abstract low-contrast region and preserving high-contrast features NC filtering edge + this edge-aware filtered image
  47. 47. 2011/12/20Stylization 2/2 Kp : #points in the moving windows On the edge : #point ↘ dark On the smooth area : #point ↗ light• Assign each output pixel a scaled version of the value of the normalization factor Kp in NC 0.11 Kp
  48. 48. 2011/12/20Joint Filtering• Alpha can be combined with other map to create some localized or selective stylization alpha map
  49. 49. 2011/12/20Joint Filtering depth-of-field effect• Smooth image A based on the edges information of image B  simulate depth-of-field (DoF) B is an alpha matte
  50. 50. 2011/12/20 Domain TransformJoint Filtering Canny edge identifies edges
  51. 51. 2011/12/20Colorization• Propagate user color S by blurring them using the edge information User color scribbles using this RF [Levin et al. 04] (σs=100, σr=.03)
  52. 52. 2011/12/20Recoloring• Soft segmentation – region Ri defined by color Ci
  53. 53. 2011/12/20Conclusions and Future Work• High-quality edge-preserving filtering of image and videos in real time • First real-time edge-preserving filter on color image at arbitrary scale • 1D smoothing filter on transformed domain ≡ Edge-preserving filter • High quality 2D edge-preserving filtering by iterating 1D-filtering • independent of the filter parameters(σs, σr) (due to moving average)• Future work • Edge-preserving applications• Limitation • Not rotationally invariant
  54. 54. 2011/12/20 Q&A

×