Fast edge-preserving/-aware high dimensional filters for image & video processing


Published on

fast Bilateral filter, fast trilateral filter, fast median filter, Fast Gauss Bilateral Filtering, bilateral grid, Gaussian kd-tree, Permutohedral Lattice, fast local histogram filter, domain transform, patchmatch, LSH, coherent sensitivity hashing, gray code kernel, propagation-assisted KD-tree, CNN, deep learning.

Published in: Technology, Business

Fast edge-preserving/-aware high dimensional filters for image & video processing

  1. 1. Fast Edge-Preserving/Aware Filters for Image/Video Processing Yu Huang Sunnyvale, California
  2. 2. Outline • Gaussian filter as weighted average • Bilateral filter (Note: anisotropic diffusion filter is equivalent) – Separable bilateral filtering (range and spatial kernels); • Fast bilateral filter – Acceleration by piecewise linear approximation; • Splatting (downsample), blurring (FFT) and slicing (upsample); – Bilateral grid (importance sampling); – Gaussian KD-tree; – Pemutohedral Lattice. • Fast Gauss (Transform) bilateral filtering; – Constant time bilateral filtering O(1).
  3. 3. Outline • Adaptive Manifold filter. • Fast median (distributed histogram) filter • Smooth local histogram filter and its acceleration • Accelerated trilateral filter (gradient preserving) • Domain Transform (DT) as a RT edge preserving filter • Patch Match • Coherent Sensitivity Hashing (CSH) • Propagation-assisted KD-tree • Patch Geodesic Paths • References
  4. 4. Gaussian Filter as Weighted Average • Weight of x depends on distance to x ),( xxf )(xI output input )(xJ x x x x
  5. 5. Bilateral Filter [Tomasi’98] • Spatial Gaussian f • Gaussian g on the intensity difference • Edge preserving filter )(xJ ),( xxf ))()(( xIIg x )(xIx)( 1 xk output input x
  6. 6. Cross-Bilateral Filter q p output input p      S IIIGG W IBF q qqp p p qp |||||| 1 ][ rs  [Eisemann’04] [Petschnigg’04]
  7. 7. Edge Preserving vs. Edge Blurring Input Image Edge-Preserving Base Image Edge-Blurring Base Image Edge-Preserving Enhanced Image Edge-Blurring Enhanced Image Halo Artifacts
  8. 8. Anisotropic Diffusion Filter  c(p, t) is large when p is not a part of an edge  c(p, t) is small when p is a part of an edge • Diffusion [Perona & Malik 1990]: – Diffusion is smoothing: pixel ; – Anisotropic kernel preserve edge: strong edge reduces kernel size; • Equivalent to bilateral filter, but not easy to accelerate it. I
  9. 9. Separable Bilateral Filtering [Pham’05]  Pros: extremely fast (fixed spatial weight + LUT for tonal weight)  Cons: effective filtering kernel is slightly distorted
  10. 10. Acceleration of Bilateral Filters • Discretize the set of possible I(x) – sparse in intensity; • Piecewise linear approximation; • Perform linear Gaussian blur (FFT); • Sub-sampling in spatial domain; • Linear interpolation in between. • k(x) is treated similarly • [Durand, Siggraph 2002] )(xJ ),( xxf ))()(( xIIg x )(xIx)( 1 xk
  11. 11. higher dimensional functions Gaussian convolution division slicing w i w D O W N S A M P L E U P S A M P L E “A Fast Approximation of the Bilateral Filter using a Signal Processing Approach” • 3-D histogram; • Splatting (spatial tonal space in high dim); • Blurring (convolution); – 3D FFT. • Division & Slicing (interpolation). [Pham, ECCV’06]
  12. 12. Bilateral Grid Bilateral Grid (importance sampling) •Bilateral grid = 3D array –x and y correspond to pixel position –z corresponds to pixel intensity –Euclidean distance accounts for edges • space distance (x, y) and intensity distance (z) •Grid can be coarsely sampled; • 70 x 70 x 10 for an 8 MB pixel image •Bilateral filter = high dim. kernel + … • Gaussian convolut. applied to grid; • Sampling and normalization; • Downsample, blur, then upsample. Standard 2D Image x y[Chen, Siggraph’07]
  13. 13. Bilateral Filter on the Bilateral Grid space intensity Image scanline Filtered scanline Slice: query grid with input image Bilateral Grid Gaussian blur grid values space intensity
  14. 14. Bilateral Grid for Color Image  T iiiiii i i i bgryxp ip iv iv  pixelofposition: pixelofueoutput val:ˆ pixelofueintput val: Image High-Dimensional Grid (5d grid) High Memory Usage Cost Bilateral Filtering Based on Luminance Bilateral Filtering Based on Color
  15. 15. Gaussian KD-Tree x y r,g,b Input Image Gaussian KD Tree High-Dimensional Space Resolution Reduction •Monte-Carlo Sampling •Weighted Importance Sampling Different Weighting to Leaf Nodes Inner Node Leaf Node [Adams, Siggraph’09] • Low Computational Complexity • Low Memory Usage • Building The Tree • Querying The Tree
  16. 16. • Gaussian KD-tree is also Important Sampling • High-dimension Gaussian filter: sampling s neighborhood • Important sampling with Gaussian KD-tree: evaluating samples as near as possible s • High dimension kernel + sampling on leaf nodes • Splatting (downsample points to leaf nodes) • Blurring (Gaussian blurring on leaf nodes) • Slicing (upsample from leaf nodes to points)
  17. 17. Difference of Gaussian Kd-Tree, Bilateral grid Naïve Bilateral Filtering 5-D Bilateral Grid 3-D Bilateral Grid KD-Tree
  18. 18. • Patch-based (not pixel-based like bilateral filter); • Averaging with nearby pixels of similar texture; – Pixel-wised – Patch-wised • More discriminative, still blurry in finer details. • Note: Intel OpenCV implements NLM for denoising. Nonlocal Means Filter [Buadess’05]
  19. 19. Non-Local Means by Gaussian Kd-Tree Gaussian KD Tree Inner Node Leaf Node Image Pixel Different Weighting to Leaf Nodes High-Dimensional Space Querying
  20. 20. Permutohedral Lattice for High Dim • A framework for high-dim Gaussian filtering (Gauss KD-tree) – Embedding the input values at positions in a high-dimensional space (splatting): bary centric coord. interpolation; – Performing a Gaussian blur in that space (blurring); – Sampling the space at the original positions (slicing). • It tessellates the high-dimensional space with uniform simplices (HD tetrahedra); • Less sparse than Gaussian kd-tree, but less memory consumption than bilateral grid, efficient for a high dim filter. [Adams, Eurographic’10]
  21. 21. Permutohedral Lattice for High Dim
  22. 22. Speed Comparison of Permutohedral Lattice with Bilateral Grid and Gaussian Kd-tree
  23. 23. Gauss Transform and its fast version • Discrete Gauss transform – v’(x) = Σ’y v(y) f(p(x) - p(y)); – Take a high-dimensional signal; – Put it into a data structure; – Perform a Gaussian blur really fast; – Read out its values. • Fast Gauss Transform [Greengard &Strain’91]: reduces computation from O(MN) to O(M + N) for N Gaussians at M points in d dimensions; – The more general “Fast Multipole Method (FMM)”. • Improved by [Yang’03]: for high dimension d, use a multivariate Taylor expansion scheme combined with the space subdivison using the k-center algorithm; – Employ “Farthest-Point Clustering”.
  24. 24. Fast Gauss Bilateral Filtering • Bilateral filtering as convolution: a weighted sum of Gaussians; • Bilateral filtering as Gauss Transform: a ratio of two (n+1)-d Gauss Transform; • Dimensional elevation [Paris’06]: clustering and Taylor series expansion; • Fast Gauss Transform: a fast multi-pole method strategy, O(N). [Yoshizawa’10]
  25. 25. Comparison of Fast Edge Preserving Filters
  26. 26. Constant Time Bilateral Filtering O(1) • [Porikli’08] • Use of integral histogram for box spatial kernel; • Use of linear filters of image powers for polynomial range kernel; • Linear filter decompositions by Taylor series for Gaussian range kernel .
  27. 27. Real-time Bilateral Filtering O(1) • [Yang et al.’09] • Bilateral filter can be decomposed into a number of O(1) spatial filters for each pixel value (quantization is moved into a range function); • Output as linear interpolation of each closest principle bilateral filtered image component pair ( I(x) = k, k+1 ); – Box filtering ->integral images; – Gaussian filtering -> recursive filtering; – Polynomial filtering -> a set of integral images; – Median filtering -> integral image.
  28. 28. Fast Median and Bilateral Filtering • Median filter: – Huang’s O(r): adjacent sliding windows overlap to some extent; • Makes use of this sequential overlap to consolidate the redundant calculations, reducing the computational complexity to O(r) – Distributive Histograms O(log(r)): If multiple columns are processed at once, the redundant calculations become sequential. • Histogram distributivity: for disjoint image regions A and B, their union’s window W compute its histogram by HW=HA + HB; • The union window’s median comes from scanning HW, splicing together from HA and HB on the fly. – High depth median filter: storage consideration • Ordinal transform: reduce the histogram size; • Compound histogram: Hc • Coarse to fine recursion [Weiss, Siggraph’06]
  29. 29. Fast Median and Bilateral Filtering O(r) O(logr)
  30. 30. Fast Median and Bilateral Filtering • Bilateral filter: – box spatial kernel: weighting function becomes constant – 3-d histogram: – logarithmic bilateral: [Weiss, Siggraph’06]
  31. 31. Smoothed Local Histogram Filter • Image filter from local histogram: – Median, Dilation/Erosion filter: 50% ,100%/0% point of the local histogram; • How to accelerate local histogram? – [Huang75] O(n), [Weiss06] O(log(n)), [Porikli05] O(1); – Center-weighted isotropic histogram neighborhood; • KDE (Parzen window) as a smoothed locally weighted histogram – K determines the frequency content of as a function of s; – W determines the spatial frequency content; – For W arbitrary kernel, the convolution can performed at the cost of O (log(n)) operations per output pixel using 2D FFT (for n neighborhood size) ; – If K, W are both Gaussian, the convolution can be done in O(1) constant time, independent of neighborhood size per pixel. • Designed new filters: – Closest mode filter: restricted mean shift; – Dominant mode filter: choose the mode corresponding to the largest population. )(ˆ sfp [Kass, ACM T-Graphics’10]
  32. 32. Smoothed Local Histogram Filter
  33. 33. Accelerating Spatially Varying Gaussian Filters • Drawback of Bilateral Filter – piecewise-flat regions, false edges, blooming at high contrast regions; – spurious detail at color edges; • Trilateral filter: Besides, the filter window is “tilted” by the gradient vector; • Acceleration schemes of S. V. (spatially varying) filters: importance sampling – Directly evaluate anisotropic filter kernel by integration for Gaussian kd-tree; – Embeds data in higher dim space to make kernels separable, i.e. dimensional elevation M:(x,y)  (x, y, x+y, x-y); – Segments dataset into overlapping regions of constant anisotropic kernel to accelerate in the permutohedral lattice; – Kernel sampling sparsely and segment by clustering, then interpolate result finally. [Baek, Siggraph Asia’10]
  34. 34. noisy signal bilateraled kernels along the gradient l x y 2D Gaussian kernel  3D Gaussian kernel ? Spatially varying Gaussian filtered The trilateral filter is a “gradient-preserving” filter
  35. 35. Comparison of Bilateral and Trilateral Bilateral filter Trilateral filter
  36. 36. Comparison of Bilateral and Trilateral Bilateral Kernel sampling Trilateral Blooming & false edge Kernel sampling
  37. 37. Domain Transform [Gastal, Siggraph’11] • Isometry (distance preserving) between curves on 2d manifold and real lines; • 1D filtering on transformed multiple-dimension domain e.g. a case of 2D filter on a color image (XY, RGB) • Real-time high-quality edge aware filtering of image/video based on a dimensionality reduction Smoothing on transformed domain Ωw ≡ Edge-preserving on Ω Ωw is ct(u) Domain Transform
  38. 38. Domain Transform [Gastal, Siggraph’11] • Implementation for image filtering with iterations – Horizontal pass along each image row – Vertical pass along each image column • Normalized Convolution (NC) • Interpolated Convolution (IC) • Recursive Filtering (RF) )()ˆ( pctpt  d ↗ ⇨ ad  0 .. stop propagation and preserve edge Halve σH at each iteration. Stop while σH is small enough )/2exp( Ha 
  39. 39. Filter Quality Comparison with Bilateral Filter, WLS, Anisotropic Diffusion and Edge-Avoid Wavelet
  40. 40. Guided Image Filtering [He et al.’10] • Problems in bilateral filter – Complexity – Gradient distortion: preserve edge, not gradient; • Guided filter: in all local windows , compute the linear coefficients and the average of in all local windows that covers pixel • Gradient preserving: q has an edge only if I has an edge • Integral images – O(1) time – Non-approximate Naturally O(N) time independent of the window radius mean variance
  41. 41. Guided Image Filtering [He et al.‘10] Linear regression Bilateral/joint bilateral filter does not have this linear model! Noise/texture
  42. 42. Adaptive Manifold Filter • Sampling the high dimensional space at scattered locations; – A reduced set of sampling points, interpolate at all N input pixels; • Computation is performed only where it is needed; • The filter response can be computed in linear time; • The number of manifolds is independent of dimensionality; • Performance vs permutohedral lattice and guided filter.
  43. 43. (a) Splatting performs a distance-weighted projection of the colors fi onto each adaptive manifold. (b) Blurring performs Gaussian filtering over each manifold, mixing the distance-weighted projections from all pixels. (c) Slicing computes the filter response for each pixel using normalized convolution. adaptive-manifold – AM, permutohedral lattice – PL, guided filter – GF. Adaptive Manifold Filter
  44. 44. Patch Match [Barnes’09] • Nearest neighbor field: nearest neighbor over all possible patch candidates for some distance function between two patches; • Naive brute force search is expensive: O(mM^2); • Dimensionality reduction is a limited solution; • Popular methods: KD-trees, Locality Sensitive Hashing (LSH); – These methods can be very accurate, but are relatively slow; • Patch Match is an iterative randomized algorithm: iteratively improve the nearest neighbor field, based on local propagation and mostly local search, until convergence; – Heavily relies on the fact that images are generally coherent; – Search space: patch offsets vs. patches; – Neighboring pixels have coherent matches; – Large number of random sampling will yield some good guesses.
  45. 45. Patch Match [Barnes’09] • Each iteration, for each pixel: – Randomly located patches (with the prob. that drops exp. with distance from current match); – Check if the offsets from neighboring patches give a better matching patch; If so, adopt neighbor’s patch offset; – Search for better patch offsets within a concentric radius around the current offset; search radius halved each time till 1 (started with the image size).
  46. 46. Generalized Patch Match [Barnes’10] • Dealing with higher dimensional spaces; • Scales and Rotations: from (x, y) to (x, y, θ, s); • Any distance function (pairwise comparisons) e.g. – SIFT, a popular computer vision descriptor; – Patches invariant to lighting changes; • k-Nearest Neighbors: heap-based. – Store k correspondences in max-heap; – If new candidate better, pop heap, push new candidate. • Outperform KD-tree and LSH by two orders of magnitude. Ground truth Gaussian noise NLM PM Generalized PM
  47. 47. Coherence Sensitive Hashing [Korman & Avidan ’11] • CSH extend LSH and Patch Match: – LSH relies on hashing by mapping similar patches to the same bin; – PM relies on images are coherent, to propagate good matches to neighbors; – CSH relies on hashing to seed the initial patch matching and on image coherence to propagate good matches (3~4 times faster than PM). • Indexing and Search: Input: images A, B, Output: nearest patch map ANNF
  48. 48. Coherence Sensitive Hashing [Korman & Avidan ’11] • Hash all patches of both images (with hash function g) – Keep k (=2) representatives of each image in each entry p Image ‘A’ Image ‘B’ k k Hash Table g g g-1 g-1 CSH Indexing
  49. 49. Coherence Sensitive Hashing [Korman & Avidan ’11] Type 1 Type 2 Type 3 Type 1 Type 2 Type 3 Uses appearance    Uses coherence  # candidates 2 4 - 6 2 Estimated contribution 20% 50% 30% (LSH) (> PM) CSH Indexing – Candidate (potential matching patches) Types appearance only appearance and coherenceappearance only
  50. 50. Coherence Sensitive Hashing [Korman & Avidan ’11] CSH Recap Preprocessing: • Project A’s and B’s patches on Walsh-Hadamard vectors Iterate (typically 1-3 times): • Hash patches into a table – Recall the hash function; – hi(v) is a binned projection of v on one of WH kernels; – Heuristics adapt to these kernels in: • number of bins (per projection); • spacing of bins. • For each patch in A – Create candidate patches in B – Pick the best one A B )()()( 1 vhvhvg n          r bva vh ii i )(
  51. 51. Propagation-Assisted KD-Tree [He & Sun ’12] • Jointly exploit distrib. of candidates and dependency of queries; • Build a kd-tree using all the patches in the Candidate Set. Each leaf contains no more than m candidates. • Scan the image A in raster order. For each patch pA(x, y) in A, do propagation-assisted kd-tree search as follows: – Descend the tree to a leaf (Leaf #0); – Propagate a leaf from left/upper, using the already matched result of pA(x- 1, y) and pA(x, y-1); – Find the NN of pA(x, y) in all the above leaves. • Check for each query its own leaf and extra leaf propagated from nearby queries after organizing the candidates in a KD-tree; • Check fewer candidates and has no backtracking; • 10~20 times than PM, 2~5 times than CSH.
  52. 52. Propagation-Assisted KD-Tree [He & Sun ’12] The joint distribution P(e0, e1) in different search strategies Propagation in Patch MatchPropagation-assisted KD-tree
  53. 53. Patch Geodesic Paths (PatchGP) • Geodesic path: – Shortest path btw pixels/patches – To apply pixel geodesic distance, compute the smoothing kernel with – Patch Geodesic Paths – Then the smooth kernel
  54. 54. Patch Geodesic Paths (PatchGP) • Minimal Hop Paths (MHP) – the path with the minimal number of hops connecting 2 nodes; – The denoising filter defined as • Weight threshold – Handle the outliers • Fast multi-scale denoising – Haarlike transform; – Hierarchical coarse to fine. Left: The MHP between p and q (red). Right: MHPs under 8 search directions. -- weight of the pixel (a) An image and and two patches (5x5) (b) and (e) the closeup. (c) and (f) patch distance map. (d) and (g) color-coded path hop maps for the PatchGPs.
  55. 55. References • Tomasi, C., Manduchi, R. “Bilateral filtering for gray and color images”. ICCV’98. • F. Durand and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range images,” ACM T-Graphics, 21(3), pp. 257–266, (SIGGRAPH 2002). • Choudhury, P., Tumblin, J., "The Trilateral Filter for High Contrast Images and Meshes," Eurographics Symp. on Rendering, pp. 186-196, 2003 • T. Pham, L. Vliet, “Separable bilateral filtering for fast video preprocessing,” ICME, 2005 • S. Paris and F. Durand, “A fast approximation of the bilateral filter using a signal processing approach,” ECCV, 2006. • B. Weiss, “Fast median and bilateral filtering,” ACM T-Graphics, 25(3), SIGGRAPH, 2006. • Porikli, F. “Constant time o(1) bilateral filtering”, IEEE CVPR’08. • Q Yang, K Tan and N Ahuja, “Real-time O(1) Bilateral Filtering”, IEEE CVPR 2009. • J. Chen, S. Paris, and F. Durand, “Real-time edge-aware image processing with the bilateral grid,” ACM Transactions on Graphics (SIGGRAPH 07). • A Adams, N. Gelfand, J. Dolson, and M. Levoy, “Gaussian KD-Trees for Fast High- Dimensional Filtering,” ACAdams M Transactions on Graphics (SIGGRAPH 2009). • S Yoshizawa, A Belyaev, H Yokota, Fast Gauss Bilateral Filtering , CG forum’10. • M Kass, J Solomon, Smoothed Local Histogram Filters, ACM T-Graphics, 2010.
  56. 56. References • A Adams, J Baek, A Davis, “Fast High-Dimensional Filtering Using the Permutohedral Lattice”, Eurographics 2010. • Baek, J., Jacobs, D. E. “Accelerating Spatially Varying Gaussian Filters”. SIGGRAPH Asia 2010. • E Gastal and M. Oliveira, Domain Transform for Edge-Aware Image and Video Processing, SIGGRAPH 2011. • Barnes, Shechtman, Finkelstein, and Goldman. Patchmatch: a randomized correspondence algorithm for structural image editing. SIGGRAPH, 2009. • C. Barnes, E. Shechtman, D. B. Goldman, and A. Finkelstein. The generalized patchmatch correspondence algorithm. ECCV, pages 29–43, 2010. • Kaiming He, Jian Sun, and Xiaoou Tang, Guided image filtering, ECCV 2010. • S. Korman and S. Avidan. “Coherency sensitive hashing”. ICCV, 2011. • K He, J Sun, Computing Nearest-Neighbor Fields via Propagation-Assisted KD-Trees, IEEE CVPR, 2012. • Gestal, Oliveira, Adaptive Manifolds for Real-Time High-Dimensional Filtering, ACM SIGGRAPH, 2012. • Chen, Kang, Yang, and Yu, Fast Patch-based Denoising Using Approximated Patch Geodesic Paths, CVPR, 2013.
  57. 57. Thanks!