IMAGE AND VIDEO ABSTRACTIONBY MULTI-SCALE ANISOTROPICKUWAHARA FILTERINGNPAR 2011JAN ERIC KYPRIANIDISHASSO-PLATTNER-INSTITU...
ABSTRACTMulti-scale Anisotropic Kuwahara Filter - a coarse-to-fine edgepreserving smoothing filter1. Strong image abstract...
OUTLINE1. Introduction          Image abstraction &                         Edge preserving smoothing filters2. Related wo...
INTRODUCTION                                                      Temporal                                                ...
PROBLEM OFSEGMENTATIONRegions segmented by mean-shift [DeCarlo & Santella 02; Wen et 1l. 06]have rough boundaries and requ...
PROBLEM OFEDGE PRESERVINGSMOOTHIHG FILTERBilateral filter or Kuwahara filter cause overblurring  • Remove detail in low-co...
PROBLEM OFANISOTROPICKUWAHARA FILTERPreserve features, directions, and is robust against high-contrastnoise   original im...
ADVANCE OFMULTI-SCALE ANISOTROPICKUWAHARA FILTERING    1.down sample to create                        2. from coarse-to-  ...
RELATED WORK1. Image pyramids2. Bilateral filter3. Kuwahara filters                                              edge pres...
IMAGEPYRAMIDS    Gaussian filter + down-sample    Gaussian filter + down-sample    Gaussian filter + down-sample
down-sampling without smoothing  aliasing
BILATERAL FILTER  [Smith 97, Tomasi 98]                                                              Input                ...
KUWAHARAFILTER[Kuwahara et al. 76]                                    [76]Select the ave. value ofsub-region whose var. is...
A tiled & aliased image filtered byKuwahara filter                       Due to                       1. Rectangular sub-r...
GENERALIZEDKUWAHARAFILTER[Papari et al. 07]                                         [76]New val. is sum of mean ofeach sub...
Fail to capture directional features &clustering artifacts                                                  non directiona...
GENERALIZEDKUWAHARAFILTER[Kyprianidis et al. PG09]                                        [76]New val. is sum of mean ofea...
ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]•   Smooth image tangents    2nd eigen vector v2 of   smoothed by       ...
ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]•         Smooth image tangents            2nd eigen vector v2 of       ...
ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]•         Smooth image tangents           2nd eigen vector v2 of   smoot...
ANISOTROPIC   KUWAHARA   FILTER   [Kyprianidis et al. PG09]   •   Smooth the tangent of image   •   Set the ellipse kernel...
ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]•   Smooth the tangent of image•   Set the ellipse kernel               ...
ANISOTROPIC  KUWAHARA  FILTER  [Kyprianidis et al. PG09]  •   Smooth the tangent of image  •   Set the ellipse kernel     ...
ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]•     Smooth the tangent of image                                      s...
MEAN CURVATURE FLOW (MCF)+ SHOCKING FILTERING[Kang and Lee. PG08]•   Mean curvature flow simplifies shape of boundaries   ...
IMAGE ABSTRACTION ONGRADIENT DOMAIN[Orzan et al. NPAR’07]            origin                                reconstruction ...
IMAGE ABSTRACTION ONGRADIENT DOMAIN1. Importance              Canny edge (small scale  large scale)     lifetimesource   ...
METHOD –     OVERVIEW1. Down sample to                     2. From coarse-to-fine, apply anisotropic   create image       ...
GRADIENTReplace (Gaussian derivatives and Sobel filter) with Jähne Filter    Jähne Filter
STRUCTURE TENSOR                   2nd eigen vector as                     image tangent
SMOOTH THESTRUCTURE OF TENSOR
Cal. v to min. err(local gradients, v)                                                     v                              ...
=         v(x) is the 1st eigen vector of local gradients    vx        The image tangent vector of x is vertical to v(x)...
Taylor series                                    If a = v2  min. c(x,y)Konstantinos G. Derpanis. “The Harris Corner Detec...
ANISOTROPYMEASURE             anisotropic  ellipse kernel              A = 1 …. λ1>> λ2             isotropic  circle ke...
ANISOTROPICKUWAHARA FILTERKernel of each sub-regionNew val. is sum ofmean of each sub-region weighted bythe inverse stdv. ...
Lower bound of stdv. to avoid1. Zero stdv. in flat region  divided by zero2. Small differences in the stdv. in large low ...
MERGE BETWEEN     SCALES1. Down sample to                     2. From coarse-to-fine, apply anisotropic   create image    ...
MULTI-SCALE ESTIMATION –MERGE TENSOR         f                     f                                              K+1     ...
MULTI-SCALE ESTIMATION –MERGE TENSOR         f                     f                                              K+1     ...
fK+1MULTI-SCALE   fKFILTERING
fK+1fK
RESULT By C++, GLSL NVDIA GTX580 512x512 … 42 ms HD 720p (1280x720) .. 150 ms
(b) the fur above                                                   the nose is less                                      ...
stronger contrast,
LIMITATIONFail to produce good-looking results                                       parts above the plant are            ...
LIMITATIONImages with high frequency texture is hard to abstract
CONCLUSION•   Avoid artifacts and smooth results    •   By adding thresholding to the weighting term•   Strong abstraction...
ANY QUESTION ?If the area of a sub-region in a Kuwahara filter is very verysmall, is it similar to a bilateral filter ?   ...
END
Upcoming SlideShare
Loading in …5
×

study Image and video abstraction by multi scale anisotropic kuwahara

1,802 views

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,802
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
39
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

study Image and video abstraction by multi scale anisotropic kuwahara

  1. 1. IMAGE AND VIDEO ABSTRACTIONBY MULTI-SCALE ANISOTROPICKUWAHARA FILTERINGNPAR 2011JAN ERIC KYPRIANIDISHASSO-PLATTNER-INSTITUT, GERMANY
  2. 2. ABSTRACTMulti-scale Anisotropic Kuwahara Filter - a coarse-to-fine edgepreserving smoothing filter1. Strong image abstraction2. Avoid artifacts in large low-contrast region original image anisotropic Kuwahara filter proposed method
  3. 3. OUTLINE1. Introduction Image abstraction & Edge preserving smoothing filters2. Related work3. Method 1. Pyramid structure : coarse  fine 2. Anisotropic Kuwahara filter 3. Merging function : (upper level, this level)4. Results5. Conclusions
  4. 4. INTRODUCTION Temporal coherence in video Segmentation NPR (mean-shift) Edge preserving smoothing filter • Bilateral • Kuwahara Image abstractionImage abstraction is useful for NPR or temporal coherence in video
  5. 5. PROBLEM OFSEGMENTATIONRegions segmented by mean-shift [DeCarlo & Santella 02; Wen et 1l. 06]have rough boundaries and require elaborate post-processing original image Mean-shift results in rough-boundary
  6. 6. PROBLEM OFEDGE PRESERVINGSMOOTHIHG FILTERBilateral filter or Kuwahara filter cause overblurring • Remove detail in low-contrast region original image Bilateral filter results in overblurring in low-contrast regions.
  7. 7. PROBLEM OFANISOTROPICKUWAHARA FILTERPreserve features, directions, and is robust against high-contrastnoise  original image Bilateral Anisotropic Kuwahara (direction) • However .. • level of abstraction depending on filter radius  • Artifacts in large low-contrast regions 
  8. 8. ADVANCE OFMULTI-SCALE ANISOTROPICKUWAHARA FILTERING 1.down sample to create 2. from coarse-to- image pyramid first fine, apply anisotropic Kawahara filter and Down Up merge the previous sample sample• Avoid artifacts and smooth results • By adding thresholding to the weighting term• Strong abstraction and avoidance of artifacts in large low-contrast regions • Coarse-to-fine from multi-scale image pyramids• Real-time processing on GPU
  9. 9. RELATED WORK1. Image pyramids2. Bilateral filter3. Kuwahara filters edge preserving4. Anisotropic Kuwahara filter smoothing filter5. Mean curvature flow + shocking filtering6. Diffusion and shock filtering7. Image abstraction on gradient domain
  10. 10. IMAGEPYRAMIDS Gaussian filter + down-sample Gaussian filter + down-sample Gaussian filter + down-sample
  11. 11. down-sampling without smoothing  aliasing
  12. 12. BILATERAL FILTER [Smith 97, Tomasi 98] Input Result edge preserving + smoothing filter Weights • Gaussian on space distance • Gaussian on range distance • sum to 1 smoothing preserve edges space rangeSylvain Paris and Frédo Durand. A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. ECCV’ 06.
  13. 13. KUWAHARAFILTER[Kuwahara et al. 76] [76]Select the ave. value ofsub-region whose var. ismin. [07]where q in the sub-regionRi of p with min. variance Anisotropic Kuwahara [09]
  14. 14. A tiled & aliased image filtered byKuwahara filter Due to 1. Rectangular sub-regions 2. Unstable if noise exits 3. Subregions have the same variance. origin Kuwahara filter
  15. 15. GENERALIZEDKUWAHARAFILTER[Papari et al. 07] [76]New val. is sum of mean ofeach sub-region weightedby the inverse stdv. [07] si : variance of sub-region i mi : mean of sub-region i Anisotropic Kuwahara [09]
  16. 16. Fail to capture directional features &clustering artifacts non directional Generalized Kuwahara Anisotropic Kuwahara
  17. 17. GENERALIZEDKUWAHARAFILTER[Kyprianidis et al. PG09] [76]New val. is sum of mean ofeach sub-region weightedby the inverse stdv. [07] si : variance of sub-region i mi : mean of sub-region i Anisotropic Kuwahara [09]
  18. 18. ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]• Smooth image tangents 2nd eigen vector v2 of smoothed by local gradients Gaussian filter• Set the ellipse kernel
  19. 19. ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]• Smooth image tangents 2nd eigen vector v2 of smoothed by local gradients Gaussian filter• Set the ellipse kernel 1. Image tangent = 2nd eigen vector of Jij structure tensor (fx, fy) = mean(local gradients (gx,gy)) Jij 2nd eigen vector Image tangent = 2nd eigen PCA of image gradients vector of local gradients
  20. 20. ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]• Smooth image tangents 2nd eigen vector v2 of smoothed by local gradients Gaussian filter• Set the ellipse kernel 1. Image tangent = 2nd eigen vector of Jij structure tensor (fx, fy) = mean(local gradients (gx,gy) Jij 2. Smooth the image tangents by Gaussian filter
  21. 21. ANISOTROPIC KUWAHARA FILTER [Kyprianidis et al. PG09] • Smooth the tangent of image • Set the ellipse kernel K0rotate K0 to generate kernel of each subregion scale & rotate as ellipse kernel
  22. 22. ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]• Smooth the tangent of image• Set the ellipse kernel K0
  23. 23. ANISOTROPIC KUWAHARA FILTER [Kyprianidis et al. PG09] • Smooth the tangent of image • Set the ellipse kernel Kirotate K0 to generate kernel of each region
  24. 24. ANISOTROPICKUWAHARAFILTER[Kyprianidis et al. PG09]• Smooth the tangent of image scale & rotate as ellipse kernel• Set the ellipse kernel1. Scale 2. Rotation α=1 Jij
  25. 25. MEAN CURVATURE FLOW (MCF)+ SHOCKING FILTERING[Kang and Lee. PG08]• Mean curvature flow simplifies shape of boundaries • Users must protect important features• Shock filter sharpens the discontinuities and flattens each homogenous regions
  26. 26. IMAGE ABSTRACTION ONGRADIENT DOMAIN[Orzan et al. NPAR’07] origin reconstruction thickness = importance
  27. 27. IMAGE ABSTRACTION ONGRADIENT DOMAIN1. Importance Canny edge (small scale  large scale) lifetimesource ∝ |gradient| ∝ thickness2. Gradient reconstruction ∝ importance • Solve Poisson eq. with constrain of gradients
  28. 28. METHOD – OVERVIEW1. Down sample to 2. From coarse-to-fine, apply anisotropic create image Kawahara filter and merge the previous pyramid firstfK+1 New image fk = Merge (fk, upsample (Filter(fk+1, Jk+1))) Down UpfK sample sample tensor Jk := covariance matrix of gradients of fk New tensor Jk = Merge (gk, upsample (smooth(Jk+1)))
  29. 29. GRADIENTReplace (Gaussian derivatives and Sobel filter) with Jähne Filter Jähne Filter
  30. 30. STRUCTURE TENSOR 2nd eigen vector as image tangent
  31. 31. SMOOTH THESTRUCTURE OF TENSOR
  32. 32. Cal. v to min. err(local gradients, v) v x = =
  33. 33. =  v(x) is the 1st eigen vector of local gradients vx  The image tangent vector of x is vertical to v(x)  The 2nd eigen vector is the image tangent vector
  34. 34. Taylor series If a = v2  min. c(x,y)Konstantinos G. Derpanis. “The Harris Corner Detector”. 2004
  35. 35. ANISOTROPYMEASURE anisotropic  ellipse kernel A = 1 …. λ1>> λ2 isotropic  circle kernel A = 0 …. λ1= λ2
  36. 36. ANISOTROPICKUWAHARA FILTERKernel of each sub-regionNew val. is sum ofmean of each sub-region weighted bythe inverse stdv. Add threshold to avoid artifacts and smooth results …. Why ?
  37. 37. Lower bound of stdv. to avoid1. Zero stdv. in flat region  divided by zero2. Small differences in the stdv. in large low contrast region  artifacts
  38. 38. MERGE BETWEEN SCALES1. Down sample to 2. From coarse-to-fine, apply anisotropic create image Kawahara filter and merge the previous pyramid firstfK+1 New image fk = Merge (fk, upsample (Filter(fk+1, Jk+1))) Down UpfK sample sample tensor Jk := covariance matrix of gradients of fk New tensor Jk = Merge (gk, upsample (smooth(Jk+1)))
  39. 39. MULTI-SCALE ESTIMATION –MERGE TENSOR f f K+1 K =Always prefer the more anisotropic tensor !ex2. Ak+1 = 0, Ak = 1  αk = 1  ~Jk = Jkex1. Ak+1 = 1, Ak = 0  αk = 0  ~Jk = Jk+1
  40. 40. MULTI-SCALE ESTIMATION –MERGE TENSOR f f K+1 KAlways prefer the more anisotropic tensor !
  41. 41. fK+1MULTI-SCALE fKFILTERING
  42. 42. fK+1fK
  43. 43. RESULT By C++, GLSL NVDIA GTX580 512x512 … 42 ms HD 720p (1280x720) .. 150 ms
  44. 44. (b) the fur above the nose is less abstracted than at the neck.very consistent strong abstraction wherelevel of abstraction slightly less abstraction above the nose
  45. 45. stronger contrast,
  46. 46. LIMITATIONFail to produce good-looking results parts above the plant are blended with the ground 
  47. 47. LIMITATIONImages with high frequency texture is hard to abstract
  48. 48. CONCLUSION• Avoid artifacts and smooth results • By adding thresholding to the weighting term• Strong abstraction and avoidance of artifacts in large low-contrast regions • Coarse-to-fine from multi-scale image pyramids• Real-time processing on GPU
  49. 49. ANY QUESTION ?If the area of a sub-region in a Kuwahara filter is very verysmall, is it similar to a bilateral filter ? Kernel of 2D Kuwahara filter  Sub-regionc can not too small Kernel of 1D bilateral filter
  50. 50. END

×