[G4]image deblurring, seeing the invisible

2,691 views
2,613 views

Published on

Published in: Technology, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,691
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
110
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

[G4]image deblurring, seeing the invisible

  1. 1. 2 CG Lab., POSTECH
  2. 2. Blur? Degrade image/video quality Shot/record under dim light circumstance  Long exposure! 3 CG Lab., POSTECH
  3. 3. Blur Various kinds of blur Camera shake (Camera motion blur) Object movements (Object motion blur) Out of focus (Defocus blur) Others (Wind, vibration, etc.) 4 CG Lab., POSTECH
  4. 4. Deblurring? Remove blur and restore a sharp image Input blurred image Deblurring result 5 CG Lab., POSTECH
  5. 5. Importance Image/video in our daily lives  But, sometimes a retake is difficult Strong demand for high quality  E.g. CCTV, car’s black box, medical imaging 6 CG Lab., POSTECH
  6. 6. Contents Fast Motion Deblurring  Sunghyun Cho and Seungyong Lee (POSTECH)  ACM SIGGRAPH Asia 2009 Text Image Deblurring Using Text-Specific Properties  Hojin Cho (POSTECH), Jue Wang (Adobe), and Seungyong Lee (POSTECH)  ECCV 2012 (to appear) 7 CG Lab., POSTECH
  7. 7. Sunghyun Cho Seungyong Lee POSTECH ACM SIGGRAPH Asia 2009
  8. 8. Motion blur Camera jitters Latent sharp image Blurred image 9 CG Lab., POSTECH
  9. 9. Image formation model Convolution  Motion blur kernel  Trace of a sensor Blurred image Latent sharp image Blur kernel * : convolution operator 10 CG Lab., POSTECH
  10. 10. Deblurring Non-blind deconvolution  Ill-posed (Due to the loss of information caused by motion blur) Blurred image Latent image PSF Blind deconvolution  Severely ill-posed Blurred image Latent image PSF 11 CG Lab., POSTECH
  11. 11. Blind deconvolution Possible solutions Severely ill-posed problem  No unique solution Blurred image 12 CG Lab., POSTECH
  12. 12. Related work Parametric kernels  Ex) 1D linear motion blur  [Yitzhakey et al. 1998], [Rav-Acha and Peleg 2005], [Cho et al. 2007], [Money and Kang 2008], … Blurred image Latent sharp image PSF 13 CG Lab., POSTECH
  13. 13. Related work More complex motion blur  [Fergus et al. 2006], [Jia 2007], [Shan et al. 2008]  Excessive amount of computation Blurred image Latent sharp image PSF 14 CG Lab., POSTECH
  14. 14. Motivation Computation time  Important for practical purpose Previous methods are slow [Fergus et al. 2006] took 1 hr 25 min. [Shan et al. 2008] took 4 min 48 sec. Our method took 5.766 sec. in CPU and 0.734 sec. using GPU accel. Image size: 640 x 480 kernel size: 25 x 25 15 CG Lab., POSTECH
  15. 15. Contributions Fast motion deblurring  Only a few sec.  Fast latent image estimation  Fast blur kernel estimation  40x ~ 60x faster than [Shan et al. 2008]  GPU acceleration  600x ~ 800x faster than [Shan et al. 2008] 16 CG Lab., POSTECH
  16. 16. Motion deblurring: Common framework Iteratively solve 1. Estimate a PSF Blurred image Latent image PSF 2. Estimate a latent sharp image using a complex image prior Blurred image Latent image PSF 17 CG Lab., POSTECH
  17. 17. Motion deblurring: Common framework Blur model B  L* K  N Blurred image B Latent image L Blur kernel K Noise N Energy function f ( L, K )  B  L * K  q( L)  r ( K ) 2 * : convolution operator q(L), r(K) : regularization terms or priors for L, K 18 CG Lab., POSTECH
  18. 18. Motion deblurring: Common frameworkBlurred image Latent image estimation Kernel estimation L  arg min B  L * K  q( L) 2 K  arg min B  L * K  r ( K ) 2 L K Deblurred result 19 CG Lab., POSTECH
  19. 19. Latent image estimation: Blurred image Latent image estimation Kernel estimation Deblurred resultAnalysis of prev. methods Two important properties  Restoration of strong edges  Inspecting around strong edges, we can find a blur kernel  Noise suppression in smooth regions  Avoids the effect of noise Blurry input Latent image estimation of on kernel estimation [Shan et al. 2008] Edge of a blurred image Edge of a sharp image 20 CG Lab., POSTECH
  20. 20. Latent image estimation: Blurred image Latent image estimation Kernel estimation Deblurred resultAnalysis of prev. methods Two important properties  Restoration of strong edges  Inspecting around strong edges, we can find a blur kernel  Noise suppression in smooth regions  Avoids the effect of noise Blurry input Latent image estimation of on kernel estimation [Shan et al. 2008] 21 CG Lab., POSTECH
  21. 21. Latent image estimation: Blurred image Latent image estimation Kernel estimation Deblurred resultAnalysis of prev. methods Two important properties  Restoration of strong edges  Inspecting around strong edges, we can find a blur kernel  Noise suppression in smooth regions  Avoids the effect of noise Blurry input Latent image estimation of on kernel estimation [Shan et al. 2008] Computationally expensive priors for q(L) L  arg min B  L * K  q( L) 2 L 22 CG Lab., POSTECH
  22. 22. Latent image estimation: Blurred image Latent image estimation Kernel estimation Deblurred resultBasic idea for accelerationWe divide… Latent image estimation Simple Deconvolution Prediction Removes blur quickly Restores strong edges Low-quality results Removes noise Simple image processing tools 23 CG Lab., POSTECH
  23. 23. Latent image estimation: Blurred image Latent image estimation Kernel estimation Deblurred resultBasic idea for acceleration Simple Deconvolution Prediction Current kernel Updated kernel 24 CG Lab., POSTECH
  24. 24. Deblurring process Blurred image Prediction Kernel estimation Deconvolution Final deconvolution Deblurred result* Deconvolution + prediction = latent image estimation 25 CG Lab., POSTECH
  25. 25. Results Blurry input Our result Blur kernel Processing time Processing time Image size Blur kernel size (CPU) (GPU) 1024 x 768 49 x 47 18.656 sec. 2.125 sec.26 26 CG Lab., POSTECH
  26. 26. Results Blurry input Our result Blur kernel Processing time Processing time Image size Blur kernel size (CPU) (GPU) 972 x 966 65 x 93 18.813 sec. 5.766 sec.27 27 CG Lab., POSTECH
  27. 27. Results Blurry input Our result Blur kernel Processing time Processing time Image size Blur kernel size (CPU) (GPU) 858 x 558 61 x 43 8.969 sec. 0.703 sec.28 28 CG Lab., POSTECH
  28. 28. Comparison (quality) Blurry input [Yuan et al. 2007] [Shan et al. 2008] our method* This method uses two input images. 29 CG Lab., POSTECH
  29. 29. Implementation Details Requirements?  Vector/matrix calculus  Fast Fourier transform (FFT)  Image filters (bilateral, shock)  Convex optimization  Solver: Conjugate gradient method 30 CG Lab., POSTECH
  30. 30. Hojin Cho1 Jue Wang2 Seungyong Lee1 1POSTECH 2Adobe12th European Conference on Computer Vision (ECCV) (to appear)
  31. 31. Introduction Common blur model b= k* l +n ∗ = Latent image l Convolution Blurred image b operator Kernel k 32 CG Lab., POSTECH
  32. 32. Introduction Image deblurring Synthetic blurred image Deblurring result of [Levin et al. CVPR 2011] 33 CG Lab., POSTECH
  33. 33. Introduction Image deblurring Synthetic blurred image Deblurring result of [Levin et al. CVPR 2011] 34 CG Lab., POSTECH
  34. 34. Difficulties  Why is handling text difficult?  (1) Different statistics against natural images  (2) Spatial distribution of text is highly regulated Log-scale Gradient Log-scale GradientNatural image Text image gradient histogram magnitude gradient histogram magnitude Directly applying a natural image deblurring method to a text image will result in an erroneous result! 35 CG Lab., POSTECH
  35. 35. Our Approach Exploit desired “general” properties of text images Introduce a new optimization framework to incorporate the text properties in deblurring  Utilizing domain-specific properties Real blurred image Our result 36 CG Lab., POSTECH
  36. 36. Desired Properties of Latent Text Images – 1/3 Text boundary  High contrasts along the texts’ boundary 37 CG Lab., POSTECH
  37. 37. Desired Properties of Latent Text Images – 2/3 Character color  Near-uniform color inside characters POSTECH 1) Near-uniform color 2) Gradients are close to zero 38 CG Lab., POSTECH
  38. 38. Desired Properties of Latent Text Images – 3/3 Background gradients  Too restrictive to assume a single background color (e.g., advertisements or posters)  Similar to the natural image 39 CG Lab., POSTECH
  39. 39. Optimization Framework Blur model b = k* l + n Deblurring problem arg min b  k * l  l (l )   k (k ) 2 l ,k l (l ) : A prior for the latent image  k (k ) : A prior for the motion blur kernel arg min b  k * l  l (l )   k (k )   a (a )   l  a 2 2 l ,k ,a  a (a) : prior for the auxiliary (reference) image a a : reflects the text properties 40 CG Lab., POSTECH
  40. 40. Deblurring Algorithm Kernel Estimation Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image b Result image l • Estimating l  Restore sharp edges along the boundaries of texts considering the text properties 1 and 2 • Estimating k  Compute the blur kernel (camera’s trajectory) • Final deconvolution  Restores texts and background using k & b 41 CG Lab., POSTECH
  41. 41. Deblurring Algorithm Kernel Estimation Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image b Result image l Blur kernel k Computing lBlurred image b Result image l Computing a 42 CG Lab., POSTECH
  42. 42. Kernel EstimationAlgorithm Details Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image Result image Step 1: Computing l  a reflects the text properties  The text properties soak into l by solving: arg min b  k * l  l (l )   l  a 2 2 l a plays a role as a guidance(reference) image 1  (k ) (b)   (a)  l    (k ) (k )   (1)  l () ()  Can be solved efficiently using FFTs and component-wise operators(multiplication/division) 43 CG Lab., POSTECH
  43. 43. Kernel Estimation Algorithm Details Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image Result image  Step 2: Computing a  Initialize a by refining l computed from Step 1  Reduce ringing artifacts and noise caused by simple deconvolution through L0 gradient minimization Bilateral filter Total variation (TV) L0 gradient minimization The result of L0 gradient minimization can provide a good initial for a  [Property 1] The boundaries of texts have large gradient values  [Property 2] Gradients inside characters are close to zero  Noise and ringing artifacts generally have small magnitudes, thus being reduced* The figures are provided in [Xu et al., SIGGRAPH Asia 2011] 44 CG Lab., POSTECH
  44. 44. Kernel EstimationAlgorithm Details Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image Result image Step 2: Computing a  With the initial a, we iteratively solve: arg min  a (a)   l  a 2 a  a (a) : a cost function which is minimized when a satisfies the text image properties l a 2 : a data term to penalize non-similarity between a and l  0 if ai =aiP For each pixel at i, a (ai )   d MAX otherwise aP : an ideal image satisfying the text image properties 1 & 2 completely. After minimizing the equation, a will reflect the text image properties while preserving the similarity against l 45 CG Lab., POSTECH
  45. 45. Kernel EstimationAlgorithm Details Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image Result image  Computing the ideal image aP:  Detect text regions from l using the stroke width transform (SWT) Typical stroke Searching in the direction Detected stroke width of the gradient at edges  For detected text regions (connected strokes), we force the pixels to have the same color value Through the SWT and re-coloring, the ideal image satisfies the following properties:  [Property 1] The large gradient around the boundaries of text characters  [Property 2] Colors are near uniform inside text characters 46 CG Lab., POSTECH
  46. 46. Kernel EstimationAlgorithm Details Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image Result image  0 if ai =aiP arg min  a (a)   l  a 2 where a (ai )   a d MAX otherwise  a (a) : a cost function which is minimized when a satisfies the text image properties l a 2 : a data term to penalize non-similarity between a and l Solution a P  if d MAX   li  aiP 2 For each pixel at i, ai   i *  li  otherwise 47 CG Lab., POSTECH
  47. 47. Kernel EstimationAlgorithm Details Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image Result image Computing k  Adopted from [Shan et al. SIGGRAPH 2008] and [Cho and Lee, SIGGRAPH Asia 2009]  ( ) *b  k * *a   k , 2 E (k )  2 *   *   { x ,  y ,  xx ,  xy ,  yy }  The auxiliary image a is used instead of the latent image l  a contains less ringing artifacts and noise due to the L0 gradient minimization 48 CG Lab., POSTECH
  48. 48. Kernel EstimationAlgorithm Details Estimating l Input Final Deconvolution Computing l Computing a Estimating k Blurred image Result image l Final deconvolution  Given the estimate kernel and the input blurred image, we minimize: (1) (2) (3) E (l )  b  k * l  1  li  2  li  ai  3  li 2 2 2 0.8 iT iT iT T  {i | a (ai )  0} : Set of pixel indices for text regions (1): [Property 1 & 2] Gradient values inside each character should be close to zero (2): [Property 1 & 2] Each character has a near-uniform color (3): [Property 3] Gradient values of background are sparse 49 CG Lab., POSTECH
  49. 49. Results Synthetic example Deblurring result of [Levin et al. CVPR 2011] Our result 50 CG Lab., POSTECH
  50. 50. Original image Synthetic blurred imageFast motion deblurring result Our result 51
  51. 51. Results on Real Photographs (1/4) Blurred image Our result 52
  52. 52. Results on Real Photographs (2/4) Blurred image Our result 53
  53. 53. Results on Real Photographs (3/4) Blurred image Our result 54
  54. 54. Results on Real Photographs (4/4) Blurred image Our result (very large blur: 105x105) 55
  55. 55. Comparison – Real ImagesBlurred image Fergus et al. Shan et al. Cho and Lee SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009 Xu and Jia Levin et al. Chen et al. Our method ECCV 2010 CVPR 2011 56 CVPR 2011 CG Lab., POSTECH
  56. 56. Comparison – Real ImagesBlurred image Fergus et al. Shan et al. Cho and Lee SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009 Xu and Jia Levin et al. Chen et al. Our method ECCV 2010 CVPR 2011 CVPR 2011 57 CG Lab., POSTECH
  57. 57. Comparison – Real Images Blurred image Fergus et al. Shan et al. Cho and Lee SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009 Not available Xu and Jia Levin et al. Chen et al. Our method ECCV 2010 CVPR 2011 CVPR 2011* Since Chen et al.’s approach requires training, re-implementing it is impossible without having the training data.
  58. 58. Comparison – Real Images Blurred image Fergus et al. Shan et al. Cho and Lee SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009 Not available Xu and Jia Levin et al. Chen et al. Our method ECCV 2010 CVPR 2011 CVPR 2011* Since Chen et al.’s approach requires training, re-implementing it is impossible without having the training data.
  59. 59. Comparison – Real Images Not available Blurred image Fergus et al. Shan et al. Cho and Lee SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009 Not available Not available Xu and Jia Levin et al. Chen et al. Our method ECCV 2010 CVPR 2011 CVPR 2011* The methods of Shan et al. and Levin et al. could not handle this image.* Since Chen et al.’s approach requires training, re-implementing it is impossible without having the training data.
  60. 60. Conclusion Previous natural image deblurring methods do not work well for text images  Due to the lack of consideration of text-specific properties We analyzed the desired properties for latent text images We proposed a novel text image deblurring algorithm which explicitly incorporates the text-specific properties into the optimization framework 61 CG Lab., POSTECH
  61. 61. Applications 62 CG Lab., POSTECH
  62. 62. Q&Ahttp://cg.postech.ac.kr

×