Transformation Guided Image Completion ICCP 2013

9,516 views

Published on

In this paper, we describe a new interactive image completion system that allows users to easily specify various forms of mid-level structures in the image. Our system supports the specification of four basic symmetric types: reflection, translation, rotation, and glide. The user inputs are automatically converted into guidance maps that encode
possible candidate shifts and, indirectly, local transformations of rotation and scale. These guidance maps are used in conjunction with a color matching cost for image
completion. We show that our system is capable of handling a variety of challenging examples.

http://www.jiabinhuang.com/

Published in: Business, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,516
On SlideShare
0
From Embeds
0
Number of Embeds
6,872
Actions
Shares
0
Downloads
35
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transformation Guided Image Completion ICCP 2013

  1. 1. Transformation Guided Image Completion Jia-Bin Huang1, Johannes Kopf2, Narendra Ahuja1, and Sing Bing Kang21 2
  2. 2. My lab mate’s picture from his honeymoon
  3. 3. Can we get rid of those annoying tourists?
  4. 4. Can we get rid of those annoying tourists?
  5. 5. Photoshop Content-Aware Fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]
  6. 6. Photoshop Content-Aware Fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]
  7. 7. Our result
  8. 8. Related Problems Texture Image Inpainting Image Completion SynthesisStationarity Yes No No Hole size Large Small LargeExample inputExample output
  9. 9. Related Work•  Statistical methods •  Good at synthesizing pure stochastic texture •  Don’t work well for structured texture•  Diffusion-based methods •  Good at filling small holes •  Don’t work well for large holes
  10. 10. Example-based methods• Idea: exploit redundancy in natural images• Previous work: •  Unit of synthesis (pixels, patches or regions) •  [Efros and Leung, ICCV 99] [Efros and Freeman, SIG 01] [Kwatra et al. SIG 03] •  Filling ordering •  [Criminisi et al., TIP 04] [Drori et al., SIG 03] [Xu et al., TIP 10] •  Global optimization •  [Wexler et al., PAMI 07] [Kwatra et al., SIG 05] [Komodakis and Tziritas, TIP 07] •  Transformations beyond translation •  [Mansfield et al., BMVC 11] [HaCohen et al. SIG 11] [Darabi et al. SIG 12] •  Semi-automatic approaches •  [Hertzmann et al. SIG 01] [Sun et al., SIG 05] [Pavic et al., VC 06] [Barnes et al. SIG 09]
  11. 11. The Key Challenge•  Geometric and photometric transformation are often required.
  12. 12. Completion using only translational patches Translational patches are not sufficient!Examplar-based inpainting Priority-BP Photoshop Content-aware fill [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07] [Wexler et al., PAMI 07] [Barnes et al. SIG 09]
  13. 13. Completion using transformed patchesDifficult to optimize! Image melding Transforming image completion [Darabi et al. SIG 12] [Mansfield et al., BMVC 11]
  14. 14. What if the computer could understand the underlying structure?Image with hole Positional guidance map Our result
  15. 15. General Framework
  16. 16. General Framework Structure in known regions Extrapolation to unknown regions
  17. 17. General Framework Structure in known regions Extrapolation to unknown regions
  18. 18. Specifying Image Structure•  How? •  Specify structure whenever possible.•  Why? •  Easy for human, but difficult for computer •  Allow humans to provide high-level expertise while the system perform low- and mid-level processing
  19. 19. Exploiting Structures in Images•  Label maps [Hertzmann et al. SIG 01] [Sun et al., SIG 05] [Barnes et al. SIG 09] •  Search sources patches only from patches with the same label (Texture-by- numbers)•  Ramp gradient •  Constraint searching along a certain direction
  20. 20. SymmetryReflection RotationTranslation Glide
  21. 21. Piecewise planar geometry
  22. 22. Translate user inputs to constraints•  Objective function•  Color matching cost•  Guidance cost: positional and non-positional
  23. 23. Positional guidance Image with missing region Positional guidance map
  24. 24. Non-positional guidance
  25. 25. Probabilistic Interpretation•  Probability factorization given user input u•  Guidance cost
  26. 26. Probabilistic interpretation forinteractive image completion methods Positional prior Non-positional priorLabel map uniform distribution with finite N/A[Hertzmann et al. SIG 01] support[Barnes et al. SIG 09]Line/contour-based degenerate distribution along N/Aconstraints the curve[Sun et al., SIG 05]Perspective N/A constant, no uncertainty[Pavic et al., VC 06] modeledOurs distribution corresponds to Laplacian distribution eps-insensitive loss
  27. 27. An image with missing region
  28. 28. Photoshop content-aware fill
  29. 29. Our result (unguided)
  30. 30. Our result (positional guide only)
  31. 31. Our result (non-positional guide only)
  32. 32. Our result (positional and non-positional)
  33. 33. Comparison with the-state-of-the-art•  Example-based Inpainting [Criminisi et al., TIP 04] •  Smart filling order selection•  Priority-BP [Komodakis and Tziritas, TIP 07] •  Global optimization with MRF model•  Photoshop content-aware fill (CS 5) •  Nonparametric formulation [Wexler et al., PAMI 07] + PatchMatch [Barnes et al. SIG 09]•  Image melding [Darabi et al. SIG 12] •  geometric and photometric transformations for patch + image gradients representation•  Transforming image completion [Mansfield et al., BMVC 11] •  Search transformed patches using generalized PatchMatch•  Ours (unguided) •  Our completion algorithm without using guidance map
  34. 34. Image with hole Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, [Wexler et al., PAMI 07] image completion TIP 07] [Barnes et al. SIG 09] [Mansfield et al., BMVC 11]
  35. 35. Guidance map Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, [Wexler et al., PAMI 07] image completion TIP 07] [Barnes et al. SIG 09] [Mansfield et al., BMVC 11]
  36. 36. Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  37. 37. Image with hole Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, [Wexler et al., PAMI 07] image completion TIP 07] [Barnes et al. SIG 09] [Mansfield et al., BMVC 11]
  38. 38. Guidance map Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, [Wexler et al., PAMI 07] image completion TIP 07] [Barnes et al. SIG 09] [Mansfield et al., BMVC 11]
  39. 39. Image with mask Our result Image melding [Darabi et al. SIG 12]Photoshop content-aware fill Priority-BP Transforming image completion [Wexler et al., PAMI 07] [Komodakis and Tziritas, TIP 07] [Mansfield et al., BMVC 11] [Barnes et al. SIG 09]
  40. 40. Guidance map Our result Image melding [Darabi et al. SIG 12]Photoshop content-aware fill Priority-BP Transforming image completion [Wexler et al., PAMI 07] [Komodakis and Tziritas, TIP 07] [Mansfield et al., BMVC 11] [Barnes et al. SIG 09]
  41. 41. Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  42. 42. Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, image completion TIP 07] [Mansfield et al., BMVC 11]
  43. 43. Image with hole Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [ [Wexler et al., PAMI 07] image completion Komodakis and Tziritas, [Barnes et al. SIG 09] [Mansfield et al., BMVC 11] TIP 07]
  44. 44. Guidance map Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [ [Wexler et al., PAMI 07] image completion Komodakis and Tziritas, [Barnes et al. SIG 09] [Mansfield et al., BMVC 11] TIP 07]
  45. 45. Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  46. 46. Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09] Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  47. 47. Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09] Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  48. 48. Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09] Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  49. 49. Image with hole Photoshop content-aware fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]Examplar-based inpainting Priority-BP [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07]
  50. 50. Image with hole Our result Image melding Transforming image completion[Darabi et al. SIG 12] [Mansfield et al., BMVC 11]
  51. 51. Guidance Map Our result Image melding Transforming image completion[Darabi et al. SIG 12] [Mansfield et al., BMVC 11]
  52. 52. Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Barnes et al. SIG 09] [Darabi et al. SIG 12]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07] image completion [Mansfield et al., BMVC 11]
  53. 53. Failure cases Image with hole Our result Photoshop content-aware fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]Examplar-based inpainting Priority-BP Transforming image completion [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07] [Mansfield et al., BMVC 11]
  54. 54. Failure cases Image with hole Our result Photoshop content-aware fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]Examplar-based inpainting Priority-BP Transforming image completion [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07] [Mansfield et al., BMVC 11]
  55. 55. Summary•  A simple user interface •  allow users to directly control the desired image completion results by specifying various image structures•  Automatically generate the transformation guidance maps •  serve as soft constraints in guiding the completion process•  An optimization framework with probabilistic interpretation •  generalize existing work using interactive techniques for image completion
  56. 56. Thank you!•  Comments or questions? Jia-Bin Huang Johannes Kopf jbhuang1@Illinois.edu kopf@microsoft.com Narendra Ahuja Sing Bing Kang n-ahuja@Illinois.edu sbkang@microsoft.com

×