Transformation Guided Image Completion ICCP 2013
Upcoming SlideShare
Loading in...5
×
 

Transformation Guided Image Completion ICCP 2013

on

  • 8,603 views

In this paper, we describe a new interactive image completion system that allows users to easily specify various forms of mid-level structures in the image. Our system supports the specification of ...

In this paper, we describe a new interactive image completion system that allows users to easily specify various forms of mid-level structures in the image. Our system supports the specification of four basic symmetric types: reflection, translation, rotation, and glide. The user inputs are automatically converted into guidance maps that encode
possible candidate shifts and, indirectly, local transformations of rotation and scale. These guidance maps are used in conjunction with a color matching cost for image
completion. We show that our system is capable of handling a variety of challenging examples.

http://www.jiabinhuang.com/

Statistics

Views

Total Views
8,603
Views on SlideShare
2,245
Embed Views
6,358

Actions

Likes
0
Downloads
18
Comments
0

41 Embeds 6,358

http://jbhuang0604.blogspot.com 2176
http://jbhuang0604.blogspot.tw 2014
http://jbhuang0604.blogspot.it 307
http://jbhuang0604.blogspot.se 294
http://jbhuang0604.blogspot.sg 287
http://jbhuang0604.blogspot.kr 192
http://jbhuang0604.blogspot.co.uk 149
http://jbhuang0604.blogspot.com.au 140
http://jbhuang0604.blogspot.hk 134
http://jbhuang0604.blogspot.ca 121
http://jbhuang0604.blogspot.jp 97
http://jbhuang0604.blogspot.fr 63
http://jbhuang0604.blogspot.de 53
http://jbhuang0604.blogspot.in 52
http://jbhuang0604.blogspot.nl 52
http://www.blogger.com 34
http://jbhuang0604.blogspot.ru 32
http://jbhuang0604.blogspot.com.es 23
http://jbhuang0604.blogspot.com.br 20
http://jbhuang0604.blogspot.ch 16
http://jbhuang0604.blogspot.co.nz 13
http://jbhuang0604.blogspot.no 11
http://jbhuang0604.blogspot.com.tr 11
http://jbhuang0604.blogspot.fi 9
http://jbhuang0604.blogspot.ie 8
http://jbhuang0604.blogspot.pt 6
http://jbhuang0604.blogspot.co.il 5
http://jbhuang0604.blogspot.be 5
http://jbhuang0604.blogspot.dk 5
http://jbhuang0604.blogspot.gr 5
http://jbhuang0604.blogspot.ae 4
http://jbhuang0604.blogspot.mx 3
http://jbhuang0604.blogspot.com.ar 3
http://digg.com 3
http://jbhuang0604.blogspot.sk 2
http://jbhuang0604.blogspot.ro 2
http://www.jbhuang0604.blogspot.com 2
http://jbhuang0604.blogspot.co.at 2
http://jbhuang0604.blogspot.nl.sixxs.org 1
http://translate.googleusercontent.com 1
http://jbhuang0604.blogspot.cz 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Transformation Guided Image Completion ICCP 2013 Transformation Guided Image Completion ICCP 2013 Presentation Transcript

  • Transformation Guided Image Completion Jia-Bin Huang1, Johannes Kopf2, Narendra Ahuja1, and Sing Bing Kang21 2
  • My lab mate’s picture from his honeymoon
  • Can we get rid of those annoying tourists?
  • Can we get rid of those annoying tourists?
  • Photoshop Content-Aware Fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]
  • Photoshop Content-Aware Fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]
  • Our result
  • Related Problems Texture Image Inpainting Image Completion SynthesisStationarity Yes No No Hole size Large Small LargeExample inputExample output
  • Related Work•  Statistical methods •  Good at synthesizing pure stochastic texture •  Don’t work well for structured texture•  Diffusion-based methods •  Good at filling small holes •  Don’t work well for large holes
  • Example-based methods• Idea: exploit redundancy in natural images• Previous work: •  Unit of synthesis (pixels, patches or regions) •  [Efros and Leung, ICCV 99] [Efros and Freeman, SIG 01] [Kwatra et al. SIG 03] •  Filling ordering •  [Criminisi et al., TIP 04] [Drori et al., SIG 03] [Xu et al., TIP 10] •  Global optimization •  [Wexler et al., PAMI 07] [Kwatra et al., SIG 05] [Komodakis and Tziritas, TIP 07] •  Transformations beyond translation •  [Mansfield et al., BMVC 11] [HaCohen et al. SIG 11] [Darabi et al. SIG 12] •  Semi-automatic approaches •  [Hertzmann et al. SIG 01] [Sun et al., SIG 05] [Pavic et al., VC 06] [Barnes et al. SIG 09]
  • The Key Challenge•  Geometric and photometric transformation are often required.
  • Completion using only translational patches Translational patches are not sufficient!Examplar-based inpainting Priority-BP Photoshop Content-aware fill [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07] [Wexler et al., PAMI 07] [Barnes et al. SIG 09]
  • Completion using transformed patchesDifficult to optimize! Image melding Transforming image completion [Darabi et al. SIG 12] [Mansfield et al., BMVC 11]
  • What if the computer could understand the underlying structure?Image with hole Positional guidance map Our result
  • General Framework
  • General Framework Structure in known regions Extrapolation to unknown regions
  • General Framework Structure in known regions Extrapolation to unknown regions
  • Specifying Image Structure•  How? •  Specify structure whenever possible.•  Why? •  Easy for human, but difficult for computer •  Allow humans to provide high-level expertise while the system perform low- and mid-level processing
  • Exploiting Structures in Images•  Label maps [Hertzmann et al. SIG 01] [Sun et al., SIG 05] [Barnes et al. SIG 09] •  Search sources patches only from patches with the same label (Texture-by- numbers)•  Ramp gradient •  Constraint searching along a certain direction
  • SymmetryReflection RotationTranslation Glide
  • Piecewise planar geometry
  • Translate user inputs to constraints•  Objective function•  Color matching cost•  Guidance cost: positional and non-positional
  • Positional guidance Image with missing region Positional guidance map
  • Non-positional guidance
  • Probabilistic Interpretation•  Probability factorization given user input u•  Guidance cost
  • Probabilistic interpretation forinteractive image completion methods Positional prior Non-positional priorLabel map uniform distribution with finite N/A[Hertzmann et al. SIG 01] support[Barnes et al. SIG 09]Line/contour-based degenerate distribution along N/Aconstraints the curve[Sun et al., SIG 05]Perspective N/A constant, no uncertainty[Pavic et al., VC 06] modeledOurs distribution corresponds to Laplacian distribution eps-insensitive loss
  • An image with missing region
  • Photoshop content-aware fill
  • Our result (unguided)
  • Our result (positional guide only)
  • Our result (non-positional guide only)
  • Our result (positional and non-positional)
  • Comparison with the-state-of-the-art•  Example-based Inpainting [Criminisi et al., TIP 04] •  Smart filling order selection•  Priority-BP [Komodakis and Tziritas, TIP 07] •  Global optimization with MRF model•  Photoshop content-aware fill (CS 5) •  Nonparametric formulation [Wexler et al., PAMI 07] + PatchMatch [Barnes et al. SIG 09]•  Image melding [Darabi et al. SIG 12] •  geometric and photometric transformations for patch + image gradients representation•  Transforming image completion [Mansfield et al., BMVC 11] •  Search transformed patches using generalized PatchMatch•  Ours (unguided) •  Our completion algorithm without using guidance map
  • Image with hole Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, [Wexler et al., PAMI 07] image completion TIP 07] [Barnes et al. SIG 09] [Mansfield et al., BMVC 11]
  • Guidance map Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, [Wexler et al., PAMI 07] image completion TIP 07] [Barnes et al. SIG 09] [Mansfield et al., BMVC 11]
  • Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  • Image with hole Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, [Wexler et al., PAMI 07] image completion TIP 07] [Barnes et al. SIG 09] [Mansfield et al., BMVC 11]
  • Guidance map Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, [Wexler et al., PAMI 07] image completion TIP 07] [Barnes et al. SIG 09] [Mansfield et al., BMVC 11]
  • Image with mask Our result Image melding [Darabi et al. SIG 12]Photoshop content-aware fill Priority-BP Transforming image completion [Wexler et al., PAMI 07] [Komodakis and Tziritas, TIP 07] [Mansfield et al., BMVC 11] [Barnes et al. SIG 09]
  • Guidance map Our result Image melding [Darabi et al. SIG 12]Photoshop content-aware fill Priority-BP Transforming image completion [Wexler et al., PAMI 07] [Komodakis and Tziritas, TIP 07] [Mansfield et al., BMVC 11] [Barnes et al. SIG 09]
  • Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  • Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, image completion TIP 07] [Mansfield et al., BMVC 11]
  • Image with hole Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [ [Wexler et al., PAMI 07] image completion Komodakis and Tziritas, [Barnes et al. SIG 09] [Mansfield et al., BMVC 11] TIP 07]
  • Guidance map Ours Ours (unguided) Image melding [Darabi et al. SIG 12]Examplar-based inpainting Priority-BP Photoshop content-aware fill Transforming [Criminisi et al., TIP 04] [ [Wexler et al., PAMI 07] image completion Komodakis and Tziritas, [Barnes et al. SIG 09] [Mansfield et al., BMVC 11] TIP 07]
  • Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  • Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09] Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  • Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09] Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  • Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Darabi et al. SIG 12] [Barnes et al. SIG 09] Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [ image completion Komodakis and Tziritas, [Mansfield et al., BMVC 11] TIP 07]
  • Image with hole Photoshop content-aware fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]Examplar-based inpainting Priority-BP [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07]
  • Image with hole Our result Image melding Transforming image completion[Darabi et al. SIG 12] [Mansfield et al., BMVC 11]
  • Guidance Map Our result Image melding Transforming image completion[Darabi et al. SIG 12] [Mansfield et al., BMVC 11]
  • Image with hole Ours Photoshop content-aware fill Image melding [Wexler et al., PAMI 07] [Barnes et al. SIG 09] [Darabi et al. SIG 12]Guidance map Examplar-based inpainting Priority-BP Transforming [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07] image completion [Mansfield et al., BMVC 11]
  • Failure cases Image with hole Our result Photoshop content-aware fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]Examplar-based inpainting Priority-BP Transforming image completion [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07] [Mansfield et al., BMVC 11]
  • Failure cases Image with hole Our result Photoshop content-aware fill [Wexler et al., PAMI 07] [Barnes et al. SIG 09]Examplar-based inpainting Priority-BP Transforming image completion [Criminisi et al., TIP 04] [Komodakis and Tziritas, TIP 07] [Mansfield et al., BMVC 11]
  • Summary•  A simple user interface •  allow users to directly control the desired image completion results by specifying various image structures•  Automatically generate the transformation guidance maps •  serve as soft constraints in guiding the completion process•  An optimization framework with probabilistic interpretation •  generalize existing work using interactive techniques for image completion
  • Thank you!•  Comments or questions? Jia-Bin Huang Johannes Kopf jbhuang1@Illinois.edu kopf@microsoft.com Narendra Ahuja Sing Bing Kang n-ahuja@Illinois.edu sbkang@microsoft.com