This document summarizes several image inpainting methods using GANs. It describes the task of image inpainting, common GAN architectures used for inpainting including conditional GANs, and several state-of-the-art models such as Partial Convolution, GLCIC, Contextual Attention, Gated Convolution, PEPSI, and PEPSI++. It also includes background information on techniques like dilated convolution, PatchGAN, cycleGAN, and WGAN used in image inpainting.

1. Image Inpainting by GAN
Presentation: X37
Date: Mar 22nd, 2020
Place: Tokyo

2. Overview
• Task overview: What’s the goal of inpainting task?
• Inpainting Task by GAN
- General structures overview
❖ Generator structure
❖ Discriminator structure
• Models
- Partial Convolution
- GLCIC; Globally and Locally Consistent Image Completion
- Contextual Attention
- Gated Convolution
- PEPSI / Diet-PEPSI

3. Image Inpainting Tasks
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T. (2018). Free-Form Image Inpainting with Gated Convolutionhttps://arxiv.org/abs/1806.03589
masked image
+ mask
Inpainting
model
image

4. GAN for Image Painting
• Original GAN:
- Generator: I / O: noise / image
- Discriminator: I / O: image / bool (real or fake)
• Conditional GAN:
- Generator: I / O: conditional information / image
- Discriminator: I / O: conditional information + image / bool (real or fake)
DZ G T / F T / FŶX ( ̂X, Y) DG ̂X

5. cGAN’s application: Image to Image translation
• Pix2pix generalize the cGAN’s application and its capability:
- synthesizing photos from label maps
- Reconstructing objects from edge maps
- Colorizing images

6. GAN on Image Inpainting tasks
• Image Inpainting is one of the Image-to-Image translation problems
- Generator: I / O: mask and masked image / inpainted image
- Discriminator: I / O: conditional information + image / bool (real or fake)
T / F
full image or
masked region
DG image
masked image
+ mask
• Several study on improvement for GAN
- WGAN, WGAN-GP [Arjovsky+ 2017] [Gulrajani+ 2017]
- Markovian GAN / PatchGAN [Wand 2016]
- Context encoding [Isola+ 2016] [Pathak+ 2016]
- Spectral Normalization / Self-Attention [Miyato+ 2018] [Zhang+ 2019]
• Several study on improvement for each component
- Generator (coarse-to-fine)
- Discriminator (GLCIC; Global and locally consistent image completion, RED)
- Contextual Attention

8. Image Inpainting for Irregular Holes Using Partial Convolutions
• Summary
- published at Dec 15, 2018
- proposed partial convolution layer
- Inpainting for irregular holes
• Model architecture
UNet-like architecture, replacing all convolutional layers with partial convolutional
layers and using nearest neighbour up-sampling in the decoding stage
• Future work: Loss is complicated, mask update is rule-based

10. Globally and Locally Consistent Image Completion
• Summary
- Proposed global and local discriminator
- Inpainting for irregular holes
• Model architecture (Auto Encoder based conditional GAN)
- Completion Network (Auto-Encoder based generator)
- Global / Local Discriminator
• Using dilated convolution layers [Yu+ 2016], weighted MSE loss [Pathak+ 2016]

12. Generative Image Inpainting with Contextual Attention
• Summary
- published at Mar 21, 2018
- proposed contextual attention layer and coarse-to-fine structure
- introduced several GAN’s techniques (GLCIC, WGAN-GP)
• Model architecture: Global and local WGANs
• To measure cosine similarity btw backgrounds and foreground

14. Free-Form Image Inpainting with Gated Convolution
• Summary
- published at Mar 21, 2018
- proposed the gated convolution for the dynamic mask update
• Model architecture: SN-PatchGAN
- Coarse-to-fine generator with contextual attention
- Spectral-normalized Markovian (Patch) discriminator

16. PEPSI : Fast Image Inpainting with Parallel Decoding Network
• Summary
- proposed PEPSI: parallel extended-decoder path for semantic inpainting
- Proposed RED(Region Ensemble Discriminator), which adds 1x1 conv on SN-Patch
discriminator
• Model architecture: modified SN-PatchGAN with contextual attention

17. PEPSI++: Fast and Lightweight Network for Image Inpainting
• Summary
- applied Rate-adaptive dilated conv layers to PEPSI

18. Appendix
• Dilated convolution
• PatchGAN / MarkovianGAN
• From Pix2pix to cycleGAN
• WGAN / WGAN-GP

19. Dilated Convolution
• To extract more global features on convolutional layers

20. PatchGAN / MarkovianGAN

21. CycleGAN