Modeling perceptual similarity and shift invariance in deep networks
•
2 likes•815 views
Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification have been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations. Despite their strong transfer performance, deep convolutional representations surprisingly lack a basic low-level property -- shift-invariance, as small input shifts or translations can cause drastic changes in the output. Commonly used downsampling methods, such as max-pooling, strided-convolution, and average-pooling, ignore the sampling theorem. The well-known signal processing fix is anti-aliasing by low-pass filtering before downsampling. However, simply inserting this module into deep networks degrades performance; as a result, it is seldomly used today. We show that when integrated correctly, it is compatible with existing architectural components, such as max-pooling and strided-convolution. We observe increased accuracy in ImageNet classification, across several commonly-used architectures, such as ResNet, DenseNet, and MobileNet, indicating effective regularization. Furthermore, we observe better generalization, in terms of stability and robustness to input corruptions. Our results demonstrate that this classical signal processing technique has been undeservingly overlooked in modern deep networks.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Modeling perceptual similarity and shift invariance in deep networks
Similar to Modeling perceptual similarity and shift invariance in deep networks (20)
More from NAVER Engineering
More from NAVER Engineering (20)
Recently uploaded
Recently uploaded (20)
Modeling perceptual similarity and shift invariance in deep networks
- 1. Modeling Perceptual Similarity and Shift-Invariance in Deep Networks Richard Zhang Adobe Research, San Francisco (formerly UC Berkeley EECS PhD) 1
- 2. Discriminative Deep Networks 2 “Rock beauty” Textures? Edges? Parts? Perceptual similarity?
- 3. Discriminative Deep Networks 3 Raw, Unlabeled Pixels Textures? Edges? Parts? Perceptual similarity?
- 4. Generative Deep Networks 4 Raw, Unlabeled Pixels Textures? Edges? Parts? Perceptual similarity?
- 5. Grayscale image: L channel Color information: ab channels abL Zhang, Isola, Efros. Colorful Image Colorization. In ECCV, 2016. c.f. Larsson, Maire, Shakhnarovich. In ECCV, 2016. 5
- 6. Concatenate (L,ab) channels 6 Grayscale image: L channel abL Zhang, Isola, Efros. Colorful Image Colorization. In ECCV, 2016. c.f. Larsson, Maire, Shakhnarovich. In ECCV, 2016.
- 7. Initial colorization attempt Colors in ab space (continuous)Loss Function 7 Training data …
- 8. Colors in ab space (continuous)Loss Function • Regression with L2 loss inadequate 8
- 9. Better Loss Function Colors in ab space (discrete) 9 • Regression with L2 loss inadequate • Use per-pixel multinomial classification
- 10. 10
- 11. Input ClassificationL2 RegressionGround Truth 11 Accounting for multimodality in the output space improves visual realism
- 12. Ansel Adams. Yosemite Valley Bridge. 13
- 13. 14 Dr. David Fleay, Thylacine (extinct in 1936).
- 14. Amateur Family Photo, 1956. 15
- 16. 17 Automatic Colorization (Informative Mistake)
- 17. Cross-Channel Encoder 256 lightness conv1 conv2 conv3 conv4 fc6conv5 64 33 17 192 256 384 17 384 17 256 4096 17 4096 17 fc7 17 fc8 ab color +L 313 Hidden Unit Activations Zhou et al. In ICLR, 2015. 18 c.f. Larsson et al. Colorization as a Proxy Task for Visual Understanding. In CVPR, 2017.
- 18. Hidden Unit (conv5) Activations sky trees water 19
- 19. faces dog faces flowers Hidden Unit (conv5) Activations 20
- 20. Egomotion Agrawal et al. ICCV 2015. Jayaraman et al. ICCV 2015. Context Noroozi and Favaro. ECCV 2016. Doersch et al. ICCV 2015. Pathak et al. CVPR 2016. Hinton & Salakhutdinov. Science 2006. Wang et al. ICCV 2015. Pathak et al. CVPR 2017. Misra et al. ECCV 2016. de Sa. NIPS 1994. Video Audio Autoencoders Denoising Autoencoders Vincent et al. ICML 2008. Goal: Set up a pre-training scheme to induce a “useful” representation Owens et al. ECCV 2016. Arandjelovic & Zisserman. ICCV 17 Generative Modeling Donahue et al. Dumoulin et al. ICLR 2017. 21
- 21. 22 X Raw Data Traditional Autoencoder Induce abstraction through compression X
- 22. 23 Denoising Autoencoder Induce abstraction through reconstruction X Corrupted Raw Data X Raw Data
- 23. 24 X Raw Data X1 X2 Induce abstraction through prediction Cross-Channel Encoder c.f. Larsson et al. Colorization as a Proxy Task for Visual Understanding. In CVPR, 2017.
- 24. 25 Zhang, Isola, Efros. Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction. In CVPR, 2017. c.f. LeCun, Denker, Solla. Optimal Brain Damage. In NIPS, 1989. X X1 X2 X1 X2 X Split-Brain Autoencoder
- 25. Input Image X Predicted Image X 26 Split-Brain Autoencoder on Images
- 26. Input RGB-HHA image RGB channels HHA depth channels HHA depth channels RGB channels Predicted RGB-HHA image 27 Split-Brain Autoencoder on RGB-D
- 27. Class supervision Pre-Trained Model How semantically organized is the feature space? Top-1ClassificationAccuracy 40 35 30 25 20 15 10 5 0 45 50 Layer conv1 pool1 conv2 pool2 conv3 conv4 conv5 pool5 227 conv1 conv2 conv3 conv4 conv5 27 13 96 256 384 13 384 13 256 Random Supervised pre-training Krizhevsky et al. In NIPS, 2012.
- 28. Performance Comparison X X X1 X2 X1 X2 29 XX XX X2 X X1 Top-1ClassificationAccuracy 40 35 30 25 20 15 10 5 0 45 50 Random Supervised pre-training Krizhevsky et al. In NIPS, 2012. Layer conv1 pool1 conv2 pool2 conv3 conv4 conv5 pool5 Doersch et al. In ICCV, 2015. Donahue et al. In ICLR, 2017.Pathak et al. In CVPR, 2016. Krähenbühl et al. In ICLR, 2016. Split-Brain Autoencoder Colorization No split Corrupted input (Regression ) Accounting for multimodality improves representation
- 29. 31 Task & Dataset Generalization Does the feature representation transfer to other tasks and datasets? Berkeley-Adobe Perceptual Patch Similarity Dataset Zhang, Isola, Efros, Shechtman, Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR, 2018. D ( , ) Rock beauty Classification Perceptual Judgments
- 30. 32 Zhang, Isola, Efros, Shechtman, Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR, 2018. How different are these patches?
- 31. Which patch is more similar to the middle? Humans L2/PSNR SSIM/FSIMc Deep Networks? < Clap >< Clap > 33
- 32. “Perceptual Losses” 34 Gatys et al. In CVPR, 2016. Johnson et al. In ECCV, 2016. Dosovitskiy and Brox. In NIPS, 2016. Chen and Koltun. In ICCV, 2017.
- 33. Deep Networks as a Perceptual Metric 𝐹 𝐹 Normalize, Subtract L2 norm, Spatial average 𝑥 𝑥0 Avg 𝑑0 (1) How well do “perceptual losses” describe perception? (2) Does it have to be VGG trained on classification? c.f. Gatys et al. CVPR 2016. Johnson et al. ECCV 2016. Dosovitskiy and Brox. NIPS 2016. 35
- 34. Original Patch (1) Traditional Distortions Distorted Patches Noise Photometric Spatial warps Compression Blur 36
- 35. Distorted Patches Original Patch Goal: Imitate CNN artifacts from real algorithms (2) CNN-Based Distortions 𝐺 Trained with random: - Input corruption - Generator architecture - Discriminator architecture - Loss function 37
- 36. 38 Which patch is more similar to the middle?
- 37. 40 % agreement with human judges Bigger/Deeper ≠ Better Networks perform strongly across supervisory signals and architectures Fitting some data is important 82.6 68.9 Low-level AlexNet (Random) AlexNet (Unsupervised) AlexNet (Self-supervised) Nets (Supervised - Imagenet classification) Human 75.7 76.8 78.0 76.876.4 75.5 74.8 70.6 70.069.7 VGG on classification (“perceptual loss”) correlates well Can we train on perceptual judgments?
- 38. Traditional Distortion Types 41 Hand-defined functions Original Patch Distorted Patches CNN-Based Distorted Patches Original Patch Family of Generators Real Algorithm Outputs Training distribution Held-out test distribution
- 39. Original Patch Tasks: superresolution, colorization, frame interpolation, video deblurring Real Algorithm Outputs Downsample + Superressolution 43
- 40. Training a Perceptual Metric 𝐹 𝐹 Normalize, Subtract L2 norm, Spatial average 𝑥 𝑥0 Avg 𝑑0 c.f. Gatys et al. CVPR 2016. Johnson et al. ECCV 2016. Dosovitskiy and Brox. NIPS 2016. 𝑤 Off-the-shelf (w=1) Frozen (w learned) Tuned (w learned)
- 41. Off-the-shelf networks already perform well Training a linear layer on top yields small performance boost. Fine-tuning through representation leads to overfitting Training distribution 82.6 76.8 80.6 Held-out distribution 69.5 65.0 64.3 Training a Perceptual Metric Off-the-shelf Tuned Human Off-the-shelf Tuned Human 78.7 Frozen 65.3 Frozen “LPIPS” metric: richzhang.github.io/PerceptualSimilarity Additionally - Ensembled-LPIPS. Kettunen, Härkönen, Lehtinen. ArXiv 2019. - Audio domain. Manocha, Finkelstein, Jin, Bryan, Zhang, Mysore. In progress.
- 42. Example classifications P(correct class) P(correct class)
- 43. Deep Networks are not Shift-Invariant P(correct class) P(correct class)
- 44. Deep Networks are not Shift-Invariant Azulay and Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? In ArXiv, 2018. Engstrom, Tsipras, Schmidt, Madry. Exploring the Landscape of Spatial Robustness. In ICML, 2019. P(correct class) P(correct class)
- 45. Azulay and Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? In ArXiv, 2018. Engstrom, Tsipras, Schmidt, Madry. A rotation and a translation suffice: Fooling cnns with simple transformations. In ArXiv, Deep Networks are not Shift-Invariant
- 46. Making Convolutional Networks Shift-Invariant Again Richard Zhang In ICML, 2019.
- 47. Why is shift-invariance lost? “Convolutions are shift-equivariant” “Pooling builds up shift-invariance” …but striding ignores Nyquist sampling theorem and aliases 51
- 67. 73 Re-examining Max-Pooling Max-pooling breaks shift-equivariance
- 68. Shift-equivariance Testbed • CIFAR • VGG network • 5 max-pools • Test shift-equivariance condition • • Circular convolution/shift pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax 32x32 1x1
- 69. Perfect shift-eq. Large deviation from shift-eq. 75 pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax Shift-equivariance, per layer Convolution is shift-equivariant
- 70. Perfect shift-eq. Large deviation from shift-eq. 76 pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax Shift-equivariance, per layer Pooling breaks shift-equivariance
- 71. 77 pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax Shift-equivariance, per layer Perfect shift-eq. Large deviation from shift-eq.
- 72. pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax Shift-equivariance, per layer Perfect shift-eq. Large deviation from shift-eq.
- 73. pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax Shift-equivariance, per layer Perfect shift-eq. Large deviation from shift-eq.
- 74. pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax Shift-equivariance, per layer Perfect shift-eq. Large deviation from shift-eq.
- 75. pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax Shift-equivariance, per layer Perfect shift-eq. Large deviation from shift-eq.
- 76. Perfect shift-eq. Large deviation from shift-eq. pixels conv1 pool1 conv2 pool2 conv3 pool3 conv4 pool4 conv5 pool5 classifier softmax Shift-equivariance, per layer Every pooling increases periodicity
- 77. Alternative downsampling methods • Blur+subsample • Antialiasing in signal processing; image processing; graphics • Max-pooling • Performs better in deep learning applications [Scherer 2010] 88
- 78. Alternative downsampling methods • Blur+subsample • Antialiasing in signal processing; image processing; graphics • Max-pooling • Performs better for deep learning [Scherer 2010] 89
- 79. Alternative downsampling methods • Blur+subsample • Antialiasing in signal processing; image processing; graphics • Max-pooling • Performs better for deep learning [Scherer 2010] 90 Reconcile antialiasing with max-pooling
- 80. Max (densely) Preserves shift-equivariance max( ) max( ) Anti-aliased pooling (MaxBlurPool) Shift-equivariance lost; heavy aliasing max( ) Strided-MaxPool max( ) Blur Preserves shift-eq. Blur Equivalent Interpretation Shift-eq. lost; heavy aliasing Shift eq. lost, but reduced aliasing Subsampling Evaluated together as “BlurPool”
- 82. 93 (2) Blur+Subsample Shift-equivariance better preserved
- 86. MaxPool (stride 2) Max (stride 1) BlurPool (stride 2) Max Pooling (VGG, AlexNet) Conv (stride 2) ReLU Conv (stride 1) ReLU BlurPool (stride 2) Strided-Convolution (ResNet, MobileNetv2) Antialiasing any downsampling layer AvgPool (stride 2) BlurPool (stride 2) Average Pooling (DenseNet)
- 91. Image-to-Image Translation 103 Input Baseline Antialias [1 1] Antialias [1 2 1] Isola et al. Pix2pix. CVPR 2017.
- 92. Striding aliases(stride=2) Add antialiasing filter + shift-equivariance + accuracy Additionally + stability to other perturbations + robustness to corruptions 104 Discussion
- 93. Discriminative Learning 105 “Rock beauty” Semantic Labels Textures? Edges? Parts? Perceptual similarity? + Solve discriminative tasks + Learn about the visual world + Force the network to learn about the visual world for free Raw Unlabeled Data Textures? Edges? Parts? Perceptual similarity? Image Synthesis + Solve graphics tasks + Engineering in inductive biases (shift-invariance) still valuable
- 94. Computer Vision · Graphics · Deep Learning · Machine Learning Human Computer Interaction · Natural Language Processing San Jose San Francisco Seattle
- 95. Thank You! Anti-aliasing CNNs Zhang. In ICML, 2019. richzhang.github.io/antialiased-cnns Perceptual Similarity Zhang, Isola, Efros, Shechtman, Wang. In CVPR, 2018. richzhang.github.io/PerceptualSimilarity Split-Brain Autoencoder Zhang, Isola, Efros. In CVPR, 2017. richzhang.github.io/splitbrainauto Colorization Zhang, Isola, Efros. ECCV, 2016. Zhang*, Zhu*, et al. SIGGRAPH, 2017. richzhang.github.io/colorization richzhang.github.io/ideepcolor BicycleGAN Zhu, Zhang, Pathak, Darrell, Efros, Wang, Shechtman. In NIPS, 2017. junyanz.github.io/BicycleGAN Detecting Photoshop. Wang, Wang, Owens, Zhang, Efros. In ICCV, 2019. peterwang512.github.io/FALdetector
- 96. Backup 108
- 98. 110
- 99. 111
- 100. 112
- 101. 113
- 102. Train without Data Augmentation Train with Data Augmentation Shift-Invariance vs Classification Accuracy Data augmentation increases both accuracy and shift-invariance Engineering in shift-invariance is “free” data augmentation Boosts shift-invariance while maintaining accuracy
- 103. Our Distortions Real Algorithm Outputs Human L2/PSNR SSIM FSIMc Gaussian K-Means Watch Obj. Split-Brain Puzzle BiGAN SqueezeNet AlexNet VGG Low-level Net (Random) Net (Unsupervised) Net (Self-supervised) Net (Supervised) Human Percep-Trained (Frozen) Percep-Trained (Tuned) Near human-level performance within our distortions Linear “calibration” on our distortions transfers successfully Training on direct task is not the complete solution 115
- 104. Failure Cases – Method can make mistakes 116
- 105. Dorothea Lange, Migrant Mother, 1936. Dorothea Lange. Migrant Mother, 1936. Library of Congress, Prints & Photographs Division, FSA/OWI Collection, reproduction number: LC-USF34-9058-C ? ? ? ? 117
- 106. Grayscale Input Larsson et al. In ECCV, 2016. Iizuka et al. In SIGGRAPH, 2016. Zhang, Isola, Efros. In ECCV, 2016. Automatic Results with Deep Networks 118Dorothea Lange. Migrant Mother, 1936. Library of Congress, Prints & Photographs Division, FSA/OWI Collection, reproduction number: LC-USF34-9058-C
- 107. 119 Dorothea Lange. Migrant Mother, 1936. Library of Congress, Prints & Photographs Division, FSA/OWI Collection, reproduction number: LC-USF34-9058-C Zhang*, Zhu*, Isola, Geng, Lin, Yu, Efros. Real-Time User-Guided Image Colorization with Learned Deep Priors. In SIGGRAPH, 2017.
- 108. 120 Zhang*, Zhu*, Isola, Geng, Lin, Yu, Efros. Real-Time User-Guided Image Colorization with Learned Deep Priors. In SIGGRAPH, 2017. Dorothea Lange. Migrant Mother, 1936. Library of Congress, Prints & Photographs Division, FSA/OWI Collection, reproduction number: LC-USF34-9058-C
- 109. 121 Zhang*, Zhu*, Isola, Geng, Lin, Yu, Efros. Real-Time User-Guided Image Colorization with Learned Deep Priors. In SIGGRAPH, 2017. Dorothea Lange. Migrant Mother, 1936. Library of Congress, Prints & Photographs Division, FSA/OWI Collection, reproduction number: LC-USF34-9058-C