Tensorflow Korea 논문읽기 모임 PR12의 113번째 발표는 The Perception Distortion Tradeoff입니다.

1. The Perception-Distortion Tradeoff

2. The Paper
The Perception-Distortion Tradeoff, CVPR 2018 Oral Presentation
Yochai Blau and Tomer Michaeli, Technion – Israel Institue of Technology
Conclusion
The Perception-Distortion Tradeoff Exists
Algorithms Cannot Achieve both Low Distortion & Good Perceptual Quality
We need to pay more attention to Evaluate Image Processing Results

3. Perception and Distortion: Metric for Image Prediction

4. Ex) Image Prediction
Slide from Y. Blau

5. Goals?
▷ Similar to GT Image = High PSNR = Low Distortion (Error)
▷ Good Perceptual Quality
𝑃𝑆𝑁𝑅 = −10 log(
1
𝐴𝑣𝑔 𝑦 𝑝𝑟𝑒𝑑 − 𝑦
2)

8. Goals?
▷ Similar to GT Image = High PSNR = Low Distortion (Error)
▷ Good Perceptual Quality
Recent Approaches
However, Is it good choice?

9. Problem Setting
▷ Original Image Distribution 𝑃𝑋
▷ Degraded Model 𝑃𝑌|𝑋
▷ Reconstruction Model 𝑃 ෠𝑋|𝑌
▷ Reconstructed Image Distribution 𝑃 ෠𝑋

10. Definition: Distortion
How Similar to Ground Truth?
𝔼 Δ 𝑋, ෠𝑋
Expectation over the Joint Distribution 𝑃 𝑋, ෠𝑋

11. Definition: Distortion
▷ How Similar to Ground Truth?
▷ Expectation over the Joint Distribution 𝑃 𝑋, ෠𝑋
▷ Full Reference Metric
𝔼 Δ 𝑋, ෠𝑋
• Mean Squared Error
• SSIM
Image Quality Assessment: From Error Visibility to Structural Similarity, TIP 2004
• MS-SSIM
Multiscale Structure Similarity for Image Quality Assessment, CSSC 2004
• IFC
An Information Fidelity Criterion for Image Quality Assessment using Natural Scene Statistics, TIP 2005
• VIF
Image Information and Visual Quality, TIP 2006
• Perceptual Loss
Perceptual Losses for Real-time Style Transfer and Super Resolution, ECCV 2016

12. • SSIM
• MS-SSIM
• Perceptual Loss
𝑆𝑆𝐼𝑀 𝑥, 𝑦 = 𝑙𝑢𝑚𝑖𝑛𝑎𝑛𝑐𝑒 𝑥, 𝑦 ∙ 𝑐𝑜𝑛𝑡𝑟𝑎𝑠𝑡 𝑥, 𝑦 ∙ 𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒(𝑥, 𝑦)
𝑀𝑢𝑙𝑡𝑖𝑆𝑐𝑎𝑙𝑒 − 𝑆𝑆𝐼𝑀 𝑥, 𝑦 = 𝑙𝑢𝑚𝑖𝑛𝑎𝑛𝑐𝑒 𝑥, 𝑦 ∙ ෍
𝑖
𝑐𝑜𝑛𝑡𝑟𝑎𝑠𝑡𝑖 𝑥, 𝑦 ∙ 𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒𝑖(𝑥, 𝑦)
𝑃𝑒𝑟𝑐𝑒𝑝𝑡𝑢𝑎𝑙 𝑥, 𝑦 = 𝑀𝑆𝐸(𝑉𝐺𝐺2,2 𝑥 − 𝑉𝐺𝐺2,2 𝑦 )

13. Definition: Perceptual Quality
The Degree to Which it Looks Like a
Natural Image
𝑑𝑖𝑣(𝑝 𝑋, 𝑝 ෠𝑋)
Perceptual Quality
∝ Human Mean Opinion Score
∝ No-Reference Metric
∝ Real & Fake Test
∝ Divergence in GANs

14. Definition: Perceptual Quality
▷ The Degree to Which it Looks Like a Natural Image
▷ Perceptual Quality
∝ Human Mean Opinion Score
∝ No-Reference Metric
∝ Real & Fake Test
∝ Divergence in GANs
𝑑𝑖𝑣(𝑝 𝑋, 𝑝 ෠𝑋)
• Total Variation
• Jenson-Shannon Divergence
Generative Adversarial Nets, NIPS 2014
• Wasserstein Distance
Wasserstein GAN, ICML 2017
• Any f-Divergence
f-GAN, NIPS 2016

15. Definition: Perceptual Quality
▷ The Degree to Which it Looks Like a Natural Image
▷ Perceptual Quality
∝ Human Mean Opinion Score
∝ No-Reference Metric (In Experiments)
∝ Real & Fake Test
∝ Divergence in GANs
𝑑𝑖𝑣(𝑝 𝑋, 𝑝 ෠𝑋)
• BRISQUE
No-Reference Image Quality Assessment in the Spatial Domain, TIP 2012
• BLIINDS-II
Blind Image Quality Assessment: A Natural Scene Stastics Approach in the DCT Domain, TIP 2012
• NIQE
Making a Completely Blind Image Quality Analyzer, IEEE SP Letters, 2013
• Ma et al.
Learning a No-Reference Quality Metric for Single Image Super-Resolution, CVIU 2017

16. • BRISQUE • NIQE
Slide from MATLAB Help

17. • Ma et al.

18. The Perception-Distortion Tradeoff
𝑑𝑖𝑣(𝑝 𝑋, 𝑝 ෠𝑋)
𝔼 Δ 𝑋, ෠𝑋

19. The Perception-Distortion Tradeoff: SISR

20. The Perception-Distortion Tradeoff: SISR

21. The Perception-Distortion Tradeoff: Proof

22. Definition 1: The Perception-Distortion Function
𝑃 𝐷 = min
𝑝෡𝑋|𝑌
𝑑(𝑝 𝑋, 𝑝෠𝑋) 𝑠. 𝑡. 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷

23. Definition 1: The Perception-Distortion Function
𝑃 𝐷 = min
𝑝෡𝑋|𝑌
𝑑(𝑝 𝑋, 𝑝෠𝑋) 𝑠. 𝑡. 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷
DistortionPerception
Recon.
Algorithm

24. Definition 1: The Perception-Distortion Function
𝑃 𝐷 = min
𝑝෡𝑋|𝑌
𝑑(𝑝 𝑋, 𝑝෠𝑋) 𝑠. 𝑡. 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷
DistortionPerception
Recon.
Algorithm

25. Definition 1: The Perception-Distortion Function
𝑃 𝐷 = min
𝑝෡𝑋|𝑌
𝑑(𝑝 𝑋, 𝑝෠𝑋) 𝑠. 𝑡. 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷
DistortionPerception
Recon.
Algorithm

26. Definition 1: The Perception-Distortion Function
𝑃 𝐷 = min
𝑝෡𝑋|𝑌
𝑑(𝑝 𝑋, 𝑝෠𝑋) 𝑠. 𝑡. 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷
DistortionPerception
Recon.
Algorithm

27. Definition 1: The Perception-Distortion Function
𝑃 𝐷 = min
𝑝෡𝑋|𝑌
𝑑(𝑝 𝑋, 𝑝෠𝑋) 𝑠. 𝑡. 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷
DistortionPerception
Recon.
Algorithm

28. Theorem 1: The Perception-Distortion Tradeoff
If 𝑑(𝑝 𝑋, 𝑝 ෠𝑋) is Convex in 𝑝 ෠𝑋,
𝑃 𝐷 is Monotonically Non-Increasing & Convex

29. Theorem 1: The Perception-Distortion Tradeoff
If 𝑑(𝑝 𝑋, 𝑝 ෠𝑋) is Convex in 𝑝 ෠𝑋,
𝑃 𝐷 is Monotonically Non-Increasing & Convex
Small Distortion+  Large Perceptual Quality Loss

30. Theorem 1: The Perception-Distortion Tradeoff
If 𝑑(𝑝 𝑋, 𝑝 ෠𝑋) is Convex in 𝑝 ෠𝑋,
𝑃 𝐷 is Monotonically Non-Increasing & Convex
Small Perceptual Quality Gain  Large Distortion ++

31. Theorem 1: The Perception-Distortion Tradeoff: Proof
If 𝑑(𝑝 𝑋, 𝑝෠𝑋) is Convex in 𝑝෠𝑋
𝑃 𝐷 is Monotonically Non-Increasing & Convex
𝑃 𝐷 = min
𝑝෡𝑋|𝑌
𝑑(𝑝 𝑋, 𝑝෠𝑋) 𝑠. 𝑡. 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷

32. Theorem 1: The Perception-Distortion Tradeoff: Proof
If 𝑑(𝑝 𝑋, 𝑝෠𝑋) is Convex in 𝑝෠𝑋
𝑃 𝐷 is Monotonically Non-Increasing & Convex
𝑃 𝐷 = min
𝑝෡𝑋|𝑌
𝑑(𝑝 𝑋, 𝑝෠𝑋) 𝑠. 𝑡. 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷
𝑃 𝐷 = Minimum 𝑑(𝑝 𝑋, 𝑝෠𝑋) ∈ 𝐷 𝐷
If 𝐷 Increases, 𝐷 𝐷 Increases, 𝑃(𝐷) is Non-Increasing

33. Theorem 1: The Perception-Distortion Tradeoff: Proof
If 𝑑(𝑝 𝑋, 𝑝෠𝑋) is Convex in 𝑝෠𝑋
𝑃 𝐷 is Monotonically Non-Increasing & Convex
𝜆𝑃 𝐷1 + 1 − 𝜆 𝑃 𝐷2 ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)

34. 𝜆𝑃 𝐷1 + 1 − 𝜆 𝑃 𝐷2 ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)
𝜆𝑑(𝑝 𝑋, 𝑝෠𝑋1
) + 1 − 𝜆 𝑑(𝑝 𝑋, 𝑝෠𝑋2
) By Definition (Optimum Estimation)

35. 𝜆𝑃 𝐷1 + 1 − 𝜆 𝑃 𝐷2 ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)
𝜆𝑑(𝑝 𝑋, 𝑝෠𝑋1
) + 1 − 𝜆 𝑑(𝑝 𝑋, 𝑝෠𝑋2
) By Definition (Optimum Estimation)
𝜆𝑑 𝑝 𝑋, 𝑝෠𝑋1
+ 1 − 𝜆 𝑑 𝑝 𝑋, 𝑝෠𝑋2
≥ 𝑑(𝑝 𝑋, 𝜆𝑝 ෠𝑋1
+ 1 − 𝜆 𝑝෠𝑋2
) 𝑑 ∙, 𝑥 is Convex

36. 𝜆𝑃 𝐷1 + 1 − 𝜆 𝑃 𝐷2 ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)
𝜆𝑑(𝑝 𝑋, 𝑝෠𝑋1
) + 1 − 𝜆 𝑑(𝑝 𝑋, 𝑝෠𝑋2
)
𝜆𝑑 𝑝 𝑋, 𝑝෠𝑋1
+ 1 − 𝜆 𝑑 𝑝 𝑋, 𝑝෠𝑋2
≥ 𝑑(𝑝 𝑋, 𝑝 ෠𝑋 𝜆
)
By Definition (Optimum Estimation)
𝜆𝑑 𝑝 𝑋, 𝑝෠𝑋1
+ 1 − 𝜆 𝑑 𝑝 𝑋, 𝑝෠𝑋2
≥ 𝑑(𝑝 𝑋, 𝜆𝑝 ෠𝑋1
+ 1 − 𝜆 𝑝෠𝑋2
) 𝑑 ∙, 𝑥 is Convex
𝑝 ෠𝑋 𝜆
= 𝜆𝑝 ෠𝑋1
+ 1 − 𝜆 𝑝෠𝑋2

37. 𝜆𝑃 𝐷1 + 1 − 𝜆 𝑃 𝐷2 ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)
𝜆𝑑(𝑝 𝑋, 𝑝෠𝑋1
) + 1 − 𝜆 𝑑(𝑝 𝑋, 𝑝෠𝑋2
)
𝜆𝑑 𝑝 𝑋, 𝑝෠𝑋1
+ 1 − 𝜆 𝑑 𝑝 𝑋, 𝑝෠𝑋2
≥ 𝑑(𝑝 𝑋, 𝑝 ෠𝑋 𝜆
)
𝑑 𝑝 𝑋, 𝑝෠𝑋 𝜆
≥ min
𝑝෡𝑋|𝑌
𝑑 𝑝 𝑋, 𝑝෠𝑋 ∶ 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷 𝜆 = 𝑃(𝐷 𝜆)
By Definition (Optimum Estimation)
𝜆𝑑 𝑝 𝑋, 𝑝෠𝑋1
+ 1 − 𝜆 𝑑 𝑝 𝑋, 𝑝෠𝑋2
≥ 𝑑(𝑝 𝑋, 𝜆𝑝 ෠𝑋1
+ 1 − 𝜆 𝑝෠𝑋2
) 𝑑 ∙, 𝑥 is Convex
𝑝 ෠𝑋 𝜆
= 𝜆𝑝 ෠𝑋1
+ 1 − 𝜆 𝑝෠𝑋2
By Definition (Not Optimum Estimation)

38. 𝜆𝑃 𝐷1 + 1 − 𝜆 𝑃 𝐷2 ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)
𝜆𝑑(𝑝 𝑋, 𝑝෠𝑋1
) + 1 − 𝜆 𝑑(𝑝 𝑋, 𝑝෠𝑋2
)
𝜆𝑑 𝑝 𝑋, 𝑝෠𝑋1
+ 1 − 𝜆 𝑑 𝑝 𝑋, 𝑝෠𝑋2
≥ 𝑑(𝑝 𝑋, 𝑝 ෠𝑋 𝜆
)
𝑑 𝑝 𝑋, 𝑝෠𝑋 𝜆
≥ min
𝑝෡𝑋|𝑌
𝑑 𝑝 𝑋, 𝑝෠𝑋 ∶ 𝔼 Δ 𝑋, ෠𝑋 ≤ 𝐷 𝜆 = 𝑃(𝐷 𝜆)
By Definition (Optimum Estimation)
𝜆𝑑 𝑝 𝑋, 𝑝෠𝑋1
+ 1 − 𝜆 𝑑 𝑝 𝑋, 𝑝෠𝑋2
≥ 𝑑(𝑝 𝑋, 𝜆𝑝 ෠𝑋1
+ 1 − 𝜆 𝑝෠𝑋2
)
𝜆𝑃 𝐷1 + 1 − 𝜆 𝑃 𝐷2 ≥ 𝑑 𝑝 𝑋, 𝑝෠𝑋 𝜆
≥ 𝑃(𝐷 𝜆)
𝑑 ∙, 𝑥 is Convex
𝑝 ෠𝑋 𝜆
= 𝜆𝑝 ෠𝑋1
+ 1 − 𝜆 𝑝෠𝑋2
By Definition (Not Optimum Estimation)

39. 𝐷 𝜆 = 𝔼 Δ 𝑋, ෠𝑋𝜆
= 𝔼 𝔼 Δ 𝑋, ෠𝑋𝜆 𝑌
= 𝔼 𝔼 𝑝 𝑥, ො𝑥 𝜆 𝑦
= 𝔼 𝔼 𝑝 ො𝑥 𝜆 𝑥, 𝑦 𝑝 𝑥 𝑦
= 𝔼 𝔼 𝜆𝑝 ො𝑥1 𝑦 + 1 − 𝜆 𝑝 ො𝑥2 𝑦 𝑝 𝑥 𝑦
= 𝔼 𝔼 𝜆𝑝 ො𝑥1 𝑦 𝑝 𝑥 𝑦 + 1 − 𝜆 𝑝 ො𝑥2 𝑦 𝑝 𝑥 𝑦
= 𝔼 𝔼 𝜆𝑝 𝑥, ො𝑥1 𝑦 + 1 − 𝜆 𝑝 𝑥, ො𝑥2 𝑦
= 𝔼 𝜆𝔼 Δ 𝑋, ෠𝑋1 𝑌 + (1 − 𝜆)𝔼 Δ 𝑋, ෠𝑋2 𝑌
= 𝜆𝔼 Δ 𝑋, ෠𝑋1 + (1 − 𝜆)𝔼 Δ 𝑋, ෠𝑋2
≤ 𝜆𝐷1 + 1 − 𝜆 𝐷2
Law of Total Expectation
By Definition of Distortion
Independence, Chain Rule
By Definition
Chain Rule
Law of Total Expectation
𝐷 𝜆 ≤ 𝜆𝐷1 + 1 − 𝜆 𝐷2
𝑃(𝐷 𝜆) ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)

40. 𝐷 𝜆 = 𝔼 Δ 𝑋, ෠𝑋𝜆
= 𝔼 𝔼 Δ 𝑋, ෠𝑋𝜆 𝑌
= 𝔼 𝔼 𝑝 𝑥, ො𝑥 𝜆 𝑦
= 𝔼 𝔼 𝑝 ො𝑥 𝜆 𝑥, 𝑦 𝑝 𝑥 𝑦
= 𝔼 𝔼 𝜆𝑝 ො𝑥1 𝑦 + 1 − 𝜆 𝑝 ො𝑥2 𝑦 𝑝 𝑥 𝑦
= 𝔼 𝔼 𝜆𝑝 ො𝑥1 𝑦 𝑝 𝑥 𝑦 + 1 − 𝜆 𝑝 ො𝑥2 𝑦 𝑝 𝑥 𝑦
= 𝔼 𝔼 𝜆𝑝 𝑥, ො𝑥1 𝑦 + 1 − 𝜆 𝑝 𝑥, ො𝑥2 𝑦
= 𝔼 𝜆𝔼 Δ 𝑋, ෠𝑋1 𝑌 + (1 − 𝜆)𝔼 Δ 𝑋, ෠𝑋2 𝑌
= 𝜆𝔼 Δ 𝑋, ෠𝑋1 + (1 − 𝜆)𝔼 Δ 𝑋, ෠𝑋2
≤ 𝜆𝐷1 + 1 − 𝜆 𝐷2
Law of Total Expectation
By Definition of Distortion
Independence, Chain Rule
By Definition
Chain Rule
Law of Total Expectation
𝐷 𝜆 ≤ 𝜆𝐷1 + 1 − 𝜆 𝐷2
𝑃(𝐷 𝜆) ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)

41. 𝜆𝑃 𝐷1 + 1 − 𝜆 𝑃 𝐷2 ≥ 𝑑 𝑝 𝑋, 𝑝෠𝑋 𝜆
≥ 𝑃(𝐷 𝜆) ≥ 𝑃(𝜆𝐷1 + 1 − 𝜆 𝐷2)

42. Traversing the Tradeoff with a GAN

45. • SRGAN
Photo-Realistic Single Image Super Resolution using a GAN, CVPR 2017
• EnhanceNet
EnhanceNet: Single Image Super Resolution Through Automated Texture Synthesis, ICCV 2017
• Inpainting-GAN
Semantic Image Inpainting with Deep Generative Models. CVPR 2017
• Denoising-GAN
Deep Generative Adversarial Compression Artifact Removal, ICCV 2017
𝐿 = 𝜙 𝐼𝑒𝑠𝑡 − 𝜙 𝐼 𝐻𝑅
2
+ 10−3
𝐿 𝐺𝐴𝑁
𝐿 = 0.2 𝜙 𝐼𝑒𝑠𝑡 − 𝜙 𝐼 𝐻𝑅
2
+ 3 ∙ 10−7
𝐺(𝜙 𝐼𝑒𝑠𝑡 ) − 𝐺(𝜙 𝐼 𝐻𝑅 ) 2
+ 2𝐿 𝐺𝐴𝑁
𝐿 = 𝑊 ⊚ (𝐼𝑒𝑠𝑡 − 𝐼 𝐻𝑅) 1 + 0.003𝐿 𝐺𝐴𝑁
𝐿 = 𝜙 𝐼𝑒𝑠𝑡 − 𝜙 𝐼 𝐻𝑅
2
+ 𝜆𝐿 𝐺𝐴𝑁

46. More Examples

47. Slide from Y. Blau

49. • Enhanced SRGAN

50. • EDSR-GAN

51. • B Dominates A
• Better Distortion, Better Perceptual Quality
• B, C and D are Admissible
• Not Dominated by Any Other Algorithm

52. Connection to Rate-Distortion Theory

53. • Rate Distortion Function 𝑅(𝐷)
• If D is a tolerable distortion,
then 𝑅 𝐷 is the Minimum Rate with which the data source can be coded
• Rate ∝ Bit per Second ∝ Source Quality
Slide from Y. Blau

54. Distortion
Rate

55. • The Optimal Algorithm is Application Dependent
• eg ) Medical Image / Personal Photos
• One Cannot Dominate the Others
Slide from http://www.screenplaysmag.com/2013/11/25/keys-to-solving-the-strategic-challenge-posed-by-msos-reliance-on-mpeg-2/

56. Discussions

57. Degradation Must be Non-Invertable

58. Anti-Correlation

59. Tradeoff Strength by Metric

60. Theorem 2
You will Never need to Degrade
More than 3dB in PSNR (or x2 in MSE)
to obtain Perpect Perceptual Quality
Slide from Y. Blau

61. If Condition is Satisfied, PD Tradeoff arises in Other Domain
𝐷𝑖𝑠𝑡𝑜𝑟𝑡𝑖𝑜𝑛 = 𝔼 Δ 𝑋, ෠𝑋
𝑃𝑒𝑟𝑐𝑒𝑝𝑡𝑖𝑜𝑛 = 𝑑𝑖𝑣(𝑝 𝑋, 𝑝 ෠𝑋)
Theorem 1: The Perception-Distortion Tradeoff
If 𝑑(𝑝 𝑋, 𝑝 ෠𝑋) is Convex in 𝑝 ෠𝑋,
𝑃 𝐷 is Monotonically Non-Increasing & Convex

62. Thank You