제가 이번에 소개드릴 논문은 Scalable Model Compression by Entropy Penalized Reparameterization이라는 논문입니다. 이전에 꾸준히 Deep Learning을 이용한 이미지 및 비디오 압축에 대해 설명드렸던 바가 있는데, 이번에는 Neural Network의 Model Parameter들을 압축하는 방법에 관한 논문입니다. 논문 링크: https://arxiv.org/abs/1906.06624 영상 링크: https://youtu.be/LJ8WD5MKA2o

1. Scalable Model Compression by
Entropy Penalized
Reparameterization
Hyeongmin Lee
PR-420
2023.2.5

2. Model Compression

3. Model Compression

4. Image Compression

5. Image Compression
 Image Compression[PR 395]
0011010100111...
Lossless Coding: bmp  Coding the Integers Itself
Lossy Coding: JPEG  Reducing the ‘Entropy’ of samples

6. Image Compression
 JPEG
8x8 cutting
Discrete Cosine Transform Quantization
Entropy Coding...
https://www.whydomath.org/node/wavlets/basicjpg.html

7. Deep Learning-based Image Compression
 Image Compression Pipeline
Transform Quantization
Entropy Coding
Decoding
Inverse
Transform

8. Deep Learning-based Image Compression
 Image Compression Pipeline
𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆

9. Deep Learning-based Image Compression
 Auto Encoder
�
𝑦𝑦 = 𝑄𝑄(𝑦𝑦)
𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔
�
𝑥𝑥 = 𝑔𝑔𝑠𝑠(�
𝑦𝑦; 𝜃𝜃𝑔𝑔)
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(�
𝑦𝑦)]
𝐷𝐷 = 𝑥𝑥 − �
𝑥𝑥 2
2
???

10. Problem of Quantization
 Adding Uniform Noise [PR328]
�
𝑦𝑦 = 𝑄𝑄 𝑦𝑦 �
𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛, 𝑛𝑛~𝑈𝑈(−
1
2
,
1
2
)
Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. "End-to-end optimized image compression." 5th International Conference on Learning Representations, ICLR 2017.
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
− log2 𝑝𝑝�
𝑦𝑦 �
𝑦𝑦 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑄𝑄(𝑔𝑔𝑎𝑎(𝑥𝑥; 𝜙𝜙𝑔𝑔)))]
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
− log2 𝑝𝑝�
𝑦𝑦 �
𝑦𝑦 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 + 𝑛𝑛)]

11. Rate Loss
 Applying Uniform Noise Approximation
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 + 𝑛𝑛)]
𝑝𝑝�
𝑦𝑦
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(�
𝑦𝑦)]
Entropy Model

12. Model Compression

13. Model Compression

14. Model Compression
 A Classifier Model

15. Model Compression
 Reparametrization
Θ = 𝐹𝐹(Φ|Ψ)

16. Model Compression
 Parameter Decoder

17. Model Compression
 Probability Model

18. Model Compression
 Discrete Optimization

19. Model Compression
 Model Partitioning

20. Results

21. Results
 Table

22. Results
 Tradeoff Analysis

23. Thank You!