The document discusses scalable model compression techniques focusing on entropy penalized reparameterization, particularly in the context of image compression. It outlines various methods including lossless and lossy coding approaches, deep learning-based pipelines, and quantization issues. Additionally, the document provides insights into reparametrization and presents results pertaining to model compression tradeoffs.

1. Scalable Model Compression by
Entropy Penalized
Reparameterization
Hyeongmin Lee
PR-420
2023.2.5

2. Model Compression

3. Model Compression

4. Image Compression

5. Image Compression
 Image Compression[PR 395]
0011010100111...
Lossless Coding: bmp  Coding the Integers Itself
Lossy Coding: JPEG  Reducing the ‘Entropy’ of samples

6. Image Compression
 JPEG
8x8 cutting
Discrete Cosine Transform Quantization
Entropy Coding...
https://www.whydomath.org/node/wavlets/basicjpg.html

7. Deep Learning-based Image Compression
 Image Compression Pipeline
Transform Quantization
Entropy Coding
Decoding
Inverse
Transform

8. Deep Learning-based Image Compression
 Image Compression Pipeline
𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆

9. Deep Learning-based Image Compression
 Auto Encoder
�
𝑦𝑦 = 𝑄𝑄(𝑦𝑦)
𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔
�
𝑥𝑥 = 𝑔𝑔𝑠𝑠(�
𝑦𝑦; 𝜃𝜃𝑔𝑔)
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(�
𝑦𝑦)]
𝐷𝐷 = 𝑥𝑥 − �
𝑥𝑥 2
2
???

10. Problem of Quantization
 Adding Uniform Noise [PR328]
�
𝑦𝑦 = 𝑄𝑄 𝑦𝑦 �
𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛, 𝑛𝑛~𝑈𝑈(−
1
2
,
1
2
)
Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. "End-to-end optimized image compression." 5th International Conference on Learning Representations, ICLR 2017.
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
− log2 𝑝𝑝�
𝑦𝑦 �
𝑦𝑦 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑄𝑄(𝑔𝑔𝑎𝑎(𝑥𝑥; 𝜙𝜙𝑔𝑔)))]
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
− log2 𝑝𝑝�
𝑦𝑦 �
𝑦𝑦 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 + 𝑛𝑛)]

11. Rate Loss
 Applying Uniform Noise Approximation
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 + 𝑛𝑛)]
𝑝𝑝�
𝑦𝑦
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(�
𝑦𝑦)]
Entropy Model

12. Model Compression

13. Model Compression

14. Model Compression
 A Classifier Model

15. Model Compression
 Reparametrization
Θ = 𝐹𝐹(Φ|Ψ)

16. Model Compression
 Parameter Decoder

17. Model Compression
 Probability Model

18. Model Compression
 Discrete Optimization

19. Model Compression
 Model Partitioning

20. Results

21. Results
 Table

22. Results
 Tradeoff Analysis

23. Thank You!