[SUB] Deep learning model to 2Bit Quantization?! BRECQ Paper review (2021 ICLR) 안녕하세요 딥러닝 논문읽기 모임 107번 째, 펀디멘탈팀 김동희님의 BRECQ 논문 리뷰 입니다. 문의 : tfkeras@kakao.com

1. BRECQ:Pushing the Limit of Post-Training Quantization
by Block Reconstruction
김동희
Accepted at ICLR 2021
고 형 권 , 김 창 연 , 송 헌 , 이 민 경 , 이 재 윤

2. Introduction
01.
2
Whyselectthispaper?

3. Introduction
01.
3
Whyselectthispaper?

4. 01. Introduction
Whyselectthispaper?
4

5. 01. Introduction
PresentationGoal
5
논문엔있지만오늘발표에는담겨져있지않은내용:
Effectofthefirstlayerandthelastlayer
Calibrationdatapoint수에따른quantization성능변화
Quantizationstepsizes를학습시키는방법
Mixedprecision을학습시키기위한geneticalgorithm
이외수식&증명…….
Quantization을위한논문의핵심접근방법에집중!

6. 01.
Overview
Introduction
6

7. 01.
PTQ QAT
X240 faster production!
Overview
Introduction
7

8. 01.
PTQ QAT
그렇다고데이터가하나도안필요한건아니고…
Overview
Introduction
8

9. Q&A

10. Background
02.
LossDegradation
Quantizedweight와Originalweight간의distance를최소화
=>2018년까지는좋은접근방법
10
BRECQstudiesthelossdegradationinpost-trainingquantizationbyapproximatingtheTaylorexpansion
andGauss-NewtonapproximationoftheHessianmatrix.

11. Background
02.
LossDegradation
lossfunction을최소화하는Quantizedweight찾기
Nahshan, Yury, et al. "Loss aware post-training quantization." arXiv preprint arXiv:1911.07190 (2019).
11
BRECQstudiesthelossdegradationinpost-trainingquantizationbyapproximatingtheTaylorexpansion
andGauss-NewtonapproximationoftheHessianmatrix.

12. Background
02.
TaylorExpansion
Hessian의complexity는O(𝑊2
)=>Hessian계산을위해두가지가정을진행
Nagel, Markus, et al. "Up or down? adaptive rounding for post-training quantization." International Conference on Machine Learning. PMLR, 2020.
https://roadcom.tistory.com/26
12
BRECQstudiesthelossdegradationinpost-trainingquantizationbyapproximatingtheTaylorexpansion
andGauss-NewtonapproximationoftheHessianmatrix.

13. Background
02.
HessianMatrix
Nagel, Markus, et al. "Up or down? adaptive rounding for post-training quantization." International Conference on Machine Learning. PMLR, 2020.
13
1.
2.
전반적으로좋은성능을보이지만INT2라는극한환경에서는성능이떨어짐
=>ΔW가커질수록위의가정이성립되지않기때문(이라고주장)
BRECQstudiesthelossdegradationinpost-trainingquantizationbyapproximatingtheTaylorexpansion
andGauss-NewtonapproximationoftheHessianmatrix.

14. Q&A

15. Proposed Method
03.
Recap
15
Weight를fullprecision과최대한따라하자
Weightㄴㄴ.loss값을비슷하게만드는quantization방법을찾아보자
근데그방법찾아보니까weight에대한Hessianmatrix를구해야하네?
Hessian직접구하기는너무힘드니까대충가정깔고들어가자
ㄴㄴ대충가정깔지말고그가정을좀더strict하게유도해보자!
(본논문의핵심아이디어)

16. Proposed Method
03.
Cross-LayerDependency
16
=>Layerdependence를고려해서Hessian을다시계산해보자
증명은논문참조…
첫번째가정
Weshowthatthecross-layerdependencecanbecomputedbymeasuringthedistanceoftheoutputof
thewholenetwork.

17. Proposed Method
03.
ApproximatingPre-activationHessian
17
저런가정은information의손실이많다!
두번째가정
이 가됨
FisherInformationMatrix를적용하면
Weshowthatthecross-layerdependencecanbecomputedbymeasuringthedistanceoftheoutputof
thewholenetwork.

18. Proposed Method
03.
BlockReconstruction
18
1. Layer-wiseReconstruction
2. Block-wiseReconstruction
3. Stage-wiseReconstruction
4. Network-wiseReconstruction
However, if more layers are considered, the distance of output will become an inaccurate signal and
prohibitstheoptimization.

19. Proposed Method
03.
BlockReconstruction
19
1. Layer-wiseReconstruction
2. Block-wiseReconstruction
3. Stage-wiseReconstruction
4. Network-wiseReconstruction
However, if more layers are considered, the distance of output will become an inaccurate signal and
prohibitstheoptimization.

20. Proposed Method
03.
BlockReconstruction
20
1. Layer-wiseReconstruction
2. Block-wiseReconstruction
3. Stage-wiseReconstruction
4. Network-wiseReconstruction
Toreachthebesttradeoffbetweenthem,weusethebuildingblockasthebasicreconstructionunit.

21. Q&A

22. Experiments
04.
Imagenet-PTQ
22

23. Experiments
04.
Imagenet-PTQ
23

24. Experiments
04.
Imagenet-QAT
24

25. Experiments
04.
MSCOCO
25

26. Thank you