NGC17 Talk @ Oracle - June 7, 2017

1. 1
Fast Algorithms for Quantized
Convolutional Neural Networks
Alessandro Pappalardo alessandro1.pappalardo@mail.polimi.it
NECSTLab, Politecnico di Milano
Oracle
06/07/2017

2. 2
Introduction
Convolutional neural
networks are at the
forefront of big data
processing.
Embedded devices and
smartphones are at the
heart of big data.
Image: Dumoulin, Vincent, and Francesco Visin. "A guide to convolution
arithmetic for deep learning." arXiv preprint arXiv:1603.07285 (2016).

3. 3
Problem
Convolutional neural networks are both
memory 💽 and computational 🖥🖥 intensive.
❤️

4. 4
💽 ➡️ Quantized Convnets
• Precision of convolution reduced to b bits.
• Arrays of unsigned integers in [0, 2b - 1].
• Quantization scheme: map minimum to 0, maximum to 2b –
1.
• Memory savings.

5. 5
🖥🖥 ➡️ Question
Can we can take advantage of the pre-
determined finiteness of the possible
values assumed by the convolution
operands to gain computational savings
at an algorithmic level

6. 6
Solution
• Number Theoretic Transforms (NTTs) are DFTs defined
on a finite field.
• NTTs hold the Circular Convolution Property (CCP) and
can be computed through FFT-like fast algorithms.
Reason in terms of finite fields.

7. 7
Supported parameters
FNT Kernel Min Block Max Block binput,max bkernel,max
F3 3x3 8x8 16x16 3 2
F3 5x5 8x8 16x16 2 2
F3 7x7,9x9 16x16 16x16 2 1
F4 3x3 8x8 32x32 6 6
F4 5x5 8x8 32x32 6 5
F4 7x7 16x16 32x32 5 5
F4 9x9 16x16 32x32 5 4
F4 11x11 32x32 32x32 5 4
RNS(F3,F4) 3x3,5x5 8x8 16x16 8 8
RNS(F3,F4) 7x7,9x9 16x16 16x16 8 8

8. 8
Theoretical costs comparison
Kernel
Naïve Int
Ops/Output Block
Valid
Output
NTT Modulo Ops/Output
Add Mul Add Mul Mul by const Shift
3x3 9 9
8x8 6x6 21.34 1.78 1 7.12
16x16 14x14 20.89 1.31 1 7.84
32x32 30x30 22.76 1.14 1 9.10
5x5 25 25
8x8 4x4 48 4 1 16
16x16 12x12 28.44 1.78 1 10.66
32x32 28x28 26.12 1.31 1 10.45
7x7 49 49
16x16 10x10 40.96 2.56 1 15.36
32x32 26x26 30.29 1.51 1 12.12
9x9 81 81
16x16 8x8 64 4 1 24
32x32 24x24 35.56 1.78 1 14.22
11x11 121 121 32x32 22x22 42.31 2.12 1 16.93

9. 9
Theoretical costs comparison
Kernel
Naïve Int
Ops/Output Block
Valid
Output
NTT Modulo Ops/Output
Add Mul Add Mul Mul by const Shift
3x3 9 9
8x8 6x6 21.34 1.78 1 7.12
16x16 14x14 20.89 1.31 1 7.84
32x32 30x30 22.76 1.14 1 9.10
5x5 25 25
8x8 4x4 48 4 1 16
16x16 12x12 28.44 1.78 1 10.66
32x32 28x28 26.12 1.31 1 10.45
7x7 49 49
16x16 10x10 40.96 2.56 1 15.36
32x32 26x26 30.29 1.51 1 12.12
9x9 81 81
16x16 8x8 64 4 1 24
32x32 24x24 35.56 1.78 1 14.22
11x11 121 121 32x32 22x22 42.31 2.12 1 16.93

10. 10
Benchmarks setting
• Qconv: a C99 compliant and portable NTT
implementation against a naïve convolution
implementation.
• Platform of choice: Raspberry Pi Zero.

11. 11
ResultsInput size 224x224x1

12. 12
Future works
• Implement SIMD subroutines for forward and inverse
transforms.
• Map element-wise products to finite field matrix
multiplication in FFLAS.
• FPGA hardware implementation.
• Optimize for 3x3 filters.

13. Question Time

14. 14
Fermat Number Transforms
FNTs are NTTs mod 𝒑 = 𝟐 𝟐 𝒕
+ 𝟏.
1. Support FFT algorithms.
2. For a length N up to 2 𝑡+1, the forward and inverse
transforms requires only modular adds and shifts.
3. Reduction mod a Fermat prime p:
logical AND, right unsigned shift, modular subtraction

15. 15
Methodology
FNT m 𝐹4 = 65537 with
blocks:
• 8x8
• 16x16
• 32x32
RNS of 𝐹3 and 𝐹4 to increase the maximum output bitwidth to:
𝑙𝑜𝑔2 𝐹3 ∙ 𝐹4 ≅ 24 bits
Overlap-and-save algorithm
FNT mod 𝐹3 = 257 with blocks:
• 8x8
• 16x16

16. 16
Optimizations
• Forward DIF FFT, inverse DIT transforms FFT.
• Precomputed power-of-two twiddle factors .
• Switch the order of the two inner-most loop of the FFT.
• Avoid useless transpositions.
• Normalize the non discarded output only.