1. The document proposes using Number Theoretic Transforms (NTTs) to accelerate quantized convolutional neural networks. NTTs allow fast convolution algorithms by exploiting the finite field of quantized values. 2. The approach applies NTTs with Fermat number transforms (FNTs) to quantized convolution operations. Benchmarks on a Raspberry Pi Zero show speedups compared to a naive convolution implementation. 3. Future work includes implementing SIMD optimizations, mapping operations to finite field matrix multiplication, an FPGA implementation, and techniques like binary segmentation to further accelerate quantized convolutional neural networks.

1. 1
Fast Algorithms for Quantized
Convolutional Neural Networks
Alessandro Pappalardo alessandro1.pappalardo@mail.polimi.it
NECSTLab, Politecnico di Milano
Lawrence Berkeley National Laboratory
06/05/2017

2. 2
Introduction
Convolutional neural
networks are at the forefront
of big data processing.
Embedded devices and
smartphones are at the
heart of big data.
Image: Dumoulin, Vincent, and Francesco Visin. "A guide to convolution
arithmetic for deep learning." arXiv preprint arXiv:1603.07285 (2016).

3. 3
Problem
Convolutional neural networks are both
memory 💽 and computational 🖥️ intensive.
❤️

4. 4
💽 ➡️ Quantized Convnets
• Precision of convolution reduced to b bits.
• Arrays of unsigned integers in [0, 2b - 1].
• Quantization scheme: map minimum to 0, maximum to 2b – 1.
• Memory savings.

5. 5
🖥️ ➡️ Question
Can we can take advantage of the pre-
determined finiteness of the possible values
assumed by the convolution operands to gain
computational savings at an algorithmic level

6. 6
Solution
Number Theoretic Transforms (NTTs) are• DFTs defined on a
finite field.
NTTs hold the• Circular Convolution Property (CCP) and can be
computed through FFT-like fast algorithms.
Reason in terms of finite fields.

7. 7
Supported parameters
FNT Kernel Min Block Max Block binput,max bkernel,max
F3 3x3 8x8 16x16 3 2
F3 5x5 8x8 16x16 2 2
F3 7x7,9x9 16x16 16x16 2 1
F4 3x3 8x8 32x32 6 6
F4 5x5 8x8 32x32 6 5
F4 7x7 16x16 32x32 5 5
F4 9x9 16x16 32x32 5 4
F4 11x11 32x32 32x32 5 4
RNS(F3,F4) 3x3,5x5 8x8 16x16 8 8
RNS(F3,F4) 7x7,9x9 16x16 16x16 8 8

8. 8
Theoretical costs comparison
Kernel
Naïve Int
Ops/Output Block
Valid
Output
NTT Modulo Ops/Output
Add Mul Add Mul Mul by const Shift
3x3 9 9
8x8 6x6 21.34 1.78 1 7.12
16x16 14x14 20.89 1.31 1 7.84
32x32 30x30 22.76 1.14 1 9.10
5x5 25 25
8x8 4x4 48 4 1 16
16x16 12x12 28.44 1.78 1 10.66
32x32 28x28 26.12 1.31 1 10.45
7x7 49 49
16x16 10x10 40.96 2.56 1 15.36
32x32 26x26 30.29 1.51 1 12.12
9x9 81 81
16x16 8x8 64 4 1 24
32x32 24x24 35.56 1.78 1 14.22
11x11 121 121 32x32 22x22 42.31 2.12 1 16.93

9. 9
Theoretical costs comparison
Kernel
Naïve Int
Ops/Output Block
Valid
Output
NTT Modulo Ops/Output
Add Mul Add Mul Mul by const Shift
3x3 9 9
8x8 6x6 21.34 1.78 1 7.12
16x16 14x14 20.89 1.31 1 7.84
32x32 30x30 22.76 1.14 1 9.10
5x5 25 25
8x8 4x4 48 4 1 16
16x16 12x12 28.44 1.78 1 10.66
32x32 28x28 26.12 1.31 1 10.45
7x7 49 49
16x16 10x10 40.96 2.56 1 15.36
32x32 26x26 30.29 1.51 1 12.12
9x9 81 81
16x16 8x8 64 4 1 24
32x32 24x24 35.56 1.78 1 14.22
11x11 121 121 32x32 22x22 42.31 2.12 1 16.93

10. 10
Benchmarks setting
Qconv• : a C99 compliant and portable NTT implementation
against a naïve convolution implementation.
Platform of choice:• Raspberry Pi Zero.

11. 11
ResultsInput size 224x224x1

12. 12
Future works
Implement SIMD subroutines for forward and inverse•
transforms.
Map element• -wise products to finite field matrix multiplication
in FFLAS.
FPGA hardware implementation.•
Binary segmentation for• 3x3 filters.

13. Question Time

14. 14
Fermat Number Transforms
FNTs are NTTs mod 𝒑 = 𝟐 𝟐 𝒕
+ 𝟏.
1. Support FFT algorithms.
2. For a length N up to 2 𝑡+1, the forward and inverse
transforms requires only modular adds and shifts.
3. Reduction mod a Fermat prime p:
logical AND, right unsigned shift, modular subtraction

15. 15
Methodology
FNT m 𝐹4 = 65537 with blocks:
• 8x8
• 16x16
• 32x32
RNS of 𝐹3 and 𝐹4 to increase the maximum output bitwidth to:
𝑙𝑜𝑔2 𝐹3 ∙ 𝐹4 ≅ 24 bits
Overlap-and-save algorithm
FNT mod 𝐹3 = 257 with blocks:
• 8x8
16• x16

16. 16
Optimizations
Forward DIF FFT, inverse DIT transforms FFT.•
Precomputed power• -of-two twiddle factors .
Switch the order of the two inner• -most loop of the FFT.
Avoid useless transpositions.•
Normalize the non discarded output only.•