This document proposes using Number Theoretic Transforms (NTTs) to accelerate quantized convolutional neural networks. NTTs, which are discrete Fourier transforms over finite fields, allow convolutional operations to be computed using fast Fourier transform-like algorithms by exploiting the circular convolution property. The approach was tested on a Raspberry Pi using Fermat number transforms to perform quantized convolutions more efficiently than a naive implementation. Future work includes optimizing for smaller kernel sizes and implementing on FPGAs.

1. 1
Fast Algorithms for Quantized
Convolutional Neural Networks
Alessandro Pappalardo alessandro1.pappalardo@mail.polimi.it
NECSTLab, Politecnico di Milano
Oracle
06/07/2017

2. 2
Introduction
Convolutional neural
networks are at the
forefront of big data
processing.
Embedded devices and
smartphones are at the
heart of big data.
Image: Dumoulin, Vincent, and Francesco Visin. "A guide to convolution
arithmetic for deep learning." arXiv preprint arXiv:1603.07285 (2016).

3. 3
Problem
Convolutional neural networks are both
memory 💽 and computational 🖥🖥 intensive.
❤️

4. 4
💽 ➡️ Quantized Convnets
• Precision of convolution reduced to b bits.
• Arrays of unsigned integers in [0, 2b - 1].
• Quantization scheme: map minimum to 0, maximum to 2b –
1.
• Memory savings.

5. 5
🖥🖥 ➡️ Question
Can we can take advantage of the pre-
determined finiteness of the possible
values assumed by the convolution
operands to gain computational savings
at an algorithmic level

6. 6
Solution
• Number Theoretic Transforms (NTTs) are DFTs defined
on a finite field.
• NTTs hold the Circular Convolution Property (CCP) and
can be computed through FFT-like fast algorithms.
Reason in terms of finite fields.

7. 7
Supported parameters
FNT Kernel Min Block Max Block binput,max bkernel,max
F3 3x3 8x8 16x16 3 2
F3 5x5 8x8 16x16 2 2
F3 7x7,9x9 16x16 16x16 2 1
F4 3x3 8x8 32x32 6 6
F4 5x5 8x8 32x32 6 5
F4 7x7 16x16 32x32 5 5
F4 9x9 16x16 32x32 5 4
F4 11x11 32x32 32x32 5 4
RNS(F3,F4) 3x3,5x5 8x8 16x16 8 8
RNS(F3,F4) 7x7,9x9 16x16 16x16 8 8

8. 8
Theoretical costs comparison
Kernel
Naïve Int
Ops/Output Block
Valid
Output
NTT Modulo Ops/Output
Add Mul Add Mul Mul by const Shift
3x3 9 9
8x8 6x6 21.34 1.78 1 7.12
16x16 14x14 20.89 1.31 1 7.84
32x32 30x30 22.76 1.14 1 9.10
5x5 25 25
8x8 4x4 48 4 1 16
16x16 12x12 28.44 1.78 1 10.66
32x32 28x28 26.12 1.31 1 10.45
7x7 49 49
16x16 10x10 40.96 2.56 1 15.36
32x32 26x26 30.29 1.51 1 12.12
9x9 81 81
16x16 8x8 64 4 1 24
32x32 24x24 35.56 1.78 1 14.22
11x11 121 121 32x32 22x22 42.31 2.12 1 16.93

9. 9
Theoretical costs comparison
Kernel
Naïve Int
Ops/Output Block
Valid
Output
NTT Modulo Ops/Output
Add Mul Add Mul Mul by const Shift
3x3 9 9
8x8 6x6 21.34 1.78 1 7.12
16x16 14x14 20.89 1.31 1 7.84
32x32 30x30 22.76 1.14 1 9.10
5x5 25 25
8x8 4x4 48 4 1 16
16x16 12x12 28.44 1.78 1 10.66
32x32 28x28 26.12 1.31 1 10.45
7x7 49 49
16x16 10x10 40.96 2.56 1 15.36
32x32 26x26 30.29 1.51 1 12.12
9x9 81 81
16x16 8x8 64 4 1 24
32x32 24x24 35.56 1.78 1 14.22
11x11 121 121 32x32 22x22 42.31 2.12 1 16.93

10. 10
Benchmarks setting
• Qconv: a C99 compliant and portable NTT
implementation against a naïve convolution
implementation.
• Platform of choice: Raspberry Pi Zero.

11. 11
ResultsInput size 224x224x1

12. 12
Future works
• Implement SIMD subroutines for forward and inverse
transforms.
• Map element-wise products to finite field matrix
multiplication in FFLAS.
• FPGA hardware implementation.
• Optimize for 3x3 filters.

13. Question Time

14. 14
Fermat Number Transforms
FNTs are NTTs mod 𝒑 = 𝟐 𝟐 𝒕
+ 𝟏.
1. Support FFT algorithms.
2. For a length N up to 2 𝑡+1, the forward and inverse
transforms requires only modular adds and shifts.
3. Reduction mod a Fermat prime p:
logical AND, right unsigned shift, modular subtraction

15. 15
Methodology
FNT m 𝐹4 = 65537 with
blocks:
• 8x8
• 16x16
• 32x32
RNS of 𝐹3 and 𝐹4 to increase the maximum output bitwidth to:
𝑙𝑜𝑔2 𝐹3 ∙ 𝐹4 ≅ 24 bits
Overlap-and-save algorithm
FNT mod 𝐹3 = 257 with blocks:
• 8x8
• 16x16

16. 16
Optimizations
• Forward DIF FFT, inverse DIT transforms FFT.
• Precomputed power-of-two twiddle factors .
• Switch the order of the two inner-most loop of the FFT.
• Avoid useless transpositions.
• Normalize the non discarded output only.