Presentation slide used at the Int'l Conference on Fied Programmable Technology 2017.

1. An Object Detector based on Multiscale
Sliding Window Search using a Fully Pipelined
Binarized CNN on an FPGA
Hiroki Nakahara, Haruyoshi Yonekawa, Shimpei Sato
Tokyo Institute of Technology, Japan
FPT2017
@Melbourne

2. Outline
• Background
• Object detector algorithm
• Fully pipelined Binarized CNN
• Experimental results
• Conclusion
2

3. Introduction
3

4. Convolutional Neural Network (CNN)
• Convolutional + fully connected + pooling layers
• State‐of‐the‐art performance in an image
recognition task
• Widely applicable
4Source: https://www.mathworks.com/discovery/convolutional‐neural‐network.html

5. Image Recognition Tasks
• Classification
• answer “category” of
the object in an image
• Object Detection
• classification + localization
• Semantic Segmentation
• Object area in pixel level
5
Easy
Hard
Children

6. Requirements in Embedded System
6
Cloud Embedded
Many classes (1000s) Few classes (<10)
Large workloads Frame rates (15‐30 FPS)
High efficiency
(Performance/W)
Low cost & low power
(1W‐5W)
Server form factor Custom form factor
J. Freeman (Intel), “FPGA Acceleration in the era of high level design”, 2017

7. Outline
• Background
• Object detector algorithm
• Fully pipelined Binarized CNN
• Experimental results
• Conclusion
7

8. Object Detection Problem
• Detecting and classifying multiple objects at the same time
• Evaluation criteria (from Pascal VOC):
8
Ground truth
annotation
Detection results:
>50% overlap of
bounding box with
ground truth
One BBox for each
object
Confidence value
for each object
Person (50%)
# .
# .
# .
#
1
11 , ∈ ,. ,…,
Average Precision (AP):

9. Proposed Object Detector
• Sliding window + Multi‐scaling + Fully pipelined BCNNs
9
...
Multi‐scale images
Wrapped
Images
by Sliding
Window
Classification
by a Fully pipelined
Binarized CNN
by Non‐maximum
Suppression

10. Sliding Window
• It is rectangular region of fixed width and height
that “slides” across an image
10

11. Multi‐Scaling (Pyramid Pooling)
• Find objects in images at different scales
• Combined with a sliding window, it can find objects
in various locations with the same window size
11

12. Non‐Maximum Suppression
• Given all scored bounding boxes in an image
• Rejects a bounding box which overlaps with a higher
scoring one considering a threshold
12

13. Quantification of Iterations
13
q
q q
q qq...
1st image
2nd image
i-th image
∆
2
• Trade‐off: Time (Iters), AP, and HW
p: Image size (given)
q: Window size
Δx: Stride
→ Find good q and Δx

14. Outline
• Background
• Object detector algorithm
• Fully pipelined Binarized CNN
• Experimental results
• Conclusion
14

15. Binarized CNN
15
x1
w0 (Bias)
fsgn(Y)
Y
z
w1
x2
w2
xn
wn
...
x1 x2 Y
‐1 ‐1 1
‐1 +1 ‐1
+1 ‐1 ‐1
+1 +1 1
x1 x2 Y
0 0 1
0 1 0
1 0 0
1 1 1

16. Optimization Techniques
• Binary CNN
• Multiple fully pipelined architecture
16
Batch normalization free
(BNF) [RAW17]
Internal FC layer replacement into a binary average
pooling [FPL17]
[FPL17] H. Nakahara, T. Fujii, S. Sato, ‘’A fully connected layer elimination for a binarized convolutional neural network
on an FPGA,’’ FPL 2017, pp. 1‐4.
[RAW17] H. Yonekawa and H. Nakahara, ‘’On‐chip memory based binarized convolutional deep neural network
applying batch normalization free technique on an FPGA,’’ IPDPS Workshops 2017, pp. 98‐105.

17. Dataflow for a 2D Convolutional Operation
17
...
...
...
...
mfeature maps
Input maps
...
...
Adder
Binarized
Weights
Sign
Shift Register
Output
maps
Integer
Bias

18. Dataflow for a 2D Convolutional Operation
18
...
...
...
...
mfeature maps
Input maps
...
...
Adder
Binarized
Weights
Sign
Shift Register
Output
maps
Integer
Bias

19. Dataflow for a 2D Convolutional Operation
19
...
...
...
...
mfeature maps
Input maps
...
...
Adder
Binarized
Weights
Sign
Shift Register
Output
maps
Integer
Bias

20. Pipelined Conv2D Circuit
x00 x01 x02 x03 x04
x10 x11 x12 x13 x14
x20 x21 x22 x23 x24
x30 x31 x32 x33 x34
x40 x41 x42 x43 x44
x22 x21 x20 x14 x13 x12 x11 x10 x04 x03 x02 x01 x00
+
Binarized
Weight
Mem.
Integer
Bias
Mem.
Write
Ctrl.
Logic
Counter
Binarized Feature Maps
(L=5, K=3)
9
Binarized MACs
(EXNORs + Adder Tree)
Sign
bit
Shift Register (2L+K bits)
Read M F.Maps at a time

21. Used CNN Model
21
Integer Conv2D
Binary Conv2D
Max Pooling
Binary Conv2D
Binary Conv2D
Binary Conv2D
Max Pooling
Binary Conv2D
Binary Conv2D
Binary Conv2D
Max Pooling
Fully Connect
Fully Connect
Fully Connect
Integer Conv2D
Binary Conv2D
Max Pooling
Binary Conv2D
Binary Conv2D
Binary Conv2D
Max Pooling
Binary Conv2D
Binary Conv2D
Binary Conv2D
Average Pooling
Fully Connect
VGG11
Our VGG
• Based on the VGG11 model
• 3x3 kernel convolution
• Replacement bottleneck
(memory intensive) layers
into an average pooling one

22. Overall Architecture
• Weight sharing
22
Pipelined
BCNN 1
FIFO
Pipelined
BCNN 2
FIFO
Pipelined
BCNN P
FIFO
AXI4 Bus
...
GPIO ARM Processor DDR Mem
Camera
Input Image
...
Weight
Mem

23. Outline
• Background
• Object detector algorithm
• Fully pipelined Binarized CNN
• Experimental results
• Conclusion
23

24. Implementation Setup
• Board: Xilinx Inc. Zynq UltraScale+
MPSoC zcu102 evaluation board
• Zynq UltraScale+ MPSoC FPGA (ZU9EG,
68,250 slices, 269,200 FFs, 1,824 BRAMs,
2,520 DSP48Es)
• FPGA design tool: Vivado HLS 2017.2
and Vivado 2017.2
• Timing constraint: 200MHz
• Deep learning framework:
Chainer 1.24.0
• Dataset: KITTI car detection
(moderate) scenario
24

25. Variation of Fully Pipelined CNNs
25
CNN Parameter Hardware Resource Accuracy Speed
Window
Size q
Stride
∆X
#18Kb
BRAMs
#FFs #LUTs #DSPs mAP FPS
96x96 24 240 194,930 114,870 0 74.36 11.10
48 71.29 45.30
64x64 16 232 189,820 169,500 0 84.80 8.70
32 82.20 34.95
48x48 12 232 171,850 172,100 0 70.50 7.80
24 64.20 31.65
32x32 8 232 169,930 178,220 0 56.32 8.55
16 52.30 34.20

26. Comparison with GPU based Detectors
26
0
10
20
30
40
50
60
70
80
90
100
0.01 0.1 1 10 100
mAP(%)
Detection Speed (FPS)
Ours
29.97
YOLOv2
MV3D(LIDAR)
SPD+RPN
Deep MANTA
RRC
FPS Acc (%)
RRC 0.27 90.22
Deep
MANTA
1.42 90.03
SDP+RPN 2.50 89.90
MV3D
(LIDAR)
4.16 79.76
YOLOv2 50.00 28.37
Proposed 34.95 82.20
YOLOv2 (GPU): 250.0 W
Proposed(FPGA): 2.5 W

27. Conclusion
• Applied a pipelined binary CNN to an object detector
• Multiple pipeline architecture
• Weight sharing
• Find good parameters for the KITTI car detection
• Better performance and accuracy than GPUs
• Future works
• Preprocessing to reduce HW and Time
• Selective search, BING, etc..
• Post‐processing to adjust bounding boxes
• SVR, another CNN, etc..
27

28. https://github.com/HirokiNakahara/GUINNESS
28