For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/01/combining-cnns-and-conventional-algorithms-for-low-compute-vision-a-case-study-in-the-garage-a-presentation-from-the-chamberlain-group/ Nathan Kopp, Principal Software Architect for Video Systems at the Chamberlain Group, presents the “Combining CNNs and Conventional Algorithms for Low-Compute Vision: A Case Study in the Garage” tutorial at the September 2020 Embedded Vision Summit. Chamberlain Group (CGI) is a global leader in access control solutions with its Chamberlain and LiftMaster garage door opener brands and myQ connected technology. In this presentation, you’ll learn how CGI is innovating to bring efficient, affordable computer vision into the garage, opening new possibilities and insights for homeowners and businesses. With constant improvements in neural network architectures and advancements in low-power edge processors, it is tempting to assume that CNNs will solve every vision problem. However, simpler “conventional” computer vision techniques continue to offer an attractive cost-to-performance ratio and require orders of magnitude less training data. Unfortunately, these algorithms often need hand-tuning of parameters, and do not generalize well to previously unseen environments. By combining CNNs with simpler algorithms into a layered, intelligent vision pipeline—and by understanding the constraints of the problem—the weaknesses of simpler algorithms can be offset by the strengths of CNNs, while still preserving their cost-saving benefits. For more information about edge AI and computer vision, please visit: https://www.edge-ai-vision.com

1. Combining CNNs and Conventional
Algorithms for Low-Compute
Vision:
A Case Study in the Garage
Nathan Kopp
Chamberlain Group
September 2020

2. © 2020 Chamberlain Group
Intro
The Question:
Are CNNs always the best choice for every computer vision problem?
The Idea:
Sometimes simpler algorithms are the better choice.
Simpler algorithms can be guided by CNNs to overcome limitations.
Simplify the problem and leverage constraints to utilize simpler algorithms.
The Story:
Creating the Smart Garage (Chapter 1)
2

3. © 2020 Chamberlain Group
Company Background
3

4. © 2020 Chamberlain Group
Creating the Smart Garage
New flagship garage door opener with camera
How do we make this product smarter?
• Know when vehicles leave and arrive.
• Know whether a specific vehicle is in the garage.
4
Vehicle Presence
&
Vehicle Re-Identification

5. © 2020 Chamberlain Group
CNN Strengths and
Weaknesses

6. © 2020 Chamberlain Group
CNN Strengths and Weaknesses
Strengths
• Generalize to unseen circumstances
• Can be trained for various tasks
• State-of-the-art results
Weaknesses
• Requires large training dataset
• High compute requirements
• High memory requirements
6

7. © 2020 Chamberlain Group
CNN-Only Solutions: Vehicle Presence
• Accuracy issues
• Need custom dataset to
improve accuracy
7
Object Detection (YOLO, SSD) Object Tracking
• Analyze video = high cost

8. © 2020 Chamberlain Group
CNN-Only Solutions: Vehicle Identification
• Accuracy Issues
• Mix-up similar-looking cars
• Datasets trained for fully-visible
vehicles
8
Make / Model Classification License Plate Recognition
• Plate not visible when parked
• Analyze video = high cost
• Blur, lighting, compression
White
Tesla
Model 3
Blue
Subaru
Forrester
Pairwise Similarity
• No truly-free pre-trained
models
• No truly free datasets
• Need a large training
dataset

9. © 2020 Chamberlain Group
More Problems: Cost Constraints
• Existing cameras have relatively low compute:
• Single core low-cost ARM
• Low memory
• Minimal cloud expenses allowed
9

10. © 2020 Chamberlain Group
Simplifying the Problem

11. © 2020 Chamberlain Group
Simplifying the Question
11
Raw Video
Process
with CNN
Rich
Metadata
Interpret
Metadata
Goal
Rich metadata is fun, but
we still must SIMPLIFY to
solve the real problem.
Vehicle Presence
&
Vehicle Re-Identification
Blue
Subaru
ForresterWhite
Tesla
Model 3
Plate: [ABC123] Plate: [XYZ987]

12. © 2020 Chamberlain Group
Simplifying the Question
12
Simplify the
question
Raw Video
Process with
Simple
Algorithm
Simple
Metadata
Goal
Raw Video
Process
with CNN
Rich
Metadata
Interpret
Metadata
Goal
Simplifying EARLIER in the
process can reduce compute.
Try to simplify the question,
instead of simplifying the
answer.
Vehicle Presence
&
Vehicle Re-Identification

13. © 2020 Chamberlain Group
Find Useful Constraints
Stuck with “bad” constraints? Look for “useful” constraints!
• Fixed camera
• Limited number of vehicles
• Fixed vehicle location (region of interest)
• Personal habits of parking location
• Temporal consistency of the background (When did you last clean your garage?)
13
Less generalization is needed!

14. © 2020 Chamberlain Group
Our Hybrid Solution

15. © 2020 Chamberlain Group
Find an Appropriate Simple Algorithm
• HOG lowers the spatial resolution: built in flexibility
• Features can be translated, to some degree
• Add a custom “attention mask”
• Easy to compare features
• Implement ”training mode” to automatically gather images
representing known conditions
15
Tesla Model 3 Present
Tesla Model 3 Absent
Selected Algorithm: Modified Histogram of Oriented Gradients (HOG)

16. © 2020 Chamberlain Group
The “Fast Scene Matcher”
• Train Mode
1. Gather images of known conditions
2. Extract features
3. Build attention masks
4. Process features into library
• Run Mode
1. Gather image of unknown condition
2. Extract features
3. Compare to library
4. Find best match, or no match
16

17. © 2020 Chamberlain Group
Combined Meta-Algorithm
Train Mode
• Multiple CNNs (PoseNet, YOLO, etc.)
• Object tracking, ROIs, line crossing
• License plate recognition
• Leverage user input to fix mistakes
Output:
• Processed “known-condition” images
17
Run Mode
• HOG-based “Fast Scene Matcher”
Output:
(for each parking spot)
• Specific vehicle present
• OR empty
• OR indeterminate
Train Mode Run Mode Retrain Mode
“Ask
for
Help”

18. © 2020 Chamberlain Group
“Guide-Follower” Design Pattern
18
Guide
Follower
Informs
• Expensive
• Slow
• Generalized
• Infrequent
• Specialized
• Frequent
• Inexpensive
• Fast
Input Output
Ask for
Help

19. © 2020 Chamberlain Group
Results

20. © 2020 Chamberlain Group
Results – Single-Frame Run Mode
Metric YOLO-COCO
256x256 1
YOLO-VEH
256x256 1
YOLO-VEH
416x416 2
FSM-HOG-4 4 FSM-HOG-7 5
Detect Any Vehicle Present 41.5% 45.5% 57.8% 86.1% 93.0%
Detect Vehicle Absent 99.4% 98.7% 99.4% 99.7% 99.4%
Re-Identify Known Vehicle n/a n/a n/a 89.2% 95.9%
Recognize Unknown Vehicle n/a n/a n/a 63.6% 80.4%
Recognize Moved Camera Position n/a n/a n/a 37.5% 29.2%
Cloud cost per 1,000,000 frames $7.38 $7.38 $19.50 $0.39 $0.39
Edge compute on existing hardware n/a n/a n/a 2 FPS 2 FPS
20
1) 256x256 YOLOv2; trained on MS COCO + VOC datasets; threshold 0.3; cost based on AWS g3s.xlarge instance size
2) 256x256 YOLOv2; trained on custom vehicle dataset; threshold 0.35; cost based on AWS g3s.xlarge instance size
3) 416x416 YOLOv2; trained on custom vehicle dataset; threshold 0.35; cost based on AWS g3s.xlarge instance size
4) Unoptimized OpenCV C++ code; 4 example images per known condition; cost based on AWS m4.xlarge instance size
5) Same as (4), but with up to 7 example images per known-vehicle.

21. © 2020 Chamberlain Group
Hybrid Solution: Combined Meta-Algorithm
21
“Inference” on the Edge“Train” in the Cloud
Guide
Follower
on already-deployed devices!

22. © 2020 Chamberlain Group
Take-Aways
1. Simplify the problem first.
• Identify your core goal.
• Simplify the question, instead of simplifying the answer.
• Leverage useful constraints.
2. Sometimes simpler algorithms are the better choice.
• Simplifying the problem allow use of a less generalized algorithm.
• Simpler algorithms can be guided by CNNs to overcome limitations.
• It is possible to preserve most of the “magic” of deep learning, at a lower cost.
22

23. © 2020 Chamberlain Group
Q & A

24. © 2020 Chamberlain Group
Resources
Chamberlain
https://www.chamberlain.com/
YOLOv2
https://pjreddie.com/darknet/yolo/
Histogram of Oriented Gradients
https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients
2020 Embedded Vision Summit
“Combining CNNs and Conventional
Algorithms for Low-Compute Vision: A
Case Study in the Garage”
Nathan Kopp
1:00pm, September 17, 2020
24