Presentation given by Nils Kucza (Bielefeld University) at the LEGaTO Final Event: Low-Energy Heterogeneous Computing Workshop on 4 September 2020 This event was collocated with FPL 2020

1. The LEGaTO project has received funding from the European Union's Horizon 2020 research and
innovation programme under the grant agreement No 780681
03.09.20
Smart Home
AI at the edge
FPL 2020
Nils Kucza, M.Sc.
Bielefeld University

2. FPL 2020
Smart Home Environments
03.09.20 2
CITEC
(Bielefeld University)
(Hospital and care facility)
(Manufacturer of domestic appliances)

3. FPL 2020
Typical smart home use cases
03.09.20
Smart Kitchen
• Gesture, speech recognition
Intelligent Chair and Fitness coach
• Gesture, face and speech recognition
Intelligent Door
• Face, Obstacle recognition
Smart Mirror
• Gesture, face, object
and speech recognition
Dialog assistance
• Speech recognition

4. FPL 2020
Main Goal of LEGaTO
Increase Energy Effiency
• Use LEGaTO Toolchain and hardware
to increase energy effiency
in Smart Home environment by x10
• Smart Mirror as Demonstrator
− Displays personalized information
− Local mashine learning frameworks
• Image: Object, face and gestures recognition
• Speech with DeepSpeech
− All detections simultaniously
• Start: 16FPS at 600W
• Goal: 10 FPS at 50W
03.09.20

5. FPL 2020
Smart Mirror - First Prototypes
03.09.20
DeepSpeech
MagicMirror²
Audio Stream
Identities
Objects &
Gestures
DisplaySpeakerMessages from/
to other devices
FaceRecognition
Camera Image
(< 1m depth range)
Camera Image
(full image range)
Object Detection
FaceNet
(face representation)
Classifier
YOLO
Gesture Detection
Smart Home
Shared Memory Object
DNN (GPU computed)
Tracker
(Kalman Filter)
WiderFace
(face detection)
Tracker
(Kalman Filter)
Person Tracking
Transcript
Peripheral Input
Camera Broadcast
Microphone Input
Combinatorial Logic
Fusion of Detections
Speech Recognition
Decision
Maker
• First Prototype
• Intel i7-7700K
• 2x GTX 1080 Ti
• 16 FPS / 600 Watt
• Second Prototype
• Intel i9-9900k
• 2x RTX 2070
• 27 FPS / 450 Watt

6. FPL 2020
Darknet/YoloV3 on different GPUs
03.09.20
TX2 (max)
Xavier (10W mode)
Xavier (15W mode)
Xavier (30W mode)
Xavier (max)
GTX 1080Ti (camera)
GTX 1080Ti
RTX 2070 (camera)
RTX 2070
0
25
50
75
100
125
150
175
200
0,00 5,00 10,00 15,00 20,00 25,00 30,00 35,00 40,00 45,00 50,00 55,00 60,00 65,00
Watt
Frames per SecondCamera
maximum

7. FPL 2020
Motivation for embedded edge Server
03.09.20
DeepSpeech
MagicMirror²
Audio Stream
Identities
Objects &
Gestures
DisplaySpeakerMessages from/
to other devices
FaceRecognition
Camera Image
(< 1m depth range)
Camera Image
(full image range)
Object Detection
FaceNet
(face representation)
Classifier
YOLO
Gesture Detection
Smart Home
Shared Memory Object
DNN (GPU computed)
Tracker
(Kalman Filter)
WiderFace
(face detection)
Tracker
(Kalman Filter)
Person Tracking
Transcript
Peripheral Input
Camera Broadcast
Microphone Input
Combinatorial Logic
Fusion of Detections
Speech Recognition
Decision
Maker

8. FPL 2020
t.RECS
• Optimized platform for
local / edge applications
• Compact dimensions (1-3 RU)
• Based on COM-HPC Microservers
− 1x COM-HPC Server
− 2x COM-HPC Client
− Also support for COM Express,
Jetson Apalis and Xavier AGX
• High-speed, low-latency communication
between microservers
• Interfaces for
− Audio / Video
− Peripheral input (USB)
04.09.2020

9. FPL 2020
Smart Mirror - Dual Xavier Testbed
03.09.20
2x Jetson AGX Xavier dev board
• Interconnected via pice
• Max power consumption 100W
DeepSpeech
MagicMirror²
Audio Stream
Identities
Objects &
Gestures
DisplaySpeakerMessages from/
to other devices
FaceRecognition
Camera Image
(< 1m depth range)
Camera Image
(full image range)
Object Detection
FaceNet
(face representation)
Classifier
YOLO
Gesture Detection
Smart Home
Shared Memory Object
DNN (GPU computed)
Tracker
(Kalman Filter)
WiderFace
(face detection)
Tracker
(Kalman Filter)
Person Tracking
Transcript
Peripheral Input
Camera Broadcast
Microphone Input
Combinatorial Logic
Fusion of Detections
Speech Recognition
Decision
Maker
Calculatedon
secondXavier

10. FPL 2020
Performance of the
Smart Mirror Prototypes
03.09.20
0 5 10 15 20 25 30
Goal (10FPS)
Dual Nvidia Xavier
Nvidia Xavier
Second Optimizations
Introduction of Tensor Cores
First Optimizations
Start Point
Frame rate / FPS
0 100 200 300 400 500 600 700
Energy Consumption / Watt

11. FPL 2020
Performance of the
Smart Mirror Prototypes
03.09.20
0 0,05 0,1 0,15 0,2 0,25 0,3
Goal (10FPS)
Dual Nvidia Xavier
Nvidia Xavier
Second Optimizations
Introduction of Tensor Cores
First Optimizations
Start Point
FPS / Watt

12. Thanks for your attention!