從Bill Gates 1995提起IoT概念之後，爾後到了2010年代的技術成熟以致商品化實現，智能化Ecosystem的建構得以在全球熱浪潮之下蓬勃發展；但也隨著智能化商品落地遇到瓶頸，智能化應用大多只存在手機App或是商場應用，除了系統集成、軟體應用等問題之外，令人詬病的問題之一就是雲運算時常不夠及時。而此時邊緣運算隨著類神經網路的崛起而彌補了雲運算的不足。 而本專題即是以raspberry pi搭載camera的架構同時進行IoT以及邊緣運算的雙重建構。使用MQTT(or line bot)建立了居家Camera及時防盜通知的IoT功能，並同時導入了類神經網路當中的MobileNet、藉著其輕巧但快速準確的能力，使raspberry pi具備了即時影像辨識的能力，使得家居攝影機功能各具全方位性能、能夠更貼近當下使用者生活所需。 2019/01/18_AIoT智能物聯網平台開發人才養成班成果展 成果展網站↗️ http://bit.ly/2MhnTkx 最新就業養成班梯次↗️ http://bit.ly/2IGot9J 或洽02-23167736

1. .
.
.
SMART Home Camera

2. Agenda
Gesture
Recognition
Motion
Detection
Background
AI
IOT Scenario
How it work
Summary

3. 3
• 即時錄像擷取
• 遠端操控鏡頭
• 遠端廣播
• 遠端警告
• 辨識影像內容
• 依照影像辨識結果進行動作
• 使用者指令辨識
使用者無法隨時隨地觀看並做出即時對應Action
camera
「邊緣智慧」可以說是AI落實真實生活應用的最後一哩路
• 占用大量傳輸資源
• 延遲時間的限制
• 防盜偵測
IOT and AIAIIOT

4. HOWITWORKS .
.
.
f ( ) = ……
…… ………
= “Master is raising hand!’
Serv
erWhen
Master
Is
Not
At
home ≠
2nd image1st image

5. Let’s take a Look at
Gesture Recognition

6. ‘0’:
Raising hand
‘1’:
Cross hand
(in-front chest)
‘2’:
Hearing
(with 2 hands)
‘3’:
Sh~
‘4’:
Touching
belly
‘5’:
Other
‘6’:
No people
Light-on Water me
Robot Arm
No Action
Activate Respective Functions
Light-off Music-on Music-off
Gesture Recognition Gesture and Function

7. Gesture Recognition How it Learn – CNN model
Fully
connection
layers
Conv2D +Pooling2D
layers
Flatten
https://pixabay.com
Built an smart model.h5
IMAGE
COLLECTI
ON
FEATURE
EXTRAYIO
NC
CLASSIFIE
R
TRAINING
WEIGHTS
COLLECTI
ON
Backpropagation
Accuracy

8. Gesture RecognitionUse trained model to Predict
Using smart model.h5
Flatten
‘0’
‘3’
‘1’
‘2’
‘4’
‘5’
‘6’
https://pixabay.com
None

9. Gesture Recognition Training set#1 – 10K data
Model #1
Shuffle color filter
while taking photos
‘0’ ‘3’‘1’ ‘2’ ‘4’ ‘5’
Heat Map Verification
Blank
Background
Model #1
Strongly depends on specific face feature and specific background
Heat Map
0
20
40
60
80
100
120
120100806040200
0.2
0.4
0.6
0.8
1.0
0.0
A class activation map (CAM) lets us see which regions in
the image were relevant to this class.
Highly related
Zero related

10. Gesture Recognition Training set#2 – 55K data
Model #2
1. Different
face
2. Different
background
3. Different
clothes
4. Add set 6 (No
people)
+Set #1
‘0’ ‘3’‘1’ ‘2’ ‘4’ ‘5’ ‘6’
Heat Map Verification
Blank
Background
Model #1
*For different FACES: Accuracy ↑ *For different BACKGROUND: Accuracy −

11. Gesture Recognition Training set#3 – 100K data
Model #3
Change
background
manually
+
Set #1 Set #2
‘0’ ‘3’‘1’ ‘2’ ‘4’ ‘5’ ‘6’
Heat Map Verification
*For different FACES: Accuracy ↑ ↑ *For different BACKGROUND: Accuracy ↑
Blank
Background
Model #1

12. Gesture Recognition Heat Map – Summary
Fix Background
Model #1
Model #2
Model #3
NA

13. Gesture Recognition Heat Map – Summary
Saliency Maps
We can use these gradients to highlight input regions that cause the most change in the output.
Intuitively this should highlight salient image regions that most contribute towards the output.
∂output
∂input

14. How about the other?
Motion Detection

15. Motion Detection Motion Detect by OpenCV
Motion Detected ROI = 1
Frame (t)
Frame (t+1)
Blur (t)
Blur (t+1)
Diff () Gray () Thresh_binary Opening x2
Generate the
contour (external)
Closing x2
If contour area >
0.2x frame area
Send http post to the web server, save the picture
and record the frame and go back to while loop
Frame (t+2) Blur (t+2)
Back to WhileLoop
OpenCV (Open Source Computer Vision Library)
is an open-source BSD-licensed library that
includes several hundreds of computer vision
algorithms.
Process Flow
Time: t Time: t+1

16. Motion Detection The First Frame(t) & Blur
First Frame (t) First Frame (t) and Blurred
Process FlowFrame (t)
Frame (t+1)
Diff () Gray () Thresh_binary Opening x2 Closing x2 Generate contour
Blur (t)
Blur (t+1)

17. Motion Detection Frame (t+1) and Difference
Frame (t+1) and Blurred Frame Diff( (t+1)- (t))
Process FlowFrame (t)
Frame (t+1)
Diff () Gray () Thresh_binary Opening x2 Closing x2 Generate contour
Blur (t)
Blur (t+1)

18. Motion Detection Gray & Threshold Binary
Gray Threshold Binary
Process FlowFrame (t)
Frame (t+1)
Diff () Gray () Thresh_binary Opening x2 Closing x2 Generate contour
Blur (t)
Blur (t+1)

19. Motion Detection Opening and Closing
Opening Closing
Process FlowFrame (t)
Frame (t+1)
Diff () Gray () Thresh_binary Opening x2 Closing x2 Generate contour
Blur (t)
Blur (t+1)

20. Motion Detection Contours and Frame Save
Generate Contours Save Frame
Process FlowFrame (t)
Frame (t+1)
Diff () Gray () Thresh_binary Opening x2 Closing x2 Generate contour
Blur (t)
Blur (t+1)
contour area > 0.2x frame area
Send http post to the web server, save the picture and
record the frame and go back to while loop
If

21. It’s How They Communicate
HTTP/Node.js/MQTT
.
.
.
Serv
er

22. System Communication
Gesture Recognition
Motion Detection
HTTP
POST
HTTP
Response
Camera
Python
WebClient
MQTT
Open CV
-
Moving object detected!
ConvNet
0123456…
Gesture classified!
Line
Serve
r
Node.Js
User’s Device
WebServer
MQTT MQTT
HTTP POST
Panel
Line bot
Listening
MQTT
Relay
GPIOI2C
Audio player
.
.
.

23. AI FunctionIOT Function
Hardware
Serv
er
Summary

24. Who Are We?
Thank you!
Python
C/Linux
C#
Hardware
Deep learning Python
OpenCV
C
Linux embedded system
Node.js
Hardware
Deep learning
Python
C
Deep learning
Presentation
Image processing
Deep learning
Python
PPT
Visualization
Bin
Robby
Ethan
Jayden