目前几乎所有的物联网解决方案都是基于感测器(sensor)，然而现实生活中，有绝大部分的与使用者的互动是基于视觉与对话的。Microsoft 的认知服务与人工智能解决方案，对于物联网来说，就像是一个大脑，让物联网解决方案可以通过人工智能，达到更多的连接，从而创造价值。 这一个场次中，讲师将使用 Microsoft 的认知服务与人工智能解决方案与物联网装置结合，当感知器启动时，透过摄像头以及 OpenCV 技术撷取环境画面之后，使用Azure 上开发好的框架判断目前的场景状态，并且透过交谈式机器人等方式，与使用者进行互动。

2. 利用电脑视觉与人工智能
创造更多物联网价值
John Chang ( 张书源 )
资深软体开发协理
john.chang@microsoft.com
IOT 302
Microsoft Tech Summit 2017
微软技术暨生态大会

3. 物联网架构
Microsoft Tech Summit 2017
微软技术暨生态大会

4. 物联网解决方案现况
依赖传感器侦测环境状态
传感器误差影响判断结果
必须依赖大量传感器
不易处理复杂的情境
需要整合不同传感器判断
需要大量投资在非本业项目

5. 理想的物联网解决方案 :
主动了解环境状态并做出反应
视觉
触觉/嗅觉/味觉
听觉
说话/动作人脑

6. 摄像头
传感器
麦克风
说话/动作物联网装置
理想的物联网解决方案 :
主动了解环境状态并做出反应

7. Custom Cloud
Gateway
Custom Cloud
Gateway
Field
Gateway
Device
VPN/ExpressRoute
OPC, HTTP, CoAP,…
Custom Protocols
Client SDK
IoT Edge
AMQP
Management app
Device registry/Management
Monitoring
Send commands
Client SDK
Management app
Device registry/Management
Monitoring
Send commands
Client SDK
Cloud protocol
gateway sample
Cloud protocol
gateway sample
AMQP, MQTT, HTTPS
Applications
- Companion apps
- Dashboards
- Controllers
- …
CoAP, AllJoyn, OPC,…
Device
Device
CoAP, AllJoyn, OPC, …
Device
Device
Device
IoT
Hub
Field
Gateway

8. Custom Cloud
Gateway
Custom Cloud
Gateway
Field
Gateway
Device
VPN/ExpressRoute
OPC, HTTP, CoAP,…
Custom Protocols
Client SDK
IoT Edge
AMQP
Management app
Device registry/Management
Monitoring
Send commands
Client SDK
Management app
Device registry/Management
Monitoring
Send commands
Client SDK
Cloud protocol
gateway sample
Cloud protocol
gateway sample
AMQP, MQTT, HTTPS
Applications
- Companion apps
- Dashboards
- Controllers
- …
CoAP, AllJoyn, OPC,…
Device
Device
CoAP, AllJoyn, OPC, …
Device
Device
Device
IoT
Hub
Field
Gateway
Data + AI

9. 结合摄像头侦测环境状态
降低传感器误判
降低传感器数量
容易处理复杂的情境
深度学习让影像辨识技术更成熟
OpenCV 可在 IoT Edge 筛选影像
Azure 认知服务支援影像分析
可与 Azure 其他服务结合

10. Sensor
Sensor
Sensor
Sensor
Sensor Blob Queue
Azure FunctionOpenCV
Computer Vision
Rule Engine
AzureIoT
Device Bot
SMS
Text
Phone
Call

11. Sensor
Sensor
Sensor
Sensor
Sensor Blob Queue
Azure FunctionOpenCV
Computer Vision
Rule Engine
AzureIoT
Device Bot
SMS
Text
Phone
Call

12. Sensor
Sensor
Sensor
Sensor
Sensor Blob Queue
Azure FunctionOpenCV
Computer Vision
Rule Engine
AzureIoT
Device Bot
SMS
Text
Phone
Call

13. Sensor
Sensor
Sensor
Sensor
Sensor Blob Queue
OpenCV
AzureIoT

14. Sensor
Sensor
Sensor
Sensor
Sensor Blob Queue
Azure FunctionOpenCV
Computer Vision
AzureIoT
失火了!

15. Sensor
Sensor
Sensor
Sensor
Sensor Blob Queue
Azure FunctionOpenCV
Computer Vision
Rule Engine
AzureIoT
Device Bot
SMS
Text
Phone
Call
失火了!

16. Demo
智慧物联网范例

17. Blob Storage Azure Functions
短訊
(Twlio)Computer Vision
API
Azure Functions

20. "faceRectangle": {"width": 193,
"height": 193,
"left": 326,
"top": 204}
“scores": { "anger": 5.182241e-8,
"contempt": 0.0000242813,
"disgust": 5.621025e-7,
"fear": 0.00115027453,
"happiness": 1.06114619e-8,
"neutral": 0.003540177,
"sadness": 9.30888746e-7,
"surprise": 0.9952837}

21. "faceRectangle": {"width": 193, "height": 193, "left": 326, "top": 204}
…
"attributes": { "age": 42, "gender": "male",
"headPose": { "roll": "8.2", "yaw": "-37.8", "pitch": "0.0" }}

24. 几个步骤建立自己的物件模型
上传照片
训练
评估
主动学习

26. Blob Storage
Azure Functions
Custom
Vision API

28. Azure Data Science虚拟机器
• 有 Windows & Linux 兩種版本
• CNTK, TensorFlow, MXNet, Caffe, Caffe2,
DIGITS, H2O, Keras, Theano, and Torch are
built, installed, and configured so they are
ready to run immediately
• NVIDIA driver, CUDA, and cuDNN are also
included
• Combine with Edge
• OpenCV 3.3 supports Caffe, TensorFlow & Torch

30. Blob Storage Azure Functions
Azure 资料科学
虚拟机器

31. 謝謝
Email: john.chang@microsoft.com

32. 讲师的激情因您的鼓励而愈发澎湃，
立即提交反馈即有机会获得精美礼品。

33. John.chang@microsoft.com