DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning
DeepRemote: A Smart Remote Controller for
Intuitive Control through Home Appliances
Recognition by Deep Learning
〇Yuta Takahashi1, Naoki Shirakura1, Kenta Toyoshima1, Takuro
Amako1, Ryota Isobe1, Jun Takamatsu1 and Keiichi Yasumoto1
1. Nara Institute of Science and Technology
The Tenth International Conference on Mobile
Computing and Ubiquitous Networking (ICMU2017)
S7: Application, October 5, 2017
Unification of remote controllers
3
Network Multiple IR controller
Increase the number of
appliances
The interface becomes
complicated
Method for selecting home appliance is important!
✔ Management cost
Home appliance selection
4
❖With special attachments
❖Without special attachments
✔ Accurate selection
Cost of devices/markers
Selectable distance
Voice [Pan 2010] Vision [Kong 2016]
Intuitiveness ✔ Intuitiveness
IR control
Burden for wearing
• IR LED [Neßelrath 2011]
• IR transmitter [Tsukada 2004]
• QR [Ullah 2012]
Dedicated device is needed
DeepRemote (Proposed)
5
❖Home appliance selection
Object recognition by deep learning
→ Intuitive and robust
❖Two units
▪ Control unit
▪ Deep learning unit
❖Network
Home network (ROS)
Control unit
6
✓ Hand-held-type remote controller
❖Main processor
▪ Raspberry Pi3 (with Wi-Fi module)
❖Capturing a home appliance
▪ Front camera
❖Control interfaces
▪ Four buttons
▪ Gestures (right & left rotation)
Deep learning unit
7
✓ Image recognition
❖Main processor
▪ Laptop PC with Core i5 (Ubuntu)
▪ Distribute the calculation load of control unit
❖Recognition Model
▪ VGG16 [Simonyan 2014]
TV
Recognition
result
Deep
learning unit
Control unit
Technique of training
❖Requirements of deep learning with zero-base
8
Huge dataset A lot of time
❖Fine tuning
Trained model Arranging model
Re-training
Experimental environment
10
✓Living room in smart-home facility
❖Five home appliances
▪ Fan (IR)
▪ Air conditioner (IR)
▪ TV (Network)
▪ Audio player (IR)
▪ Air purifier (Network)
❖Training data
▪ Took 20 images each appliances at P1, P2 and P3
▪ 20 × 5 (appliances) × 3 (positions) = 300 images
Example of training data
11
P1
P2
P3
Air
purifier
Air
conditioner
Audio
player Fan TV
Training
❖Model
▪ VGG16 trained ImageNet
❖Dataset
▪ Three hundred images
▪ Five classes
12
❖Optimizer
▪ SDG
❖Loss function
▪ Categorical cross entropy
Evaluations
1. Classification accuracy
▪ Captured 50 images of each appliance in each position
▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images
13
2. Response time
▪ Measured time since pushing button until return the result
▪ Evaluated at the same time as 1.
3. User test
▪ Verified control time of home appliances
Appliance Precision [%] Recall [%] F-measure [%]
P1
Air purifier 100.00 68.00 80.95
Audio player 83.33 90.00 86.54
TV 96.15 100.00 98.04
Air conditioner 57.47 100.00 72.99
Fan 86.96 40.00 54.80
Average 84.78 79.60 78.66
P2
Air purifier 81.63 80.00 80.81
Audio player 100.00 98.00 98.99
TV 92.59 100.00 96.15
Air conditioner 79.25 84.00 81.55
Fan 100.00 90.00 94.74
Average 90.69 90.40 90.45
P3
Air purifier 76.36 84.00 80.00
Audio player 100.00 58.00 73.42
TV 71.43 80.00 75.47
Air conditioner 53.17 84.00 65.12
Fan 100.00 62.00 76.54
Average 80.19 73.60 74.11
1. Classification accuracy of home appliances
14
P1:Effect of
black door?
P3: Too near?
Evaluations
1. Classification accuracy
▪ Captured 50 images of each appliance in each position
▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images
15
2. Response time
▪ Measured time since pushing button until return the result
▪ Evaluated at the same time as 1.
3. User test
▪ Verified control time of home appliances
Evaluations
1. Classification accuracy
▪ Captured 50 images of each appliance in each position
▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images
17
2. Response time
▪ Measured time since pushing button until return the result
▪ Evaluated at the same time as 1.
3. User test
▪ Verified control time of home appliances
3. User test to assess control time (1/3)
Experimental conditions
Five participants
Position: P1
Targets: Fan, Air conditioner, TV and Audio player
18
❖Control time ❖Comparison
Holding Power on DeepRemote Original
vs
3. User test to assess control time (2/3)
19
F:86% F:98% F:77% F:55%
High → ✔ about 5 seconds
Lower than the 85% → over 10 seconds
Accuracy
Accuracy requires
85% over
3. User test to assess control time (3/3)
20
DeepRemote: Sum of each test > All
Original: Sum of each test < All
Changing time
DeepRemote < Original
Sum Sum
31.533.7
11.7
7.7
(- 2.1)
(+ 3.0)
Conclusions
❖DeepRemote
▪ Smart device for intuitively control the home appliances
▪ Deep learning method for home appliance selection
❖Results of evaluation
▪ 81.07% classification accuracy on average
▪ Average of response time is 1.97 seconds
▪ Time of power on an appliance takes 5 seconds
▪ 85% accuracy is required
▪ Time of changing target is lower than original remote
controller
21
Future works
❖Improving recognition accuracy
▪ Cropping object
❖Investigating energy consumption
❖Accuracy of distinguishing similar appliances
❖Experiment in long-term usage
▪ User teaches correct label when the system recognizes
wrong
23
Detail conditions of user test
❖Compare DeepRemote and original remote controller
▪ measure control time
▪ Remove “air conditioner” (no original controller)
▪ Five participants (males in the 20s)
▪ Perform three times in each measurement (one participant performs
30 times)
▪ Participant’s position: P1
24
1) Measure one appliance control time
(start) → (power on an appliance)
2) Measure all appliances control time
(start) → (power on the fan)
→・ ・ ・ → (power on the audio player)
Confusion matrix of P1
25
Classified
Air purifier
Air
conditioner
Audio
player
Fan TV
True
Air purifier 34 6 9 0 1
Air
conditioner 0 50 0 0 0
Audio player 0 1 45 3 1
Fan 0 30 0 20 0
TV 0 0 0 0 50
Confusion matrix of P2
26
Classified
Air purifier
Air
conditioner
Audio
player
Fan TV
True
Air purifier 40 6 0 0 4
Air
conditioner 8 42 0 0 0
Audio player 1 0 49 0 0
Fan 0 5 0 45 0
TV 0 0 0 0 50
Confusion matrix of P3
27
Classified
Air purifier
Air
conditioner
Audio
player
Fan TV
True
Air purifier 42 5 0 0 3
Air
conditioner 7 42 0 0 1
Audio player 3 6 29 0 12
Fan 0 19 0 31 0
TV 3 7 0 0 40