DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning

DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning
DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning 〇Yuta Takahashi1, Naoki Shirakura1, Kenta Toyoshima1, Takuro Amako1, Ryota Isobe1, Jun Takamatsu1 and Keiichi Yasumoto1 1. Nara Institute of Science and Technology The Tenth International Conference on Mobile Computing and Ubiquitous Networking (ICMU2017) S7: Application, October 5, 2017
Background ❖Increasing home appliances 2 ❖Many IR remote controllers ▪ Need to learn how to use ▪ High management costs Increase user’s burden
Unification of remote controllers 3 Network Multiple IR controller Increase the number of appliances The interface becomes complicated Method for selecting home appliance is important! ✔ Management cost
Home appliance selection 4 ❖With special attachments ❖Without special attachments ✔ Accurate selection Cost of devices/markers Selectable distance Voice [Pan 2010] Vision [Kong 2016] Intuitiveness ✔ Intuitiveness IR control Burden for wearing • IR LED [Neßelrath 2011] • IR transmitter [Tsukada 2004] • QR [Ullah 2012] Dedicated device is needed
DeepRemote (Proposed) 5 ❖Home appliance selection Object recognition by deep learning → Intuitive and robust ❖Two units ▪ Control unit ▪ Deep learning unit ❖Network Home network (ROS)
Control unit 6 ✓ Hand-held-type remote controller ❖Main processor ▪ Raspberry Pi3 (with Wi-Fi module) ❖Capturing a home appliance ▪ Front camera ❖Control interfaces ▪ Four buttons ▪ Gestures (right & left rotation)
Deep learning unit 7 ✓ Image recognition ❖Main processor ▪ Laptop PC with Core i5 (Ubuntu) ▪ Distribute the calculation load of control unit ❖Recognition Model ▪ VGG16 [Simonyan 2014] TV Recognition result Deep learning unit Control unit
Technique of training ❖Requirements of deep learning with zero-base 8 Huge dataset A lot of time ❖Fine tuning Trained model Arranging model Re-training
Experiment 9
Experimental environment 10 ✓Living room in smart-home facility ❖Five home appliances ▪ Fan (IR) ▪ Air conditioner (IR) ▪ TV (Network) ▪ Audio player (IR) ▪ Air purifier (Network) ❖Training data ▪ Took 20 images each appliances at P1, P2 and P3 ▪ 20 × 5 (appliances) × 3 (positions) = 300 images
Example of training data 11 P1 P2 P3 Air purifier Air conditioner Audio player Fan TV
Training ❖Model ▪ VGG16 trained ImageNet ❖Dataset ▪ Three hundred images ▪ Five classes 12 ❖Optimizer ▪ SDG ❖Loss function ▪ Categorical cross entropy
Evaluations 1. Classification accuracy ▪ Captured 50 images of each appliance in each position ▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images 13 2. Response time ▪ Measured time since pushing button until return the result ▪ Evaluated at the same time as 1. 3. User test ▪ Verified control time of home appliances
Appliance Precision [%] Recall [%] F-measure [%] P1 Air purifier 100.00 68.00 80.95 Audio player 83.33 90.00 86.54 TV 96.15 100.00 98.04 Air conditioner 57.47 100.00 72.99 Fan 86.96 40.00 54.80 Average 84.78 79.60 78.66 P2 Air purifier 81.63 80.00 80.81 Audio player 100.00 98.00 98.99 TV 92.59 100.00 96.15 Air conditioner 79.25 84.00 81.55 Fan 100.00 90.00 94.74 Average 90.69 90.40 90.45 P3 Air purifier 76.36 84.00 80.00 Audio player 100.00 58.00 73.42 TV 71.43 80.00 75.47 Air conditioner 53.17 84.00 65.12 Fan 100.00 62.00 76.54 Average 80.19 73.60 74.11 1. Classification accuracy of home appliances 14 P1:Effect of black door? P3: Too near?
2. Response time 16 (n=750) Maximum time: 3.07 [sec] Minimum time: 1.72 [sec] Stable recognition about two seconds
3. User test to assess control time (1/3) Experimental conditions Five participants Position: P1 Targets: Fan, Air conditioner, TV and Audio player 18 ❖Control time ❖Comparison Holding Power on DeepRemote Original vs
3. User test to assess control time (2/3) 19 F:86% F:98% F:77% F:55% High → ✔ about 5 seconds Lower than the 85% → over 10 seconds Accuracy Accuracy requires 85% over
3. User test to assess control time (3/3) 20 DeepRemote: Sum of each test > All Original: Sum of each test < All Changing time DeepRemote < Original Sum Sum 31.533.7 11.7 7.7 (- 2.1) (+ 3.0)
Conclusions ❖DeepRemote ▪ Smart device for intuitively control the home appliances ▪ Deep learning method for home appliance selection ❖Results of evaluation ▪ 81.07% classification accuracy on average ▪ Average of response time is 1.97 seconds ▪ Time of power on an appliance takes 5 seconds ▪ 85% accuracy is required ▪ Time of changing target is lower than original remote controller 21
Appendix 22
Future works ❖Improving recognition accuracy ▪ Cropping object ❖Investigating energy consumption ❖Accuracy of distinguishing similar appliances ❖Experiment in long-term usage ▪ User teaches correct label when the system recognizes wrong 23
Detail conditions of user test ❖Compare DeepRemote and original remote controller ▪ measure control time ▪ Remove “air conditioner” (no original controller) ▪ Five participants (males in the 20s) ▪ Perform three times in each measurement (one participant performs 30 times) ▪ Participant’s position: P1 24 1) Measure one appliance control time (start) → (power on an appliance) 2) Measure all appliances control time (start) → (power on the fan) →・・・ → (power on the audio player)
Confusion matrix of P1 25 Classified Air purifier Air conditioner Audio player Fan TV True Air purifier 34 6 9 0 1 Air conditioner 0 50 0 0 0 Audio player 0 1 45 3 1 Fan 0 30 0 20 0 TV 0 0 0 0 50

Check these out next

DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning

Recommended

More Related Content

Slideshows for you(19)

Viewers also liked(6)

Similar to DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning(20)

More from Yuta Takahashi(16)

Recently uploaded(20)

DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning