Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning

13,610 views

Published on

The Tenth International Conference on Mobile Computing and Ubiquitous Networking (ICMU2017)
S7: Application, October 5, 2017

Published in: Technology
  • Be the first to comment

  • Be the first to like this

DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning

  1. 1. DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning 〇Yuta Takahashi1, Naoki Shirakura1, Kenta Toyoshima1, Takuro Amako1, Ryota Isobe1, Jun Takamatsu1 and Keiichi Yasumoto1 1. Nara Institute of Science and Technology The Tenth International Conference on Mobile Computing and Ubiquitous Networking (ICMU2017) S7: Application, October 5, 2017
  2. 2. Background ❖Increasing home appliances 2 ❖Many IR remote controllers ▪ Need to learn how to use ▪ High management costs Increase user’s burden
  3. 3. Unification of remote controllers 3 Network Multiple IR controller Increase the number of appliances The interface becomes complicated Method for selecting home appliance is important! ✔ Management cost
  4. 4. Home appliance selection 4 ❖With special attachments ❖Without special attachments ✔ Accurate selection Cost of devices/markers Selectable distance Voice [Pan 2010] Vision [Kong 2016] Intuitiveness ✔ Intuitiveness IR control Burden for wearing • IR LED [Neßelrath 2011] • IR transmitter [Tsukada 2004] • QR [Ullah 2012] Dedicated device is needed
  5. 5. DeepRemote (Proposed) 5 ❖Home appliance selection Object recognition by deep learning → Intuitive and robust ❖Two units ▪ Control unit ▪ Deep learning unit ❖Network Home network (ROS)
  6. 6. Control unit 6 ✓ Hand-held-type remote controller ❖Main processor ▪ Raspberry Pi3 (with Wi-Fi module) ❖Capturing a home appliance ▪ Front camera ❖Control interfaces ▪ Four buttons ▪ Gestures (right & left rotation)
  7. 7. Deep learning unit 7 ✓ Image recognition ❖Main processor ▪ Laptop PC with Core i5 (Ubuntu) ▪ Distribute the calculation load of control unit ❖Recognition Model ▪ VGG16 [Simonyan 2014] TV Recognition result Deep learning unit Control unit
  8. 8. Technique of training ❖Requirements of deep learning with zero-base 8 Huge dataset A lot of time ❖Fine tuning Trained model Arranging model Re-training
  9. 9. Experiment 9
  10. 10. Experimental environment 10 ✓Living room in smart-home facility ❖Five home appliances ▪ Fan (IR) ▪ Air conditioner (IR) ▪ TV (Network) ▪ Audio player (IR) ▪ Air purifier (Network) ❖Training data ▪ Took 20 images each appliances at P1, P2 and P3 ▪ 20 × 5 (appliances) × 3 (positions) = 300 images
  11. 11. Example of training data 11 P1 P2 P3 Air purifier Air conditioner Audio player Fan TV
  12. 12. Training ❖Model ▪ VGG16 trained ImageNet ❖Dataset ▪ Three hundred images ▪ Five classes 12 ❖Optimizer ▪ SDG ❖Loss function ▪ Categorical cross entropy
  13. 13. Evaluations 1. Classification accuracy ▪ Captured 50 images of each appliance in each position ▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images 13 2. Response time ▪ Measured time since pushing button until return the result ▪ Evaluated at the same time as 1. 3. User test ▪ Verified control time of home appliances
  14. 14. Appliance Precision [%] Recall [%] F-measure [%] P1 Air purifier 100.00 68.00 80.95 Audio player 83.33 90.00 86.54 TV 96.15 100.00 98.04 Air conditioner 57.47 100.00 72.99 Fan 86.96 40.00 54.80 Average 84.78 79.60 78.66 P2 Air purifier 81.63 80.00 80.81 Audio player 100.00 98.00 98.99 TV 92.59 100.00 96.15 Air conditioner 79.25 84.00 81.55 Fan 100.00 90.00 94.74 Average 90.69 90.40 90.45 P3 Air purifier 76.36 84.00 80.00 Audio player 100.00 58.00 73.42 TV 71.43 80.00 75.47 Air conditioner 53.17 84.00 65.12 Fan 100.00 62.00 76.54 Average 80.19 73.60 74.11 1. Classification accuracy of home appliances 14 P1:Effect of black door? P3: Too near?
  15. 15. Evaluations 1. Classification accuracy ▪ Captured 50 images of each appliance in each position ▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images 15 2. Response time ▪ Measured time since pushing button until return the result ▪ Evaluated at the same time as 1. 3. User test ▪ Verified control time of home appliances
  16. 16. 2. Response time 16 (n=750) Maximum time: 3.07 [sec] Minimum time: 1.72 [sec] Stable recognition about two seconds
  17. 17. Evaluations 1. Classification accuracy ▪ Captured 50 images of each appliance in each position ▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images 17 2. Response time ▪ Measured time since pushing button until return the result ▪ Evaluated at the same time as 1. 3. User test ▪ Verified control time of home appliances
  18. 18. 3. User test to assess control time (1/3) Experimental conditions Five participants Position: P1 Targets: Fan, Air conditioner, TV and Audio player 18 ❖Control time ❖Comparison Holding Power on DeepRemote Original vs
  19. 19. 3. User test to assess control time (2/3) 19 F:86% F:98% F:77% F:55% High → ✔ about 5 seconds Lower than the 85% → over 10 seconds Accuracy Accuracy requires 85% over
  20. 20. 3. User test to assess control time (3/3) 20 DeepRemote: Sum of each test > All Original: Sum of each test < All Changing time DeepRemote < Original Sum Sum 31.533.7 11.7 7.7 (- 2.1) (+ 3.0)
  21. 21. Conclusions ❖DeepRemote ▪ Smart device for intuitively control the home appliances ▪ Deep learning method for home appliance selection ❖Results of evaluation ▪ 81.07% classification accuracy on average ▪ Average of response time is 1.97 seconds ▪ Time of power on an appliance takes 5 seconds ▪ 85% accuracy is required ▪ Time of changing target is lower than original remote controller 21
  22. 22. Appendix 22
  23. 23. Future works ❖Improving recognition accuracy ▪ Cropping object ❖Investigating energy consumption ❖Accuracy of distinguishing similar appliances ❖Experiment in long-term usage ▪ User teaches correct label when the system recognizes wrong 23
  24. 24. Detail conditions of user test ❖Compare DeepRemote and original remote controller ▪ measure control time ▪ Remove “air conditioner” (no original controller) ▪ Five participants (males in the 20s) ▪ Perform three times in each measurement (one participant performs 30 times) ▪ Participant’s position: P1 24 1) Measure one appliance control time (start) → (power on an appliance) 2) Measure all appliances control time (start) → (power on the fan) →・ ・ ・ → (power on the audio player)
  25. 25. Confusion matrix of P1 25 Classified Air purifier Air conditioner Audio player Fan TV True Air purifier 34 6 9 0 1 Air conditioner 0 50 0 0 0 Audio player 0 1 45 3 1 Fan 0 30 0 20 0 TV 0 0 0 0 50
  26. 26. Confusion matrix of P2 26 Classified Air purifier Air conditioner Audio player Fan TV True Air purifier 40 6 0 0 4 Air conditioner 8 42 0 0 0 Audio player 1 0 49 0 0 Fan 0 5 0 45 0 TV 0 0 0 0 50
  27. 27. Confusion matrix of P3 27 Classified Air purifier Air conditioner Audio player Fan TV True Air purifier 42 5 0 0 3 Air conditioner 7 42 0 0 1 Audio player 3 6 29 0 12 Fan 0 19 0 31 0 TV 3 7 0 0 40

×