Published on Feb. 7, 2018 This slide was presented at Augmented Human 2018. http://www.sigah.org/AH2018/ Telewheelchair: the Remote Controllable Electric Wheelchair System combined Human and Machine Intelligence https://dl.acm.org/citation.cfm?id=3174914 【Project page】 http://digitalnature.slis.tsukuba.ac.jp/2017/03/telewheelchair/ 【Project movie】 https://www.youtube.com/watch?v=e9bcp0elNFs 【Presenter】 Satoshi Hashizume (橋爪智) University of Tsukuba, Digital Nature Group (Yoichi Ochiai) 【Abstract】 Wheelchairs are essential means of transport for the elderly people and the physically challenged. However, wheelchairs need to be accompanied by caregivers. As society ages and the number of care recipients increases, the burden on caregivers is expected to increase. In order to reduce the burden on caregivers, we present Telewheelchair, an electric wheelchair equipped with a remote control function and computational operation assistance function. The caregiver can remotely control the Telewheelchair by means of a head mounted display (HMD). In addition, the proposed system is equipped with a human detection system to stop the wheelchair automatically and avoid collisions. We conducted a user study on the wheelchair in four types of systems and investigated the time taken to achieve tasks. Telewheelchair will enhance geriatric mobility and improve society by combining human intelligence and machine intelligence.

1. Telewheelchair
the Remote Controllable Electric Wheelchair
System combined Human and Machine Intelligence
1
Satoshi Hashizume, Ippei Suzuki, KazukiTakazawa
Ryuichiro Sasaki,Yoichi Ochiai
University ofTsukuba, AISIN Seiki Co.,Ltd.

2. Introduction
Wheelchairs are essential means of transport.
It is always necessary for carers to be located
near the wheelchair.
2

3. Introduction
3
Problem
Aging society
Lack of caregivers
Strain of caregiver is increasing.

4. Introduction
Strain of caregiver is increasing.
We tried to reduce the strain on caregivers
by developing a new electric wheelchair system.
4

5. Introduction
Method to reduce the strain on caregivers.
Remote control
Manipulation aid with automatic operation
5

6. Related Work: automatic operation
Applications of virtual reality technology to
wheelchair remote steering systems
Remote control electric wheelchair using VR.
Complex configuration.
6
RT Gundersen, Stephen J Smith, and Ben AAbbott. 1996. Applications of virtual reality technology to wheelchair remote
steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.

7. Related Work: automatic operation
Robotic Wheelchair Easy to Move and
Communicate with Companions
Using a laser range sensor to move with the
companion.
7
Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki.
2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors
in Computing Systems. ACM, 3079–3082.

8. Related Work : manipulation method
Electrooculography (EOG)
Electromyograph (EMG)
Voice
Hand gesture
8

9. Related Work: manipulation method
EOG guidance of a wheelchair using neural networks
Identify Electrooculography using a neural network to operate a
wheelchair.
9
Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair
using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.

10. Related Work: manipulation method
Electronic control of a wheelchair guided by voice
commands
Using the voice commands to operate the wheelchair.
10
PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995),
665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE,
2007 annual conference. IEEE, 336–340.

11. Implementation
11

12. Implementation : electric wheelchair
Based on TAO LIGHT II-m of AISIN SEIKI CO,.
LTD.
22 inch, Max speed 6 km/h
width 70cm, length 100cm, height 135cm
12

13. Implementation : electric wheelchair
We recorded the image of the wheelchair
viewpoint with the omnidirectional
camera.
A microcomputer is connected to the
controller.
13

14. Implementation : electric wheelchair
14

15. Implementation : base station
15
Driver watched a image of a wheelchair viewpoint using HMD.
Display an arrow corresponding to the direction of the
wheelchair movement in VR.

16. Implementation : base station
16
Using HDMI splitter to share the image of the omnidirectional camera.
HDMI Extender
HDMI Splitter
(b)Linux
HDMI HDMI
HDMI
UDP
(c) Linux
Environment Recognition
(LSD-SLAM; ROS)
Object Recognition
(YOLO; CUDA+OpenCV)
(a) Windows
VR Controll
(Unity)
Wheelchair Controll
(Processing)

17. Implementation : remote control
Wireless transfer of omnidirectional image and operation signal.
17
Video
CW-1 (IDX Company, Ltd.)
Full HD, Max 30m
less than 1ms latency
Operation signal
Xbee ZB S2C
(Digi International K.K.)
Max 60m

18. Implementation : operation assistance
Automatic stop by object identification.
Using YOLO which is the real-time
object detection system.
18
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016
IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI:
http://dx.doi.org/10.1109/CVPR.2016.91

19. Implementation : operation assistance
Environmental map creation by SLAM.
19
Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer
International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54

20. Evaluation
Purpose｜ To investigate the operability of the remote operation.
Design ｜We set 6 tasks and 4 operation methods.
We measured the operation time for each task.
Recording the position of the wheelchair using motion capture.
Condition｜Weight of 52.5kg that same as average body weight of 60 year
old women is loaded on wheelchair.
20

21. Evaluation
Four operation methods.
21

22. Evaluation: Four operation methods
Normal mode
Operating Telewheelchair
with the handle.
22

23. Evaluation: Four operation methods
Standby mode
Operating with the controller
by standing next to wheelchair.
23

24. Evaluation: Four operation methods
Display mode
Participants watched the
display image of only the
front part of the
omnidirectional image.
Do not look the wheelchair
directly.
24

25. Evaluation: Four operation methods
HMD mode
Wear HMD and operate
wheelchair while watching
the omnidirectional image.
25

26. Evaluation
Six tasks of course
Straight
Right turn
Left turn
Rotation
Door
Obstacle
26

27. Evaluation
Outline of course
Tasks set in the course in
a row.
Based on equipment
standards of nursing
care facilities stipulated
in Japan.
27
START
GOAL
0.8m
1.4m

28. 28

29. Result: operation time of tasks
29
Straight Right turn Door Left turn Rotation Obstacle

30. Result: operation path of wheelchair
30

31. Result
Operations using display mode are more difficult / discomfort than
normal mode.
31
Difficulty of the operation
Feeling of discomfort between
movement and operation.

32. Discussion
Users operated to turn squarely because they could not look
around wheelchair in display mode.
→ Display mode takes a longer time.
32
Rotation Obstacle

33. Discussion
Stand by and HMD modes have similar difficulty of control.
We can choose the operation mode depends on the use case.
Case of remote control → use HMD mode.
Case of control near the wheelchair → use stand by mode.
33

34. Limitation
Latency
Specification of wheelchair and processing delay.
Transmission distance
Specification of wireless device.
Monitoring passenger
Passenger's fear
34

35. Contributions
1. We conducted experiments on the operability of the
wheelchair using an HMD.
2. For remote operation, an immersive operation method using
an HMD improved the stability more than an operation using a
large display.
3. In realizing the semi-automatic operation of an electric
wheelchair, we explored functions to be developed in future
work.
35

36. Research Member
36
Satoshi Hashizume1
Ryuichiro Sasaki2
1University of Tsukuba, 2AISIN Seiki Co., Ltd.
Ippei Suzuki1 Kazuki Takazawa1
Yoichi Ochiai1