This document proposes using a Leap Motion Controller to intuitively operate a robotic arm to help elderly and disabled people perform tasks. It describes mapping a user's hand gestures detected by the Leap Motion Controller to control movement of the robotic arm in real time. This aims to simplify human-robot interaction by replacing keyboards or joysticks with natural hand motions. The system was implemented and tested for applications like tele-operation of robots and detection of hand tremors.

1. Intuitive and Adaptive Robotics Arm
Manipulation
using Leap Motion Controller (LMC)
D. Bassily, C. Georgoulas, J. Güttler, T. Linner, T. Bock
ISR/Robotik 2014; 41st International Symposium on Robotics, Munich, Germany
Quan Le
HCI Lab
04th
, Feb 2015

2. ▧ Introduction
▧ Proposed concepts
▧ Implementation
▧ Research projects
▧ Conclusion
Contents
Fig1. The 6-DOF Jaco robotic arm
2

3. - Populations around the world are rapidly ageing.
- Ageing society faces numerous challenges in performing
simple task in Activities of Daily Living (ADLs).
Introduction3
- Caring of elderly and handicapped people becomes more
and more important.

4. - Many technical aids have been developed, but provided
limited functionality.
Introduction
Fig3. Mobile arm support (MAS)Fig2. JAECO/Rancho Multilink MAS
Mobile arm supports: evidence
based benefits and criteria for use, Journal of Spinal
Cord Medicine, 31 (2008) 388-393
4

5. BIG CONCEPT
5

6. Proposed concepts
Supporting elderly and disabled people in home
environment
- Reduce the involved operation complexity of some
common human-robot interactions.
- The Leap Motion Controller (LMC) is used to operate the
Jaco Robot in intuitive way.
- Keyboard or joystick is replaced by LMC.
- Mapping user’s real arm and the Jaco arm.
- Estimate and filter out the unwanted oscillation.
6

7. Proposed concepts
Tele-Operation
- A method to operate a robot, while still being at a distance
from it.
- Dangerous and risky application areas (space exploration,
surveillance, surgery, nuclear plants and underwater
operation).
7

8. Proposed concepts
Tremor Detection
- The sub-millimeter accuracy of the
controller could be seen as an
opportunity for detecting symptoms
related to hand tremor.
- Detect and process the user palm,
sent out-put data to robotic arm,
filter out unwanted oscillation,
enable smooth operations.
- LMC can be efficiently used to
validate and quantify the tremor and
bradykinesia.
8

9. Coordinate
system
transformation
Mapping
Algorithm
Connect to
Arduino
micro-
controller
Steps of development9

10. Coordinate system transformation
- Relate two systems together by performing some
rotations.
Steps of development
Fig4. Coordinate system transformation
10

11. Mapping algorithm
Steps of development
Fig5. The mapping algorithm
11

12. Calibration process
- Performs a specific gesture.
- Fix hand in 10 ~15 sec.
- Threshold values ensure that the
hand tremor is not reflect to the
robotic arm.
Steps of development
Fig6. The Calibration process
12

13. Connect to Arduino micro-controller
- Enhance the functionality of the system.
Steps of development
Fig7. Grasp and Release operation
13

14. Overall flow diagram
Steps of development
Fig8. The information flow diagram
14

15. ALL flat
Research
USA2
Projects
Research projects15

16. - Offers possibilities for elderly people to work at their
home environment and thus contribute to the society.
USA2
projects
Fig9. The USA² project demonstrator
Fig10. The schematic diagram of the system
16

17. - Contribute the quality of living of the ageing society and
impaired people, and assist them in maintaining an
independent lifestyle in the home environment.
- Implement system in AAL* fat, also to mobility such as
wheelchairs, rollators,…
AAL flat research
*Ambient Assisted Living
17

18. AAL flat research
Fig11. The bedroom terminal
Fig13. The kitchen terminal
Fig12. The Lisa Smart Wall
18

19. - The intuitive manipulation of robotics arm uses a new
gesture and LMC tracking system with sub-millimeter
accuracy.
- Introduce a simple and straighforward robotics arm
minipulation scheme.
- Enable incorporation of robotics system into the home
environment.
- LMC technology would be benefit to enhance the
independence and autonomy of individuals or montitors
and prevents some diseases, such as Parkinson.
Conclusion19

20. Thanks!
Any questions?