This was part of my master's thesis focused on using hand gestures to fly a Parrot Bebop Drone. The motion sensing device used here is a Leap Motion.
As the next step, I am going to integrate Kinect and Myo with the system.
Ch10-Global Supply Chain - Cadena de Suministro.pdf
Using hand gestures to fly UAVs
1. Exploring Alternative
Control Modalities for
Unmanned Aerial Vehicles
Thesis Presentation
David Qorashi
Grand Valley State University
April 2015
Committee Members
Dr. Engelsma, Dr. Alsabbagh, Dr. Dulimarta
3. –Business Insider (2014)
“We predict that 12% of an estimated $98 billion
in cumulative global spending on aerial drones
over the next decade will be for commercial
purposes.”
5. Problems with current
approaches
• Steep Learning Curve
• Failing to pilot the drone in initial attempts
• Damage to drone/environment
• Comprise safety of pilot and/or bystanders
7. Endangers People’s Safety
Roman Pirozek was killed while flying a
remote-controlled model helicopter on
September 2013 (Source: Wall Street
Journal).
10. Human robot interaction: A
new field in HCI
• Goodtich & Olsen defined a general Interface on designing effective UIs
• Dury et.al defined a set of HRI taxonomies
• Quigley et al. used a numeric parameter-based interface for controlling
• Quigley et al. used voice control interaction
• Natural User Interfaces
• Gesture-based Interaction for collocated ground robots (Rogolla et al.
2002)
• Gesture-based Interaction for collocated Drones (Ng et al. 2011)
13. Microsoft
Kinect
Source: Microsoft Corp. (2014)
New Approach: Using motion sensing input devices
A Kinect-Base Natural Interface for Quadrotor Control (Sanna et al. 2012)
14. Challenges using Body Parts
Movements
• Not very comforting
• Not very receptive by the pilots
• Very hard to do in a long run
24. • Sample: A subset of students
• First, they were asked to complete a very simple mission
in an specific route using multi-touch approach
• Second, we ask them to pilot the drone in the same
route using the implemented approach
• Finally, we ask them to fill a questionnaire to rate each
approach.
• Also, the trainer examined the accuracy of landings after
each flight
27. What needs to be done?
• Controller for sending the commands to the drone
• Gesture Recognition System
• Interfacing Gesture system with Drone Controller
28. Drone Controller
• Bebop Drone released in December 2014
• Documentations for the new API: Extremely weak
• A high-level wrapper was created around C API
• https://github.com/gvsucis/c-bebop-drone
40. Every ML algorithm needs
some features
• Features are extracted from the data
• Features in use for this application:
• Tips’ positions
• Center of Palm position
• Velocities
41. Leap Motion API provided
us with positions of tips,
palm and velocities
44. With huge set of data all these methods converge
Domingos, Pedro. "A few useful things to know about machine
learning." Communications of the ACM 55.10 (2012): 78-87.
45. For this App, we chose $P
• Why?
• We didn’t have lots of training data
• $P is optimized for gesture recognition
er: a 2-D gesture recognizer designed for rapid prototyping of ges
46. As the last step we connected
gesture recognizer to drone
controller
57. Why?
• Gesture controller involves nondeterministic factors
• Quality of the sensor
• Quality of data gathered during training phase
• Lighting conditions of the environment during
the test
• Suitableness of algorithm used for training
60. Raised Questions During
Research??
• How should we handle combinatorial commands?
• How can we make GR 100% accurate and more
robust?
• How can we define more intuitive gestures?
61. Future work?
• Use another kind of sensor for gathering data
• Run image processing algorithms instead of relying
on the API provided by Leap
• Make a huge data set instead of using just two
people’s hand gestures
• Considering multimodal controls
UAVs are defined as an aircraft without a human pilot on board.
They are currently popular in military, but using them for commercial purposes is ramping up. They can revolutionize various business sectors.
For now it is not allowed to use them for business purposes in the U.S., though lawmakers plan to phase in commercial drone flights beginning in 2015, starting with limited flights of small drones
Despite all the improvements for overall controllability of drones, most of them are still quite challenging for inexperienced users to pilot.
Users will buy drones, unbox them, and attempt to fly them immediately. The problem is that they are not able to fly them smoothly in their initial efforts. :)
. Another common issue is that novice users lose control of drones very easily. Drones will fly away and will never come back. Running a simple query on youtube for ‘drone flies away’ returns up to 59,000 results. It’s a common problem ;)
And most importantly, they can harm pilot or the other people in the area.
The left news belongs to two days ago in one of the branches of TGI Friday in Manchester, UK. The operator wanted to show the poor customer that he can land the drone on her hand, but … accidents happen. He crashed the drone into woman’s face and cuts it open in restaurant.
The worst accident happened last year. The drone chopped the head off of this young man due to problems in controllability.
On this thesis we will focus on investigating alternate control modalities that can mitigate these problems.
Human Robot Interaction (HRI) is a new field of study in HCI.
most of these approaches focus on ground-based vehicles. some of them aren’t applicable to the drones, due to the fact that drones are in 3D environments and not 2D.
Quigley investigated several different interfaces for controlling drones, from an input interface where the user should enter numerical values for navigating the UAV, to utilizing a physical model of the drone as the controller. They utilized a PDA to control the drone. Their focus in their studies was to control a remote drone rather than a collocated one. Also, they used a voice control interaction approach. They used talking mechanism to a PDA in order to control a remote drone
NUI is the common word used by designers and developers of human-machine interfaces to refer to a user interface that is effectively invisible, and remains invisible as the user continuously learns increasingly complex interactions.
Limitation: Most of gesture-based recognition research use robot’s built-in camera to run some image-processing task to recognize gestures.There are some issues with these approaches: complex backgrounds, different lighting conditions and also real-time execution limitations.
. Using this kind of motion sensing devices abate the problem of gesture recognition. The researcher can rely on the data provided by Kinect and they do not need to run image-processing algorithms for recognizing gestures because the problem is already solved with Kinect.
The goal is to employ and test an end-to-end UAV system that provides an easy-to-use control interface for novice drone users.
The expectation is that by implementing gesture-based navigation, the novice user will have an enjoyable and safe experience quickly learning how to navigate a drone with ease, and also avoid losing or damaging the vehicle while they are still on the initial learning curve.
We will focus to pilot a specific kind of drone called quadcoptors. Though, the mechanisms we use are general and extendable to the other types as well. A quadcopter is an aerial vehicle propelled by four rotors. They have a fixed pitch, which makes them mechanically simpler than a typical helicopter
The drone uses an open software stack which makes it easily programmable.
motion sensing input device
it supports hand and finger motions as input, analogous to a mouse, but requiring no hand contact or touching
it captures hand motions.
deliver them to a computer
computer interprets the gestures and maps them to appropriate commands
and sends them to the drone via wifi connection
drone will execute the command
After implementing, efficiency of solution will get examined
To make sure that the pilots are maneuvering on the exact same route using each controlling approach
Using the official API the programmer has access to all features of drone including sensors data, video stream and navigation channel data.
To make sure that the pilots are maneuvering on the exact same route using each controlling approach