Using hand gestures to fly UAVs
•Download as PPTX, PDF•
5 likes•2,188 views
This was part of my master's thesis focused on using hand gestures to fly a Parrot Bebop Drone. The motion sensing device used here is a Leap Motion. As the next step, I am going to integrate Kinect and Myo with the system.
More Related Content
What's hot
What's hot (20)
Similar to Using hand gestures to fly UAVs
Similar to Using hand gestures to fly UAVs (20)
Recently uploaded
Recently uploaded (20)
Using hand gestures to fly UAVs
- 1. Exploring Alternative Control Modalities for Unmanned Aerial Vehicles Thesis Presentation David Qorashi Grand Valley State University April 2015 Committee Members Dr. Engelsma, Dr. Alsabbagh, Dr. Dulimarta
- 2. Unmanned Aerial Vehicles (UAVs) • Primary Usage • Defense & Security Sectors • Potential Usage • Private Security • News Gathering (CNN) • Agricultural Practices • Product Transport • Aerial Advertising • … Source: New York Daily News (2014)
- 3. –Business Insider (2014) “We predict that 12% of an estimated $98 billion in cumulative global spending on aerial drones over the next decade will be for commercial purposes.”
- 4. Challenge Hard to fly for inexperienced users
- 5. Problems with current approaches • Steep Learning Curve • Failing to pilot the drone in initial attempts • Damage to drone/environment • Comprise safety of pilot and/or bystanders
- 7. Endangers People’s Safety Roman Pirozek was killed while flying a remote-controlled model helicopter on September 2013 (Source: Wall Street Journal).
- 8. Purpose of this study Alternative Control Mechanisms
- 10. Human robot interaction: A new field in HCI • Goodtich & Olsen defined a general Interface on designing effective UIs • Dury et.al defined a set of HRI taxonomies • Quigley et al. used a numeric parameter-based interface for controlling • Quigley et al. used voice control interaction • Natural User Interfaces • Gesture-based Interaction for collocated ground robots (Rogolla et al. 2002) • Gesture-based Interaction for collocated Drones (Ng et al. 2011)
- 11. Most gesture-based approaches use drone’s built-in camera for image recognition
- 12. Source: Collocated Interaction with Flying Robots (Ng et al. 2011)
- 13. Microsoft Kinect Source: Microsoft Corp. (2014) New Approach: Using motion sensing input devices A Kinect-Base Natural Interface for Quadrotor Control (Sanna et al. 2012)
- 14. Challenges using Body Parts Movements • Not very comforting • Not very receptive by the pilots • Very hard to do in a long run
- 16. Hand Gestures
- 17. The purpose of study Develop an improved & more intuitive method of piloting
- 18. Thesis Hypothesis • Increasing efficiency in piloting • Adding overall enjoyable experience in piloting
- 19. Quadcopters • Also known as quadrotor • 4 rotors • Fixed pitch • Controlling happens via Wi-Fi
- 20. Bebop AR-Drone An affordable Quadcopter • Rich Programming API • Affordable Price
- 21. Leap Motion Controller Source: Forbes Magazine (2014)
- 23. Core Experiment Implemented gesture-based interface vs. conventional multi-touch methods Comparative Analysis
- 24. • Sample: A subset of students • First, they were asked to complete a very simple mission in an specific route using multi-touch approach • Second, we ask them to pilot the drone in the same route using the implemented approach • Finally, we ask them to fill a questionnaire to rate each approach. • Also, the trainer examined the accuracy of landings after each flight
- 25. Defined Mission
- 26. Implementation
- 27. What needs to be done? • Controller for sending the commands to the drone • Gesture Recognition System • Interfacing Gesture system with Drone Controller
- 28. Drone Controller • Bebop Drone released in December 2014 • Documentations for the new API: Extremely weak • A high-level wrapper was created around C API • https://github.com/gvsucis/c-bebop-drone
- 29. Implemented Functions • ARDrone3SendSettingsAllSettings • ARDrone3SendCommonAllStates • ARDrone3SendTakeoffCommand • ARDrone3SendLandCommenad • ARDrone3SendPCMD • ARDrone3SendSendSpeedSettingsHullProtection • ARDrone3SendPilotingFlatTrim • ARDrone3SendYawRightCommand • ARDrone3SendYawLeftCommand • ARDrone3SendAscendCommand • ARDrone3SendDescendCommand • ARDrone3SendHoverCommand • ARDrone3SendMoveForwardCommand • ARDrone3SendMoveBackwardCommand
- 30. Using the implemented wrapper we were able to send commands to the Bebop
- 32. Gesture Recognition System • Q: What method of implementation would be better? (better here, means general and accurate) • A: Machine Learning
- 33. ML Applications • Classification • Grouping • Ranking
- 35. Gesture Recognizer • Method in use: Supervised Machine Learning • Steps: • Sample Collection And Tagging • Machine Leaning Recognizer Code
- 40. Every ML algorithm needs some features • Features are extracted from the data • Features in use for this application: • Tips’ positions • Center of Palm position • Velocities
- 41. Leap Motion API provided us with positions of tips, palm and velocities
- 44. With huge set of data all these methods converge Domingos, Pedro. "A few useful things to know about machine learning." Communications of the ACM 55.10 (2012): 78-87.
- 45. For this App, we chose $P • Why? • We didn’t have lots of training data • $P is optimized for gesture recognition er: a 2-D gesture recognizer designed for rapid prototyping of ges
- 46. As the last step we connected gesture recognizer to drone controller
- 47. Voila!
- 48. Experiment
- 49. Asked subjects to try out both approaches
- 50. Mission
- 51. Data collected
- 52. Results
- 56. Surprise? Does it mean multi-touch method is better?
- 57. Why? • Gesture controller involves nondeterministic factors • Quality of the sensor • Quality of data gathered during training phase • Lighting conditions of the environment during the test • Suitableness of algorithm used for training
- 58. As expected, majority of people thought that GESTURE method is more intuitive
- 60. Raised Questions During Research?? • How should we handle combinatorial commands? • How can we make GR 100% accurate and more robust? • How can we define more intuitive gestures?
- 61. Future work? • Use another kind of sensor for gathering data • Run image processing algorithms instead of relying on the API provided by Leap • Make a huge data set instead of using just two people’s hand gestures • Considering multimodal controls
- 62. Thank you for your time Questions?
Editor's Notes
- UAVs are defined as an aircraft without a human pilot on board. They are currently popular in military, but using them for commercial purposes is ramping up. They can revolutionize various business sectors. For now it is not allowed to use them for business purposes in the U.S., though lawmakers plan to phase in commercial drone flights beginning in 2015, starting with limited flights of small drones
- Despite all the improvements for overall controllability of drones, most of them are still quite challenging for inexperienced users to pilot.
- Users will buy drones, unbox them, and attempt to fly them immediately. The problem is that they are not able to fly them smoothly in their initial efforts. :)
- . Another common issue is that novice users lose control of drones very easily. Drones will fly away and will never come back. Running a simple query on youtube for ‘drone flies away’ returns up to 59,000 results. It’s a common problem ;)
- And most importantly, they can harm pilot or the other people in the area. The left news belongs to two days ago in one of the branches of TGI Friday in Manchester, UK. The operator wanted to show the poor customer that he can land the drone on her hand, but … accidents happen. He crashed the drone into woman’s face and cuts it open in restaurant. The worst accident happened last year. The drone chopped the head off of this young man due to problems in controllability.
- On this thesis we will focus on investigating alternate control modalities that can mitigate these problems.
- Human Robot Interaction (HRI) is a new field of study in HCI. most of these approaches focus on ground-based vehicles. some of them aren’t applicable to the drones, due to the fact that drones are in 3D environments and not 2D. Quigley investigated several different interfaces for controlling drones, from an input interface where the user should enter numerical values for navigating the UAV, to utilizing a physical model of the drone as the controller. They utilized a PDA to control the drone. Their focus in their studies was to control a remote drone rather than a collocated one. Also, they used a voice control interaction approach. They used talking mechanism to a PDA in order to control a remote drone NUI is the common word used by designers and developers of human-machine interfaces to refer to a user interface that is effectively invisible, and remains invisible as the user continuously learns increasingly complex interactions.
- Limitation: Most of gesture-based recognition research use robot’s built-in camera to run some image-processing task to recognize gestures.There are some issues with these approaches: complex backgrounds, different lighting conditions and also real-time execution limitations.
- . Using this kind of motion sensing devices abate the problem of gesture recognition. The researcher can rely on the data provided by Kinect and they do not need to run image-processing algorithms for recognizing gestures because the problem is already solved with Kinect.
- The goal is to employ and test an end-to-end UAV system that provides an easy-to-use control interface for novice drone users.
- The expectation is that by implementing gesture-based navigation, the novice user will have an enjoyable and safe experience quickly learning how to navigate a drone with ease, and also avoid losing or damaging the vehicle while they are still on the initial learning curve.
- We will focus to pilot a specific kind of drone called quadcoptors. Though, the mechanisms we use are general and extendable to the other types as well. A quadcopter is an aerial vehicle propelled by four rotors. They have a fixed pitch, which makes them mechanically simpler than a typical helicopter
- The drone uses an open software stack which makes it easily programmable.
- motion sensing input device it supports hand and finger motions as input, analogous to a mouse, but requiring no hand contact or touching
- it captures hand motions. deliver them to a computer computer interprets the gestures and maps them to appropriate commands and sends them to the drone via wifi connection drone will execute the command
- After implementing, efficiency of solution will get examined
- To make sure that the pilots are maneuvering on the exact same route using each controlling approach
- Using the official API the programmer has access to all features of drone including sensors data, video stream and navigation channel data.
- To make sure that the pilots are maneuvering on the exact same route using each controlling approach