Downloaded 45 times
Uploaded byDavid Qorashi
Using hand gestures to fly UAVs
This thesis explores alternative control modalities for unmanned aerial vehicles (UAVs) to reduce the steep learning curve faced by inexperienced users. The research proposes a gesture-based control interface using motion sensing input devices, specifically a Leap Motion controller, to enhance the piloting experience and efficiency of quadcopters. The findings indicate that while users generally find the gesture-based approach more intuitive, challenges such as variability in sensor performance and training data quality need to be addressed in future work.
More Related Content
Similar to Using hand gestures to fly UAVs
![[Deck] What's New in Spark-Iceberg Integration via DSV2.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/deckwhatsnewinspark-icebergintegrationviadsv2-260210005337-25955b12-thumbnail.jpg?width=640&height=640&fit=bounds)
Using hand gestures to fly UAVs
- 1. Exploring Alternative Control Modalitiesfor Unmanned Aerial Vehicles Thesis Presentation David Qorashi Grand Valley State University April 2015 Committee Members Dr. Engelsma, Dr. Alsabbagh, Dr. Dulimarta
- 2. Unmanned Aerial Vehicles (UAVs) •Primary Usage • Defense & Security Sectors • Potential Usage • Private Security • News Gathering (CNN) • Agricultural Practices • Product Transport • Aerial Advertising • … Source: New York Daily News (2014)
- 3. –Business Insider (2014) “Wepredict that 12% of an estimated $98 billion in cumulative global spending on aerial drones over the next decade will be for commercial purposes.”
- 4. Challenge Hard to flyfor inexperienced users
- 5. Problems with current approaches •Steep Learning Curve • Failing to pilot the drone in initial attempts • Damage to drone/environment • Comprise safety of pilot and/or bystanders
- 6.
- 7. Endangers People’s Safety RomanPirozek was killed while flying a remote-controlled model helicopter on September 2013 (Source: Wall Street Journal).
- 8. Purpose of thisstudy Alternative Control Mechanisms
- 9.
- 10. Human robot interaction:A new field in HCI • Goodtich & Olsen defined a general Interface on designing effective UIs • Dury et.al defined a set of HRI taxonomies • Quigley et al. used a numeric parameter-based interface for controlling • Quigley et al. used voice control interaction • Natural User Interfaces • Gesture-based Interaction for collocated ground robots (Rogolla et al. 2002) • Gesture-based Interaction for collocated Drones (Ng et al. 2011)
- 11. Most gesture-based approaches usedrone’s built-in camera for image recognition
- 12. Source: Collocated Interactionwith Flying Robots (Ng et al. 2011)
- 13. Microsoft Kinect Source: Microsoft Corp.(2014) New Approach: Using motion sensing input devices A Kinect-Base Natural Interface for Quadrotor Control (Sanna et al. 2012)
- 14. Challenges using BodyParts Movements • Not very comforting • Not very receptive by the pilots • Very hard to do in a long run
- 15.
- 16.
- 17. The purpose ofstudy Develop an improved & more intuitive method of piloting
- 18. Thesis Hypothesis • Increasingefficiency in piloting • Adding overall enjoyable experience in piloting
- 19. Quadcopters • Also knownas quadrotor • 4 rotors • Fixed pitch • Controlling happens via Wi-Fi
- 20. Bebop AR-Drone An affordable Quadcopter •Rich Programming API • Affordable Price
- 21. Leap Motion Controller Source:Forbes Magazine (2014)
- 23. Core Experiment Implemented gesture-basedinterface vs. conventional multi-touch methods Comparative Analysis
- 24. • Sample: Asubset of students • First, they were asked to complete a very simple mission in an specific route using multi-touch approach • Second, we ask them to pilot the drone in the same route using the implemented approach • Finally, we ask them to fill a questionnaire to rate each approach. • Also, the trainer examined the accuracy of landings after each flight
- 25.
- 26.
- 27. What needs tobe done? • Controller for sending the commands to the drone • Gesture Recognition System • Interfacing Gesture system with Drone Controller
- 28. Drone Controller • BebopDrone released in December 2014 • Documentations for the new API: Extremely weak • A high-level wrapper was created around C API • https://github.com/gvsucis/c-bebop-drone
- 29. Implemented Functions • ARDrone3SendSettingsAllSettings •ARDrone3SendCommonAllStates • ARDrone3SendTakeoffCommand • ARDrone3SendLandCommenad • ARDrone3SendPCMD • ARDrone3SendSendSpeedSettingsHullProtection • ARDrone3SendPilotingFlatTrim • ARDrone3SendYawRightCommand • ARDrone3SendYawLeftCommand • ARDrone3SendAscendCommand • ARDrone3SendDescendCommand • ARDrone3SendHoverCommand • ARDrone3SendMoveForwardCommand • ARDrone3SendMoveBackwardCommand
- 30. Using the implemented wrapperwe were able to send commands to the Bebop
- 31.
- 32. Gesture Recognition System •Q: What method of implementation would be better? (better here, means general and accurate) • A: Machine Learning
- 33. ML Applications • Classification •Grouping • Ranking
- 34.
- 35. Gesture Recognizer • Methodin use: Supervised Machine Learning • Steps: • Sample Collection And Tagging • Machine Leaning Recognizer Code
- 36.
- 37.
- 38.
- 40. Every ML algorithmneeds some features • Features are extracted from the data • Features in use for this application: • Tips’ positions • Center of Palm position • Velocities
- 41. Leap Motion APIprovided us with positions of tips, palm and velocities
- 43.
- 44. With huge setof data all these methods converge Domingos, Pedro. "A few useful things to know about machine learning." Communications of the ACM 55.10 (2012): 78-87.
- 45. For this App,we chose $P • Why? • We didn’t have lots of training data • $P is optimized for gesture recognition er: a 2-D gesture recognizer designed for rapid prototyping of ges
- 46. As the laststep we connected gesture recognizer to drone controller
- 47.
- 48.
- 49. Asked subjects totry out both approaches
- 50.
- 51.
- 52.
- 56. Surprise? Does itmean multi-touch method is better?
- 57. Why? • Gesture controllerinvolves nondeterministic factors • Quality of the sensor • Quality of data gathered during training phase • Lighting conditions of the environment during the test • Suitableness of algorithm used for training
- 58. As expected, majorityof people thought that GESTURE method is more intuitive
- 59.
- 60. Raised Questions During Research?? •How should we handle combinatorial commands? • How can we make GR 100% accurate and more robust? • How can we define more intuitive gestures?
- 61. Future work? • Useanother kind of sensor for gathering data • Run image processing algorithms instead of relying on the API provided by Leap • Make a huge data set instead of using just two people’s hand gestures • Considering multimodal controls
- 62. Thank you foryour time Questions?
Editor's Notes
- #3 UAVs are defined as an aircraft without a human pilot on board. They are currently popular in military, but using them for commercial purposes is ramping up. They can revolutionize various business sectors. For now it is not allowed to use them for business purposes in the U.S., though lawmakers plan to phase in commercial drone flights beginning in 2015, starting with limited flights of small drones
- #5 Despite all the improvements for overall controllability of drones, most of them are still quite challenging for inexperienced users to pilot.
- #6 Users will buy drones, unbox them, and attempt to fly them immediately. The problem is that they are not able to fly them smoothly in their initial efforts. :)
- #7 . Another common issue is that novice users lose control of drones very easily. Drones will fly away and will never come back. Running a simple query on youtube for ‘drone flies away’ returns up to 59,000 results. It’s a common problem ;)
- #8 And most importantly, they can harm pilot or the other people in the area. The left news belongs to two days ago in one of the branches of TGI Friday in Manchester, UK. The operator wanted to show the poor customer that he can land the drone on her hand, but … accidents happen. He crashed the drone into woman’s face and cuts it open in restaurant. The worst accident happened last year. The drone chopped the head off of this young man due to problems in controllability.
- #9 On this thesis we will focus on investigating alternate control modalities that can mitigate these problems.
- #11 Human Robot Interaction (HRI) is a new field of study in HCI. most of these approaches focus on ground-based vehicles. some of them aren’t applicable to the drones, due to the fact that drones are in 3D environments and not 2D. Quigley investigated several different interfaces for controlling drones, from an input interface where the user should enter numerical values for navigating the UAV, to utilizing a physical model of the drone as the controller. They utilized a PDA to control the drone. Their focus in their studies was to control a remote drone rather than a collocated one. Also, they used a voice control interaction approach. They used talking mechanism to a PDA in order to control a remote drone NUI is the common word used by designers and developers of human-machine interfaces to refer to a user interface that is effectively invisible, and remains invisible as the user continuously learns increasingly complex interactions.
- #12 Limitation: Most of gesture-based recognition research use robot’s built-in camera to run some image-processing task to recognize gestures.There are some issues with these approaches: complex backgrounds, different lighting conditions and also real-time execution limitations.
- #14 . Using this kind of motion sensing devices abate the problem of gesture recognition. The researcher can rely on the data provided by Kinect and they do not need to run image-processing algorithms for recognizing gestures because the problem is already solved with Kinect.
- #18 The goal is to employ and test an end-to-end UAV system that provides an easy-to-use control interface for novice drone users.
- #19 The expectation is that by implementing gesture-based navigation, the novice user will have an enjoyable and safe experience quickly learning how to navigate a drone with ease, and also avoid losing or damaging the vehicle while they are still on the initial learning curve.
- #20 We will focus to pilot a specific kind of drone called quadcoptors. Though, the mechanisms we use are general and extendable to the other types as well. A quadcopter is an aerial vehicle propelled by four rotors. They have a fixed pitch, which makes them mechanically simpler than a typical helicopter
- #21 The drone uses an open software stack which makes it easily programmable.
- #22 motion sensing input device it supports hand and finger motions as input, analogous to a mouse, but requiring no hand contact or touching
- #23 it captures hand motions. deliver them to a computer computer interprets the gestures and maps them to appropriate commands and sends them to the drone via wifi connection drone will execute the command
- #24 After implementing, efficiency of solution will get examined
- #26 To make sure that the pilots are maneuvering on the exact same route using each controlling approach
- #33 Using the official API the programmer has access to all features of drone including sensors data, video stream and navigation channel data.
- #51 To make sure that the pilots are maneuvering on the exact same route using each controlling approach