Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
www.euka.org
DRONE CONVENTION 2015
#DroneConEU
www.droneconvention.eu
21 April 2015, C-Mine, Genk
INTELLIGENTE VISIE MAAKT...
Intelligent vision making
your drones autonomous
Prof. dr. ir. Toon Goedemé
EAVISE – KU Leuven
Introduction
• UAV’s are used in a variety of tasks
• Downside: RPAS requires skilled pilot
=> make them autonomous!
www.e...
How to make drones autonomous?
• Intelligent mission planning
• Path planning
• Autonomous motor control
• Obstacle avoida...
Why cameras?
• Enormously rich environment sensor
o Humans (and most animals)
rely for a great percentage
on visual input ...
What can cameras do for your UAV?
• Mapping
o Building a 3D map of a (formerly unknown) environment
• Localisation
o Recog...
Main vision
technologies
Main 2D vision technologies
A. SLAM
o Simultaneous Localisation And Mapping
• GPS alternative
• 3D map building
• Obstacle...
A. Single camera Visual SLAM
www.eavise.be 9
A glance on Monocular SLAM
• Can we track the motion of a camera while it is moving?
• Pick natural scene features to serv...
Prominent Monocular SLAM systems
• Algorithms ready in the Vision&Robotics research
communities. But ready to leave the la...
Challenges for UAV visual SLAM
• Fast motion
• Large scales
• Robustness
• Rich maps
• Low computation
• Sensor failures
•...
Successfull UAV visual SLAM approaches
https://www.youtube.com/watch?v=zeeSQsN8p6Y https://www.youtube.com/watch?v=59TWljD...
B. Object detection
www.eavise.be 14
Object recognition  object detection
Object recognition
Object identification
Object detection
Object categorisation
Obj...
Object recognition  object categorization
• Object categorization – focus on a complete object class
• Variation in the ...
Object recognition  object categorization
• Object categorization gets harder the moment there is
more and more variatio...
Object recognition  object categorization
• Object categorization – general approach
o Learning from examples of class i...
Viola & Jones object categorization framework
• Open source framework available in OpenCV
• Famous from face location in c...
Viola & Jones object categorization framework
• Used approach
o Transform image to integral image
o On that image
• Calcul...
Viola & Jones object categorization framework
• Used approach
o CASCADE: combining weak classifiers into a strong
classifi...
Other object detection algorithms
• Dalal & Triggs – HOG for human detection – HOG + SVM
www.eavise.be 22
Other object detection algorithms
• Felzenszwalb – Cascade object detection with deformable
parts AKA LatentSVM.
www.eavis...
Other object detection algorithms
• Dollár– Integral channel features
www.eavise.be 24
Example autonomous
UAV projects
• A UAV that flies autonomously through the
tree corridors of an ochard
• Inspection of
o Number of flowers
o Fruit quanti...
UAV inspector
Test UAV: AR Drone
Camera verticaal
Ultrasone sensoren
Camera horizontaal
• Processor: ARM9, 468MHz + DCT mo...
UAV inspector
Vision algorithm
Visie
Roll setpoint
Yaw setpoint
Camerabeeld
MG-punt = Roll setpoint
Vanishing point = Yaw ...
UAV inspector
Vision algorithm
www.eavise.be
UAV inspector
Vision algorithm
Good
Threshold
Bad
Threshold
Automatische threshold:
• Score berekenen voor
beste threshold...
Orchard monitoring results
www.eavise.be
Example project #2: Cametron
Flying autonomous camera crew
a system that, with minimal human intervention,
produces high q...
= a system that, with minimal human intervention, produces
high quality audiovisual (AV) productions
o a virtual camera cr...
• Autonomous UAV
• Real-time persoon detectie and –tracking on embedded
hardware
• Topological localisation
• Visual servo...
Cametron realisations
• 3D localisation system, for visual
localisation of the UAV in a room
• Person detection and tracki...
Detecting a person
• Viola and Jones (Haar/Adaboost)
• HOG
• ICF
• DPM
Person detection and tracking framework
with PTZ ca...
Detecting persons
Person detection and tracking framework
with PTZ camera
V&J
HOG full body
model
ICF upper body
model
DPM...
• Rules making it more enjoyable looking at the video material
o Nose room
o Head room
o Rule of thirds
Cinematographic ru...
Detecting gaze direction
Face detection
Frontal model
Face detection
Left looking model
Face detection
Right looking model...
Difficult because camera is moving
Detection of direction of movement
Determination of
movement
direction
Positioning of
p...
Results on PTZ camera system
Person
detection
Determination of
gaze or movement
direction
Apply
cinematographic
rules
Pers...
• In literature: Processing done mostly offline
• Which hardwareplatform is best suited to run a certain
image processing ...
• Key criterium: maximizing endurance
• Embedded processing board
o has certain weight, to be carried as payload
o consume...
Selection tool for embedded processing unit
for complex on-board UAV computer vision
processor memory Power
(Watt)
Weight
...
First experiments
2 benchmark algorithms on each platform
www.eavise.be 45
First cametron results on UAV
• See our demo at DroneCon
• Tracking an actor from a flying UAV while automatically
keeping...
Conclusions
• Future of UAVs lie in fully autonomous applications
• Camera is a very powerful and lightweight sensor
• Ima...
More information?
• Autonomous orchard inspection
[Dries Hulens, Maarten Vandersteegen and Toon Goedemé, UAV autonoom late...
Upcoming SlideShare
Loading in …5
×

Intelligente visie maakt drones autonoom

886 views

Published on

Presentation of prof. dr. ir. Toon Goedemé KULeuven

Published in: Technology
  • Be the first to comment

Intelligente visie maakt drones autonoom

  1. 1. www.euka.org DRONE CONVENTION 2015 #DroneConEU www.droneconvention.eu 21 April 2015, C-Mine, Genk INTELLIGENTE VISIE MAAKT DRONES AUTONOOM Prof. dr. ir. Toon Goedemé www.eavise.be 1
  2. 2. Intelligent vision making your drones autonomous Prof. dr. ir. Toon Goedemé EAVISE – KU Leuven
  3. 3. Introduction • UAV’s are used in a variety of tasks • Downside: RPAS requires skilled pilot => make them autonomous! www.eavise.be 3
  4. 4. How to make drones autonomous? • Intelligent mission planning • Path planning • Autonomous motor control • Obstacle avoidance • … • All of the above rely on GOOD SENSORS o GPS o Proximity sensors o Cameras • 2D • 3D www.eavise.be 4
  5. 5. Why cameras? • Enormously rich environment sensor o Humans (and most animals) rely for a great percentage on visual input for navigation • Very lightweight • Very power efficient • Very small • Cheap • … • But: very difficult to process huge data stream o Modern image processing algorithms can do a lot o but are computationally very demanding o Difficult to make it • real time • lightweight • power efficient www.eavise.be 5
  6. 6. What can cameras do for your UAV? • Mapping o Building a 3D map of a (formerly unknown) environment • Localisation o Recognizing the place where the UAV flies o As an alternative to GPS o Or in cooperation with GPS (and INS) • Obstacle avoidance o Detecting obstacles and steer UAV around it o “sense and avoid” • Object/person detection o Where are persons or other objects? • Object/person recognition o Who is it? Which object is it? • Object/person tracking and visual servoing o Steering the UAV in order to follow a moving person or an object www.eavise.be 6
  7. 7. Main vision technologies
  8. 8. Main 2D vision technologies A. SLAM o Simultaneous Localisation And Mapping • GPS alternative • 3D map building • Obstacle detection B. Object detection o Detect where objects appear in the image regardless of: • Appearance variances • Illumination changes • Occlusion • Viewpoint changes www.eavise.be 8
  9. 9. A. Single camera Visual SLAM www.eavise.be 9
  10. 10. A glance on Monocular SLAM • Can we track the motion of a camera while it is moving? • Pick natural scene features to serve as landmarks o Range sensing (laser/sonar): points, line segments, 3D planes, corners o Vision: point features, lines, textured surfaces • Key: features must be distinctive and recognizable from different viewpoints www.eavise.be 10
  11. 11. Prominent Monocular SLAM systems • Algorithms ready in the Vision&Robotics research communities. But ready to leave the lab & perform every day tasks? www.eavise.be 11
  12. 12. Challenges for UAV visual SLAM • Fast motion • Large scales • Robustness • Rich maps • Low computation • Sensor failures • Handle larger amounts of data effectively • Competing goals: • Key: agile manipulation of information www.eavise.be 12
  13. 13. Successfull UAV visual SLAM approaches https://www.youtube.com/watch?v=zeeSQsN8p6Y https://www.youtube.com/watch?v=59TWljDYmB8 https://www.youtube.com/watch?v=84jZg7hXEVc https://www.youtube.com/watch?v=ThnI-dcJL4E www.eavise.be 13
  14. 14. B. Object detection www.eavise.be 14
  15. 15. Object recognition  object detection Object recognition Object identification Object detection Object categorisation Object classification ≠ ???? www.eavise.be 15
  16. 16. Object recognition  object categorization • Object categorization – focus on a complete object class • Variation in the class itself: cars, cows, … • Overal appearance of the object is still the same o Cars 4 wheels – front and side windows - … o Cows 4 legs – robust rectangular body - … www.eavise.be 16
  17. 17. Object recognition  object categorization • Object categorization gets harder the moment there is more and more variation introduced • Challenges for robustness: illumination, object pose, clutter, occlusions, intra class appearance, viewpoint, … www.eavise.be 17
  18. 18. Object recognition  object categorization • Object categorization – general approach o Learning from examples of class instances o Use the learned model for detecting new objects + + + + + + - - - www.eavise.be 18
  19. 19. Viola & Jones object categorization framework • Open source framework available in OpenCV • Famous from face location in consumer photo cameras • [P. Viola and M. Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, IEEE Conference on Computer Vision and Pattern Recognition, 2001] www.eavise.be 19
  20. 20. Viola & Jones object categorization framework • Used approach o Transform image to integral image o On that image • Calculate possible features • HAAR Wavelets / Local Binary Patterns o Put all features in a boosting process • Machine learning technique • AdaBoost approach www.eavise.be 20
  21. 21. Viola & Jones object categorization framework • Used approach o CASCADE: combining weak classifiers into a strong classifier o Sliding window approach for feature calculation on a single image www.eavise.be 21
  22. 22. Other object detection algorithms • Dalal & Triggs – HOG for human detection – HOG + SVM www.eavise.be 22
  23. 23. Other object detection algorithms • Felzenszwalb – Cascade object detection with deformable parts AKA LatentSVM. www.eavise.be 23
  24. 24. Other object detection algorithms • Dollár– Integral channel features www.eavise.be 24
  25. 25. Example autonomous UAV projects
  26. 26. • A UAV that flies autonomously through the tree corridors of an ochard • Inspection of o Number of flowers o Fruit quantity o Fruite ripeness o Diseases o … • Why a UAV? o No problem with uneven ground o Greater speed possible than rover • Drone under development by Jon Verbeke, UAV research, KU Leuven Campus Oostende • Image processing done by EAVISE UAV inspectorExample project #1: Autonomous orchard inspection www.eavise.be
  27. 27. UAV inspector Test UAV: AR Drone Camera verticaal Ultrasone sensoren Camera horizontaal • Processor: ARM9, 468MHz + DCT module • MDDR RAM 128MB • Camera hor: 640 x 480, 93° • Camera vert: 176 x 144, 64° • 3 assige accelerometer • 3 assige gyroscoop • Wifi module www.eavise.be
  28. 28. UAV inspector Vision algorithm Visie Roll setpoint Yaw setpoint Camerabeeld MG-punt = Roll setpoint Vanishing point = Yaw setpoint www.eavise.be
  29. 29. UAV inspector Vision algorithm www.eavise.be
  30. 30. UAV inspector Vision algorithm Good Threshold Bad Threshold Automatische threshold: • Score berekenen voor beste thresholdwaarde • In het begin van de vlucht • Tijdens de vlucht www.eavise.be
  31. 31. Orchard monitoring results www.eavise.be
  32. 32. Example project #2: Cametron Flying autonomous camera crew a system that, with minimal human intervention, produces high quality audiovisual (AV) productions www.eavise.be 32
  33. 33. = a system that, with minimal human intervention, produces high quality audiovisual (AV) productions o a virtual camera crew o coordinated by a virtual director o with postprocessing by a virtual editor o following cinematographic rules Cametron www.eavise.be 33
  34. 34. • Autonomous UAV • Real-time persoon detectie and –tracking on embedded hardware • Topological localisation • Visual servoing Cametron virtual camera man Left of axis Drone A Drone B Right of axis Action axis Actor A Actor B www.eavise.be 34
  35. 35. Cametron realisations • 3D localisation system, for visual localisation of the UAV in a room • Person detection and tracking framework with PTZ camera • Selection tool of embedded processing unit for UAV www.eavise.be 35
  36. 36. Detecting a person • Viola and Jones (Haar/Adaboost) • HOG • ICF • DPM Person detection and tracking framework with PTZ camera www.eavise.be 36
  37. 37. Detecting persons Person detection and tracking framework with PTZ camera V&J HOG full body model ICF upper body model DPM upper body model ICF full body model ICF face model Choice of detector depending in desired shot typr Detection of actor www.eavise.be 37
  38. 38. • Rules making it more enjoyable looking at the video material o Nose room o Head room o Rule of thirds Cinematographic rules www.eavise.be 38
  39. 39. Detecting gaze direction Face detection Frontal model Face detection Left looking model Face detection Right looking model Determination of gaze direction 0,8 5 0,1 Positioning of person on horizontal axis www.eavise.be 39
  40. 40. Difficult because camera is moving Detection of direction of movement Determination of movement direction Positioning of person on horizontal axis Vp > 0 Vp < 0 www.eavise.be 40
  41. 41. Results on PTZ camera system Person detection Determination of gaze or movement direction Apply cinematographic rules Person tracking PID control www.eavise.be 41
  42. 42. • In literature: Processing done mostly offline • Which hardwareplatform is best suited to run a certain image processing task on board a UAV? Selection tool for embedded processing unit for complex on-board UAV computer vision www.eavise.be 42
  43. 43. • Key criterium: maximizing endurance • Embedded processing board o has certain weight, to be carried as payload o consumes electrical power from on-board battery Selection tool for embedded processing unit for complex on-board UAV computer vision Moeilijkheid algoritme • Fps simpel alg • Fps moeilijk alg • Fps nieuw alg Verbruik van elk bordje• fps nodig • Propeller specs Motor efficiëntie Vliegtijd per bordjeVliegtijd schatter • Gewicht elk bordje • Batterij Welke bordjes kunnen alg draaien www.eavise.be 43
  44. 44. Selection tool for embedded processing unit for complex on-board UAV computer vision processor memory Power (Watt) Weight (gram) NUC Intel I5 dual core 8G exp to 16G 20,1 550 ITX Intel I7 quad core 16G 68 684 Jetson ARM quad core A15 2G exp to 16G 12,5 185 ATOM Intel I5 dual 8G exp to 16G 23,5 427 RPI ARM11 512Mb 3,6 69 XU3 SD card Cortex A15 quad core 2G 11 70 U3 SD card ARM processor 2G 6,7 52 brix Intel I7 quad core 8G exp to 16 26 172 www.eavise.be 44
  45. 45. First experiments 2 benchmark algorithms on each platform www.eavise.be 45
  46. 46. First cametron results on UAV • See our demo at DroneCon • Tracking an actor from a flying UAV while automatically keeping it correctly framed www.eavise.be 46
  47. 47. Conclusions • Future of UAVs lie in fully autonomous applications • Camera is a very powerful and lightweight sensor • Image processing algorithms have huge potential o 3D SLAM o Object/person detection and tracking • Complex processing can be done on-board with recent hardware www.eavise.be 47
  48. 48. More information? • Autonomous orchard inspection [Dries Hulens, Maarten Vandersteegen and Toon Goedemé, UAV autonoom laten vliegen door een boomgaard, master’s thesis, KU Leuven Campus De Nayer, 2012] • Automatic lecturer recording with a PTZ camera while obeying cinematographic rules [Dries Hulens, Tom Rumes and Toon Goedemé, Autonomous lecture recording with a PTZ camera while complying with cinematographic rules, CRV 2014, the Eleventh Conference on Computer and Robot Vision, Montréal, Quebec, May 7-9, 2014.] • Selection tool for embedded platforms for on-board a UAV computer vision processing [Dries Hulens, Jon Verbeke and Toon Goedemé, How to choose the best embedded processing platform for on-board UAV image processing?,10th International Conference on Computer Vision Theory and Applications, VISAPP 2015, Berlin, Germany, March 11-14, 2015] • On-board real-time tracking of pedestrians on a UAV [Floris De Smedt, Dries Hulens and Toon Goedemé, On-board real-time tracking of pedestrians on a UAV, EVW workshop co-located with CVPR 2015, Boston USA, 2015.] • These papers and much more on www.eavise.be www.eavise.be 48

×