Machine Vision for Intelligent Drones: An Overview

MACHINE VISION FOR INTELLIGENT
DRONES: AN OVERVIEW
APRIL 27TH 2016
KEVIN HEFFNER SAMUEL FOUCHER
RESEARCH ASSOCIATE, CRIM DIRECTEUR, VISION AND IMAGING TEAM
PEGASUS RESEARCH & TECHNOLOGIES CRIM

2
Outline
Introduction to UAVs
Intelligent Drones
Machine Vision Technologies
Example Applications
Trends and Challenges

3
noun: drone; plural noun: drones
a male bee in a colony of social bees, which does no work but can fertilize a queen.
Source: Steven Zaloga, Unmanned Aerial Vehicles: Robotic Air Warfare 1917-2007, Osprey Publishing, 2008
In 1935, U.S. Adm. William H. Standley saw a British demonstration of the Royal
Navy's DH 82B Queen Bee.
Commander D Fahrney was tasked to develop something similar for the US Navy.
"Fahrney adopted the name ‘drone’ in homage to the Queen Bee”
DRONE TERMINOLOGY

4
TYPES OF DRONES BY SIZE
UAV ACRONYMS
Unmanned Aerial Vehicle (UAV)
Micro Aerial Vehicle (MAV)
Miniature Aerial Vehicle (MAV)
Small UAS (SUAS)
Medium Altitude Long Endurance (MALE)
High Altitude Long Endurance (HALE)
Remotely Piloted Aircraft Systems (RPAS)

5
Range (miles)
Flightduration(hours)
MaxOperatingAltitude(feet)
1 10 100 500 1K 10K
64
32
16
8
4
2
1
40K
20K
10K
5K
1K
Maximum Weight (lbs)
1 10 50 100 1K 2K 5K 10K 20K
UAV CLASSIFICATIONS
HOW FAR, HOW HIGH, HOW LONG, HOW HEAVY ?

6
Range (miles)
Flightduration(hours)
MaxOperatingAltitude(feet)
1 10 100 500 1K 10K
64
32
16
8
4
2
1
40K
20K
10K
5K
1K
Maximum Weight (lbs)
1 10 50 100 1K 2K 5K 10K 20K
UAV CLASSIFICATIONS: SOME EXAMPLES

7
The majority of private sector applications utilize drones as sensing platforms
FUNCTIONS AND MISSIONS
Site inspection (sensing)
Energy sector (sensing)
Geomatics (sensing)
Firefighting (sensing)
Search & Rescue (sensing)
Surveillance (sensing)
Agriculture (sensing)
Agriculture (intervention)
Transport (packages)
Recreational
PRIVATE SECTOR
Intelligence gathering
Weather
Weapons
Search & Rescue
Medical Evacuation
Communications
Transport
Target practice
Decoy
MILITARY

9
Which rules apply to me ?
http://www.tc.gc.ca/media/documents/ca-standards/Info_graphic_-
_Flying_an_umanned_aircraft_-_Find_out_if_you_need_permission_from_TC.pdf
START
NO
I have read and can meet
the exemption conditions
for UAVs < 2 kg
I have read and can meet
the exemption conditions
for UAVs from2 to 25 kg
You don’t
need permission,
but you must meet
the exemption
conditions
You must
apply for a
Special Flight
Operations
Certificate
You don’t
need permission, but
you do have to fly
safely
You don’t
need permission, but you
must meet the exemption
conditions AND notify
Transport Canada with:
1.Contact info
2.UAV model
3.Description of operation
4.Geographic boundaries
of operation
NO
It weighs > 35 Kg
NO
I use my aircraft for
WORK or RESEARCH
YES
YES NO
It weighs < 2 Kg
NO YES YESNO
YES YES
It weighs > 25 Kg
CANADIAN UAV REGULATION

10
http://www.tc.gc.ca/eng/civilaviation/standards/standards-4179.html
• Fly your drone during daylight and in good weather (not in clouds or fog).
• Keep your drone in sight, where you can see it with your own eyes – not only
through an on-board camera, monitor or SmartPhone.
• Make sure your drone is safe for flight before take-off. Ask yourself, for
example, are the batteries fully charged? Is it too cold to fly?
• Know if you need to apply for a Special Flight Operations Certificate
• Respect the privacy of others – avoid flying over private property or taking
photos or videos without permission.
• Closer than 9 km from any airport, heliport, or aerodrome.
• Higher than 90 metres above the ground.
• Closer than 150 metres from people, animals, buildings, structures, or vehicles.
• In populated areas or near large groups of people, including sporting events,
concerts, festivals, and firework shows.
• Near moving vehicles, highways, bridges, busy streets or anywhere you could
endanger or distract drivers.
• Within restricted and controlled airspace, including near or over military bases,
prisons, and forest fires.
• Anywhere you may interfere with first responders.
Do
Don’t fly

11
Currently there is no requirement for a Pilot or Operator License.
Transport Canada has communicated their intent to institute new
regulatory requirements that will address licensing and training of
drone operators for drones up to 25 kg.
These requirements also establish how the UAV should be marked
and registered.

13
DRONE EDUCATION
1. Make drones easier and safer to operate
2. To increase their autonomy and decrease human supervisory
control requirements
3. So that they can perform complex functions
4. So they can collaborate with other systems (including drones)
Why do we want drones to become more intelligent ?
Making decisions
Reasoning
Prioritizing tasks
Detecting differences
Finding similarities
Learning from experience
Characteristics of Intelligence

14
*See Ingrand and Ghallab - http://homepages.laas.fr/felix/publis-pdf/aicom13.pdf
Cognitive Robotics – Breakdown of intelligence into functions
ROBOT INTELLIGENCE
Planning off-line prediction of feasible actions;
Acting task execution; refines planned actions;
Observing detects and recognizes;
Monitoring comparison of predicted versus observed;
Goal reasoning monitoring at mission level;
Learning acquire, adapt and improve through experience;
DELIBERATION FUNCTIONS*

15
ENVIRONMENT
Task Execution
Low-level instructions
Hardware Interface
Robot Platform
Sensory
Functions
Motor
Functions
ENVIRONMENT
Deliberation
Robot Platform
Task Execution
Hardware Interface
Sensory
Functions
Motor
Functions
Robot Architecture Cognitive Robot Architecture
ROBOT ARCHITECTURES

16
ENVIRONMENT
Deliberation
Robot Platform
Task Execution
Hardware Interface
Sensory
Functions
Motor
Functions
Symbolic
reasoning
Sub-symbolic
processing
Adjust speed to 40 knots.
Change heading to 75 deg.
…
Follow route A.
Return to base.
…
COGNITIVE ROBOT ARCHITECTURES

17
ENVIRONMENT
Deliberation
Robot Platform
Task Execution
Hardware Interface
Symbolic
reasoning
Sub-symbolic
processing
Adjust speed to 40 knots.
Change heading to 75 deg.
…
Follow route A.
Return to base.
…
COGNITIVE ROBOT ABSTRACT MACHINE
CRAM Kernel
ROS Interface
Computable
Predicates
Perception Navigation Manipulation Code OWL
LISP
SWI Prolog
CPL - CRAM Plans
Belief State
- Object designators
- Entity designators
- KnowRob knowledge base
KnowRob Reasoner
CPL
Extension
Modules
KnowRob
Extension
Modules

18
INTELLIGENT DRONE APPLICATION DEVELOPMENT

19
Machine Vision Technologies

20
COMPUTER VISION
 Object or place recognition
 Visual odometry
 3D reconstruction
 Target tracking
 Anomaly detection
Many applications
 OpenCV
 PCL
 ArrayFire
 …
Open-source libraries
The main goal of Computer Vision is to enable a computer to analyze, process
and understand one or more images taken by a vision system.
Development of MV Applications has exploded recently thanks to the advent of
high-res digital cameras along with increased computing power & machine learning.

21
PLATFORM
Ubiquity of low-
cost prosumer
drones
SENSORS
Smarter, Smaller,
Integrated
Processors
PERFECT STORM OF TECHNOLOGIES
 Size
 Weight
and
 Power
 Price
(low) SWaPP
COMPUTING
RESOURCES
Embedded, cloud
Machine Vision
Libraries

22
ON-BOARD COMPUTER VISION
 The combination of computer vision and IMU data (altitude, acceleration, etc.)
can improve the precision and robustness of drone navigation.
 Vision-based solutions are interesting for small drones (< 2.0 kg) for indoor
navigation, obstacle avoidance and other problems.
 Navigation
 Path Planning
 Localization
 Geo-fencing
 Environment
 GPS-denied
 Obstacle avoidance
 Swarming
 Automatic Landing and takeoff
 Toward increasing autonomy
Mission critical aspects
 Environment mapping
 Object detection
 Anomaly detection
 Video enhancement:
 De-hazing
 Image stabilization
 Super-resolution
 Contrast enhancement
 Data compression
Application related processing

23
SIMULTANEOUS LOCALIZATION AND MAPPING (SLAM)
Example of an application for an on-board vision system
 Precise drone localization
 Environment mapping around the drone
 Sensing with a sonar, a laser or a camera
 Object and place recognition
Problems to solve
Motion
Initial
position
prediction
Image
Keypoint
Extraction
Matching
Revised
Prediction

24
KEYPOINTS DETECTION
 Robustness
 Appearance (illumination, occlusion, etc.)
 Viewpoint
 Local and distinct description
 Quantity
 Low computation cost
Ideal properties
Keypoints are sailient points in an image
 Harris
 SURF
 SIFT
 ORBE
 FAST
Many different techniques
Detection
Description
Matching

25
VISUAL SLAM
Camera + SLAM = monocular visual SLAM (vSLAM) exploits visual information
(keypoints)
RGB-D or stereo cameras directly minimize the photometric error
Matching between successive images Mapping

26
VISUAL SLAM FOR GPS-DENIED ENVIRONMENTS
https://www.youtube.com/watch?v=VWWvjSHZCNo
 High precision intertial navigation
units are costly and heavy
 Laser range finders (costly and
heavy)
 Cameras (visual odometry)
Possible solutions
 Precision: +/- 2.5m
 Not accurate in urban environment
 Not possible to use indoors
GPS navigation limitations

27
Machine Vision Applications
for
Intelligent Drones

28
A LIST OF DRONE APPLICATIONS
Analysis of first 3136 exemption requests
authorized by the FAA for small drones (<25 Kg)

29
NATURAL RESOURCE MANAGEMENT
RESEARCH APPLICATIONS
Source: Shahbazi, M., Théau, J. et P. Ménard (2014) Recent applications of unmanned aerial imagery in natural
resource management. GIScience & Remote Sensing 51
Based on review of 150 articles about the use of UAV imagery

30
 Improve water use
 Pest management
 Weed-identification
 Irrigation and water distribution
 Manned aircraft: ~$1,000/hour
 Satellite Imagery: $$$
 Multispectral sensors:
• Visible (R,G,B)
• Near-Infrared (NIR)
• Thermal Infrared (TIR)
• Resolution < 1 cm, > 12 Mpixels
 Hyperspectral: 640 bands
 LIDAR
AGRICULTURE APPLICATIONS

31
PRECISION AGRICULTURE
Source: http://www.pole-pegase.com/documents/Actualites/6_-_presentation_Exametrics.pdf
Source : http://www.icv.fr/conseil-viticulture-oenologie/oenoview
Resolution(m)
10
1
0.1
0.01
0.001
$$$
$$
$

32
AGRICULTURE APPLICATIONS
 Information gathering
 NDVI
 Multispectral
 Hyperspectral
 Intervention
 Irrigation control
 Spraying
 Harvesting
AGRIBOTICS APPICATIONS

33
GLACIER MASS BALANCE APPLICATION
Source: Christophe Kinnard, Atelier sur la sécurité civile, AQT’2015
Orthomosaic (10 cm) + MNE
~US$10,600

34
ENERGY SECTOR INSPECTION & SITE SECURITY
Crack detection in dams
and reactors
A security drone that will
chase down intruders.

35
COFFEE BREAK DETECTION APPLICATION

36
HOTSPOT DETECTION USING OPTIMAL SEARCH PATTERN
Pegasus Research and Technologies Research Project
 Rapid deployment
 Low-cost lightweight IR sensors
 Optimized search path
 Applications for
 For missing persons
 Remnant forest fire detection
 Security applications
HOTSPOT DETECTION

37
OPEN-FIELD HOTSPOT DETECTION TRIAL
FOR LOCALIZED SEARCH AND RESCUE
1. Determine presence and approximate
location of candidate “missing person”.
2. High resolution camera is then pointed
at location and fed to SAR team.
3. System queries operator for next step
4. Future capability for automatic object
recognition.
HOTSPOT DETECTION

38
SEMI-AUTOMATED ROOF INSPECTION APPLICATION
Pegasus Research and Technologies Research Project

39
ROOF INSPECTION APPLICATION
1. Automated data collection
a. 3D Model construction images
b. High-resolution texture images
2. 3D Model construction
3. Roof extraction and zone detection
4. High-resolution texture generation
5. Inspection annotations
6. Report generation
ROOF INSPECTION PIPELINE

40
(define-policy start-execution()
"MISSION STARTED ?"
(:init (format t "Initializing policy to start the execution ~%") t)
(:check (setq execution_param (roslisp:get-param "/execute_mission"))
(print execution_param)
(cond ((eql execution_param 1))))
(:recover (format t "Running recovery mechanisms ~%"))
(:clean-up (format t "Running clean up ~%")))
(def-top-level-cram-function execute_misson ()
"MISSION EXECUTION"
(roslisp:start-ros-node "execute_mission")
(mav:init-ros-mav "hummingbird")
(roslisp:set-param "/execute_mission" 0)
(roslisp:set-param "/pause_execution" 0)
(roslisp:set-param "/roof_detected" 0)
(setf foo 1)
(setf counter 1)
(with-failure-handling
((policy-check-condition-met (f)
(print "starting execution") (return)))
(with-named-policy 'start-execution ()
(loop while (= counter 1)
do(print "waiting for execution")
(setq counter 1)
(sleep 3))))
(with-failure-handling
((policy-check-condition-met (f)
(print "test condition met")
(return)))
(with-named-policy 'my-policy ()
(defparameter *transform-listener* (make-instance 'cl-tf:transform-listener))
(sleep 5)
(setq trans (cl-tf:lookup-transform *transform-listener* :source-frame "/hummingbird/ground_truth" :target-frame "/world"))
AUTOMATED IMAGE COLLECTION USING DELIBERATION

44
ROOF ZONE DETECTION AND MEASUREMENT

45
HIGH RESOLUTION IMAGES FOR DETAILED INSPECTION

46
1
2
ANNOTATIONS AND REPORT GENERATION

48
Source: Graves Presentation: NASA Aeronautics Strategic Thrust: Assured Autonomy for Aviation Transformation, Vision & Roadmap, March 9th 2016
TRENDS AND CHALLENGES

49
Source: Graves Presentation: NASA Aeronautics Strategic Thrust: Assured Autonomy for Aviation Transformation, Vision & Roadmap, March 9th 2016

50
?
Drone
? ?
?
 Airworthiness considerations
 Platform health monitoring & failure prediction
 Fault detection & failure recovery
 Intelligent operator aids
 Operator training
 Navigation warnings and notifications
 Obstacle avoidance
 Night-time operations
Drone Safety
 Rogue drones
 Drone pirates
 Drone data protection (Cybersecurity)
 Privacy
Drone Security

51
Personal Drones
TRENDS AND CHALLENGES: DRONE HYPE CYCLE ?
Agribotics
Cinema
Geomatics
PublicSafety
EnergySector
SiteInspection

52
Advances in remote sensing, embedded processing and cloud
computing technologies have created opportunities for new
applications.
The availability of low-cost prosumer drones adds another dimension
of opportunity.
Currently most drones are flying sensors.
Adding intelligence to drones can improve safety and allow for drones
to participate in data gathering and processing as part of an
information ecosystem, e.g. Internet of Things (IOT).
Regulation and safety aspects will be main factors in the adoption and
practical use of drones for commercial purposes.
CONCLUSIONS

WWW.CRIM.CA
Suivez-nous
Dialoguez avec nous
Suivez-nous
#CRIM_ca
wwwCRIMca
Tous droits réservés © 2016 CRIM. 405, avenue Ogilvy, bureau 101, Montréal (Québec) H3N 1M3/514 840-1234/1 877 840-2746
Kevin Heffner
Research Associate, CRIM
President, Pegasus Research & Technologies
K.Heffner@peretec.com
Samuel Foucher
Équipe Vision et Imagerie
CRIM – Centre de recherche informatique de Montréal
Samuel.Foucher@crim.ca
Principal partenaire financierLe CRIM est un centre de recherche appliquée en TI qui développe, en mode collaboratif avec ses clients et partenaires, des
technologies innovatrices et du savoir-faire de pointe, et les transfère aux entreprises et aux organismes québécois afin de les
rendre plus productifs et plus compétitifs localement et mondialement. Le CRIM dispose de quatre équipes de recherche en TI
de calibre mondial, d’un centre de tests et d’interopérabilité considéré comme une référence neutre au Québec ainsi qu’un
centre de formations de pointe en TI. Le CRIM œuvre principalement dans les domaines des interactions et interfaces personne-
système, de l’analytique avancée et des architectures et technologies avancées de développement et tests. Détenteur d’une
certification ISO 9001:2008, son action s’inscrit dans les politiques et stratégies pilotées par le ministère de l'Économie, de
l'Innovation et des Exportations (MEIE), son principal partenaire financier.

Machine Vision for Intelligent Drones: An Overview

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Machine Vision for Intelligent Drones: An Overview

Similar to Machine Vision for Intelligent Drones: An Overview (20)

Recently uploaded

Recently uploaded (20)

Machine Vision for Intelligent Drones: An Overview