Vision-Based Obstacle Recognition and Autonomous Flights through Gate Structures with an Unmanned Helicopter

Vision-Based Obstacle Recognition and Autonomous
Flights through Gate Structures with an Unmanned
Helicopter
Florian Adolf, Franz Andert, Lukas Goormann, Jörg Dittrich
Institute of Flight Systems, Dept. Unmanned Aircraft, DLR Braunschweig
AHS 65th Annual Forum, Grapevine TX
Paper No. 080406

Florian Adolf > Unmanned Aircraft > Institute of Flight Systems > German Aerospace Center (DLR) > 29th May.2009
AHS 65th Annual Forum & Technology Display > Slide 2
Test site “Rosenkrug”
Motivation
Flying VTOL UAVs in urban environments is of high interest, e.g. for
surveillance and reconnaissance missions
Images: Courtesy of AutoFlug
UAVs usually rely on the full situational awareness of remote operator:
Start
Goal

Challenges
Problems:
Available GPS accuracy of ~6 m vs. <6 m wide gates
Data link quality and robustness for video-based remote control
Given:
• Expected GPS position of
narrow passage (e.g. entrance)
• Expected fly-through direction
Autonomous flight through narrow passages
?
?
?
Desired:
• Center position of narrow passage
• “Best” direction of flight through passage

Approach: Vision-based Autonomy
Waypoint navigation towards narrow passage
At goal location:
Passage detection and GPS-
referencing
Measurement corrects mission plan
Start
Gate
GPS-referenced map
of known environent
Expected position
and direction of
narrow passage
Path planned by
operator before
mission start
Essential requirements:
Onboard gate detection in real-time
Automatic onboard replanning
“Hands-Off” mission execution:
(e.g. from take-off to successful flight through
narrow passage)

The VTOL UAV Systems
ARTIS UAV Family
Magnetometer
Power Supply
IMUGPS
Telemetry
Flight Control
Camera
Computer Vision
Sonar

Example Scenario
Based on EMAV’08 outdoor dynamic flight competition1)
Given:
Gate GPS position and
expected orientation
Gate appearance (shape,
size etc.)
1) www.dgon.de/emav2008
Example gate, 6m x 6m on EMAV 2008

Gate Recognition
Tests with monocular color camera (Imaging Source DFx 31F03)
1024x768 pixels; 24 bit color; max. 30 fps; 60°x48° FOV
Color segmentation
(HSV)
 binary image
Blob detection
(white pixel clusters)
 List of objects
Blob filtering
(w.r.t. size and shape)

Gate Localization
Necessary for mission planning:
Based on new measurements, generate corrected waypoints online
zC
yf
zf
xf
xC
yCTransform detected
gate center into UAV‘s
reference frame
Compensate measurement errors:
Image processing
Camera alignment
Vehicle state estimates
e.g. Vehicle heading estimation error
~4° at 20m distance in front of gate
→ 1.40m gate center displacement!
Reference frame center
not precisely known w.r.t.
to camera position.

Gate Localization (cont.)
“Target fitting algorithm”
Corrects camera vs
state estimates reference frame
Iterative procedure beginning
with initial camera alignment
Minimizes target
deviation from multiple images
using multivariate Newton-
Raphson approximation
Best setup: Optimize camera
heading and pitch only;
no roll angle or translation.
viewed
gate posts
view with updated
camera alignment
flight
trajectory
flight
trajectory

Flight Mission Planning
Requirements for autonomous flight:
Precise vehicle control
Waypoint navigation
Image processing & gate estimation
as presented
Mission planning as follows:
Initial flight towards gate
approx. position given
If gate found, generate
waypoints crossing the gate
Waypoint-based flight
Consider tasks if gate search
was not successful,
e.g. fly additional attempts
or abort mission
a-priori gate knowledge
estimated true gate position
vehicle
position
waypoints to
fly through gate
initial
waypoints
p3
p4
p1 p2

Reactive Layer: Basic Movement Capabilities
Path Command
Sequencing Layer: Compile behaviors into plans
Hover Hover To Hover Hover ToTake Off Land
Gate Detection and Calibration
Mission Time
t0 t1 t2a t3 t5 t6 t7t2b t4bt4a
Flight Controller
Deliberate Layer
Take off Hover To Land
Onboard High-Level Control Architecture
Gate Passing
Gate Mission Behavior
Parameters position
and heading are
unknown
Sequence of behavior commands
ID 20070510
TO -5
...
HV 37.1415 48.688 -10 90
HV 37.1415 48.688 -10 180
HV 31.81 48.688 -10 180
HV 26.4785 1.63484 -10 180
HV 26.4785 1.63484 -10 90
HV 26.4785 48.688 -10 90
HV 26.4785 48.688 -10 180
HV 21.147 48.688 -10 180
...
LD
WO
REACTIVE PLANNING [Jensen 1989]
Rune M. Jensen and Manuela M. Veloso.
Interleaving deliberative and reactive planning in dynamic multi-agent domains.
In Proceedings of the AAAI Fall Symposium on on Integrated Planning for
Autonomous Agent Architectures. AAAI Press, October 1998.

Three phases:
Approach a-priori estimated start position
(Re)detect gate until detection confidence is high
& continously adapt linear trajectory through gate point
Fly through gate
believed center
position and
orientation
Gate poleStart
UAV
End
UAV
UAV
Approaching
Gate (Re)detection
Fly Through
detected
center position
and orientation
Gate Passing Behavior

Mission Execution
Gate Mission Behavior
Objective:
Update position using measured gate position
Avoid collision with gate poles while flying through gate
TO z
GM xgate ygate psigate
… Gate not precisely detected
time
HV xs1 ys1 zs1 psigate
HV xs2 ys2 zs2 psigate
Calibration
HV xs1‘ ys1‘ zs1‘ psigate‘
HV xs2‘ ys2‘ zs2‘ psigate‘
Re-calibration
HV xs3‘‘ ys3‘‘ zs3‘‘ psigate‘‘
…
Gate detected
Fly through
Expected gate
position and fly-
through direction
Measured gate
position and fly-
through direction
Based on
expected
position
Based either on
expectation or
measurement

Mission Planning
Planning steps before the mission start:
Load geo-referenced environment
Setup estimated gate using GPS
coordinates
Automated path planner generates
mission plan for waypoint navigationid 1242048571
COORDINATES: Local_Cartesian
DATUM: WGE
#_ELLIPSOID: WE
ORIGIN_LATITUDE: 52.41999
ORIGIN_LONGITUDE: 10.438
ORIGIN_HEIGHT: 63.9
ORIENTATION: 0.00000
TO -3
GM 50 100 63.43495
Translate mission goal into command(s)

Flight Test Results (1)
Test of Integrated System
• Passing gate at 1.5 m/s
• Gate localization at 15 Hz

-10 0 10 20 30 40 50 60
-30
-20
-10
0
10
20
-10 0 10 20 30 40 50 60
-30
-20
-10
0
10
20
x[m]
y [m]
x[m]
y [m]
initial flight
gate search
gate crossing
gate posts
generated
waypoints
gate
search
path correction
switch to manual flight
(a) (b)
Initial waypoint accurately known (a)
Initial waypoint translated sideeways ≈4m (b)
Error cases in expected gate location

-10 0 10 20 30 40 50 60
-30
-20
-10
0
10
20
-10 0 10 20 30 40 50 60
-30
-20
-10
0
10
20
x[m]
y [m]
x[m]
y [m]
(c) (d)
path
correction
Initial waypoint rotated ≈20° (c)
Initial waypoint translated sideways by ≈3m + rotation ≈20° (d)
Gate center well corrected,
orientation error hardly
compensated

-10 0 10 20 30 40 50 60
-30
-20
-10
0
10
20
-10 0 10 20 30 40 50 60
-30
-20
-10
0
10
20
x[m]
y [m]
x[m]
y [m]
emergency switch
to manual flight
no detection and
additional search
(e) (f)
Failure: wrong gate estimation; manual interruption (e)
Failure: no gate estimated; successful automatic interruption (f)
Recording a video dropped frame rate:
 too few images and/or distance
between each image too large
Reason: Gate markers
were not within camera
FOV in most images

Test results from 16 flights:
13 successful gate passes
2 successful automatic mission
interruptions due to no gate
detection
1 time: false estimation (too
high), safety pilot aborted
No UAV crash
Summary

Summary
Onboard reactive planning to account for gate position updates
No „generic“ approach:
Knowledge about gate dimensions required
Independent of remote control data link
Real-time gate detection
Complex mission is easy to specify
+
-
Future Work:
Additional sensors (stereo, laser scanner, etc.)
Arbitrary gates with less a-priori knowledge
Feasibility of overall approach shown in flight tests
Target fitting algorithm:
Improve manual camera calibration
Useful if initial alignment is not accurate or may drift

Questions?
Thank you for your attention!

BACKUP

EMAV 2008 competition setup
“Outdoor Dynamic Flight Competition”

Gate Mission
believed center
position and
orientation
detected
center position
and orientation
Arch poleStart
UAV
REACTIVE PLANNING [Jensen 1989]
Rune M. Jensen and Manuela M. Veloso.
Interleaving deliberative and reactive planning in dynamic
multi-agent domains.
In Proceedings of the AAAI Fall Symposium on on
Integrated Planning for Autonomous Agent Architectures.
AAAI Press, October 1998.

Experiment
Scenario Test

Related Work
“Fibre Optic Transmission System for Micro UAVs“ [Schulze 2007]
Study investigated fibre optic data links for robust high bandwidth UAV
control
Result: Not jammable, reliable data link
Pros:
High bandwith link
Manual remote control possible
Cons:
Limited to 1000m in trajectory length
Additional weight of cable and transmission electronic
Hard to recover after mission (forests, power lines etc)
What happens if cable breaks?
Hardware solution, low level of system autonomy

Related Work
AI Concepts for “Assistive Intelligence“1
System good and adaptive at particular tasks
“Intelligence without Reason“
and “Intelligence without Representation“ [Brooks 1991]
Bottom-Up approach (“behavior-based AI“)
Simple, task oriented modules solve problems
without explicit world model
1) Nigel Shadbolt, British Computer Society President and ECS Professor of Artificial Intelligence
Pros:
Avoids explicit notion of uncertainty in future environment states
Applications to domain of (semi-)autonomous UAV missions exist
Cons:
Direct links to sensors imply reliable sensor information

Approach
Overview
Given: Common components available on most modern UAVs:
Offline mission planning
Online, onboard waypoint navigation
Rapid onboard image processing (e.g. Blob-Tracking)
Increase UAV Autonomy:
Approach “rough“ target gate (e.g. GPS position)
Localize exact target gate location
Fly through gate, no collsion!
Use Behavior-based AI
Make “gate sensor” and way it’s used robust

Behavior State Chart
Init
StandBy
GateMissionInit
GateMissionStart
GateMissionStandBy
GateMissionStop
GateMissionTerminate
VisionServiceMsg sent
InitServiceCommunication
done
acknowledge received &&
VisionServiceMsg sent
acknowledge received
sent Msg == Stop
no Communication
no Communication
no Communication
no Communication
sent Msg == Terminate
sent Msg == Start
no Communication
no Communication
CommunicationVisService BehaviorGateMission
Init
CalibrateStart
CalibrateStop
FlyThroughStart
FlyThroughStop
FlyThroughFinalClimb
Terminate
S_2_i + heading reached
&& VisionServiceStandBy
S_2 + heading reached
&& VisionServiceInit done
&& VisionServiceConfidence ok
S_f_i reached
StopRequest ||
DirectCommand
StopRequest ||
DirectCommand
StopRequest ||
DirectCommand
StopRequest ||
DirectCommand
StopRequest ||
DirectCommand
StopRequest ||
DirectCommand
StopRequest ||
DirectCommand

Image Processing Algorithm (2)
Camera calibration:
find camera parameters
(image center, focal length, distortion)
Use chessboard calibration pattern
Calibration allows usage
of simple pinhole-camera model
Camera resectioning:
estimate rotation and translation
between gate and camera
Known gate markers
Estimator based on perspective
transformations
x
y
z
p6
x
y
z

zC
yf
zf
xf
xC
yC
Gate Position Estimation (1)
“Classic Approach”
Transform camera-relative gate position into global position
Requirements:
Camera alignment on vehicle
Global vehicle position
Global gate position can be estimated, but:
Errors:
Image processing
Camera alignment
Vehicle state
Better solution: estimate biases using multiple images

“Trajectory fitting algorithm”
General idea: vehicle moves during gate approach
Measured with GPS/INS (navigation coordinate system)
Measured with image sequences (camera-relative gate position moves)
Both is the same movement! (trajectories are rotated and translated)
This redundancy is used to minimize alignment errors
Solution: try to fit both navigation-based and image-based trajectories,
Estimate rotation and translation between both
This results in transformation between nav. and image coordinates

Fits image-based and navigation-based trajectories
Based on least-squares fitting algorithm
Result: optimized gate orientation as flight target
g1
h1
R t,
R t,
hn gn
viewed
gate posts
gate center/
direction
flight target/
heading

Problem: „straight“ trajectories (foremost: roll angle)
Solution with additional points based on attitude angles
R t, R t,
g1
h1
h4
g4
h1
g1
h20
g20
flight
trajectory
(3-DOF)
image-
based
positions
(3-DOF)
flight
trajectory
(6-DOF)
image-
based
orientation
(6-DOF)

General idea: gate position is constant while vehicle moves
Measured gate position will move if transformations are wrong
Assume correct camera resectioning
With correct alignment: estimated gate position would not move
(only influenced by noise)
After taking a few images:
Compare estimated gate positions based on single images
Estimate alignment where these positions match
Result: better alignment
Use this new alignment to get gate position

Estimate “best” camera
alignment
Iterative procedure beginning
with initial camera alignment
Minimizing target
deviation from multiple images
using multivariate Newton-
Raphson approximation
Best results when optimizing
only camera heading and pitch;
no roll angle or translation
R thc hc,
viewed
gate posts
wiew with updated
camera alignment
new R thc hc,
flight
trajectory
flight
trajectory

Gate Position Estimation
Classic approach:
Error of less than 40 centimeters
May be sufficient if hand-measured alignment is accurate
Trajectory fitting algorithm:
Risk of high variations; very dependent on flown trajectory
Target fitting algorithm:
Improves hand-measured camera alignment
Useful if initial alignment is not accurate or may drift
Target Fitting algorithm used in autonomous flight tests

Manual Flight Test Results (1)
-20 0 20 40 60 80 100
-40
-30
-20
-10
0
10
20
30
40
50
60
1st gate
2nd gate
flight path
images taken
... ...
y [m]
x[m]
classic approach
-20 0 20 40 60 80 100
-40
-30
-20
-10
0
10
20
30
40
50
60
x[m]
y [m]
trajectory fitting algorithm
target fitting algorithm
Comparison of gate position estimations with manual flight tests
Classic algorithm
Trajectory fitting
Target fitting
Reference: flown trajectory

46 47 48 49 50 51 52
15
15.5
16
16.5
17
17.5
18
18.5
zoom: 1st gate
y [m]
x[m]
0 1 2 3 4 5
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
zoom: 2nd gate
y [m]
x[m]
Estimated gate positions (zoom)
Classic algorithm
Trajectory fitting
Target fitting
Reference: flown trajectory

0 50 100 150 200 250
0
5
10
15
20
25
error[m]
image frames
1st gate
0 50 100 150 200 250
0
5
10
15
20
25
image frames
error[m]
2nd gate
Errors (distances) between estimated gate positions and flown trajectory
Classic algorithm
Trajectory fitting
Target fitting

0 50 100 150 200
230
240
250
260
image frames
Ψ[deg];atan2(y,x)[deg]
..
heading Ψ
flight path azimuth
0 50 100 150 200 250
-10
-5
0
5
10
15
20
25
height of 1st gate
height of 2nd gate
z[m]
image frames
Estimated gate height vs. flown height
Estimated gate direction vs. flight direction and vehicle heading

Vision-Based Obstacle Recognition and Autonomous Flights through Gate Structures with an Unmanned Helicopter

Recommended

Recommended

More Related Content

What's hot

What's hot (11)

Similar to Vision-Based Obstacle Recognition and Autonomous Flights through Gate Structures with an Unmanned Helicopter

Similar to Vision-Based Obstacle Recognition and Autonomous Flights through Gate Structures with an Unmanned Helicopter (20)

More from Florian-Michael Adolf

More from Florian-Michael Adolf (7)

Recently uploaded

Recently uploaded (20)

Vision-Based Obstacle Recognition and Autonomous Flights through Gate Structures with an Unmanned Helicopter