Using Camera,Extract Image,GPS system

1. Vision – based RunwayVision – based Runway
Recognition foR uaVRecognition foR uaV
autonomous Landingautonomous Landing
Neeraj TiwariNeeraj Tiwari
INDIAN INSTITUTE OF SPACE SCIENCEINDIAN INSTITUTE OF SPACE SCIENCE
AND TECHNOLOGYAND TECHNOLOGY
15May201715May2017

2. objectiVe and goaLobjectiVe and goaL
►Low cost for UAV.Low cost for UAV.
►Light weight.Light weight.
►Unmanned System.Unmanned System.
►Easily Installation.Easily Installation.

3. LandingLanding

4. Human operator
for high level
control
Laptop Computer for
Vision Processing
and Control
Algorithms
Ground Station
bLock diagRambLock diagRam
RC Plane with Camera
30 frames per second
720x480 pixels RGB
Downsample to 360x240
Remote Control
(72 Mhz)
Analog Video
(NTSC 900 Mhz)

5. oVeRViewoVeRView
►System for landing a UAV on a runwaySystem for landing a UAV on a runway
 Small RC airplaneSmall RC airplane
 Only sensor is a fixed, forward-looking cameraOnly sensor is a fixed, forward-looking camera
 Finds the runway using Hough Transform.Finds the runway using Hough Transform.
 Linear control systemLinear control system
►ExperimentsExperiments
 Microsoft Flight Simulator (no flight model)Microsoft Flight Simulator (no flight model)
 Partial implementation on a real UAVPartial implementation on a real UAV

6. aiRPoRt tRackingaiRPoRt tRacking
gPs tRackinggPs tRacking

7. HoRiZon detectionHoRiZon detection
►The horizon profile shape can be used forThe horizon profile shape can be used for
attitude determination and other localizationattitude determination and other localization
process.process.
►This method is based on edge basedThis method is based on edge based
horizon detection or image segmentationhorizon detection or image segmentation
based horizon detection methodbased horizon detection method
►This technique work when vehicle is closeThis technique work when vehicle is close
to the Ground and there is a strong horizonto the Ground and there is a strong horizon
edge.edge.

8. HoRiZon detectionHoRiZon detection
►An edge strength histogram is computedAn edge strength histogram is computed
based on the edge strength produced by thebased on the edge strength produced by the
canny edge detector and the top p% of thecanny edge detector and the top p% of the
points are obtained as possible points thatpoints are obtained as possible points that
comprise the horizon.comprise the horizon.
►The Standard Hough Transform is thenThe Standard Hough Transform is then
applied to fit probable lines in the binaryapplied to fit probable lines in the binary
edge map.edge map.
►The horizon will be the largest edgeThe horizon will be the largest edge
strength.strength.

9. HoRiZon detection matLabHoRiZon detection matLab
outPutoutPut

10. HoRiZon detection matLabHoRiZon detection matLab
outPutoutPut

11. HORIZON DETECTION WITH HIGHHORIZON DETECTION WITH HIGH
PROCESSING SYSTEMPROCESSING SYSTEM

12. MaIN STEPSMaIN STEPS
1.1. Locate the runway in each video frameLocate the runway in each video frame
2.2. Estimate the attitude of the UAVEstimate the attitude of the UAV
3.3. Steer the UAV towards the runwaySteer the UAV towards the runway
maintaining the correct glideslopemaintaining the correct glideslope

13. LOCaTE THE RuNWaYLOCaTE THE RuNWaY
IMAGE
DILATION
IMAGE
THRESHOL
DING
EDGE
DETECTIO
N
SMALL
REGION
RMOVAL
CONVOLUTIO
N
OPERATION
HOUGH
TRANSFOR
M
IDENTIFY PEAK
AND EXTRACT
LINES
RUNWAY
IMAGE
SUPERIMPOS
E LINES ON
THE
ORIGINAL
RUNWAY
IMAGE

14. IMaGE DILaTIONIMaGE DILaTION
►Dilation adds pixels to the boundaries of theDilation adds pixels to the boundaries of the
objects in an image.objects in an image.
►The number of pixels added or removedThe number of pixels added or removed
from the objects in an image depends onfrom the objects in an image depends on
the size and shape of the structuringthe size and shape of the structuring
element used in image process.element used in image process.
►The runway markings are highlighted usingThe runway markings are highlighted using
this technique.this technique.

15. IMaGE DILaTIONIMaGE DILaTION
ORIGINAL IMAGE DILATED IMAGE

16. IMaGE THRESHOLDINGIMaGE THRESHOLDING
►Thresholding is the process of mappingThresholding is the process of mapping
pixels to produce a two level image.pixels to produce a two level image.
►Here OTSU’s thresholding appears moreHere OTSU’s thresholding appears more
appropriate.appropriate.
►OTSU’s method chooses the thresholdOTSU’s method chooses the threshold
value in such a way that the intra classvalue in such a way that the intra class
variance of black and white pixels isvariance of black and white pixels is
minimized.minimized.

17. IMaGE THRESHOLDINGIMaGE THRESHOLDING
GRAY IMAGE OTSU’s THRESHOLDING

18. THRESHOLDINGTHRESHOLDING
GRAY IMAGE LOCAL THRESHOLDING

19. EDGE DETECTIONEDGE DETECTION
►Sobel edge operator is selected for thisSobel edge operator is selected for this
operation because of its optimal solution foroperation because of its optimal solution for
low defective edge rate , localization oflow defective edge rate , localization of
edge and giving one response for singleedge and giving one response for single
edge.edge.

20. EDGE DETECTIONEDGE DETECTION
CANNY OPERATION SOBEL OPERATOR

21. REMOVAL OF SMALL REGIONREMOVAL OF SMALL REGION
MAGE BEFORE SMALL REGION REMOVAL
AFTER SMALL REGION REMOVAL

22. LINE DETECTION TECHNIQUESLINE DETECTION TECHNIQUES
►It used to extract the pair of linesIt used to extract the pair of lines
representing the runway boundaries.representing the runway boundaries.
►Once the runway boundaries are roughlyOnce the runway boundaries are roughly
obtained, straight line detection is to beobtained, straight line detection is to be
performed.performed.
►Image convolution can be used to easilyImage convolution can be used to easily
detect lines, only vertical, 45’ and 135’detect lines, only vertical, 45’ and 135’
masks are sufficient.masks are sufficient.

23. LINE DETECTION TECHNIQUESLINE DETECTION TECHNIQUES
-1 2 -1
-1 2 -1
-1 2 -1
-1 -1 2
-1 2 -1
2 -1 -1
2 -1 -1
-1 2 -1
-1 -1 2
VERTICAL LINES
45’ LINES
135’ LINES
CONVOLUTION MASKS FOR STRAIGHT LINE DETECTION

24. LINE FITTING USING HOUGHLINE FITTING USING HOUGH
TRANSFORMTRANSFORM
►Hough transform is a robust method used toHough transform is a robust method used to
extract arbitrary shapes such as lines ,extract arbitrary shapes such as lines ,
circles, ellipses, out of an image.circles, ellipses, out of an image.
►Straight lines are parameterized in the formStraight lines are parameterized in the form
ρ = x.cos= x.cos ɵ + y.sin+ y.sin ɵ
► In Hough space , lines are mapped to aIn Hough space , lines are mapped to a
point such that a point represents allpoint such that a point represents all
possible lines.possible lines.

25. LINE FITTING USING HOUGHLINE FITTING USING HOUGH
TRANSFORMTRANSFORM

26. STACk OF REFERENCE FRAMESSTACk OF REFERENCE FRAMES

27. ESTIMATE THE UAV ATTITUDEESTIMATE THE UAV ATTITUDE
► 6 Degrees of Freedom6 Degrees of Freedom
 Pitch, Bank, Heading, Elevation, Distance, CoursePitch, Bank, Heading, Elevation, Distance, Course
► StrategyStrategy
 IgnoreIgnore DistanceDistance
 FindFind PitchPitch andand BankBank from the horizon line (x-axis)from the horizon line (x-axis)
 FindFind ElevationElevation,, HeadingHeading,, CourseCourse from the runwayfrom the runway

28. INTUITIVE GEOMETRyINTUITIVE GEOMETRy
►Relationship between runway appearanceRelationship between runway appearance
and UAV attitudeand UAV attitude
 This is how human pilots land visuallyThis is how human pilots land visually
Too High On Target Too Far Right

29. FORMAL GEOMETRyFORMAL GEOMETRy
► 3D Projection3D Projection
 C = Internal CalibrationC = Internal Calibration
 R = External CalibrationR = External Calibration
► Small Angle ApproximationSmall Angle Approximation
 Assume the UAV is flying smooth and levelAssume the UAV is flying smooth and level

30. ESTIMATE THE UAV ATTITUDEESTIMATE THE UAV ATTITUDE
►Recover the orientation parametersRecover the orientation parameters
►Vanishing point of the runwayVanishing point of the runway
►Beginning of the runwayBeginning of the runway

31. Control the UAVControl the UAV
► Cascaded Linear FeedbackCascaded Linear Feedback
ControllerController
 Two separate chainsTwo separate chains
 Two gainsTwo gains
► ProportionalProportional
► IntegralIntegral
► IntuitiveIntuitive
 If UAV is too far right, steer leftIf UAV is too far right, steer left
 If UAV is too high, pitch downIf UAV is too high, pitch down
 Bank angle is derivative ofBank angle is derivative of
heading, heading is derivativeheading, heading is derivative
of courseof course
 Pitch is derivative of elevationPitch is derivative of elevation
PI 1
PI 1
PI 2
PI 3
PI 2
Course
Heading
Bank
Elevation
Pitch

32. AUtopilot GUiAUtopilot GUi

33. AlGorithm performAnCeAlGorithm performAnCe
►Multiple stagesMultiple stages
 Control loops run at 50 HzControl loops run at 50 Hz
►Integrates smoothly even while input stays sameIntegrates smoothly even while input stays same
 Horizon detection runs at 10 HzHorizon detection runs at 10 Hz
►Pitch and bank are the most sensitivePitch and bank are the most sensitive
 Runway detection runs at 2 HzRunway detection runs at 2 Hz
►Elevation and course are the least sensitiveElevation and course are the least sensitive

34. SimUlAtor reSUltSSimUlAtor reSUltS
►Microsoft Flight SimulatorMicrosoft Flight Simulator
►Simulator-in-the-loop (separate computer)Simulator-in-the-loop (separate computer)
 ICRA08_1140_VI_fi.mp4ICRA08_1140_VI_fi.mp4
 Horizon 2007 09 Sep 11 Tue 08.13pm.aviHorizon 2007 09 Sep 11 Tue 08.13pm.avi

35. SimUlAtor reSUltSSimUlAtor reSUltS
► Error from earth curvatureError from earth curvature

36. ACtUAl UAV experimentSACtUAl UAV experimentS
►Only using partial implementationOnly using partial implementation
 Horizon stabilizationHorizon stabilization
 Road following (no runway available)Road following (no runway available)
 Only brief periods of autonomous controlOnly brief periods of autonomous control

37. reAl time VieWreAl time VieW

38. ConClUSionSConClUSionS
►Successful but imprecise landingsSuccessful but imprecise landings
►Performance is applicable to ARDUINOPerformance is applicable to ARDUINO
 Slower and more stable than actual UAVsSlower and more stable than actual UAVs
►Assumption of linear system is notAssumption of linear system is not
applicable near the runwayapplicable near the runway
 This is why the aircraft oscillates before landingThis is why the aircraft oscillates before landing
►Future workFuture work
 Incorporate flight model into controller designIncorporate flight model into controller design

39. thAnK-YoUthAnK-YoU