The document describes an image exploitation system developed for airborne surveillance using unmanned aerial vehicles. The system processes real-time video and flight data acquired by the UAV. It uses image processing algorithms like enhancement, filtering, target tracking, and geo-referencing of targets. The system displays mission parameters in real-time on multiple displays. It was found to be qualified for surveillance applications. Sample results are presented.

1. 2007-01-3912
Image Exploitation System for Airborne Surveillance
Karunakaran. P, Frederick Mathews, S.H .Padmanabhan, T.K.Sateesh
HCL Technologies, India
Copyright © 2007 SAE International
ABSTRACT
The operational design of Image exploitation system to
process real time video and flight data acquired by an
Unmanned Aerial Vehicle is presented. The system
comprises of image processing tasks along with image
processing applications embedded on vision processor
board. Image exploitation makes use of image
processing algorithms for information extraction using
selective and adaptive techniques such as image
enhancements, filtering, automatic target tracking, and
computation of geo-reference location of target. The
system operates in real time mode to display mission
parameters on a multi display system. The results
indicated the system is qualified to be used for
surveillance applications. A sample set of the results are
presented.
INTRODUCTION
Airborne surveillance has been widely used in different
range of applications in civilian and military applications,
such as search and rescue missions, border security,
resource exploration, wildfire and oil spill detection,
target tracking, surveillance, etc [1]. The unmanned
airborne vehicle is equipped with special sensors (day/
night) to image objects in ground and assigns the actual
recognition task (surveillance) to the crew or record
image data and analyze them off-line on the ground.
Pilot less airborne vehicle with sensors carrying
platforms transmitting data to a ground control station for
analysis and data interpretation.
Description of Image exploitation system
developed to acquire and process real time video with
flight data of a pilot less vehicle while flying over a
specified area is presented. The system is generic in
nature for image processing algorithms and techniques
essentially for any airborne vehicles.
IMAGE EXPLOITATION
Image exploitation is a process carried out by acquisition
and processing of sensory information of a scene or
targets for surveillance applications .This process
involves processing large volume of image data acquired
by multi sensor such as optical, infrared and radar data.
For Information extraction and quick analysis the image
data has to be processed using fast computational
image processing algorithms and efficient embedded
processor [2, 3]. This involves development of
automated image exploitation tools such as selective
and adaptive image enhancements ,filtering ,boundary
detection of targets, feature extraction ,target acquisition
and tracking , identifying geo location of the target,
Zooming ,target area measurements, etc. The results of
processing are stored in a mission data base for further
analysis and interpretation by decision makers.
Image exploitation system developed consists of
hardware and exploitation software for processing aerial
vehicle data. The hardware system consists of the
following components: an Industrial PC, Frame grabber,
embedded vision processor and a graphic card for
multiple displays. The image exploitation system has
built in tools with functional features to acquire, store,
retrieve, process, analyze, interpret and display
information from imagery during a vehicle mission .The
data captured by the camera is transferred to work
station and analyzed to provide imagery information .
Imagery information is in the form of video clips, video
frames and the corresponding flight data, the calculated
location of the targets and related information .The
extraction and exploitation of imagery intelligence from
aerial surveillance enhances understanding and
interpretation of scene contents, allows vehicle to see
distant targets, and enhances surveillance capabilities.
The block diagram of the image exploitation system is
given in figure 1.

2. Figure 1 Block diagram of imaging system
SYSTEM DEVELOPMENT
The hardware architecture of Image exploitation
system core consists of a Computing System (Industrial
PC) with multiple imaging boards, frame grabber card,
graphics card to support multiple displays, and network
interface card along with supporting peripherals. The
Host PC is connected to 3 Embedded Vision Processor
(EVP) cards (256 MB RAM) with add-on modules and a
Frame Grabber card (FG) .The EVPs and the FG boards
are interconnected through the on-board Auxiliary Bus to
ensure data transfers are carried out independently of
the PCI bus as given below in Figure 2.
Figure 2 Host Interface
The Frame Grabber digitizes the incoming PAL analog
video (color/grayscale) of 25 frames/sec (1 Frame
768x576) and transfers it (120 MB/sec) to Host PC via
PCI bus or to embedded processor via dedicated data
bus called AB (Auxiliary Bus). AB is a dedicated bus
that makes transfer of video from Frame grabber to
embedded processor at a very high speed (200 MB/sec)
and bandwidth with out depending or affecting the Host
PC. The output of the video capture and processing are
viewed on the HOST PC’s monitor through the HOST
PC’s Graphics card (Dual display 256 MB DDR memory)
and the output of each embedded processor is viewed
on separate monitors connected and controlled by
graphics card on embedded processor board. Real-time
imaging applications require high data bandwidth from
the video source to the display output [4]. These
systems also demand very low latencies from the
processing system because a human operator is badly
affected by any apparent lag between the input image
and the presented image when in a moving platform
(such as an aircraft). This means that the processing
delays in the system need to be minimized and often
kept below 12.5 fps (which is a typical frame time). Add
to this further complex operations (Image processing
algorithms) need to be applied to each and every pixel,
perhaps several times. Real time Video image
processing is realized by using the EVP, optimized
image processing library and distributing the image
processing functions to multiple EVP. Image acquisition
and processing is done in parallel and the data goes to
host as well as EVP. One such framework is to use the
processing ability of multiple embedded processors
where 80 ms is the required time to process 1 frame and
3 embedded boards are used for fast processing of
images. Further an auxiliary bus with multicast mode
allows images to be sent for concurrent processing on
two or more embedded processors .The Software
architecture of the system with application layer,
hardware layer and interface layer is given in Figure 3 as
below.
Figure 3 Software Architecture Diagram
APPLICATION SOFTWARE
Image exploitation system contains application software
which extracts and exploits the information of aerial
imagery obtained from onboard sensors mounted on
UAV or other reconnaissance platforms. The output is
obtained from the exploitation of aerial video imagery
captured by camera mounted on UAV with Flight
telemetry data .Imagery intelligence is obtained by
applying image processing techniques and algorithms

3. where target feature information is extracted
automatically from real time video data for image
analysis. The basic input to the application software is
the flight video image and data captured by the
unmanned mission. The system processes the images
using Image processing application software which
displays input and processed images for analysis. Some
of the processing functions required by users are as
follows:
1. Mission settings, acquisition of video flight data
and image display.
2. Video Image analysis using Image Processing
functions such as Enhancements, edge
detection, filtering, zooming, etc.
3. User interaction and processing such as single
frame and multiple frame target calculation,
terrain measurements, area based retrieval, etc.
4. Real time target tracking from video images.
5. Real time Plotting of flight parameters (Health).
6. Creating ,updating and retrieval of Imagery
intelligence database
7. Printing and saving the image file data in a CD.
METHODOLGY
The methodology is based on the development of
fully automated decision criteria tools [3, 5 and 6] for
feature information extraction from video images which
involves various image processing algorithms such as:
1. Video Image enhancement techniques for selective
(linear and non-linear enhancements) and
automatic enhancements to search for a target
region in the video imagery. In selective mode of
enhancement user has an option to select standard
enhancement techniques such as Histogram
equalization, contrast stretching, contrast stretching
with clipping population on a single tail or both tails
of the image histogram. Enhancement techniques
are also applied on a moving window of user
specified size in the video image to assist user in
selection of target. Once the target is selected by
user, various parameters for input image are
calculated such as average intensity, standard
deviation and Intensity histogram. Adaptive video
Image enhancement techniques are applied for real
time enhancement using lookup table [LUTs].
Lookup table is created based on derived Image
statistics [Range, entropy, coefficient of variation,
mean and contrast] obtained from real time video
Image data as given in table 1.
Table 1 Image Information Statistics
Analyzing various data sets it is found that contrast
information derived is more stable than other image
statistics as it depends more on grey level range in
the scene data. Contrast information is derived from
real time video image data as : Contrast (%) = Max
GL –Min GL / Max GL + Min GL where Max GL and
Min GL are the maximum and minimum gray level
from input image. LUTs are created based on the
input image contrast computed and corresponding
LUTs are assigned in real time for Image
enhancement.
2. Target region processing {TRP] tools to process
user selected region of the video frame .Option to
select various Image filtering techniques (Low
pass and high pass, median) ,image sharpening
tools, selected area zooming , Edge boundary
information of the target area by various edge
detection (Sobel , Canny ,Laplace ,Robert, etc )
and to display all the processed output .
3 Target tracking tool to track a particular target in the
video frame using template pattern matching
algorithm .The algorithm is based on normalized
cross correlation method. The Position of target in
the first image is identified as “Window area” and the
target in the subsequent frame is identified as
“Search area” .The problem is then to estimate the
position of the template in subsequent images.
Normalized correlation template matching algorithm
is used to search target in the subsequent frame and
to estimate the position of the target accurately.
4 Algorithm for computation of geo –referenced
position of the target selected by the user using
the flight data. By click of mouse on target, the
look angle, slant range, easting/Northing or
Latitude and longitude of the selected area is
displayed on Map.
5 Algorithm for retrieval of area based target within
a user specified area. Retrieves the targets that lie
within the user defined area or the maximum
Range of search. Area to be specified in terms of
Circular range around the current UAV position.
6 Tool for Real time Flight parameters display such
as Roll, Pitch, azimuth, altitude and heading angle
of the Vehicle.
7 Algorithm for Terrain measurement and to
measure specific terrain features. Measures
includes linear distance on ground, tracing of
curvilinear path, area coverage of interesting
region and perimeter.
RESULTS
The basic input to the Image exploitation system
consists of mission path, probable target locations and
map data. During a mission Image exploitation system

4. acquires image data from the camera and flight
parameters from onboard systems. Frame grabber
digitizes image data and transfers it to Host PC and
multiple embedded processor boards. Application
software in host uses multiple embedded processor
boards to achieve real time image processing, analysis
and display of various parameters to user during
mission. Application software is tested using the
playback flight video image and data captured by the
unmanned mission in a form of DVD connected to the
embedded vision system. The system processes the
images using Image processing application software in
the form GUI menu which displays input and processed
images for analysis.
Video data with flight parameters is processed in real
time where Roll, Pitch, heading, height of the the vehicle,
azimuth, elevation are displayed in meter display as
given in Figure 4.
.
Figure 4 Real Time Video Image display
. A sample result of a moving window area enhancement
is given in figure 5.
Figure 5. Moving window Area Enhancement
Target region processing of a target region in the
video frames with variable window sizes and image
Processing algorithms such as edge detection
/enhancement/ Zooming/sharpening tools, etc is
applied for selected area .A sample results of
processing is given in figure 6.
Figure 6 Selected area Image Processing
The geo-referenced location of the target position on the
ground is computed from a single and multiple frames
associated with the target using flight data parameters.
Once the user clicks a Target, the Look Angle, Slant
Range, Easting/ Northing or Latitude and Longitude of
the selected target is displayed with the target image as
given in figure 7.
Figure 7 Target Position computation
A Target Tracking function is developed to track the
particular target(s) in the video frame using template
matching techniques. Window frame area of target frame
is matched with searched area window of the next frame
using normalized cross correlation as a similarity
measure. A sample result of tracking is given in Figure 8
as below.
Figure 8 Sample Target Tracking
Target area measurements on ground are computed
using the geo-referenced values. Target area is selected
by the user on a freezed frame. Distance, area and
perimeter of the target are computed using Euclidian
distance and it is displayed on map/ zoomed map. User
has an option to select and view previous video frames
of the incoming video with specified frame interval for
analysis. Whenever user acquires a target in the host,
this module displays the list of last few target images.
Further user can retrieve targets that lay within the user
defined or the maximum range of search .When user
access this module, dialog is displayed with UAV current
position and all target locations on the map. Retrieve
target list is maintained where user can see the target
details: Target name, easting, Northing, distance from
the current location, look angle ,azimuth and elevation
with the target images in the form of context diagram as
given in figure 9

5. .
Figure 9 Target Image Retrieval
An Image intelligence database is generated for Targets
such as geo-location of the target, normalized view of
the target, history information, Video clipping, and
auxiliary data like survey maps, satellite image and
digital elevation model, etc.
CONCLUSION
This study demonstrate the development of a
image exploitation system with potential of using image
processing tools for processing real time video and flight
data for an unmanned aerial vehicle. The image
exploitation system is developed using multiple
embedded processing boards connected to the host and
results of processing is displayed in multiple displays
using graphic card. Real time Video Image acquisition
and image analysis using multiple embedded processor
with optimized image algorithms library is realized .The
work involves the design and development of the system
meeting project specific requirements and integrating the
hardware, software features. The system is tested using
simulated flight data collected for a surveillance mission.
The advantages of this architecture being scalable and
flexible to increase embedded processor boards as per
requirement. To maximize productivity and minimize the
decision making cycle of the request, algorithm
parallelism effort will seek to achieve better latency.
Further exploration of multiple sensor and study of fusion
techniques may improve target detection performance.
ACKNOWLEDGMENTS
Authors gratefully acknowledge Dr. Jharna Majumdar
for her constant encouragement and technical guidance
throughout the project.
REFERENCES
[1] M.Kontitsis, Kimon P. Valavanis et al. A UAV Vision
System for Airborne Surveillance, Proceedings of
the 2004 IEEE, International Conference on Robotics
& Automation, New Orleans, LA, April 2004 P-77-83.
[2] P. Doherty et. al, The WITAS unmanned aerial vehicle
Project .In. Proc. of the 14th
European conf. of. Artificial
Intelligence, 2000.
[3] Brian Hoerl, ARIES migrates Image Exploitation to
UAVs, VME bus Systems, Mercury /Aug 2005
[4] Jamie Heather & Moira Smith, Analysis of Registration
requirements & Techniques for Imaging Sensor Suites
on UVs, 1st SEAS DTC Technical Conference
Edinburgh 2006,
[5] R. Gonzalez and R. Woods, Digital image
Processing, Addison Wesley, 1992.
[6] Paul Robertson et al. Adaptive image analysis for
Aerial Surveillance, IEEE Intelligent systems,
IEEE 1999, P-1094