1. DRAFT
Real-time UAS Video Analytics
In Mission Exploitation of ISR Image and Full Motion Video Content
Gregory B. Pepus
Tuesday, February 22, 2012
Version 1.2
Globally, public sector customers are spending $6.5 billion dollars or more annually to deploy Unmanned Aerial Systems (UAS)
which are a combination of Unmanned Aerial Vehicles (UAVs) and associated ground station technology and personnel. Flex
Analytics and its partner piXlogic offer the means to automatically exploit UAV (and other) collected image and full motion
video in real-time both during and after the mission to produce actionable intelligence.

2. Synopsis ........................................................................................................................................................ 1
UAS ISR Background .................................................................................................................................. 1
piXlogic’s Image and Video Analytics ........................................................................................................ 2
Other Capabilities ..................................................................................................................................... 4
Classification, Matching and Tagging ........................................................................................................ 4
Text in the Image ...................................................................................................................................... 6
piXserve for UAS and Manned Aircraft ISR Missions ................................................................................ 6
Concept of Operations for UAS Image/Video Alerting ............................................................................. 7
UAS Bandwidth Considerations for ISR Video and Imagery ..................................................................... 8
Summary ................................................................................................................................................... 9
i

3. Synopsis
On a global basis, military, police, emergency and other public service providers are spending in excess
of $6.5 billion dollars or more annually to deploy a range of Unmanned Aerial Systems (UAS) which are a
combination of Unmanned Aerial Vehicles (UAVs) and associated ground station communications,
computer technology and personnel. The single most important use for UAS is the Intelligence,
Surveillance, and Reconnaissance (ISR) mission.
UAS ISR Background
The principle means by which ISR is carried out via UAS is through Full Motion Video (FMV) and/or still
image collection as seen in Figure 1. In practice the ISR function is used for immediate mission support;
unfortunately, most of the collected surveillance material (over 99%) remains unexploited and
unexploitable in any large scale automated sense and this is a global problem.
Figure 1 – UAS ISR – UAV Based Image and Full Motion Video Sensors
Collected image and video content is most certainly NOT used in any great scale for real-time generation
of in-mission actionable intelligence except for the” immediate” limited surveillance needs of the
current mission while flying the aircraft. This is because the UAS have not, as a rule, included the
analytical technology, either in the bird or on the ground, necessary to support automated near real-
time exploitation of the incredible volume of video and image material being collected.
Globally customers are buying the air platforms, cameras and storage facilities to generate, view and
store this information; but have, in practice, spent almost nothing on automated exploitation of that
information particularly in near real-time during actual mission operations. The limited amount of video
and image exploitation that is done is carried out by people in an attempt to analyze less than 1% of the
overall image and full motion video content that’s collected via UAVs as shown in Figure 2. The principal
problem is that human’s only have limited ability to actively analyze video before there are significant
decreases in concentration, object recognition and cognitive performance.
1

4. Figure 2 –UAS Ground Control Systems
Why don’t customers usually consider automated image and video analytical systems? Most don’t know
the art of the possible and those that have experience with related technologies have found these
systems to be highly inaccurate producing unreliable analytical results. However, the need is real and
ever growing. It is compounded because most UAS have multiple cameras and individual humans cannot
actively split their focus in a detailed way on multiple video feeds and effectively capture actionable
intelligence from those feeds for a sustainable period.
Fortunately, the state of the art for automated exploitation and analysis for image and video content is
now highly reliable and the technology is most definitely ready for primetime. Flex Analytics LLC and its
partner piXlogic, Inc. offer automated image and video technology solutions that are immediately
available to help the government analyze image and video content in order to produce actionable
intelligence during and after UAS mission operations. These solutions allow “accurate” automated
targeting and tagging of mission specific objects of interest - leading to immediate actionable
intelligence.
Additionally, state-of-the-art analytic solutions for ISR FMV and image allow customers to dramatically
reduce the amount of stored content because analytics would identify the immediate images and/or
video content that contained important material and record ONLY that information in a searchable
index. This approach could lead to a 25% to 50% or higher reduction in the amount of video and/or
image content retained – immediately saving customers millions of dollars in storage costs and allowing
a complete and immediate exploitation of collected content.
piXlogic’s Image and Video Analytics
piXlogic Inc. offers piXserve an image and video search and analytics platform that automatically indexes
the objects within an image or frame of video. Think “Google for image and video” where no manual
intervention is required to automatically tag objects of interest for immediate actionable alerting, search
2

5. and analysis. Using image processing search and analysis algorithms that work similarly to human
vision, piXserve is able to identify particular specific objects at the pixel level in image and video content.
For example, a user can provide an image of a minivan and the software will match it to all occurrences
of the minivan found in the current corpus of image or video content as shown in Figure 3.
Figure 3 – Searching for an Object (e.g. Minivan) And Getting Results Back
piXserve identifies and then describes in an index the object’s shape and immediately attempts to
classify that shape and also to match it with specific library of objects of interest to get not only a
classification but also a specific match. When the software identifies an object it creates a mathematical
geometric description of that object recognizing all sorts of details such as edges, texture, color, lighting
and many other factors and those facets of information are inserted into the index and made
searchable.
When it is able to classify an object or match it to a specific library item it then tags1 the object in the
search index. Once indexed, piXserve allows image and video content to be “searched” using either an
image of the object of interest (or something similar), or using a keyword describing that object or by
using other search criteria to be described later in this document. Figure 4 shows how the software
might be used to find a ship of interest by searching using parts of that ship which lead to a match. For
example, the software might identify key markings, flags2, text on the object or other particular features
and apply metadata to that image or video which is stored in the searchable index.
1
Tags refer to the process of automatically, without human intervention, applying labels to objects in image or
video content.
2
The flags are too small in this image for the software to accurately identify
3

6. Metadata
Flag
Flag
Mark
Ship
Line
Ship
Name
Figure 4 – Finding an Object
Other Capabilities
The software also has the capability to do “facial biometrics” to National Institute of Standards and
Technology (NIST) specification. NIST has a full facial biometrics program and a range of data and
testing material to support biometric facial recognition software capabilities. NIST has had several
industry competitions and piXserve has developed its facial biometrics based on the data and
requirements put forward NIST’s Facial Recognition Grand Challenge (FRGC)3.
Additionally, the software has the ability to recognize “text” in image or video. It is able to identify not
only the language but also the “text string” on an image. This capability is not subject to the technical
limitations of optical character recognition (OCR) because it uses different technology. That is, it uses
piXserve’s ability to identify and recognize specific shapes applied to text in more than 28 languages
including “Chinese, Korean, Russian and a range of Western languages”.
Classification, Matching and Tagging
As mentioned above the software has the ability to generally classify an object using a concept piXlogic
calls a notion4. This capability allows piXserve to have a notion for and identify the broad class of things
to which an object it has identified belongs. For example, piXserve is able to classify things like, sky, sea,
tree, person, building, window and many other classes of objects.
3
http://www.nist.gov/itl/iad/ig/frgc.cfm
4
Notions are essentially an “ontology” broadly identifying a class to which an object belongs – i.e. that green thing
is a tree, the blue stuff is sea or sky etc.
4

7. Tree Person/Face Sea/Sky Building Window
Figure 5 – Notions for Identified Objects of Interest
When the software recognizes a specific class of object it is able to “tag” that object with a keyword. So
as shown in Figure 5 the software would try to tag pictures of trees, person, face, sea, sky, building,
window accordingly.
However, the software can go beyond that. For example, if you were trying to find a specific vehicle in a
UAV video, one could develop a library5 containing a couple of example pictures of that vehicle from
different angles. The software would use those examples to specifically match all instances of that
vehicle in that video and tag that vehicle according to the library images filename. In this case the
examples of the vehicle might have the filenames Minivan,1, Minivan,2, Minivan, 3 etc.
Minivan, 1 Minivan, 2 Minivan, 3
Figure 6 – Small Library of a Minivan
For example, as shown in Figure 6, we could create a library of a few images of a minivan as indicated
above then when the video of interest is processed (indexed) by piXserve it attempts to label all
instances of the minivan that occur in that video as Minivan. Then, instead of using a picture of a
minivan to search with, users would simply use piXserve’s keyword search function typing in the word
“Minivan” and getting a list of all search results back as shown below in Figure 7.
Furthermore, it will tag all occurrences of that person in incoming new images or video and facial
recognition in this case approaches 98% accuracy on average based on NIST standards.
5
Library in piXlogic parlance is a folder with images to be matched whereby the filenames are used to provide
tagging of matches in the incoming corpus of image or video content.
5

8. Figure 7 – Finding the Minivan Using Keywords/Results Shown to Right
Text in the Image
piXserve, as mentioned above, has the ability to recognize text in the image. Unlike OCR technology
which is highly sensitive to page color, font color and font type, piXserve can identify text in image under
almost any condition of foreground or background color or font. When an image or video is indexed
with the alphanumeric option set, users can search an index of image or video content using the text in
image option. This allows video to be searched by its banners (e.g. a news broadcast), or text on any
object such as a street signs or building names or characters on a t-shirt, or license tags, or writing on a
ship, truck, train or plane etc. as shown in Figure 8.
Pacific Line Maryland Free Meals United Way … China Shipping … N65482
Figure 8 - Finding Text on the Image
piXserve for UAS and Manned Aircraft ISR Missions
Implementing a piXserve solution for UAS and manned aircraft ISR missions makes a tremendous
amount of sense. piXserve can be used in different contexts to help produce actionable intelligence
either during or after the mission based on several different approaches used during the UAS and / or
manned ISR missions. These approaches include:
6

9. 1) Image and video is collected in real-time from the aircraft because communications links are
robust enough to handle the higher bandwidth demands of image or video content;
2) A targeted subset of image and video content can be collected in real-time from the aircraft
because communications links are robust enough to handle some but NOT all of the higher
bandwidth demands of image or video content;
3) A targeted subset of image and video content can NOT be collected in real-time from the
aircraft because communications links are NOT robust enough to handle some but not all of the
higher bandwidth demands of image or video content;
In approach 1 the communications links are robust enough so that image and video content can be
transferred from the vehicle to the ground-station in near-real time. Server infrastructure at the
ground-station can process incoming video so that alerts for targeted information of interest can be
processed and real actionable intelligence generated from video and image content during the mission.
In approach 2 the communications links are not robust enough to handle the immediate download of all
image and/or video content to the ground station. However, in aircraft that are large enough to handle
a payload of about 7lbs, a server the size of a laptop can run the piXserve software in order to allow
important targeted image or video content to be downloaded to the ground station for immediate
action.
In approach 3 the communications links are not robust enough to handle the immediate download of
any image and/or video content to the ground station. However, in aircraft that are large enough to
handle a payload of about 7lbs, a server the size of a laptop can run the piXserve software in order to
filter results, tag the image and video and allow a text messages about a specific object of interest at a
specific geo-coordinate be sent to the ground station so that immediate action can be taken as
demonstrated in Figure 9.
UAS Ground Computing Systems Actionable Intel
Figure 9 - UAS Enabled to Give Real-time Actionable Intel from ISR Data
In all of the aforementioned approaches, piXserve has the ability to manage and change specific target
objects on the fly even over very low bandwidth links so that image and/or video content alerting can be
changed dynamically as needed. That is, new objects of interest can be uploaded to the server for
automatic tagging during the actual mission timespan.
Concept of Operations for UAS Image/Video Alerting
In a typical scenario where UAS ISR is the prime mission objective and piXserve is being used to
automatically identify priority intelligence requirements (PIRs) then piXserve can be deployed on larger
7

10. UAVs via a very low power, light-weight, compact micro-server stack that will allow image and video
processing on the aircraft itself as shown in Figure 10.
Prior to mission initiation, a list of PIRs would be compiled and source images matching the items of
interest would be gathered by the mission planners. These source images, which piXlogic calls 2D non-
transformable objects (i.e. objects that have a fixed 2D shape), would have their file names changed to
whatever taxonomic label the government wanted to tag incoming images or video with and placed into
the mission’s library folder on the server residing on the UAV.
Figure 10 – Micro-server Running piXserve Aboard UAV
During operations, as video was collected from the UAS ISR imaging and video sensors, piXserve would
dynamically match the 2D non-transformable objects (i.e. the list of image files of things we want to
find) in the mission library and tag the image and video content appropriately. Alerts would be
automatically sent to the ground station immediately indicating to ground personnel that a PIR of
interest was just identified by the UAS and that some action could be taken as shown in Figure 11.
Figure 11 – UAS – Sending Alerts from piXlogic During ISR Mission Operations
UAS Bandwidth Considerations for ISR Video and Imagery
In some low-bandwidth situations where the UAS doesn’t have sufficient bandwidth to “normally”
transfer actionable video or imagery to the ground station another technology that Flex Analytics has
available is a product called FLUME. FLUME, by Saratoga Data Systems Inc., is a file transfer protocol
that allows data to be transferred over poor quality low bandwidth links at very high speeds with high
reliability.
8

11. In a recent demonstration for Special Operations Command (SOCOMM) Technical Network Test-bed
(TNT) at the Army Urban Warfare Training Center in Muscatatuck, Indiana, Flex Analytics and its
partners including Cloud Front Group, Saratoga Data Systems Inc. and piXlogic demonstrated using
piXlogic in combination with FLUME as part of the overall UAS package. The goal was to emulate
piXserve and FLUME running on the UAV and transferring video clips of PIRs of interest to the ground
station over poor quality communications links so that commanders on the ground could take
advantage of immediately actionable intelligence as shown below in Figure 12.
Figure 12 - TNT CONPS, Demonstration for SOCOMM - piXserve and FLUME as part of a UAS
During the test, piXlogic successfully alerted on pre-mission identified PIRs which resulted in small 6Mb
to 8Mb clips of the objects of interest being transferred to the ground station over 2MB/sec links with a
relatively high error rates. Ordinarily, using standard TCP/IP File Transfer Protocols (FTP or SCP6) video
clips of 6Mb to 8Mb in size would take about 7.5 minutes over a typical 2Mb/sec error prone link.
However, in combination with FLUME the 6Mb to 8MB video clip data was transferred in under a
minute.
As a result, the test for SOCOMM showing piXserve and FLUME as part of a UAS system was highly
successful. It clearly demonstrated that piXserve and FLUME could:
1. Identify PIRs of interest accurately
2. Transfer targeted video clips rather than the entire video stream to the ground
3. Provide fast and accurate data transfer rates over low bandwidth error prone comms links
4. Allow commanders on the ground to receive accurate, timely information about multiple PIRs
and thereby providing immediately actionable intelligence
Overall piXserve and FLUME met and exceed all mission objectives.
Summary
Today, the means exists for the UAS customers to acquire software technology that would allow them to
automatically index and make searchable and analyzable all UAS collected image and video content.
The means exists immediately to employ that technology at the edge of the collection envelope where
6
Another FTP services is the FTP over Secured Communications Protocol (SCP)
9

12. the UAV aircraft are deploying their video and image sensors on ISR duties in combat, law-enforcement,
rescue, intelligence and other settings.
Government and civilian UAS customers are collecting Pedabytes of video and image content (its REALLY
BIG DATA). In the past they didn’t have the means to reasonably, cost-effectively and reliably exploit
that information. However, technology immediately available from Flex Analytics and its partner piXlogic
can change that situation. piXserve, the image and video search and analytics software can be used to
help the government find mission objects of interest resulting in immediately actionable intelligence –
right on the aircraft and/or at the ground station.
10