This document provides an overview of the piXserve solution for intelligent image and video search capabilities. piXserve can automatically index and analyze image and video content to generate searchable metadata without manual intervention. It allows users to perform visual searches using images, video frames, objects, faces, text, or complex multi-modal queries. piXserve also enables powerful automated alerting when matches to specified criteria are found. The solution is scalable and can process live and archived video and image sources for applications in defense, security and intelligence.

1. A piXlogic White Paper
Sponsored by Flex Analytics
4984 El Camino Real
Suite 205
Los Altos, CA 94022
T. 650-967-4067
info@piXlogic.com
www.piXlogic.com
Intelligent Image and Video Search
for Defense Applications
Government Reseller for piXlogic
10314 Thornbush Lane
Bethesda, MD 20814
(301) 787-2989
Jul-2012

2. July 2012 pg. 2

3. Contents Introduction
Images and videos have always been key
Introduction 3 elements of intelligence and defense
Problem Statement 3 operations. In recent years, the scope and
diversity of digital imagery has greatly
Previous Options 4
increased in every area: ground, satellite,
The piXserve Solution 5 UAV, surveillance, broadcast, etc. The
Key Features of piXserve 5 volume of material being acquired and
Security Applications 8 stored is staggering, with no visible plateau
in sight. Traditional methods of organizing,
Implementation 9
cataloguing, and distributing this material to
Summary 9 analysts and the war-fighter are becoming
About piXlogic 10 impractical due to the scale involved. On
the other hand, timely access to nuggets of
vital information contained in images/videos
is key to operational success. The ability to
cross-correlate the information, whether it’s
being obtained from live sources or from
archived repositories, is more important than
ever.
In this environment, image/video search and
retrieval has become the new “must have”
element of any comprehensive solution.
Unfortunately, today’s image/video
management systems are not well suited to
help make sense of the data collected, and
can only provide, at best, very limited search
and retrieval capabilities.
Problem Statement
Most video management systems offer
limited options for automating processes
such as searching archived footage, or
generating alerts from live video. For the
most part, these features are either not
available, or only available in a very limited
sense. Often, a significant amount of
manpower is required to carry out even
simple search tasks. This is well known in
the field. Correlating visual data from
July 2012 pg. 3

4. different sources is another very generally in the same location on the image.
challenging task, mostly done Both of these requirements limit the scope
manually today. Automated change of applications possible with such systems.
detection is yet another largely
elusive goal. Face Recognition: Much as in the ALPR
case, a big hurdle is to know where the face
Industry/government efforts during to be measured is on the image. To solve
the last few years have focused on this, typical systems require that the distance
building infrastructure and have between the camera and the subject be
resulted in great improvements in the within a predefined range. Lighting
ability to acquire higher resolution variations are also critical which is why the
imagery/full motion video, moving more successful implementations are limited
this material around the network to indoor, entry-way, type of set-ups.
efficiently, and storing it. These are Outdoor video in unconstrained
great accomplishments, but by environments presents a challenge that is
themselves they are not enough. outside the realm of most commercial
Now is the time to leverage previous solutions available today.
investments and provide a much
needed level of automation so that Object Detection: The ability to
analysts can deal with the size and detect/recognize/search for specific objects
scale of the problems they face. in a video or an image is not usually
However, for most solution available. Some attempts have been made
providers, this remains a significant for video, but the methods used are overly
technical challenge. simplistic and unreliable. A typical
technique relies on “frame differencing” to
Previous Options separate moving things from a stationary
When automated video analysis tools background. The idea is simple but
are available, they tend to be single- unfortunately it only works in trivial
purpose with limited scope of situations. If the camera is moving, the
applicability and stringent operating background will move as well and frame
requirements. Consider the differencing techniques won’t work.
following three examples: Turning off a light, a cloud passing in the
sky, a moving shadow, these are all things
Automated License Plate that can yield undesired results. Even when
Recognition: For most systems, the the background and the camera are
hurdle is to know where the license stationary, the amount of information that is
plate is in the image being analyzed. obtained is limited. If the camera is
To circumvent this problem, solution calibrated, some guess about the size of the
providers either require the use of object can be made and from this an
specialized cameras (infrared) or that inference can be derived about what is in the
the cameras be placed such that the scene (perhaps an adult, may be not a dog),
license plate to be recognized is but even this too can be quite unreliable (is
July 2012 pg. 4

5. it a dog, a tumbleweed, or a far away
person). Crowded environments
present a critical challenge to today’s 2. piXserve reasons about what it "saw"
systems. in the image and develops an initial
level of "understanding" about
content and context. Where it can, it
The piXserve Solution automatically creates searchable
piXserve is a general-purpose "tags" for what it saw in the image
image/video search and alerting (piXlogic calls these tags "Notions").
solution. Breakthrough technology For example, it can detect the
developed by piXlogic allows the presence of things such as: sky,
software to automatically “see” the vegetation, flower, face, building,
contents of an image/video frame car, map, airplane, helicopter, etc.
and create a searchable index and
uses this information so that users 3. piXserve uses all the information
can search and create alerts in a very calculated from the image to make
natural and logical way. comparisons between a search image
and previously indexed
1. piXserve automatically images/videos so that users can find
“segments” an image in a results that most closely match what
way that discerns the they are looking for.
individual objects in the
image. It creates a 4. piXserve can "see" not only visual
mathematical description of objects but also text strings that may
the appearance of these appear anywhere in the field of view
objects "on the fly", and of the image. This text is also
stores it as a searchable index indexed and made searchable.
in a database. piXserve works with text from many
languages (alphanumeric/latin-
character based languages, Japanese,
Korean, Chinese, etc.)
5. Depending on the quality of the
imagery involved and the type of
search being done, piXserve has
been designed to achieve accuracies
in excess of 85%.
Key Features of piXserve
 Automatic Indexing
Point piXserve to a repository of
July 2012 pg. 5

6. images/video files or to a live the mouse to point to an area of
video feed, and automatically the query image to indicate
index content. No manual which specific item(s) should
intervention or data entry be searched for.
required.
3. Browse the contents of existing
 Powerful Search databases, grab a frame “on the
Through a web browser fly” from a video that is
interface, users login to playing, and use that frame to
piXserve, connect to formulate a visual search
available databases and query.
formulate search queries to
retrieve desired images/video 4. Search images and videos by
segments: object class ("Notion")
1. Use an arbitrary image 5. Type a text string to search
to search for pictures/videos where that
images/video segments string appears in the field of
that contain the same or view (a license plate, a street
similar items sign, a name tag, etc.)
6. Search for faces of specific
individuals
7. Perform not only simple but
also complex multi-modal
searches. (Example: find video
sequences where something
like the bag in this picture
AND this face from this other
picture AND this text string I
just typed all appear in the field
of view at the same time.)
Use AND, OR, and NOT
operators to combine up to
6 criteria in a single query.
2. U
s
e
July 2012 pg. 6

7. 8. Search by file name and naming)
9. Search by keyword or  Powerful Alerts
other external metadata, Create alert criteria just as you would
if available. formulate a search query. piXserve-
ALERT keeps track of what
10. Submit sample images piXserve machines on the network
of non-deformable are indexing and when a match is
objects of interest and made consistent with what the user
automatically tag specified, it generates a signal. The
images/video frames user receives an e-mail with a link to
when these items are the alert results. A JMS (Java
visible. Messaging Service) signal is also
generated to pass the alert on to other
 Powerful Automated Tagging systems and applications for further
action.
1. Automatically tag
images/video frames  Powerful Metadata
with the name of The richness of metadata calculated
recognized by piXserve about each image/video
individuals that frame processed (objects and tags),
appear therein can be exploited to enable
(automated face customized applications that are of
naming). high value in a variety of settings
such as:
2. Suggest keywords to
describe the contents 1. Automatic determination of
of a picture/video change detection when
frame (automated videos taken at different
keyword times from different angles
recommendations) are compared.
2. Determining which portions
3. Submit sample of a video archive contain
images of non- useful information, and
deformable objects of which could be safely deleted
interest and to minimize storage
automatically tag requirements.
images/video frames
when these items are  Scaleable Architecture
visible. (automated piXserve is a multi-threaded, J2EE
2D-object detection scalable application that is suitable
for the most demanding
July 2012 pg. 7

8. implementations. piXserve extends the capabilities of today’s
systems by adding the ability to
 Web Services API automatically analyze the video that is being
A REST-based API package collected and stored. These video streams
is available to support can be intercepted by piXserve and analyzed
integrations with third party for alerting purposes. Similarly, recorded
applications and workflow video can be analyzed, searched and
environments. correlated using piXserve. The analytical
capabilities in piXserve support: facial
Security Applications recognition, general purpose object
If you are concerned with the cost, detection and recognition, text recognition,
speed, and accuracy of your video license plate recognition, automatic tagging,
investigative work, whether it be and more. All the indexing work is done
forensic in nature or dealing with automatically, server side, in the
live situations, then you should background. Users are then free to create
consider piXserve as a “must-have” visually-based search criteria and navigate
add-on to your current system. the body of accumulated material. They can
do all of this “on the fly”, as they see fit at
Conventional systems focus on the moment, based on whatever problem or
managing and manipulating cameras situation they are dealing with.
and storage devices. Unfortunately,
they only provide limited capabilities The piXserve search environment is
for searching the captured video: intuitive and productive, and the user
time, date, motion, transaction interface is through a web browser (Internet
trigger…these are among the more Explorer, Mozilla Firefox, Safari, Google
common set of options available. Chrome, or equivalent). Users can drag-
While useful, these features alone are and-drop a picture from anywhere to
inadequate to support a productive formulate a similarity search query, or pause
workflow and significant manpower a video while it’s playing, and use that
effort is required even for the simpler frame to create a new search criteria or
tasks. Common situations involve refine an existing one. This latter capability
several operators having to stare at a greatly simplifies the discovery process
bank of monitors for hours on end in precisely in those situations when the user
order to catch an event of interest, or isn’t quite sure what they are looking for and
having to wade through hundreds of are working in an investigative/exploratory
hours of video from many cameras mode.
looking for a specific event or trying
to correlate separate ones. These
situations are labor intensive, error
prone, and do not scale well.
July 2012 pg. 8

9. Implementation
piXserve can process videos in a
variety of formats (MPEG-1, MPEG-
2, MPEG-4, H-263, H-264, etc.).
piXserve can also process still
images in over 90 different formats
(jpeg, tiff, png, bmp, psd, etc.)
piXserve can index both archived
video as well as live video broadcast
from Multicast IP cameras. piXserve
and piXserve-ALERT run on
standard 2-CPU rack servers (multi-
core Intel-Xeon processors or
equivalent), in a Windows Server
2003 or 2008 environment.
Customers typically choose Dell or
HP hardware for implementation. parallelize throughput and serve growing
piXserve is available in both x32 and needs.
x64 bit versions.
In order to index archived video The metadata created by piXserve is stored
piXserve requires that the storage in an RDBMs (Oracle or MS-SQL are
device be accessible via a network supported, PostgreSQL is bundled with
share (Linux/Unix/Windows). piXserve). The data and the piXserve output
Further, the stored video should not can be integrated/correlated to that from
be in a proprietary, non-standard other systems that the customer may be
format. using. The alerting functionality is provided
by piXserve-ALERT. A single instance of
A single server can process large piXserve-ALERT can serve many users and
amounts of archived material, or live monitor potentially thousands of alert
video from multiple feeds/sources. criteria. Here too scaling is achieved by
The higher the number of cores on adding additional ALERT servers. In
the server, the higher the number of configurations were several hundreds or
hours of video per day that can be thousands of individuals will be searching
processed by a single machine. piXserve generated data, the use of
piXserve implementations can range piXserve-Enterprise Edition is
in size, from as little as a single recommended.
server to scalable multi-server and
distributed configurations. The Summary
architecture of the product is such Images and videos are a critical element of
that as the needs of the customer defense and intelligence operations. It is
grow, hardware can be added to very difficult to deal with an ever-growing
amount of captured video without
July 2012 pg. 9

10. automation. The alternatives to  Content Auto-tagging (automatically
automation are expensive, time label an image/video)
consuming, and prone to errors. At  Content Alerting (automatically inform
the same time, there is a lack of users when items of interest appear in a
suitable tools to provide a
live video stream or web crawl)
meaningful level of real-world
automation.  Content Change Detection
(automatically compare images and
piXserve provides an unparalleled video segments to detect changes at
level of image analysis and the object level)
understanding. In a single tool it
provides capabilities that span: piXlogic serves the needs of government
object detection and recognition, and industrial customers. piXlogic sells its
face recognition, license plate products directly and through a network of
recognition, text recognition, resellers in the US, the UK, Japan, Australia,
automatic tagging, and more. In Argentina, Israel, and Italy.
each of these areas, piXserve
redefines the state of the art and can Corporate
help your meet the efficiency and piXlogic, Inc. T. +1-650-967-4067
4984 El Camino Real E. info@piXlogic.com
effectiveness goals that you have set Suite 205 W. www.piXlogic.com
for yourself. Los Altos, CA 94022
Flex Analytics is a systems integrator and
About piXlogic software reseller in the U.S. Intelligence
piXlogic is a privately held company Community. It supports the sale,
located in Los Altos, CA, USA, the implementation and customization of
heart of Silicon Valley. piXlogic is piXserve in government installations.
an In-Q-Tel portfolio company (a
venture capital organization that Government Sales +1-301-787-2989
Flex Analytics LLC gpepus@flexanalytics.com
serves the needs of the US
10314 Thornbush Ln www.flexanalytics.com
Intelligence Community). The Bethesda, MD 20814
company’s flagship products are
piXserve and piXserve-ALERT.
The software enables:
 Content Discovery (find
pictures/videos that contain
specific objects, scenes, text, or
people of interest)
July 2012 pg. 10