ACM Multimedia 2011, Scottsdale, Arizona, USA, 28 Nov - 01 Dec 2011. http://eprints.soton.ac.uk/273040/ OpenIMAJ and ImageTerrier are recently released open- source libraries and tools for experimentation and devel- opment of multimedia applications using Java-compatible programming languages. OpenIMAJ (the Open toolkit for Intelligent Multimedia Analysis in Java) is a collection of libraries for multimedia analysis. The image libraries con- tain methods for processing images and extracting state- of-the-art features, including SIFT. The video and audio libraries support both cross-platform capture and process- ing. The clustering and nearest-neighbour libraries contain efficient, multi-threaded implementations of clustering al- gorithms. The clustering library makes it possible to easily create BoVW representations for images and videos. OpenI- MAJ also incorporates a number of tools to enable extremely- large-scale multimedia analysis using distributed computing with Apache Hadoop. ImageTerrier is a scalable, high-performance search engine platform for content-based image retrieval applications using features extracted with the OpenIMAJ library and tools. The ImageTerrier platform provides a comprehensive test- bed for experimenting with image retrieval techniques. The platform incorporates a state-of-the-art implementation of the single-pass indexing technique for constructing inverted indexes and is capable of producing highly compressed index data structures.

1. ImageTerrier
www.imageterrier.org
OpenIMAJIntelligent Multimedia Analysis In Java
www.openimaj.org
OpenIMAJ and ImageTerrier
Java Libraries and Tools for Scalable Multimedia
Analysis and Indexing of Images
Jonathon Hare, Sina Samangooei & David Dupplaw
The University of Southampton

2. What is OpenIMAJ?
Scalable
Clustering
Scalable
KNN
OpenIMAJIntelligent Multimedia Analysis In Java
Image
Analysis
Libraries
Command-line Tools
Hadoop Map-Reduce Implementations
Hardware
Machine
Learning
Video
Web Audio
Analysis
AnalysisProcessing
ProcessingProcessing
Analysis Rendering
Kinect Audio &
Video
Capture
Exotic
Hardware

3. Libraries
Command-line Tools
Hadoop Map-Reduce Implementations
Efficient,
Compressed
Inverted
Index
Query
Engine
Term
Metadata
Integration
Based on
Terrier Constructable
from Images
Distance
Metrics
scale
Re-Ranking affine
orientation
Compressed
Extensible
OpenIMAJIntelligent Multimedia Analysis In Java
Scalable
e.g SIFT BoVW
(x, y, scale, etc)

4. What is ImageTerrier?
Efficient inverted index construction for image/
video features (and potentially audio features too)
Augmented inverted index, capable of
storing almost any kind of meta information
per term
Easily extensible
Provided implementations
contain things like (x,y,scale)
for SIFT BoWV
Indexing is scalable across machines using
Hadoop
Complete OpenIMAJ integration
Included tools can build an index
straight from a set of images using
common techniques like SIFT BoVW
or read features extracted from other
tools.
This can even be done on
Hadoop!
Easy to define searching and re-ranking
strategies
Various distances, weightings and
constraints (such as Affine &
Homographic) are provided
Tested with corpuses in excess of 10M
images

5. A Little history...
IAMMediaEngine
c. 2005
MapSnap
c. 2006
JCVMLTK
c. 2009
OpenIMAJ &
ImageTerrier
May 2011
MoVE
c. 2010
Pure-Java SIFT &
Matching
Extensible Java (&
JNI) Content-
Based Retrieval
System
Refactoring of older code,
addition of face detection
and machine learning
components
Initial attempts at scalable
BoWV indexing using terrier,
shape analysis, etc
Many new features added.
Improvements to
ImageTerrier, face pipeline,
demos, video capture and
hardware, ...
PhotoCopain
c. 2005
Image classifiers
SMS-NLP
c. 2010
Machine learning,
clustering
Massive code cleanup and
documentation effort. First
Open Source Release

6. It’s pure Java...
(with the exception of thin interfaces for hardware)
Isn’t it slow?

7. Why another set of libraries?
Scalability + Portability.
OpenIMAJ projects work on:
Android mobile devices
Hadoop clusters
...without recompilation!
Easy to use and extend
Implement many important
Multimedia algorithms
While allowing easy adaption
and extensions.
Software engineering principles
adopted
clear separations of
concerns
maintainable, understandable
We simplify data representations:
Images are 2D-arrays of
pixels;
Videos are iterable streams
of Images
Audio is an array of samples

8. Both OpenIMAJ and
ImageTerrier have been
designed from the ground-up
to be easily extensible.
We’ve made it easy to modify
and understand algorithms
and techniques by separating
different components.

10. Dominant(orienta+on(of(an(
orienta+on(histograms(

11. Replace'with'a'Null'
orienta1on'extractor.'

12. Regular(SIFT(feature(
provider(

13. Replaced(with(Irregular(
Binning(

14. Rapid Application Development
“Make it easy to use”
This philosophy lets researchers build complex applications quickly
Example: Southampton Goggles
synchronised: 6 cameras, GPS device, compass
feature extraction and index construction
mobile integration
…all programmed over 2 days!
Find out more at: http://bit.ly/SOTONGOG

16. Student projects
Our students use OpenIMAJ
(…for 2 years now!)
OpenIMAJ lets them:
Get something basic working quickly
Understand the algorithms they are
using
Implement new algorithms in a
structured way
Contribute to a living, open source
project
To support our students we wrote a
comprehensive OpenIMAJ tutorial.

17. Research applications
Through combination of OpenIMAJ and ImageTerrier various
multimedia research applications can be addressed
- We have done so for over 6 years!
1)Guess The Building - part of the LiveMemories project
Estimate geolocation and semantic information for an un-
annotated image of a landmark
2)Stock Photo Finder - part of the Living Knowledge project
Find the source of a web image in a stock photo database

18. Any Questions?
Try the OpenIMAJ tutorial for yourself:
http://openimaj.org/tutorial.pdf