The system provides a service-oriented architecture that allows for multiple viewpoints of multimedia data inside repositories. The analysis layer is responsible for extracting low-level features and semantic annotations from media files through various processing pipelines. Annotation of visual and audio content is performed using bag-of-visual words, MSER, SURF, SIFT features, and SVM classifiers. The system was evaluated for usability, allowing users to search, annotate, and interact with videos and interfaces.

1. A web system for ontology-based multimedia annotation,
browsing and search
M. Bertini, G. Becchi, A. Del Bimbo, A. Ferracani, D. Pezzatini
University of Florence - MICC
System architecture
The system provides a service-oriented architecture Interface layer Authoring layer
(SOA) that allows for multiple viewpoints of multimedia data Corporate CMS
Authoring
environment
inside repositories, providing better ways to reuse, repurpose End-user
interfaces
and share rich media.
SOA Architecture layer
The analysis layer is responsible for extracting low level System
repository
features and semantic annotations from media files, System SOA
through a series of processing pipelines that can be executed File
storage
in a cloud of servers, orchestrated by apposite services.
Annotation of visual content is performed using the Video/Image
analysis pipeline
Bag-of-Visual-Words paradigm, based on a fusion of MSER, Local SOA architecture
SURF and SIFT features and the Pyramid Matching Kernel. Database
Analysis
Audio analysis
pipeline
Audio annotation is based on a fusion of features like ZCR, Semantic
search / browse
Syntactic search /
browse
user-defined
MFCCs, chroma and spectral features and SVM classifiers. Media
storage
pipeline
CBIR retrieval is performed using rhythm and pitch features Analysis layer
for audio and MPEG-7 features for visual data.
Semantinc annotation, browsing and search tools
All the web applications of the system have
been developed according to the RIA
paradigm. In particular the applications of the
Interface and Authoring layers are developed
in AJAX and Flash/Flex, while data is
exchanged using SOAP, RSS and JSON for
metadata and RTMP for video streaming.
Interfaces allow to perform manual
annotation (to check automatic
annotations, add metadata or create ground
truth annotations to train new automatic
concept detectors), browse , search
(semantic and content based) and tagging.
Usability evaluation
The goal of the field trials was to assess the usability of the system, in particular letting the users to interact with
the search engine and its interfaces, to pose semantic and syntactic-level queries, but also to annotate, automatically
and manually, some videos.
http://www.micc.unifi.it/ This work was partially supported by the EU IST IM3I project (contract FP7-222267). http://www.im3i.eu