ACM Multimedia 2007 Guo-Jun Qi, Xian-Sheng Hua , Yong Rui, Jinhui Tang, Tao Mei and Hong-Jiang Zhang Microsoft Research Asia September 25, 2007

Motivation Correlative Multi-Label Annotation Modeling correlations Learning the classifier Connections to Gibbs Random Field Experiments Live Demo

Motivation

Correlative Multi-Label Annotation

Modeling correlations

Learning the classifier

Connections to Gibbs Random Field

Experiments

Live Demo

How many images and videos in the world? May 2007 : 500 millions Aug. 2007 : 1 billion 2000 images /minute Sep. 2007 : 84 millions

How many images and videos in the world?

Year Manual Automatic Learning-Based 1970 1980 1990 2000 70 - 80’ Manual Labeling 90’ Pure Content Based (QBE) Now Automated Annotation

Learning-based video annotation schemes New sample Lake? Now Automated Annotation Learning-Based Modeling and Learning Classifier Training samples Features Person Grass Tree Building Road Face

A typical strategy – Individual Concept Detection Annotate multiple concepts separately Low-Level Features Outdoor Face Person People-Marching Road Walking- Running -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1

A typical strategy – Individual Concept Detection

Annotate multiple concepts separately

√ Person √ Street √ Building × Beach × Mountain √ Crowd √ Outdoor √ Walking/Running √ Marching ? Marching

Low-Level Features Outdoor Face Person People-Marching Road Walking- Running -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 Low-Level Features Outdoor Face Person People-Marching Road Walking- Running -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 Concept Model Vector Score Score Score Score Score Score Concept Fusion Another typical strategy – Fusion-Based Context Based Concept fusion (CBCF)

Another typical strategy – Fusion-Based

Context Based Concept fusion (CBCF)

Low-Level Features Outdoor Face Person People-Marching Road Walking- Running -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 Concept Fusion Concept Model Vector Score Score Score Score Score Score

Our strategy – Integrated Concept Detection Correlative Multi-Label Learning (CML) -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 -1 / 1 Low-Level Features Outdoor People-Marching Road Face Person Walking- Running

Our strategy – Integrated Concept Detection

Correlative Multi-Label Learning (CML)

Multi-Label Annotation No correlation Has Correlations, but uses a second step Model concepts and correlations in one step Individual Detectors Fusion Based Integrated 1 st Paradigm 2 nd Paradigm 3 rd Paradigm

Our strategy – Integrated Concept Detection Correlative Multi-Label Learning (CML)

Our strategy – Integrated Concept Detection

Correlative Multi-Label Learning (CML)

How to model concepts and the correlations among concept in a single step

How to model concepts and the correlations among concept in a single step

Notations

Notations

Modeling concept and correlations simultaneously - 1 1

Modeling concept and correlations simultaneously

Modeling concept and correlations simultaneously - 1 1

Modeling concept and correlations simultaneously

Modeling concept and correlations

Modeling concept and correlations

Learning the classifier Misclassification Error Loss function Empirical risk Regularization Introduce slack variables Lagrange dual Find solution by SMO

Learning the classifier

Connection to Gibbs Random Field Define a random field Rewrite the classifier Define energy function Define GRF is a random field consists of all adjacent sites, that is, this RF is fully connected

Connection to Gibbs Random Field

Connection to Gibbs Random Field Rewrite the classifier Define energy function Intuitive explanation of CML Define a random field Define GRF is a random field consists of all adjacent sites, that is, this RF is fully connected

Connection to Gibbs Random Field

Experiments TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%)

Experiments

TRECVID 2005 dataset (170 hours)

39 concepts (LSCOM-Lite)

Training (65%), Validation (16%), Testing (19%)

Experiments TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%) CML ( MAP=0.290 ) improves IndSVM ( MAP=0.246 ) 17% and CBCF ( MAP=0.253 ) 14% CML CBCF SVM SVM  CML ↑ 17% CBCF  CML ↑ 14%

Experiments

TRECVID 2005 dataset (170 hours)

39 concepts (LSCOM-Lite)

Training (65%), Validation (16%), Testing (19%)

CML ( MAP=0.290 ) improves IndSVM ( MAP=0.246 ) 17% and CBCF ( MAP=0.253 ) 14%

Experiments TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%) CML ( MAP=0.290 ) improves IndSVM ( MAP=0.246 ) 17% and CBCF ( MAP=0.253 ) 14% CML CBCF SVM SVM  CML ↑ 131% CBCF  CML ↑ 128%

Experiments

TRECVID 2005 dataset (170 hours)

39 concepts (LSCOM-Lite)

Training (65%), Validation (16%), Testing (19%)

CML ( MAP=0.290 ) improves IndSVM ( MAP=0.246 ) 17% and CBCF ( MAP=0.253 ) 14%

Experiments TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%) CML ( MAP=0.290 ) improves IndSVM ( MAP=0.246 ) 17% and CBCF ( MAP=0.253 ) 14% CML CBCF SVM CML CBCF SVM CML CBCF SVM

Experiments

TRECVID 2005 dataset (170 hours)

39 concepts (LSCOM-Lite)

Training (65%), Validation (16%), Testing (19%)

CML ( MAP=0.290 ) improves IndSVM ( MAP=0.246 ) 17% and CBCF ( MAP=0.253 ) 14%

Experiments TRECVID 2005 dataset (170 hours) 39 concepts (LSCOM-Lite) Training (65%), Validation (16%), Testing (19%) CML ( MAP=0.290 ) improves IndSVM ( MAP=0.246 ) 17% and CBCF ( MAP=0.253 ) 14%

Experiments

TRECVID 2005 dataset (170 hours)

39 concepts (LSCOM-Lite)

Training (65%), Validation (16%), Testing (19%)

CML ( MAP=0.290 ) improves IndSVM ( MAP=0.246 ) 17% and CBCF ( MAP=0.253 ) 14%

Correlative Multi-Label Video Annotation A new paradigm for multi-label annotation Models correlations and concepts simultaneously Has a close connection to Gibbs Random Field

Correlative Multi-Label Video Annotation

A new paradigm for multi-label annotation

Models correlations and concepts simultaneously

Has a close connection to Gibbs Random Field

Multi-Instance Multi-Label Annotation Exploit correlations among concepts and among instances at the same time Not only can get image/frame level annotation, but also can get region level annotation Sky Mountain Water Sands Scenery

Multi-Instance Multi-Label Annotation

Exploit correlations among concepts and among instances at the same time

Not only can get image/frame level annotation, but also can get region level annotation

Correlative Multi-Label Video Annotation A new paradigm for multi-label annotation Models correlations and concepts simultaneously Has a close connection to Gibbs Random Field

Correlative Multi-Label Video Annotation

A new paradigm for multi-label annotation

Models correlations and concepts simultaneously

Has a close connection to Gibbs Random Field