CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications

Semi-Supervised Learning in Computer Vision
Part II

Amir Saffari,Christian Leistner,Horst Bischof

Institute for Computer Graphics and Vision, Graz University of Technology

June 18th, 2010

SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line Manifo

Outline

1 SemiBoost & Visual Similarity Learning

2 On-line Semi-supervised Boosting
Tracking

3 Semi-Supervised Random Forests
MILForests
On-line Random Forests

4 On-line Manifold Regularization

5 Conclusion & Outlook

Graz University of Technology

Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II


SemiBoost
[Mallapragada et al.,PAMI’09] [Leistner et al.,CVPR’08]

Loss function
(x,y)∈XL e −yF (x) +

F (x)
λu s(x, x ) cosh(F (x) − F (x )) + λl s(x, x )e −2y
x∈XU x ∈XU (x ,y )∈XL

Optimization Problem

arg min = s(x, x )e −2y(F (x )+αf (x ))
f (x),α x ∈XU (x,y)∈XL

+λu s(x, x )e ((F (x )−F (x)) e α(f (x)−f (x ))
x ∈XU Graz University of Technology



SemiBoost

λu
px = λl I(y = 1)s(x, x )e −2F (x ) + s(x, x )e F (x )−F (x)
2
(x ,y )∈XL x∈XU

and
λu
qx = λl I(y = −1)s(x, x )e −2F (x ) + s(x, x )e F (x)−F (x )
2
(x ,y )∈XL x∈XU




SemiBoost

Pseudo Labels and Weights

ˆ
yx = sign(px − qx )
wx = |px − qx |

Optimal α
1 x∈XU pi I(f (x) = 1) + qi I(f (x) = −1)
α= ln
4 x∈XU pi I(f (x) = −1) + qi I(f (x) = 1)




SemiBoost

labeled training data (x, y) ∈ XL and unlabeled data x ∈ XU
Similarity measure s(x, x )
Weak learners fi
weight parameters λu , λl
max iterations T

1 For t = 1, 2, . . . , T
2 Compute pi and qi for every given sample
3 ˆ
yx = sign(px − qx )
4 wx = |px − qx |
5 Train weak classiﬁer ft (x)
6 Compute αt
7 F (x) ← F (x) + αt ft (x)
8 EndFor Graz University of Technology



SemiBoost with learned Similarities
[Hertz et al.,CVPR’04]

Radial Basis Function [Zhu et al.,ICML’03]
d(x,x )2
−
σ2
s(x, x ) = e

d(x, x ) . . . distance between points




Learning Distance Functions

Idea
Learn distance or metric function on labeled data which then
can discriminatively support task-speciﬁc classiﬁcation.

Distance Function
F d : X × X → Y = [−1 1]

Training Pairs of “same” or “different” [Hertz et al.,CVPR’04]

Dd = {(x, x , +1)|y = y , x, x ∈ DL } ∪
∪{(x, x , −1)|y y , x, x ∈ DL }




SemiBoost with learned Distance Functions

Number of Training Pairs (Symmetric case)
n·(n−1)
2

?
+- ? +
?
+ - SemiBoost
?
+ - ?




Using Arbitrary Classiﬁers

Approximate pair-wise classiﬁer
|F (x, x )| ≈ |F (x) − F (x )|

+

+ ?

?
-




Reusing Prior Classiﬁers
[Schapire et al,ML’02]

Classiﬁer Combination
F C (x) = α0 F P (x) + F (x)

?
? +
SemiBoost ?
?
- ?




SemiBoost Applications

Car Detection




Similarity Performance

Accuracy depending on the number of samples





Face Detection

(a) prior (b) trained (c) combined




Simple Data mining method
[Levin et al.,ICCV’03][Rosenberg et al.,2005]

1 Labeled training data (x, y) ∈ XL
2 Train cascaded detector F P (x) on XL using [Viola & Jones,2001]
3 Use a web image search engine in order to collect huge
amounts of possibly useful images XU ; pass phrases that are
much likely related to your target object
4 Apply F P (x) in a sliding window manner on XU and copy all
∗
detections to XU
∗
5 Train a SemiBoost classifier F (x) on XL and XU using F P (x)
as prior
6 Output the final classifier F (x)





Transfer Learning




On-line Boosting




Tracking

[Oza,PhD-Thesis’01], [Grabner & Bischof,CVPR’06]




Tracking




Tracking

Tracking is an One-Shot Semi-supervised Learning Problem




On-line SemiBoost

P
−Fn−1 (x) −Fn−1 (x) + e −Fn−1 (x) e F (x)
˜
px ≈ e S(x, xi ) ≈ e F (x) ≈ F P (x) P
xi ∈X+
e + e −F (x)

P
e Fn−1 (x) e −F (x)
qx ≈ e Fn−1 (x)
˜ S(x, xi ) ≈ e Fn−1 (x) F − (x) ≈ P P
xi ∈X−
e F (x) + e −F (x)

sinh(F P (x) − Fn−1 )
˜ ˜
pn (x)−qn (x) = = tanh(F P (x))−tanh(Fn−1 (x))
cosh(F P (x))




Tracking

Problem: Rapid Appearance Changes




Exploration-Exploitation Dilemma

Convex Trade-off
(F (x)) = (1 − α) l (F (x)) + α u (F (x))

We need more Robustness when minimizing the labeled loss!




Loss Functions

Random classiﬁcation noise defeats all convex potential boosters
[Long and Servidio,ICML’08]




On-line Gradient Boost

Gradient Descent Functional Gradient Descent

GradientBoost [Friedman et al.,Annals of Statistics’01]
ft (x) = arg max − LT f (x)
f (x)




On-line Gradient Boost
A training sample: (xn , yn ), A differentiable loss function (·)
Number of selectors M , Number of weak learners per selector K
1 Set F0 (xn ) = 0.
2 Set the initial weight wn = − (0).
3 For m = 1 to M
4 For k = 1 to K
5 Train k th weak learner fm (x) with sample (xn , yn ) and weight wn .
k

k k k
6 em ← em + wn I(sign(fm (xn )) yn ) //Compute the error
7 EndFor
8 Find the best weak learner with the least total weighted error:
k
j = arg min em .
k
j
9 Set fm (xn ) = fm (xn ).
10 Set Fm (xn ) = Fm−1 (xn ) + fm (xn ).
11 Set the weight wn = − (yn Fm (xn )).
12 EndFor Graz University of Technology

13


Weight Updates




Co-Training of Pedestrian Detectors

Exponential Loss Logit Loss




SERBoost

Expectation Regularization [Mann and MacCallum,ICML’07]
Penalize model predictions on unlabeled data that deviate from
certain expectation.

SERBoost [Saffari et al.,ECCV’08]
L(H (x), X) = Ll (H (x), Xl ) + βLu (H (x), Xu )

L(H (x), X) = e −yH (x) + e −yp H (x) cosh(H (x))
x∈XL x∈XU

Pseudo Label
+
yp = 2Pp (x) − 1



On-line SERBoost with logistic loss

Supervised Loss

Ll (XL ) = log 1 + e −2yF (x)
(x,y)∈Xl

= log e −yF (x) (e yF (x) + e −yF (x) )
(x,y)∈XL

= −yF (x) + log e F (x) + e −F (x) .
(x,y)∈XL





Minimize the cross entropy

H (Pp , P) = − Pp (y = z|x) log P(y = z|x)
z∈{−1,1}

= − 2Pp (y = 1|x) − 1 F (x) + log e F (x) + e −F (x)
yp (x)





Unsupervised Loss

Lu (XU ) = ˆ
H (Pp , P) = −yp (x)F (x) + log e F (x) + e −F (x)
x∈XU x∈XU





Unsupervised Loss

Lu (XU ) = ˆ
H (Pp , P) = −yp (x)F (x) + log e F (x) + e −F (x)
x∈XU x∈XU

Unlabeled Update

∀ x ∈ XU :wx = yp (x) − tanh(F (x))
ˆ
yx = sign yp (x) − tanh(F (x))




OSER Tracking

λ = 0.5




Inﬂuence of convex combination




Multiple Instance Boosting
[Viola et al.,NIPS’05][Babenko et al.,CVPR’09]




Multiple Instance Boosting
[Viola et al.,NIPS’05][Babenko et al.,CVPR’09]

Bags
{(B1 , y1 ), . . . , (Bn , yn )}
Bi = {xi1 , xi2 , . . . , xini }

Minimize binary log-likelihood

log L= (yi log p(yi ) + (1 − yi ) log p(yi ))
i




Semi-Supervised Multiple Instance Boosting
[Zeisl et al.,CVPR’10]

Combine beneﬁts of MIL and SSL





Unlabeled Loss of the Bags
Nu
Lu (XB ) = −
u Pp (z|Bu ) log(P(z|Bu ))
i i
i=1 z∈Y

Approximate max with geometric mean
NBi
1/NBi
P(y = 1|Bi ) = 1 − 1 − P(y = 1|xij )
j=1





Experimental Results

Sequence MILSER MIL OSB OAB
sylv 0.64 0.61 0.46 0.50
david 0.71 0.54 0.31 0.32
faceocc2 0.78 0.65 0.63 0.64
coke11 0.18 0.29 0.12 0.20
tiger1 0.60 0.51 0.17 0.27
tiger2 0.46 0.50 0.08 0.25
faceocc1 0.68 0.63 0.71 0.47
girl 0.64 0.53 0.69 0.38




On-line Co-Training
[Liu et al.,ICCV’09][Saffari et al.,ECCV’10]

Performance measured in average location center errors in pixels

Approach sylv david faceocc2 tiger1 tiger2 coke faceocc1 girl
MV-GPBoost 17 20 10 15 16 20 12 15
CoBoost 15 33 11 22 19 14 13 17
SemiBoost 22 59 43 46 53 85 41 52
MILBoost 11 23 20 15 17 21 27 32




End Part I




Random Forests

[Breiman,ML’01]

Ensemble of n decision trees
N
F (x) = n=1 f (x)

Information Gain
|Il | |Ir |
∆H = − |I |+|Ir | H (Il ) − |Il |+|Ir | H (Ir )
l




Random Forests

Advantages:
speed
parallelism
noise robust
inherently multi-class




Random Forests

Advantages:
speed
parallelism
noise robust
Applications:
Object Detection, Semantic Segmentation, Categorization,
Tracking, etc.




Random Forests

Advantages:
speed
parallelism
noise robust
Applications:
Object Detection, Semantic Segmentation, Categorization,
Tracking, etc.
Disadvantage:
RFs demand a huge amount of data in order to leverage their
full potential [Caruana et al.,ICML’08]




Optimization

Incorporate labels for the unlabeled data as additional
optimization variables!




Optimization

Incorporate labels for the unlabeled data as additional
optimization variables!

Deterministic Annealing [Rose,IJCNN’98]

p ∗ = arg minEp (F(y)) − T H(p)
p∈P

T0 > T1 > . . . > T∞ = 0

p ∗ . . . distributions over the label predictions




Two Step Optimization

First Stage

1
f∗ = arg min
n (fy (x))+
f |Xl |
(x,y)∈Xl
α
+ (fyu (x))
ˆ
|Xu |
x∈Xu




Finding the optimal Distributions

Take the derivate w.r.t. each class
ˆ ˆ ˆ
hi (p, x) = p(i|x)(α (gi (x)) + T log(p(i|x))) (1)
dhi
ˆ
= α (gi (x)) + T log(p(i|x)) + T (2)
ˆ
d pi

Optimal Distribution
p ∗ (i|x) = exp(− α
ˆ (fi (x))+T
T )/Z (x)

K
Z (x) = ˆ∗
i=1 p (i|x) is the partition function



Experiments

Classiﬁcation Accuracy in %

Method SVM TSVM SER RMSB RF DAS-RF
g50c 91.7 93.1 91.9 94.2 89.1 93.3
Letter 70.3 65.9 76.5 79.9 76.4 79.7
SensIt 80.2 79.9 81.9 83.7 76.5 84.3

Train and Test time in Seconds

Method SVM TSVM SER RMSB RF DAS-RF GPU
Letter 25 74 3124 125 35 72 29
SensIt 195 687 1158 514 125 410 137



Caltech-101

binary classiﬁcation error
Class RF DAS-RF Improvement
C4 0.0081 0.0033 58%
C5 0.0078 0.002 65%
C20 0.011 0.0013 87.5%
C33 0.007 0.003 52%
C81 0.0027 0.001 62.5%
classiﬁcation error over different numbers of labeled samples
Algorithm l = 15 l = 30
RF 0.72 0.64
DAS-RF 0.70 0.60
LinSVM 0.74 0.65

improvement 2% 4%


Prior Regularization

Potential Information Gain
|Il | |Ir |
∆H = − |I |+|Ir | H (Il ) − |Il |+|Ir | H (Ir )
l

Kullback-Leibler Divergence
DKL (q p) = H (q, p) − H (q)
1
DSKL (q p) = 2 (DKL (q p) + DKL (p q))

Prior-regularized node score
∆H ∗ = ∆H + λ∆DSKL (q p)
ˆ




Airbag

m−1 m
OOBE : eF − eF




Multiple Instance Forests
[Leistner et al.,ECCV’10]

-
-
- -
+ -

-
+ -

- -
-
+
-

[Dietterich,AI’97]

Content-based Image Retrieval
Object Detection and Categorization
Tracking
Action Recognition




Multiple Instance Learning is a special case of semi-supervised
Learning!





Multi-class Instance Classiﬁer
F (x) : X → Y = {1, . . . , K }
{(B1 , y1 ), . . . , (Bn , yn )}, where yi ∈ {1, . . . , K }

Objective Function

n ni
j j
({yi }∗ , F ∗ ) =arg min (Fy j (xi ))
j i
{yi },F (·) i=1 j=1
ni
j
s.t. ∀i : I(yi = arg max Fk (xi )) 1.
j=1 k∈Y





DA Loss Function

n ni K n
j j
LDA (F , p) =
ˆ ˆ
p(k|xi ) (Fk (xi )) +T ˆ
H (pi )
i=1 j=1 k=1 i=1

Entropy of the distribution inside a bag
ni K
j j
ˆ
H (pi ) = − ˆ ˆ
p(k|xi ) log(p(k|xi ))
j=1 k=1




Evaluation

Method Elephant Fox Tiger Musk1 Musk2
RandomForest[Breiman,2001] 74 60 77 85 78
MILForest 84 64 82 85 82
MI-Kernel[Andrews,2003] 84 60 84 88 89
MI-SVM[Zhou,2009] 81 59 84 78 84
mi-SVM[Zhou,2009] 82 58 79 87 84
MILES[Chen,2006] 81 62 80 88 83
SIL-SVM[Bunescu,2007] 85 53 77 88 87
AW-SVM[Gehler,2007] 82 64 83 86 84
AL-SVM[Gehler,2007] 79 63 78 86 83
EM-DD[Zhang,2001] 78 56 72 85 85
MILBoost-NOR[Viola,2006] 73 58 56 71 61




Corel Data Set




Corel Data Set

Results for the COREL image categorization benchmark

Method Corel-1000 Corel-2000 Testing[sec.] Training[sec.]
MILForest 59 66 4.6 22.0
MILES 58 67 180 960




Semantic Segmentation

[Vezhnevets & Buhmann,CVPR’10]




Interactive Segmentation




Tracking with On-line RF




Tracking

RT ∩ RGT /RT ∪ RGT

Sequence OSERB MILBoost OSB OAB ORF MILForest RF
sylv 0.64 0.61 0.46 0.50 0.53 0.59 0.50
david 0.69 0.54 0.31 0.32 0.69 0.72 0.32
faceocc2 0.77 0.65 0.63 0.64 0.72 0.77 0.79
tiger1 0.65 0.51 0.17 0.27 0.38 0.55 0.34
tiger2 0.42 0.50 0.08 0.25 0.43 0.53 0.32
coke 0.2 0.33 0.08 0.25 0.35 0.35 0.15
faceocc1 0.77 0.63 0.71 0.47 0.71 0.77 0.77
girl 0.77 0.53 0.69 0.38 0.70 0.71 0.74




On-line Manifold Regularization
[Goldberg et al.,ECML’08]

Based on Convex Programming in kernel space using
stochastic gradient descent
Random Projection Trees [Dasgupta & Freund, TR, 2007]




On-line Manifold Regularization
[Goldberg et al.,ECML’08]




On-line Graph-based SSL
[Kveton et al.,OLCV’10]

Harmonic Function Solution





Merge the two most similar vertices and add the new vertex





Face recognition of 8 people




Conclusion

Semi-supervised Learning is a powerful learning paradigm with
many potential applications in Computer Vision




Conclusion

Semi-supervised Learning is a powerful learning paradigm with
many potential applications in Computer Vision

It is often also the way how learning is done in nature
It can be applied virtually everywhere where classiﬁers are
applied
On-line SSL can be used in order to make
tracking-by-detection systems more robust




Outlook

We need to increase the robustness of SSL algorithms in order to
leverage more applications




Outlook

We need to increase the robustness of SSL algorithms in order to
leverage more applications

Demand for more on-line Semi-Supervised Methods
SSL from weakly-related unlabeled data




References
Books
O. Chapelle and B. Schoelkopf and A. Zien, “Semi-Supervised Learning”, The MIT Press, 2006
Xiaojin Zhu and Andrew B. Goldberg, “Introduction to Semi-Supervised Learning”, Morgan & Claypool, 2009
Papers and Articles
C. Leistner, A. Saffari and H. Bischof, “MILForests: Multiple-Instance Learning with Randomized
Trees”,ECCV’10
C. Leistner, A. Saffari, J. Santner and H. Bischof: “,Semi-Supervised Random Forests”,ICCV’09
C. Leistner, A. Saffari, P Roth and H. Bischof: “On Robustness of On-line Boosting – A Competitive
.M.
Study”,(ICCV) OLCV’09
H. Grabner, C. Leistner and H. Bischof: “On-line Semi-Supervised Boosting for Robust Tracking”,ECCV’08
B. Zeisl, C. Leistner, A. Saffari and H. Bischof: “On-line Semi-supervised Multiple-Instance Boosting”,CVPR’10
C. Leistner, “Semi-Supervised Ensemble Methods for Computer Vision”, PhD-Thesis, Graz University of
Technology, 2010
A. Saffari, C. Leistner, M. Godec, J. Santner and H. Bischof, “On-line Random Forests”, (ICCV) OLCV’09
A. Saffari, C. Leistner, M. Godec and H. Bischof, “Robust Multi-View Multi-Class Boosting with Priors”,ECCV’10
B. Kveton, M. Valko, M. Philipose and L. Huang, “Online Semi-Supervised Perception: Real-Time Learning
without Explicit Feedback”, (CVPR) OLCV’10
A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09
C. Leistner, H. Grabner and H. Bischof, “Semi-Supervised Boosting using Visual Similarity Learning”,CVPR’08
A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09
A. Saffari, H. Grabner and H. Bischof, “SERBoost: Semi-supervised Boosting with Expectation Graz University of Technology
Regularization”,ECCV’08


CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications

More Related Content

What's hot

Viewers also liked

Similar to CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications

More from zukun

Recently uploaded

CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications