Upcoming SlideShare
×

# CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications

1,293 views

Published on

Published in: Education
1 Like
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
1,293
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
69
0
Likes
1
Embeds 0
No embeds

No notes for slide

### CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications

1. 1. Semi-Supervised Learning in Computer Vision Part II Amir Saffari,Christian Leistner,Horst Bischof Institute for Computer Graphics and Vision, Graz University of Technology June 18th, 2010
2. 2. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
3. 3. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost[Mallapragada et al.,PAMI’09] [Leistner et al.,CVPR’08] Loss function (x,y)∈XL e −yF (x) + F (x) λu s(x, x ) cosh(F (x) − F (x )) + λl s(x, x )e −2y x∈XU x ∈XU (x ,y )∈XL Optimization Problem arg min = s(x, x )e −2y(F (x )+αf (x )) f (x),α x ∈XU (x,y)∈XL +λu s(x, x )e ((F (x )−F (x)) e α(f (x)−f (x )) x ∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
4. 4. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost λu px = λl I(y = 1)s(x, x )e −2F (x ) + s(x, x )e F (x )−F (x) 2 (x ,y )∈XL x∈XU and λu qx = λl I(y = −1)s(x, x )e −2F (x ) + s(x, x )e F (x)−F (x ) 2 (x ,y )∈XL x∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
5. 5. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Pseudo Labels and Weights ˆ yx = sign(px − qx ) wx = |px − qx | Optimal α 1 x∈XU pi I(f (x) = 1) + qi I(f (x) = −1) α= ln 4 x∈XU pi I(f (x) = −1) + qi I(f (x) = 1) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
6. 6. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost labeled training data (x, y) ∈ XL and unlabeled data x ∈ XU Similarity measure s(x, x ) Weak learners fi weight parameters λu , λl max iterations T 1 For t = 1, 2, . . . , T 2 Compute pi and qi for every given sample 3 ˆ yx = sign(px − qx ) 4 wx = |px − qx | 5 Train weak classiﬁer ft (x) 6 Compute αt 7 F (x) ← F (x) + αt ft (x) 8 EndFor Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
7. 7. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost with learned Similarities[Hertz et al.,CVPR’04] Radial Basis Function [Zhu et al.,ICML’03] d(x,x )2 − σ2 s(x, x ) = e d(x, x ) . . . distance between points Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
8. 8. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoLearning Distance Functions Idea Learn distance or metric function on labeled data which then can discriminatively support task-speciﬁc classiﬁcation. Distance Function F d : X × X → Y = [−1 1] Training Pairs of “same” or “different” [Hertz et al.,CVPR’04] Dd = {(x, x , +1)|y = y , x, x ∈ DL } ∪ ∪{(x, x , −1)|y y , x, x ∈ DL } Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
9. 9. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost with learned Distance Functions Number of Training Pairs (Symmetric case) n·(n−1) 2 ? +- ? + ? + - SemiBoost ? + - ? Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
10. 10. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoUsing Arbitrary Classiﬁers Approximate pair-wise classiﬁer |F (x, x )| ≈ |F (x) − F (x )| + + ? ? - Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
11. 11. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoReusing Prior Classiﬁers[Schapire et al,ML’02] Classiﬁer Combination F C (x) = α0 F P (x) + F (x) ? ? + SemiBoost ? ? - ? Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
12. 12. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Car Detection Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
13. 13. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSimilarity Performance Accuracy depending on the number of samples Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
14. 14. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Car Detection Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
15. 15. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Face Detection (a) prior (b) trained (c) combined Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
16. 16. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSimple Data mining method[Levin et al.,ICCV’03][Rosenberg et al.,2005] 1 Labeled training data (x, y) ∈ XL 2 Train cascaded detector F P (x) on XL using [Viola & Jones,2001] 3 Use a web image search engine in order to collect huge amounts of possibly useful images XU ; pass phrases that are much likely related to your target object 4 Apply F P (x) in a sliding window manner on XU and copy all ∗ detections to XU ∗ 5 Train a SemiBoost classiﬁer F (x) on XL and XU using F P (x) as prior 6 Output the ﬁnal classiﬁer F (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
17. 17. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Transfer Learning Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
18. 18. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Transfer Learning Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
19. 19. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
20. 20. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Boosting Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
21. 21. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking [Oza,PhD-Thesis’01], [Grabner & Bischof,CVPR’06] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
22. 22. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
23. 23. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
24. 24. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
25. 25. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Tracking is an One-Shot Semi-supervised Learning Problem Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
26. 26. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SemiBoost P −Fn−1 (x) −Fn−1 (x) + e −Fn−1 (x) e F (x) ˜ px ≈ e S(x, xi ) ≈ e F (x) ≈ F P (x) P xi ∈X+ e + e −F (x) P e Fn−1 (x) e −F (x) qx ≈ e Fn−1 (x) ˜ S(x, xi ) ≈ e Fn−1 (x) F − (x) ≈ P P xi ∈X− e F (x) + e −F (x) sinh(F P (x) − Fn−1 ) ˜ ˜ pn (x)−qn (x) = = tanh(F P (x))−tanh(Fn−1 (x)) cosh(F P (x)) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
27. 27. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
28. 28. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
29. 29. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Problem: Rapid Appearance Changes Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
30. 30. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
31. 31. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoExploration-Exploitation Dilemma Convex Trade-off (F (x)) = (1 − α) l (F (x)) + α u (F (x)) We need more Robustness when minimizing the labeled loss! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
32. 32. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoLoss Functions Random classiﬁcation noise defeats all convex potential boosters [Long and Servidio,ICML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
33. 33. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Gradient Boost Gradient Descent Functional Gradient Descent GradientBoost [Friedman et al.,Annals of Statistics’01] ft (x) = arg max − LT f (x) f (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
34. 34. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Gradient Boost A training sample: (xn , yn ), A differentiable loss function (·) Number of selectors M , Number of weak learners per selector K 1 Set F0 (xn ) = 0. 2 Set the initial weight wn = − (0). 3 For m = 1 to M 4 For k = 1 to K 5 Train k th weak learner fm (x) with sample (xn , yn ) and weight wn . k k k k 6 em ← em + wn I(sign(fm (xn )) yn ) //Compute the error 7 EndFor 8 Find the best weak learner with the least total weighted error: k j = arg min em . k j 9 Set fm (xn ) = fm (xn ). 10 Set Fm (xn ) = Fm−1 (xn ) + fm (xn ). 11 Set the weight wn = − (yn Fm (xn )). 12 EndFor Graz University of Technology 13 Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
35. 35. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoWeight Updates Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
36. 36. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCo-Training of Pedestrian Detectors Exponential Loss Logit Loss Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
37. 37. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSERBoost Expectation Regularization [Mann and MacCallum,ICML’07] Penalize model predictions on unlabeled data that deviate from certain expectation. SERBoost [Saffari et al.,ECCV’08] L(H (x), X) = Ll (H (x), Xl ) + βLu (H (x), Xu ) L(H (x), X) = e −yH (x) + e −yp H (x) cosh(H (x)) x∈XL x∈XU Pseudo Label + yp = 2Pp (x) − 1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
38. 38. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Supervised Loss Ll (XL ) = log 1 + e −2yF (x) (x,y)∈Xl = log e −yF (x) (e yF (x) + e −yF (x) ) (x,y)∈XL = −yF (x) + log e F (x) + e −F (x) . (x,y)∈XL Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
39. 39. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Minimize the cross entropy H (Pp , P) = − Pp (y = z|x) log P(y = z|x) z∈{−1,1} = − 2Pp (y = 1|x) − 1 F (x) + log e F (x) + e −F (x) yp (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
40. 40. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Unsupervised Loss Lu (XU ) = ˆ H (Pp , P) = −yp (x)F (x) + log e F (x) + e −F (x) x∈XU x∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
41. 41. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Unsupervised Loss Lu (XU ) = ˆ H (Pp , P) = −yp (x)F (x) + log e F (x) + e −F (x) x∈XU x∈XU Unlabeled Update ∀ x ∈ XU :wx = yp (x) − tanh(F (x)) ˆ yx = sign yp (x) − tanh(F (x)) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
42. 42. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOSER Tracking λ = 0.5 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
43. 43. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoInﬂuence of convex combination Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
44. 44. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Boosting[Viola et al.,NIPS’05][Babenko et al.,CVPR’09] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
45. 45. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Boosting[Viola et al.,NIPS’05][Babenko et al.,CVPR’09] Bags {(B1 , y1 ), . . . , (Bn , yn )} Bi = {xi1 , xi2 , . . . , xini } Minimize binary log-likelihood log L= (yi log p(yi ) + (1 − yi ) log p(yi )) i Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
46. 46. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Combine beneﬁts of MIL and SSL Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
47. 47. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Unlabeled Loss of the Bags Nu Lu (XB ) = − u Pp (z|Bu ) log(P(z|Bu )) i i i=1 z∈Y Approximate max with geometric mean NBi 1/NBi P(y = 1|Bi ) = 1 − 1 − P(y = 1|xij ) j=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
48. 48. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Gradient for NOR and geometric mean 2 z − P(y = 1|Bi ) aij (z) = P(y = 1|xij ) NBi P(y = 1|Bi ) Pseudo Labels and Weights wij =β Pp (z|Bu )aij (z) i z∈Y yij =I β Pp (z|Bu )aij (z) > 0 i z∈Y Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
49. 49. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Experimental Results Sequence MILSER MIL OSB OAB sylv 0.64 0.61 0.46 0.50 david 0.71 0.54 0.31 0.32 faceocc2 0.78 0.65 0.63 0.64 coke11 0.18 0.29 0.12 0.20 tiger1 0.60 0.51 0.17 0.27 tiger2 0.46 0.50 0.08 0.25 faceocc1 0.68 0.63 0.71 0.47 girl 0.64 0.53 0.69 0.38 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
50. 50. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Co-Training[Liu et al.,ICCV’09][Saffari et al.,ECCV’10] Performance measured in average location center errors in pixels Approach sylv david faceocc2 tiger1 tiger2 coke faceocc1 girl MV-GPBoost 17 20 10 15 16 20 12 15 CoBoost 15 33 11 22 19 14 13 17 SemiBoost 22 59 43 46 53 85 41 52 MILBoost 11 23 20 15 17 21 27 32 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
51. 51. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line Manifo End Part I Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
52. 52. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests [Breiman,ML’01] Ensemble of n decision trees N F (x) = n=1 f (x) Information Gain |Il | |Ir | ∆H = − |I |+|Ir | H (Il ) − |Il |+|Ir | H (Ir ) l Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
53. 53. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
54. 54. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Applications: Object Detection, Semantic Segmentation, Categorization, Tracking, etc. Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
55. 55. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Applications: Object Detection, Semantic Segmentation, Categorization, Tracking, etc. Disadvantage: RFs demand a huge amount of data in order to leverage their full potential [Caruana et al.,ICML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
56. 56. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Random Forests Random Forests maximize the margin ml (x, y) = p(y|x) − max p(k|x) k∈Y k y Unlabeled Margin mu (xu ) = max fi (xu ) i∈Y Semi-supervised Loss 1 λ L(f) = (fy (x)) + (mu (x)) |Xl | |Xu | (x,y)∈Xl x∈Xu Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
57. 57. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization Incorporate labels for the unlabeled data as additional optimization variables! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
58. 58. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization Incorporate labels for the unlabeled data as additional optimization variables! Deterministic Annealing [Rose,IJCNN’98] p ∗ = arg minEp (F(y)) − T H(p) p∈P T0 > T1 > . . . > T∞ = 0 p ∗ . . . distributions over the label predictions Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
59. 59. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization DA-Loss for Semi-supervised Random Forests 1 LDA (f, p) = ˆ (fy (x))+ |Xl | (x,y)∈Xl K α + ˆ p(i|x) (fi (x))+ |Xu | x∈Xu i=1 K T + ˆ H (p) |Xu | x∈Xu i=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
60. 60. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTwo Step Optimization First Stage 1 f∗ = arg min n (fy (x))+ f |Xl | (x,y)∈Xl α + (fyu (x)) ˆ |Xu | x∈Xu Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
61. 61. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTwo Step Optimization Second Stage K α p∗ =arg min ˆ ˆ p(i|x) (fi (x))+ ˆ p |Xu | x∈Xu i=1 K T + ˆ ˆ p(i|x) log(p(i|x)) |Xu | x∈Xu i=1 p ∗ (i|x) = exp(− α ˆ (fi (x))+T T )/Z (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
62. 62. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoFinding the optimal Distributions Take the derivate w.r.t. each class ˆ ˆ ˆ hi (p, x) = p(i|x)(α (gi (x)) + T log(p(i|x))) (1) dhi ˆ = α (gi (x)) + T log(p(i|x)) + T (2) ˆ d pi Optimal Distribution p ∗ (i|x) = exp(− α ˆ (fi (x))+T T )/Z (x) K Z (x) = ˆ∗ i=1 p (i|x) is the partition function Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
63. 63. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoExperiments Classiﬁcation Accuracy in % Method SVM TSVM SER RMSB RF DAS-RF g50c 91.7 93.1 91.9 94.2 89.1 93.3 Letter 70.3 65.9 76.5 79.9 76.4 79.7 SensIt 80.2 79.9 81.9 83.7 76.5 84.3 Train and Test time in Seconds Method SVM TSVM SER RMSB RF DAS-RF GPU Letter 25 74 3124 125 35 72 29 SensIt 195 687 1158 514 125 410 137 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
64. 64. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCaltech-101 binary classiﬁcation error Class RF DAS-RF Improvement C4 0.0081 0.0033 58% C5 0.0078 0.002 65% C20 0.011 0.0013 87.5% C33 0.007 0.003 52% C81 0.0027 0.001 62.5% classiﬁcation error over different numbers of labeled samples Algorithm l = 15 l = 30 RF 0.72 0.64 DAS-RF 0.70 0.60 LinSVM 0.74 0.65 Graz University of Technology improvement 2% 4% Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
65. 65. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoPrior Regularization Potential Information Gain |Il | |Ir | ∆H = − |I |+|Ir | H (Il ) − |Il |+|Ir | H (Ir ) l Kullback-Leibler Divergence DKL (q p) = H (q, p) − H (q) 1 DSKL (q p) = 2 (DKL (q p) + DKL (p q)) Prior-regularized node score ∆H ∗ = ∆H + λ∆DSKL (q p) ˆ Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
66. 66. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoAirbag m−1 m OOBE : eF − eF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
67. 67. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoAirbag m−1 m OOBE : eF − eF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
68. 68. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
69. 69. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests[Leistner et al.,ECCV’10] - - - - + - - + - - - - + - [Dietterich,AI’97] Content-based Image Retrieval Object Detection and Categorization Tracking Action Recognition Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
70. 70. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests Multiple Instance Learning is a special case of semi-supervised Learning! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
71. 71. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests Multi-class Instance Classiﬁer F (x) : X → Y = {1, . . . , K } {(B1 , y1 ), . . . , (Bn , yn )}, where yi ∈ {1, . . . , K } Objective Function n ni j j ({yi }∗ , F ∗ ) =arg min (Fy j (xi )) j i {yi },F (·) i=1 j=1 ni j s.t. ∀i : I(yi = arg max Fk (xi )) 1. j=1 k∈Y Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
72. 72. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests DA Loss Function n ni K n j j LDA (F , p) = ˆ ˆ p(k|xi ) (Fk (xi )) +T ˆ H (pi ) i=1 j=1 k=1 i=1 Entropy of the distribution inside a bag ni K j j ˆ H (pi ) = − ˆ ˆ p(k|xi ) log(p(k|xi )) j=1 k=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
73. 73. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoEvaluation Method Elephant Fox Tiger Musk1 Musk2 RandomForest[Breiman,2001] 74 60 77 85 78 MILForest 84 64 82 85 82 MI-Kernel[Andrews,2003] 84 60 84 88 89 MI-SVM[Zhou,2009] 81 59 84 78 84 mi-SVM[Zhou,2009] 82 58 79 87 84 MILES[Chen,2006] 81 62 80 88 83 SIL-SVM[Bunescu,2007] 85 53 77 88 87 AW-SVM[Gehler,2007] 82 64 83 86 84 AL-SVM[Gehler,2007] 79 63 78 86 83 EM-DD[Zhang,2001] 78 56 72 85 85 MILBoost-NOR[Viola,2006] 73 58 56 71 61 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
74. 74. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCorel Data Set Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
75. 75. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCorel Data Set Results for the COREL image categorization benchmark Method Corel-1000 Corel-2000 Testing[sec.] Training[sec.] MILForest 59 66 4.6 22.0 MILES 58 67 180 960 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
76. 76. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemantic Segmentation [Vezhnevets & Buhmann,CVPR’10] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
77. 77. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
78. 78. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Random Forests On-line Bagging [Oza,PhD-Thesis’01] → Poisson(λ) On-line recursive splitting is hard → Tree Growing Info Gain |Rjls | |Rjrs | ∆L(Rj , s) = L(Rj ) − L(Rjls ) − L(Rjrs ) |Rj | |Rj | Splitting Rules |Rj | > α and ∃s ∈ S : ∆L(Rj , s) > β On-line DA → Annealing Schedule for each sample xi Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
79. 79. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Random Forests Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
80. 80. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoInteractive Segmentation Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
81. 81. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking with On-line RF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
82. 82. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
83. 83. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking RT ∩ RGT /RT ∪ RGT Sequence OSERB MILBoost OSB OAB ORF MILForest RF sylv 0.64 0.61 0.46 0.50 0.53 0.59 0.50 david 0.69 0.54 0.31 0.32 0.69 0.72 0.32 faceocc2 0.77 0.65 0.63 0.64 0.72 0.77 0.79 tiger1 0.65 0.51 0.17 0.27 0.38 0.55 0.34 tiger2 0.42 0.50 0.08 0.25 0.43 0.53 0.32 coke 0.2 0.33 0.08 0.25 0.35 0.35 0.15 faceocc1 0.77 0.63 0.71 0.47 0.71 0.77 0.77 girl 0.77 0.53 0.69 0.38 0.70 0.71 0.74 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
84. 84. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Manifold Regularization[Goldberg et al.,ECML’08] Based on Convex Programming in kernel space using stochastic gradient descent Random Projection Trees [Dasgupta & Freund, TR, 2007] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
85. 85. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Manifold Regularization[Goldberg et al.,ECML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
86. 86. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Harmonic Function Solution Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
87. 87. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Merge the two most similar vertices and add the new vertex Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
88. 88. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Face recognition of 8 people Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
89. 89. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoConclusion Semi-supervised Learning is a powerful learning paradigm with many potential applications in Computer Vision Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
90. 90. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoConclusion Semi-supervised Learning is a powerful learning paradigm with many potential applications in Computer Vision It is often also the way how learning is done in nature It can be applied virtually everywhere where classiﬁers are applied On-line SSL can be used in order to make tracking-by-detection systems more robust Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
91. 91. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutlook We need to increase the robustness of SSL algorithms in order to leverage more applications Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
92. 92. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutlook We need to increase the robustness of SSL algorithms in order to leverage more applications Demand for more on-line Semi-Supervised Methods SSL from weakly-related unlabeled data Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
93. 93. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoReferences Books O. Chapelle and B. Schoelkopf and A. Zien, “Semi-Supervised Learning”, The MIT Press, 2006 Xiaojin Zhu and Andrew B. Goldberg, “Introduction to Semi-Supervised Learning”, Morgan & Claypool, 2009 Papers and Articles C. Leistner, A. Saffari and H. Bischof, “MILForests: Multiple-Instance Learning with Randomized Trees”,ECCV’10 C. Leistner, A. Saffari, J. Santner and H. Bischof: “,Semi-Supervised Random Forests”,ICCV’09 C. Leistner, A. Saffari, P Roth and H. Bischof: “On Robustness of On-line Boosting – A Competitive .M. Study”,(ICCV) OLCV’09 H. Grabner, C. Leistner and H. Bischof: “On-line Semi-Supervised Boosting for Robust Tracking”,ECCV’08 B. Zeisl, C. Leistner, A. Saffari and H. Bischof: “On-line Semi-supervised Multiple-Instance Boosting”,CVPR’10 C. Leistner, “Semi-Supervised Ensemble Methods for Computer Vision”, PhD-Thesis, Graz University of Technology, 2010 A. Saffari, C. Leistner, M. Godec, J. Santner and H. Bischof, “On-line Random Forests”, (ICCV) OLCV’09 A. Saffari, C. Leistner, M. Godec and H. Bischof, “Robust Multi-View Multi-Class Boosting with Priors”,ECCV’10 B. Kveton, M. Valko, M. Philipose and L. Huang, “Online Semi-Supervised Perception: Real-Time Learning without Explicit Feedback”, (CVPR) OLCV’10 A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09 C. Leistner, H. Grabner and H. Bischof, “Semi-Supervised Boosting using Visual Similarity Learning”,CVPR’08 A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09 A. Saffari, H. Grabner and H. Bischof, “SERBoost: Semi-supervised Boosting with Expectation Graz University of Technology Regularization”,ECCV’08 Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II