CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications

  • 845 views
Uploaded on

 

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
845
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
46
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Semi-Supervised Learning in Computer Vision Part II Amir Saffari,Christian Leistner,Horst Bischof Institute for Computer Graphics and Vision, Graz University of Technology June 18th, 2010
  • 2. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 3. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost[Mallapragada et al.,PAMI’09] [Leistner et al.,CVPR’08] Loss function (x,y)∈XL e −yF (x) + F (x) λu s(x, x ) cosh(F (x) − F (x )) + λl s(x, x )e −2y x∈XU x ∈XU (x ,y )∈XL Optimization Problem arg min = s(x, x )e −2y(F (x )+αf (x )) f (x),α x ∈XU (x,y)∈XL +λu s(x, x )e ((F (x )−F (x)) e α(f (x)−f (x )) x ∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 4. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost λu px = λl I(y = 1)s(x, x )e −2F (x ) + s(x, x )e F (x )−F (x) 2 (x ,y )∈XL x∈XU and λu qx = λl I(y = −1)s(x, x )e −2F (x ) + s(x, x )e F (x)−F (x ) 2 (x ,y )∈XL x∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 5. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Pseudo Labels and Weights ˆ yx = sign(px − qx ) wx = |px − qx | Optimal α 1 x∈XU pi I(f (x) = 1) + qi I(f (x) = −1) α= ln 4 x∈XU pi I(f (x) = −1) + qi I(f (x) = 1) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 6. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost labeled training data (x, y) ∈ XL and unlabeled data x ∈ XU Similarity measure s(x, x ) Weak learners fi weight parameters λu , λl max iterations T 1 For t = 1, 2, . . . , T 2 Compute pi and qi for every given sample 3 ˆ yx = sign(px − qx ) 4 wx = |px − qx | 5 Train weak classifier ft (x) 6 Compute αt 7 F (x) ← F (x) + αt ft (x) 8 EndFor Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 7. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost with learned Similarities[Hertz et al.,CVPR’04] Radial Basis Function [Zhu et al.,ICML’03] d(x,x )2 − σ2 s(x, x ) = e d(x, x ) . . . distance between points Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 8. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoLearning Distance Functions Idea Learn distance or metric function on labeled data which then can discriminatively support task-specific classification. Distance Function F d : X × X → Y = [−1 1] Training Pairs of “same” or “different” [Hertz et al.,CVPR’04] Dd = {(x, x , +1)|y = y , x, x ∈ DL } ∪ ∪{(x, x , −1)|y y , x, x ∈ DL } Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 9. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost with learned Distance Functions Number of Training Pairs (Symmetric case) n·(n−1) 2 ? +- ? + ? + - SemiBoost ? + - ? Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 10. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoUsing Arbitrary Classifiers Approximate pair-wise classifier |F (x, x )| ≈ |F (x) − F (x )| + + ? ? - Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 11. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoReusing Prior Classifiers[Schapire et al,ML’02] Classifier Combination F C (x) = α0 F P (x) + F (x) ? ? + SemiBoost ? ? - ? Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 12. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Car Detection Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 13. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSimilarity Performance Accuracy depending on the number of samples Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 14. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Car Detection Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 15. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Face Detection (a) prior (b) trained (c) combined Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 16. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSimple Data mining method[Levin et al.,ICCV’03][Rosenberg et al.,2005] 1 Labeled training data (x, y) ∈ XL 2 Train cascaded detector F P (x) on XL using [Viola & Jones,2001] 3 Use a web image search engine in order to collect huge amounts of possibly useful images XU ; pass phrases that are much likely related to your target object 4 Apply F P (x) in a sliding window manner on XU and copy all ∗ detections to XU ∗ 5 Train a SemiBoost classifier F (x) on XL and XU using F P (x) as prior 6 Output the final classifier F (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 17. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Transfer Learning Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 18. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Transfer Learning Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 19. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 20. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Boosting Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 21. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking [Oza,PhD-Thesis’01], [Grabner & Bischof,CVPR’06] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 22. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 23. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 24. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 25. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Tracking is an One-Shot Semi-supervised Learning Problem Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 26. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SemiBoost P −Fn−1 (x) −Fn−1 (x) + e −Fn−1 (x) e F (x) ˜ px ≈ e S(x, xi ) ≈ e F (x) ≈ F P (x) P xi ∈X+ e + e −F (x) P e Fn−1 (x) e −F (x) qx ≈ e Fn−1 (x) ˜ S(x, xi ) ≈ e Fn−1 (x) F − (x) ≈ P P xi ∈X− e F (x) + e −F (x) sinh(F P (x) − Fn−1 ) ˜ ˜ pn (x)−qn (x) = = tanh(F P (x))−tanh(Fn−1 (x)) cosh(F P (x)) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 27. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 28. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 29. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Problem: Rapid Appearance Changes Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 30. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 31. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoExploration-Exploitation Dilemma Convex Trade-off (F (x)) = (1 − α) l (F (x)) + α u (F (x)) We need more Robustness when minimizing the labeled loss! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 32. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoLoss Functions Random classification noise defeats all convex potential boosters [Long and Servidio,ICML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 33. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Gradient Boost Gradient Descent Functional Gradient Descent GradientBoost [Friedman et al.,Annals of Statistics’01] ft (x) = arg max − LT f (x) f (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 34. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Gradient Boost A training sample: (xn , yn ), A differentiable loss function (·) Number of selectors M , Number of weak learners per selector K 1 Set F0 (xn ) = 0. 2 Set the initial weight wn = − (0). 3 For m = 1 to M 4 For k = 1 to K 5 Train k th weak learner fm (x) with sample (xn , yn ) and weight wn . k k k k 6 em ← em + wn I(sign(fm (xn )) yn ) //Compute the error 7 EndFor 8 Find the best weak learner with the least total weighted error: k j = arg min em . k j 9 Set fm (xn ) = fm (xn ). 10 Set Fm (xn ) = Fm−1 (xn ) + fm (xn ). 11 Set the weight wn = − (yn Fm (xn )). 12 EndFor Graz University of Technology 13 Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 35. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoWeight Updates Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 36. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCo-Training of Pedestrian Detectors Exponential Loss Logit Loss Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 37. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSERBoost Expectation Regularization [Mann and MacCallum,ICML’07] Penalize model predictions on unlabeled data that deviate from certain expectation. SERBoost [Saffari et al.,ECCV’08] L(H (x), X) = Ll (H (x), Xl ) + βLu (H (x), Xu ) L(H (x), X) = e −yH (x) + e −yp H (x) cosh(H (x)) x∈XL x∈XU Pseudo Label + yp = 2Pp (x) − 1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 38. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Supervised Loss Ll (XL ) = log 1 + e −2yF (x) (x,y)∈Xl = log e −yF (x) (e yF (x) + e −yF (x) ) (x,y)∈XL = −yF (x) + log e F (x) + e −F (x) . (x,y)∈XL Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 39. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Minimize the cross entropy H (Pp , P) = − Pp (y = z|x) log P(y = z|x) z∈{−1,1} = − 2Pp (y = 1|x) − 1 F (x) + log e F (x) + e −F (x) yp (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 40. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Unsupervised Loss Lu (XU ) = ˆ H (Pp , P) = −yp (x)F (x) + log e F (x) + e −F (x) x∈XU x∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 41. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Unsupervised Loss Lu (XU ) = ˆ H (Pp , P) = −yp (x)F (x) + log e F (x) + e −F (x) x∈XU x∈XU Unlabeled Update ∀ x ∈ XU :wx = yp (x) − tanh(F (x)) ˆ yx = sign yp (x) − tanh(F (x)) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 42. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOSER Tracking λ = 0.5 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 43. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoInfluence of convex combination Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 44. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Boosting[Viola et al.,NIPS’05][Babenko et al.,CVPR’09] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 45. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Boosting[Viola et al.,NIPS’05][Babenko et al.,CVPR’09] Bags {(B1 , y1 ), . . . , (Bn , yn )} Bi = {xi1 , xi2 , . . . , xini } Minimize binary log-likelihood log L= (yi log p(yi ) + (1 − yi ) log p(yi )) i Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 46. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Combine benefits of MIL and SSL Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 47. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Unlabeled Loss of the Bags Nu Lu (XB ) = − u Pp (z|Bu ) log(P(z|Bu )) i i i=1 z∈Y Approximate max with geometric mean NBi 1/NBi P(y = 1|Bi ) = 1 − 1 − P(y = 1|xij ) j=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 48. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Gradient for NOR and geometric mean 2 z − P(y = 1|Bi ) aij (z) = P(y = 1|xij ) NBi P(y = 1|Bi ) Pseudo Labels and Weights wij =β Pp (z|Bu )aij (z) i z∈Y yij =I β Pp (z|Bu )aij (z) > 0 i z∈Y Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 49. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Experimental Results Sequence MILSER MIL OSB OAB sylv 0.64 0.61 0.46 0.50 david 0.71 0.54 0.31 0.32 faceocc2 0.78 0.65 0.63 0.64 coke11 0.18 0.29 0.12 0.20 tiger1 0.60 0.51 0.17 0.27 tiger2 0.46 0.50 0.08 0.25 faceocc1 0.68 0.63 0.71 0.47 girl 0.64 0.53 0.69 0.38 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 50. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Co-Training[Liu et al.,ICCV’09][Saffari et al.,ECCV’10] Performance measured in average location center errors in pixels Approach sylv david faceocc2 tiger1 tiger2 coke faceocc1 girl MV-GPBoost 17 20 10 15 16 20 12 15 CoBoost 15 33 11 22 19 14 13 17 SemiBoost 22 59 43 46 53 85 41 52 MILBoost 11 23 20 15 17 21 27 32 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 51. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line Manifo End Part I Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 52. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests [Breiman,ML’01] Ensemble of n decision trees N F (x) = n=1 f (x) Information Gain |Il | |Ir | ∆H = − |I |+|Ir | H (Il ) − |Il |+|Ir | H (Ir ) l Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 53. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 54. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Applications: Object Detection, Semantic Segmentation, Categorization, Tracking, etc. Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 55. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Applications: Object Detection, Semantic Segmentation, Categorization, Tracking, etc. Disadvantage: RFs demand a huge amount of data in order to leverage their full potential [Caruana et al.,ICML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 56. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Random Forests Random Forests maximize the margin ml (x, y) = p(y|x) − max p(k|x) k∈Y k y Unlabeled Margin mu (xu ) = max fi (xu ) i∈Y Semi-supervised Loss 1 λ L(f) = (fy (x)) + (mu (x)) |Xl | |Xu | (x,y)∈Xl x∈Xu Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 57. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization Incorporate labels for the unlabeled data as additional optimization variables! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 58. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization Incorporate labels for the unlabeled data as additional optimization variables! Deterministic Annealing [Rose,IJCNN’98] p ∗ = arg minEp (F(y)) − T H(p) p∈P T0 > T1 > . . . > T∞ = 0 p ∗ . . . distributions over the label predictions Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 59. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization DA-Loss for Semi-supervised Random Forests 1 LDA (f, p) = ˆ (fy (x))+ |Xl | (x,y)∈Xl K α + ˆ p(i|x) (fi (x))+ |Xu | x∈Xu i=1 K T + ˆ H (p) |Xu | x∈Xu i=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 60. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTwo Step Optimization First Stage 1 f∗ = arg min n (fy (x))+ f |Xl | (x,y)∈Xl α + (fyu (x)) ˆ |Xu | x∈Xu Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 61. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTwo Step Optimization Second Stage K α p∗ =arg min ˆ ˆ p(i|x) (fi (x))+ ˆ p |Xu | x∈Xu i=1 K T + ˆ ˆ p(i|x) log(p(i|x)) |Xu | x∈Xu i=1 p ∗ (i|x) = exp(− α ˆ (fi (x))+T T )/Z (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 62. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoFinding the optimal Distributions Take the derivate w.r.t. each class ˆ ˆ ˆ hi (p, x) = p(i|x)(α (gi (x)) + T log(p(i|x))) (1) dhi ˆ = α (gi (x)) + T log(p(i|x)) + T (2) ˆ d pi Optimal Distribution p ∗ (i|x) = exp(− α ˆ (fi (x))+T T )/Z (x) K Z (x) = ˆ∗ i=1 p (i|x) is the partition function Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 63. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoExperiments Classification Accuracy in % Method SVM TSVM SER RMSB RF DAS-RF g50c 91.7 93.1 91.9 94.2 89.1 93.3 Letter 70.3 65.9 76.5 79.9 76.4 79.7 SensIt 80.2 79.9 81.9 83.7 76.5 84.3 Train and Test time in Seconds Method SVM TSVM SER RMSB RF DAS-RF GPU Letter 25 74 3124 125 35 72 29 SensIt 195 687 1158 514 125 410 137 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 64. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCaltech-101 binary classification error Class RF DAS-RF Improvement C4 0.0081 0.0033 58% C5 0.0078 0.002 65% C20 0.011 0.0013 87.5% C33 0.007 0.003 52% C81 0.0027 0.001 62.5% classification error over different numbers of labeled samples Algorithm l = 15 l = 30 RF 0.72 0.64 DAS-RF 0.70 0.60 LinSVM 0.74 0.65 Graz University of Technology improvement 2% 4% Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 65. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoPrior Regularization Potential Information Gain |Il | |Ir | ∆H = − |I |+|Ir | H (Il ) − |Il |+|Ir | H (Ir ) l Kullback-Leibler Divergence DKL (q p) = H (q, p) − H (q) 1 DSKL (q p) = 2 (DKL (q p) + DKL (p q)) Prior-regularized node score ∆H ∗ = ∆H + λ∆DSKL (q p) ˆ Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 66. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoAirbag m−1 m OOBE : eF − eF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 67. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoAirbag m−1 m OOBE : eF − eF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 68. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 69. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests[Leistner et al.,ECCV’10] - - - - + - - + - - - - + - [Dietterich,AI’97] Content-based Image Retrieval Object Detection and Categorization Tracking Action Recognition Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 70. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests Multiple Instance Learning is a special case of semi-supervised Learning! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 71. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests Multi-class Instance Classifier F (x) : X → Y = {1, . . . , K } {(B1 , y1 ), . . . , (Bn , yn )}, where yi ∈ {1, . . . , K } Objective Function n ni j j ({yi }∗ , F ∗ ) =arg min (Fy j (xi )) j i {yi },F (·) i=1 j=1 ni j s.t. ∀i : I(yi = arg max Fk (xi )) 1. j=1 k∈Y Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 72. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests DA Loss Function n ni K n j j LDA (F , p) = ˆ ˆ p(k|xi ) (Fk (xi )) +T ˆ H (pi ) i=1 j=1 k=1 i=1 Entropy of the distribution inside a bag ni K j j ˆ H (pi ) = − ˆ ˆ p(k|xi ) log(p(k|xi )) j=1 k=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 73. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoEvaluation Method Elephant Fox Tiger Musk1 Musk2 RandomForest[Breiman,2001] 74 60 77 85 78 MILForest 84 64 82 85 82 MI-Kernel[Andrews,2003] 84 60 84 88 89 MI-SVM[Zhou,2009] 81 59 84 78 84 mi-SVM[Zhou,2009] 82 58 79 87 84 MILES[Chen,2006] 81 62 80 88 83 SIL-SVM[Bunescu,2007] 85 53 77 88 87 AW-SVM[Gehler,2007] 82 64 83 86 84 AL-SVM[Gehler,2007] 79 63 78 86 83 EM-DD[Zhang,2001] 78 56 72 85 85 MILBoost-NOR[Viola,2006] 73 58 56 71 61 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 74. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCorel Data Set Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 75. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCorel Data Set Results for the COREL image categorization benchmark Method Corel-1000 Corel-2000 Testing[sec.] Training[sec.] MILForest 59 66 4.6 22.0 MILES 58 67 180 960 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 76. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemantic Segmentation [Vezhnevets & Buhmann,CVPR’10] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 77. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 78. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Random Forests On-line Bagging [Oza,PhD-Thesis’01] → Poisson(λ) On-line recursive splitting is hard → Tree Growing Info Gain |Rjls | |Rjrs | ∆L(Rj , s) = L(Rj ) − L(Rjls ) − L(Rjrs ) |Rj | |Rj | Splitting Rules |Rj | > α and ∃s ∈ S : ∆L(Rj , s) > β On-line DA → Annealing Schedule for each sample xi Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 79. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Random Forests Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 80. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoInteractive Segmentation Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 81. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking with On-line RF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 82. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 83. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking RT ∩ RGT /RT ∪ RGT Sequence OSERB MILBoost OSB OAB ORF MILForest RF sylv 0.64 0.61 0.46 0.50 0.53 0.59 0.50 david 0.69 0.54 0.31 0.32 0.69 0.72 0.32 faceocc2 0.77 0.65 0.63 0.64 0.72 0.77 0.79 tiger1 0.65 0.51 0.17 0.27 0.38 0.55 0.34 tiger2 0.42 0.50 0.08 0.25 0.43 0.53 0.32 coke 0.2 0.33 0.08 0.25 0.35 0.35 0.15 faceocc1 0.77 0.63 0.71 0.47 0.71 0.77 0.77 girl 0.77 0.53 0.69 0.38 0.70 0.71 0.74 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 84. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Manifold Regularization[Goldberg et al.,ECML’08] Based on Convex Programming in kernel space using stochastic gradient descent Random Projection Trees [Dasgupta & Freund, TR, 2007] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 85. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Manifold Regularization[Goldberg et al.,ECML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 86. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Harmonic Function Solution Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 87. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Merge the two most similar vertices and add the new vertex Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 88. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Face recognition of 8 people Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 89. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoConclusion Semi-supervised Learning is a powerful learning paradigm with many potential applications in Computer Vision Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 90. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoConclusion Semi-supervised Learning is a powerful learning paradigm with many potential applications in Computer Vision It is often also the way how learning is done in nature It can be applied virtually everywhere where classifiers are applied On-line SSL can be used in order to make tracking-by-detection systems more robust Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 91. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutlook We need to increase the robustness of SSL algorithms in order to leverage more applications Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 92. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutlook We need to increase the robustness of SSL algorithms in order to leverage more applications Demand for more on-line Semi-Supervised Methods SSL from weakly-related unlabeled data Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  • 93. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoReferences Books O. Chapelle and B. Schoelkopf and A. Zien, “Semi-Supervised Learning”, The MIT Press, 2006 Xiaojin Zhu and Andrew B. Goldberg, “Introduction to Semi-Supervised Learning”, Morgan & Claypool, 2009 Papers and Articles C. Leistner, A. Saffari and H. Bischof, “MILForests: Multiple-Instance Learning with Randomized Trees”,ECCV’10 C. Leistner, A. Saffari, J. Santner and H. Bischof: “,Semi-Supervised Random Forests”,ICCV’09 C. Leistner, A. Saffari, P Roth and H. Bischof: “On Robustness of On-line Boosting – A Competitive .M. Study”,(ICCV) OLCV’09 H. Grabner, C. Leistner and H. Bischof: “On-line Semi-Supervised Boosting for Robust Tracking”,ECCV’08 B. Zeisl, C. Leistner, A. Saffari and H. Bischof: “On-line Semi-supervised Multiple-Instance Boosting”,CVPR’10 C. Leistner, “Semi-Supervised Ensemble Methods for Computer Vision”, PhD-Thesis, Graz University of Technology, 2010 A. Saffari, C. Leistner, M. Godec, J. Santner and H. Bischof, “On-line Random Forests”, (ICCV) OLCV’09 A. Saffari, C. Leistner, M. Godec and H. Bischof, “Robust Multi-View Multi-Class Boosting with Priors”,ECCV’10 B. Kveton, M. Valko, M. Philipose and L. Huang, “Online Semi-Supervised Perception: Real-Time Learning without Explicit Feedback”, (CVPR) OLCV’10 A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09 C. Leistner, H. Grabner and H. Bischof, “Semi-Supervised Boosting using Visual Similarity Learning”,CVPR’08 A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09 A. Saffari, H. Grabner and H. Bischof, “SERBoost: Semi-supervised Boosting with Expectation Graz University of Technology Regularization”,ECCV’08 Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II