CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications

1,293 views

Published on

Published in: Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,293
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
69
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applications

  1. 1. Semi-Supervised Learning in Computer Vision Part II Amir Saffari,Christian Leistner,Horst Bischof Institute for Computer Graphics and Vision, Graz University of Technology June 18th, 2010
  2. 2. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  3. 3. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost[Mallapragada et al.,PAMI’09] [Leistner et al.,CVPR’08] Loss function (x,y)∈XL e −yF (x) + F (x) λu s(x, x ) cosh(F (x) − F (x )) + λl s(x, x )e −2y x∈XU x ∈XU (x ,y )∈XL Optimization Problem arg min = s(x, x )e −2y(F (x )+αf (x )) f (x),α x ∈XU (x,y)∈XL +λu s(x, x )e ((F (x )−F (x)) e α(f (x)−f (x )) x ∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  4. 4. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost λu px = λl I(y = 1)s(x, x )e −2F (x ) + s(x, x )e F (x )−F (x) 2 (x ,y )∈XL x∈XU and λu qx = λl I(y = −1)s(x, x )e −2F (x ) + s(x, x )e F (x)−F (x ) 2 (x ,y )∈XL x∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  5. 5. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Pseudo Labels and Weights ˆ yx = sign(px − qx ) wx = |px − qx | Optimal α 1 x∈XU pi I(f (x) = 1) + qi I(f (x) = −1) α= ln 4 x∈XU pi I(f (x) = −1) + qi I(f (x) = 1) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  6. 6. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost labeled training data (x, y) ∈ XL and unlabeled data x ∈ XU Similarity measure s(x, x ) Weak learners fi weight parameters λu , λl max iterations T 1 For t = 1, 2, . . . , T 2 Compute pi and qi for every given sample 3 ˆ yx = sign(px − qx ) 4 wx = |px − qx | 5 Train weak classifier ft (x) 6 Compute αt 7 F (x) ← F (x) + αt ft (x) 8 EndFor Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  7. 7. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost with learned Similarities[Hertz et al.,CVPR’04] Radial Basis Function [Zhu et al.,ICML’03] d(x,x )2 − σ2 s(x, x ) = e d(x, x ) . . . distance between points Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  8. 8. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoLearning Distance Functions Idea Learn distance or metric function on labeled data which then can discriminatively support task-specific classification. Distance Function F d : X × X → Y = [−1 1] Training Pairs of “same” or “different” [Hertz et al.,CVPR’04] Dd = {(x, x , +1)|y = y , x, x ∈ DL } ∪ ∪{(x, x , −1)|y y , x, x ∈ DL } Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  9. 9. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost with learned Distance Functions Number of Training Pairs (Symmetric case) n·(n−1) 2 ? +- ? + ? + - SemiBoost ? + - ? Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  10. 10. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoUsing Arbitrary Classifiers Approximate pair-wise classifier |F (x, x )| ≈ |F (x) − F (x )| + + ? ? - Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  11. 11. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoReusing Prior Classifiers[Schapire et al,ML’02] Classifier Combination F C (x) = α0 F P (x) + F (x) ? ? + SemiBoost ? ? - ? Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  12. 12. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Car Detection Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  13. 13. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSimilarity Performance Accuracy depending on the number of samples Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  14. 14. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Car Detection Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  15. 15. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Face Detection (a) prior (b) trained (c) combined Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  16. 16. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSimple Data mining method[Levin et al.,ICCV’03][Rosenberg et al.,2005] 1 Labeled training data (x, y) ∈ XL 2 Train cascaded detector F P (x) on XL using [Viola & Jones,2001] 3 Use a web image search engine in order to collect huge amounts of possibly useful images XU ; pass phrases that are much likely related to your target object 4 Apply F P (x) in a sliding window manner on XU and copy all ∗ detections to XU ∗ 5 Train a SemiBoost classifier F (x) on XL and XU using F P (x) as prior 6 Output the final classifier F (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  17. 17. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Transfer Learning Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  18. 18. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemiBoost Applications Transfer Learning Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  19. 19. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  20. 20. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Boosting Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  21. 21. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking [Oza,PhD-Thesis’01], [Grabner & Bischof,CVPR’06] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  22. 22. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  23. 23. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  24. 24. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  25. 25. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Tracking is an One-Shot Semi-supervised Learning Problem Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  26. 26. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SemiBoost P −Fn−1 (x) −Fn−1 (x) + e −Fn−1 (x) e F (x) ˜ px ≈ e S(x, xi ) ≈ e F (x) ≈ F P (x) P xi ∈X+ e + e −F (x) P e Fn−1 (x) e −F (x) qx ≈ e Fn−1 (x) ˜ S(x, xi ) ≈ e Fn−1 (x) F − (x) ≈ P P xi ∈X− e F (x) + e −F (x) sinh(F P (x) − Fn−1 ) ˜ ˜ pn (x)−qn (x) = = tanh(F P (x))−tanh(Fn−1 (x)) cosh(F P (x)) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  27. 27. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  28. 28. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  29. 29. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Problem: Rapid Appearance Changes Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  30. 30. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  31. 31. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoExploration-Exploitation Dilemma Convex Trade-off (F (x)) = (1 − α) l (F (x)) + α u (F (x)) We need more Robustness when minimizing the labeled loss! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  32. 32. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoLoss Functions Random classification noise defeats all convex potential boosters [Long and Servidio,ICML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  33. 33. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Gradient Boost Gradient Descent Functional Gradient Descent GradientBoost [Friedman et al.,Annals of Statistics’01] ft (x) = arg max − LT f (x) f (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  34. 34. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Gradient Boost A training sample: (xn , yn ), A differentiable loss function (·) Number of selectors M , Number of weak learners per selector K 1 Set F0 (xn ) = 0. 2 Set the initial weight wn = − (0). 3 For m = 1 to M 4 For k = 1 to K 5 Train k th weak learner fm (x) with sample (xn , yn ) and weight wn . k k k k 6 em ← em + wn I(sign(fm (xn )) yn ) //Compute the error 7 EndFor 8 Find the best weak learner with the least total weighted error: k j = arg min em . k j 9 Set fm (xn ) = fm (xn ). 10 Set Fm (xn ) = Fm−1 (xn ) + fm (xn ). 11 Set the weight wn = − (yn Fm (xn )). 12 EndFor Graz University of Technology 13 Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  35. 35. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoWeight Updates Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  36. 36. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCo-Training of Pedestrian Detectors Exponential Loss Logit Loss Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  37. 37. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSERBoost Expectation Regularization [Mann and MacCallum,ICML’07] Penalize model predictions on unlabeled data that deviate from certain expectation. SERBoost [Saffari et al.,ECCV’08] L(H (x), X) = Ll (H (x), Xl ) + βLu (H (x), Xu ) L(H (x), X) = e −yH (x) + e −yp H (x) cosh(H (x)) x∈XL x∈XU Pseudo Label + yp = 2Pp (x) − 1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  38. 38. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Supervised Loss Ll (XL ) = log 1 + e −2yF (x) (x,y)∈Xl = log e −yF (x) (e yF (x) + e −yF (x) ) (x,y)∈XL = −yF (x) + log e F (x) + e −F (x) . (x,y)∈XL Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  39. 39. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Minimize the cross entropy H (Pp , P) = − Pp (y = z|x) log P(y = z|x) z∈{−1,1} = − 2Pp (y = 1|x) − 1 F (x) + log e F (x) + e −F (x) yp (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  40. 40. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Unsupervised Loss Lu (XU ) = ˆ H (Pp , P) = −yp (x)F (x) + log e F (x) + e −F (x) x∈XU x∈XU Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  41. 41. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line SERBoost with logistic loss Unsupervised Loss Lu (XU ) = ˆ H (Pp , P) = −yp (x)F (x) + log e F (x) + e −F (x) x∈XU x∈XU Unlabeled Update ∀ x ∈ XU :wx = yp (x) − tanh(F (x)) ˆ yx = sign yp (x) − tanh(F (x)) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  42. 42. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOSER Tracking λ = 0.5 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  43. 43. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoInfluence of convex combination Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  44. 44. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Boosting[Viola et al.,NIPS’05][Babenko et al.,CVPR’09] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  45. 45. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Boosting[Viola et al.,NIPS’05][Babenko et al.,CVPR’09] Bags {(B1 , y1 ), . . . , (Bn , yn )} Bi = {xi1 , xi2 , . . . , xini } Minimize binary log-likelihood log L= (yi log p(yi ) + (1 − yi ) log p(yi )) i Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  46. 46. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Combine benefits of MIL and SSL Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  47. 47. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Unlabeled Loss of the Bags Nu Lu (XB ) = − u Pp (z|Bu ) log(P(z|Bu )) i i i=1 z∈Y Approximate max with geometric mean NBi 1/NBi P(y = 1|Bi ) = 1 − 1 − P(y = 1|xij ) j=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  48. 48. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Gradient for NOR and geometric mean 2 z − P(y = 1|Bi ) aij (z) = P(y = 1|xij ) NBi P(y = 1|Bi ) Pseudo Labels and Weights wij =β Pp (z|Bu )aij (z) i z∈Y yij =I β Pp (z|Bu )aij (z) > 0 i z∈Y Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  49. 49. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10] Experimental Results Sequence MILSER MIL OSB OAB sylv 0.64 0.61 0.46 0.50 david 0.71 0.54 0.31 0.32 faceocc2 0.78 0.65 0.63 0.64 coke11 0.18 0.29 0.12 0.20 tiger1 0.60 0.51 0.17 0.27 tiger2 0.46 0.50 0.08 0.25 faceocc1 0.68 0.63 0.71 0.47 girl 0.64 0.53 0.69 0.38 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  50. 50. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Co-Training[Liu et al.,ICCV’09][Saffari et al.,ECCV’10] Performance measured in average location center errors in pixels Approach sylv david faceocc2 tiger1 tiger2 coke faceocc1 girl MV-GPBoost 17 20 10 15 16 20 12 15 CoBoost 15 33 11 22 19 14 13 17 SemiBoost 22 59 43 46 53 85 41 52 MILBoost 11 23 20 15 17 21 27 32 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  51. 51. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line Manifo End Part I Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  52. 52. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests [Breiman,ML’01] Ensemble of n decision trees N F (x) = n=1 f (x) Information Gain |Il | |Ir | ∆H = − |I |+|Ir | H (Il ) − |Il |+|Ir | H (Ir ) l Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  53. 53. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  54. 54. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Applications: Object Detection, Semantic Segmentation, Categorization, Tracking, etc. Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  55. 55. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoRandom Forests Advantages: speed parallelism noise robust inherently multi-class Applications: Object Detection, Semantic Segmentation, Categorization, Tracking, etc. Disadvantage: RFs demand a huge amount of data in order to leverage their full potential [Caruana et al.,ICML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  56. 56. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemi-Supervised Random Forests Random Forests maximize the margin ml (x, y) = p(y|x) − max p(k|x) k∈Y k y Unlabeled Margin mu (xu ) = max fi (xu ) i∈Y Semi-supervised Loss 1 λ L(f) = (fy (x)) + (mu (x)) |Xl | |Xu | (x,y)∈Xl x∈Xu Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  57. 57. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization Incorporate labels for the unlabeled data as additional optimization variables! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  58. 58. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization Incorporate labels for the unlabeled data as additional optimization variables! Deterministic Annealing [Rose,IJCNN’98] p ∗ = arg minEp (F(y)) − T H(p) p∈P T0 > T1 > . . . > T∞ = 0 p ∗ . . . distributions over the label predictions Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  59. 59. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOptimization DA-Loss for Semi-supervised Random Forests 1 LDA (f, p) = ˆ (fy (x))+ |Xl | (x,y)∈Xl K α + ˆ p(i|x) (fi (x))+ |Xu | x∈Xu i=1 K T + ˆ H (p) |Xu | x∈Xu i=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  60. 60. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTwo Step Optimization First Stage 1 f∗ = arg min n (fy (x))+ f |Xl | (x,y)∈Xl α + (fyu (x)) ˆ |Xu | x∈Xu Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  61. 61. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTwo Step Optimization Second Stage K α p∗ =arg min ˆ ˆ p(i|x) (fi (x))+ ˆ p |Xu | x∈Xu i=1 K T + ˆ ˆ p(i|x) log(p(i|x)) |Xu | x∈Xu i=1 p ∗ (i|x) = exp(− α ˆ (fi (x))+T T )/Z (x) Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  62. 62. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoFinding the optimal Distributions Take the derivate w.r.t. each class ˆ ˆ ˆ hi (p, x) = p(i|x)(α (gi (x)) + T log(p(i|x))) (1) dhi ˆ = α (gi (x)) + T log(p(i|x)) + T (2) ˆ d pi Optimal Distribution p ∗ (i|x) = exp(− α ˆ (fi (x))+T T )/Z (x) K Z (x) = ˆ∗ i=1 p (i|x) is the partition function Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  63. 63. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoExperiments Classification Accuracy in % Method SVM TSVM SER RMSB RF DAS-RF g50c 91.7 93.1 91.9 94.2 89.1 93.3 Letter 70.3 65.9 76.5 79.9 76.4 79.7 SensIt 80.2 79.9 81.9 83.7 76.5 84.3 Train and Test time in Seconds Method SVM TSVM SER RMSB RF DAS-RF GPU Letter 25 74 3124 125 35 72 29 SensIt 195 687 1158 514 125 410 137 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  64. 64. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCaltech-101 binary classification error Class RF DAS-RF Improvement C4 0.0081 0.0033 58% C5 0.0078 0.002 65% C20 0.011 0.0013 87.5% C33 0.007 0.003 52% C81 0.0027 0.001 62.5% classification error over different numbers of labeled samples Algorithm l = 15 l = 30 RF 0.72 0.64 DAS-RF 0.70 0.60 LinSVM 0.74 0.65 Graz University of Technology improvement 2% 4% Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  65. 65. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoPrior Regularization Potential Information Gain |Il | |Ir | ∆H = − |I |+|Ir | H (Il ) − |Il |+|Ir | H (Ir ) l Kullback-Leibler Divergence DKL (q p) = H (q, p) − H (q) 1 DSKL (q p) = 2 (DKL (q p) + DKL (p q)) Prior-regularized node score ∆H ∗ = ∆H + λ∆DSKL (q p) ˆ Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  66. 66. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoAirbag m−1 m OOBE : eF − eF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  67. 67. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoAirbag m−1 m OOBE : eF − eF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  68. 68. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  69. 69. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests[Leistner et al.,ECCV’10] - - - - + - - + - - - - + - [Dietterich,AI’97] Content-based Image Retrieval Object Detection and Categorization Tracking Action Recognition Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  70. 70. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests Multiple Instance Learning is a special case of semi-supervised Learning! Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  71. 71. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests Multi-class Instance Classifier F (x) : X → Y = {1, . . . , K } {(B1 , y1 ), . . . , (Bn , yn )}, where yi ∈ {1, . . . , K } Objective Function n ni j j ({yi }∗ , F ∗ ) =arg min (Fy j (xi )) j i {yi },F (·) i=1 j=1 ni j s.t. ∀i : I(yi = arg max Fk (xi )) 1. j=1 k∈Y Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  72. 72. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoMultiple Instance Forests DA Loss Function n ni K n j j LDA (F , p) = ˆ ˆ p(k|xi ) (Fk (xi )) +T ˆ H (pi ) i=1 j=1 k=1 i=1 Entropy of the distribution inside a bag ni K j j ˆ H (pi ) = − ˆ ˆ p(k|xi ) log(p(k|xi )) j=1 k=1 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  73. 73. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoEvaluation Method Elephant Fox Tiger Musk1 Musk2 RandomForest[Breiman,2001] 74 60 77 85 78 MILForest 84 64 82 85 82 MI-Kernel[Andrews,2003] 84 60 84 88 89 MI-SVM[Zhou,2009] 81 59 84 78 84 mi-SVM[Zhou,2009] 82 58 79 87 84 MILES[Chen,2006] 81 62 80 88 83 SIL-SVM[Bunescu,2007] 85 53 77 88 87 AW-SVM[Gehler,2007] 82 64 83 86 84 AL-SVM[Gehler,2007] 79 63 78 86 83 EM-DD[Zhang,2001] 78 56 72 85 85 MILBoost-NOR[Viola,2006] 73 58 56 71 61 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  74. 74. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCorel Data Set Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  75. 75. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoCorel Data Set Results for the COREL image categorization benchmark Method Corel-1000 Corel-2000 Testing[sec.] Training[sec.] MILForest 59 66 4.6 22.0 MILES 58 67 180 960 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  76. 76. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoSemantic Segmentation [Vezhnevets & Buhmann,CVPR’10] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  77. 77. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutline 1 SemiBoost & Visual Similarity Learning 2 On-line Semi-supervised Boosting Tracking 3 Semi-Supervised Random Forests MILForests On-line Random Forests 4 On-line Manifold Regularization 5 Conclusion & Outlook Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  78. 78. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Random Forests On-line Bagging [Oza,PhD-Thesis’01] → Poisson(λ) On-line recursive splitting is hard → Tree Growing Info Gain |Rjls | |Rjrs | ∆L(Rj , s) = L(Rj ) − L(Rjls ) − L(Rjrs ) |Rj | |Rj | Splitting Rules |Rj | > α and ∃s ∈ S : ∆L(Rj , s) > β On-line DA → Annealing Schedule for each sample xi Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  79. 79. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Random Forests Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  80. 80. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoInteractive Segmentation Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  81. 81. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking with On-line RF Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  82. 82. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  83. 83. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoTracking RT ∩ RGT /RT ∪ RGT Sequence OSERB MILBoost OSB OAB ORF MILForest RF sylv 0.64 0.61 0.46 0.50 0.53 0.59 0.50 david 0.69 0.54 0.31 0.32 0.69 0.72 0.32 faceocc2 0.77 0.65 0.63 0.64 0.72 0.77 0.79 tiger1 0.65 0.51 0.17 0.27 0.38 0.55 0.34 tiger2 0.42 0.50 0.08 0.25 0.43 0.53 0.32 coke 0.2 0.33 0.08 0.25 0.35 0.35 0.15 faceocc1 0.77 0.63 0.71 0.47 0.71 0.77 0.77 girl 0.77 0.53 0.69 0.38 0.70 0.71 0.74 Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  84. 84. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Manifold Regularization[Goldberg et al.,ECML’08] Based on Convex Programming in kernel space using stochastic gradient descent Random Projection Trees [Dasgupta & Freund, TR, 2007] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  85. 85. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Manifold Regularization[Goldberg et al.,ECML’08] Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  86. 86. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Harmonic Function Solution Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  87. 87. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Merge the two most similar vertices and add the new vertex Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  88. 88. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOn-line Graph-based SSL[Kveton et al.,OLCV’10] Face recognition of 8 people Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  89. 89. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoConclusion Semi-supervised Learning is a powerful learning paradigm with many potential applications in Computer Vision Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  90. 90. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoConclusion Semi-supervised Learning is a powerful learning paradigm with many potential applications in Computer Vision It is often also the way how learning is done in nature It can be applied virtually everywhere where classifiers are applied On-line SSL can be used in order to make tracking-by-detection systems more robust Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  91. 91. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutlook We need to increase the robustness of SSL algorithms in order to leverage more applications Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  92. 92. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoOutlook We need to increase the robustness of SSL algorithms in order to leverage more applications Demand for more on-line Semi-Supervised Methods SSL from weakly-related unlabeled data Graz University of Technology Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II
  93. 93. SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line ManifoReferences Books O. Chapelle and B. Schoelkopf and A. Zien, “Semi-Supervised Learning”, The MIT Press, 2006 Xiaojin Zhu and Andrew B. Goldberg, “Introduction to Semi-Supervised Learning”, Morgan & Claypool, 2009 Papers and Articles C. Leistner, A. Saffari and H. Bischof, “MILForests: Multiple-Instance Learning with Randomized Trees”,ECCV’10 C. Leistner, A. Saffari, J. Santner and H. Bischof: “,Semi-Supervised Random Forests”,ICCV’09 C. Leistner, A. Saffari, P Roth and H. Bischof: “On Robustness of On-line Boosting – A Competitive .M. Study”,(ICCV) OLCV’09 H. Grabner, C. Leistner and H. Bischof: “On-line Semi-Supervised Boosting for Robust Tracking”,ECCV’08 B. Zeisl, C. Leistner, A. Saffari and H. Bischof: “On-line Semi-supervised Multiple-Instance Boosting”,CVPR’10 C. Leistner, “Semi-Supervised Ensemble Methods for Computer Vision”, PhD-Thesis, Graz University of Technology, 2010 A. Saffari, C. Leistner, M. Godec, J. Santner and H. Bischof, “On-line Random Forests”, (ICCV) OLCV’09 A. Saffari, C. Leistner, M. Godec and H. Bischof, “Robust Multi-View Multi-Class Boosting with Priors”,ECCV’10 B. Kveton, M. Valko, M. Philipose and L. Huang, “Online Semi-Supervised Perception: Real-Time Learning without Explicit Feedback”, (CVPR) OLCV’10 A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09 C. Leistner, H. Grabner and H. Bischof, “Semi-Supervised Boosting using Visual Similarity Learning”,CVPR’08 A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09 A. Saffari, H. Grabner and H. Bischof, “SERBoost: Semi-supervised Boosting with Expectation Graz University of Technology Regularization”,ECCV’08 Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II

×