Relevance Filtering meets Active Learning
                     — Improving Web-based Concept Detectors —


               ...
Outline



     Introduction


     Approach: Active Relevance Filtering


     Experimental Results


     Summary




D....
Digital Video




D.Borth: : Relevance Filtering meets Active Learning   3   March 29 2010
Digital Video

     ”...about 24 hours of video is uploaded every minute, 1 billion views per day...”

                   ...
Digital Video

     ”...about 24 hours of video is uploaded every minute, 1 billion views per day...”

                   ...
Digital Video

     ”...about 24 hours of video is uploaded every minute, 1 billion views per day...”

                   ...
Digital Video

     ”...about 24 hours of video is uploaded every minute, 1 billion views per day...”

                   ...
Concept Detection - Framework




             unknown video shot X
             concept vocabulary t1 ...tn
             ...
Concept Detection - Framework




             expert labels are used as training data
             time consuming effort [...
Concept Detection - Framework




             propose web video as training source      [Ulges07]

             use tags ...
Concept Detection - Framework




             label noise problem
                     subjective
                     co...
Concept Detection - Framework




             relevance filtering
                     adapt concept learning to noisy lab...
Relevance Filtering Approaches

                                               Relevance Filtering
                    (¸)...
Automatic Relevance Filtering



     Idea
             take label noise into account during model training
             i...
Automatic Relevance Filtering



     Idea
             take label noise into account during model training
             i...
Automatic Relevance Filtering


     Approach         [Ulges10]


             training data:            X = {x1 , . . . ,...
Automatic Relevance Filtering


     Approach         [Ulges10]


             training data:            X = {x1 , . . . ,...
Active Learning

     Idea
             select informative samples for manual labeling




D.Borth: : Relevance Filtering ...
Active Learning

     Idea
             select informative samples for manual labeling

     Related Work
             tex...
Active Learning

     Idea
             select informative samples for manual labeling

     Related Work
             tex...
Active Learning




             pool-based active learning
             selects label according to model
             new...
Our Approach: Active Relevance Filtering




             active learning + auto. relevance filtering
             selects ...
Experiments




D.Borth: : Relevance Filtering meets Active Learning   15   March 29 2010
Experiments

 YouTube-22Concepts-Dataset
        100 videos per concept
        keyframes extracted
        features:
    ...
Experiments

 YouTube-22Concepts-Dataset
        100 videos per concept
        keyframes extracted
        features:
    ...
Experiments

 YouTube-22Concepts-Dataset
        100 videos per concept
        keyframes extracted
        features:
    ...
Experiments - Impact of Label Noise

                                                               Relevance Filtering


...
Experiments - Relevance Filtering
                                         Active Learning                                ...
Experiments - Relevance Filtering
                                         Active Learning                                ...
Experiments - Top Ranked Keyframes

                                               concept: basketball




       a) no re...
Experiments - Top Ranked Keyframes

                                               concept: basketball




       a) no re...
Experiments - Top Ranked Keyframes

                                               concept: basketball




       a) no re...
Experiments - Top Ranked Keyframes

                                               concept: eiffeltower




       a) no re...
Experiments - Top Ranked Keyframes

                                               concept: eiffeltower




       a) no re...
Experiments - Top Ranked Keyframes

                                               concept: eiffeltower




       a) no re...
Discussion


     Contributions
             concept learning from noisy (= weakly labeled) web video
             evaluat...
Discussion


     Contributions
             concept learning from noisy (= weakly labeled) web video
             evaluat...
Questions?




D.Borth: : Relevance Filtering meets Active Learning        26   March 29 2010
Upcoming SlideShare
Loading in …5
×

Relevance Filtering meets Active Learning: Improving Web-based Concept Detectors

1,006 views

Published on

Talk at the ACM Int. Conference on Multimedia Information Retrieval (MIR) in Philadelphia, USA

Link to original publication: http://madm.dfki.de/publication&pubid=4535

Published in: Technology, Education
  • Be the first to comment

Relevance Filtering meets Active Learning: Improving Web-based Concept Detectors

  1. 1. Relevance Filtering meets Active Learning — Improving Web-based Concept Detectors — Damian Borth*, Adrian Ulges, Thomas M. Breuel German Research Center for Artificial Intelligence (DFKI) & University of Kaiserslautern, Germany March 29 2010 D.Borth: : Relevance Filtering meets Active Learning 1 March 29 2010
  2. 2. Outline Introduction Approach: Active Relevance Filtering Experimental Results Summary D.Borth: : Relevance Filtering meets Active Learning 2 March 29 2010
  3. 3. Digital Video D.Borth: : Relevance Filtering meets Active Learning 3 March 29 2010
  4. 4. Digital Video ”...about 24 hours of video is uploaded every minute, 1 billion views per day...” , 2010 D.Borth: : Relevance Filtering meets Active Learning 3 March 29 2010
  5. 5. Digital Video ”...about 24 hours of video is uploaded every minute, 1 billion views per day...” , 2010 ”...TV, video on demand, Internet video, and P2P video will account for over 91 percent of global consumer traffic by 2013...” , 2009 D.Borth: : Relevance Filtering meets Active Learning 3 March 29 2010
  6. 6. Digital Video ”...about 24 hours of video is uploaded every minute, 1 billion views per day...” , 2010 ”...TV, video on demand, Internet video, and P2P video will account for over 91 percent of global consumer traffic by 2013...” , 2009 Information Overload vs. Video Retrieval high demand for automatic machine indexing D.Borth: : Relevance Filtering meets Active Learning 3 March 29 2010
  7. 7. Digital Video ”...about 24 hours of video is uploaded every minute, 1 billion views per day...” , 2010 ”...TV, video on demand, Internet video, and P2P video will account for over 91 percent of global consumer traffic by 2013...” , 2009 Information Overload vs. Video Retrieval high demand for automatic machine indexing one solution: concept detection [Snoek09], [Smeaton09], ... → as key building block of CBVR D.Borth: : Relevance Filtering meets Active Learning 3 March 29 2010
  8. 8. Concept Detection - Framework unknown video shot X concept vocabulary t1 ...tn statistical model estimating concept presence P(ti |X ) D.Borth: : Relevance Filtering meets Active Learning 4 March 29 2010
  9. 9. Concept Detection - Framework expert labels are used as training data time consuming effort [Ayache07] → datasets are limited in vocabulary size [Hauptmann07], overfit [Yang08] and narrowed in its flexibility D.Borth: : Relevance Filtering meets Active Learning 5 March 29 2010
  10. 10. Concept Detection - Framework propose web video as training source [Ulges07] use tags as class labels allows autonomous concept learning D.Borth: : Relevance Filtering meets Active Learning 6 March 29 2010
  11. 11. Concept Detection - Framework label noise problem subjective coarse D.Borth: : Relevance Filtering meets Active Learning 7 March 29 2010
  12. 12. Concept Detection - Framework relevance filtering adapt concept learning to noisy labels perform label refinement D.Borth: : Relevance Filtering meets Active Learning 8 March 29 2010
  13. 13. Relevance Filtering Approaches Relevance Filtering (¸) ˚ (¸) : automatic Relevance Filtering ¸ (¸) ˚ (¸) Ä manual annotation with Active Learning ˚ (¸) ¸ Active (¸) (¸) Ä :+ Relevance Filtering ˚ ¸ weak labels filtered labels auto. relevance filtering active learning combination of both → active relevance filtering D.Borth: : Relevance Filtering meets Active Learning 9 March 29 2010
  14. 14. Automatic Relevance Filtering Idea take label noise into account during model training identify false positive and filter them D.Borth: : Relevance Filtering meets Active Learning 10 March 29 2010
  15. 15. Automatic Relevance Filtering Idea take label noise into account during model training identify false positive and filter them Related Work joint probabilities of tags and content [Bernard03], [Feng04] neighbor voting [Snoek09] samples reweighting according to inferred relevance [Ulges08] D.Borth: : Relevance Filtering meets Active Learning 10 March 29 2010
  16. 16. Automatic Relevance Filtering Approach [Ulges10] training data: X = {x1 , . . . , xn } training labels: Y = {y1 , . . . , yn } (known) true labels: Y = {y1 , . . . , yn } (unknown) yi = −1 → yi = −1 yi = 1 → yi ∈ {1, −1} (true pos. or false pos.) D.Borth: : Relevance Filtering meets Active Learning 11 March 29 2010
  17. 17. Automatic Relevance Filtering Approach [Ulges10] training data: X = {x1 , . . . , xn } training labels: Y = {y1 , . . . , yn } (known) true labels: Y = {y1 , . . . , yn } (unknown) yi = −1 → yi = −1 yi = 1 → yi ∈ {1, −1} (true pos. or false pos.) statistical model: kernel densities infer yi by estimating relevance scores βi = P(yi |xi , yi = 1) fitted by EM model extension: φ(X , Y ) → φ(X , Y , β) D.Borth: : Relevance Filtering meets Active Learning 11 March 29 2010
  18. 18. Active Learning Idea select informative samples for manual labeling D.Borth: : Relevance Filtering meets Active Learning 12 March 29 2010
  19. 19. Active Learning Idea select informative samples for manual labeling Related Work text classification [Lewis94], [Tong02], ... image retrieval [Tong01], [Chang05], ... video data labeling [Ayache07], [Hua08], ... D.Borth: : Relevance Filtering meets Active Learning 12 March 29 2010
  20. 20. Active Learning Idea select informative samples for manual labeling Related Work text classification [Lewis94], [Tong02], ... image retrieval [Tong01], [Chang05], ... video data labeling [Ayache07], [Hua08], ... Sample Selection Methods 1. most relevant sampling 2. uncertainty sampling 3. most relevant sampling + density weighted repulsion (DWR) D.Borth: : Relevance Filtering meets Active Learning 12 March 29 2010
  21. 21. Active Learning pool-based active learning selects label according to model new labeled sample helps further selection D.Borth: : Relevance Filtering meets Active Learning 13 March 29 2010
  22. 22. Our Approach: Active Relevance Filtering active learning + auto. relevance filtering selects label according to filtered model new labeled sample helps further filtering & selection D.Borth: : Relevance Filtering meets Active Learning 14 March 29 2010
  23. 23. Experiments D.Borth: : Relevance Filtering meets Active Learning 15 March 29 2010
  24. 24. Experiments YouTube-22Concepts-Dataset 100 videos per concept keyframes extracted features: SIFT [Lowe99] ”swimming” ”cats” visual words [Sivic03] D.Borth: : Relevance Filtering meets Active Learning 15 March 29 2010
  25. 25. Experiments YouTube-22Concepts-Dataset 100 videos per concept keyframes extracted features: SIFT [Lowe99] ”swimming” ”cats” visual words [Sivic03] Setup subset of 10 concepts trained on: 500 noisy pos. samples 1000 neg. samples tested on: 500 pos. samples 1500 neg. samples D.Borth: : Relevance Filtering meets Active Learning 15 March 29 2010
  26. 26. Experiments YouTube-22Concepts-Dataset 100 videos per concept keyframes extracted features: SIFT [Lowe99] ”swimming” ”cats” visual words [Sivic03] Setup Noisy Pos. Samples subset of 10 concepts label precision of web trained on: video: 20 − 50% [Ulges10] 500 noisy pos. samples for this experiments: 20% 1000 neg. samples 500 noisy pos. samples: tested on: 100 true pos. samples 500 pos. samples 400 false pos. samples 1500 neg. samples D.Borth: : Relevance Filtering meets Active Learning 15 March 29 2010
  27. 27. Experiments - Impact of Label Noise Relevance Filtering 0.60 no relevance filtering automatic relevance filt. ground truth mean avg. precision 0.40 0.50 0.30 System Performance Mean Average Precision (MAP) system MAP noisy data 0.455 auto. relevance filtering helps auto. relevance filtering 0.482 potential gap of improvement ground truth 0.557 remains D.Borth: : Relevance Filtering meets Active Learning 16 March 29 2010
  28. 28. Experiments - Relevance Filtering Active Learning Active Relevance Filtering DWR ground truth labels ground truth labels random 0.54 0.54 mean avg. precision mean avg. precision most relevant uncertainty 0.50 0.50 automatic relevance filtering automatic relevance filtering DWR most relevant 0.46 0.46 no relevance filtering no relevance filtering uncertainty random 0 100 200 300 400 500 0 100 200 300 400 500 labeled samples labeled samples Active Learning Active Rel. Filtering active learning can initial auto. relevance outperform random filtering helps selection improves active learning further D.Borth: : Relevance Filtering meets Active Learning 17 March 29 2010
  29. 29. Experiments - Relevance Filtering Active Learning Active Relevance Filtering DWR ground truth labels ground truth labels random 0.54 0.54 mean avg. precision mean avg. precision most relevant uncertainty 0.50 0.50 automatic relevance filtering automatic relevance filtering DWR most relevant 0.46 0.46 no relevance filtering no relevance filtering uncertainty random 0 100 200 300 400 500 0 100 200 300 400 500 labeled samples labeled samples Direct Comparison DWR sampling approach # refined samples AL ARF 0 0.455 0.482 50 0.474 0.541 250 0.525 0.557 D.Borth: : Relevance Filtering meets Active Learning 18 March 29 2010
  30. 30. Experiments - Top Ranked Keyframes concept: basketball a) no relevance filtering, b) automatic relevance filtering, c) active relevance filtering D.Borth: : Relevance Filtering meets Active Learning 19 March 29 2010
  31. 31. Experiments - Top Ranked Keyframes concept: basketball a) no relevance filtering, b) automatic relevance filtering, c) active relevance filtering D.Borth: : Relevance Filtering meets Active Learning 20 March 29 2010
  32. 32. Experiments - Top Ranked Keyframes concept: basketball a) no relevance filtering, b) automatic relevance filtering, c) active relevance filtering D.Borth: : Relevance Filtering meets Active Learning 21 March 29 2010
  33. 33. Experiments - Top Ranked Keyframes concept: eiffeltower a) no relevance filtering, b) automatic relevance filtering, c) active relevance filtering D.Borth: : Relevance Filtering meets Active Learning 22 March 29 2010
  34. 34. Experiments - Top Ranked Keyframes concept: eiffeltower a) no relevance filtering, b) automatic relevance filtering, c) active relevance filtering D.Borth: : Relevance Filtering meets Active Learning 23 March 29 2010
  35. 35. Experiments - Top Ranked Keyframes concept: eiffeltower a) no relevance filtering, b) automatic relevance filtering, c) active relevance filtering D.Borth: : Relevance Filtering meets Active Learning 24 March 29 2010
  36. 36. Discussion Contributions concept learning from noisy (= weakly labeled) web video evaluation of different refinement approaches proposed approach: active relevance filtering D.Borth: : Relevance Filtering meets Active Learning 25 March 29 2010
  37. 37. Discussion Contributions concept learning from noisy (= weakly labeled) web video evaluation of different refinement approaches proposed approach: active relevance filtering Experimental Results automatic relevance filtering helps but is limited active learning is outperforming random selection active relevance filtering is able to improves active learning auto. relevance filtering + active learning D.Borth: : Relevance Filtering meets Active Learning 25 March 29 2010
  38. 38. Questions? D.Borth: : Relevance Filtering meets Active Learning 26 March 29 2010

×