Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Video concept detection by learning from web images

1,055 views

Published on

Video concept detection by learning from web images

Published in: Technology, Education
  • Brent Forsman is paralyzed from a hunting accident. He now uses the Demolisher system to generate income. He says: "I believe there is no way this system can fail!" ◆◆◆ http://t.cn/A6zP2GDT
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Best survey site online! $1,500 a month thanks to you guys! Without a doubt the best paid surveys site online!I have made money from other survey sites but made double or triple with GoldOpinions.com for the same time and effort. The variety and number of daily paid surveys I get from them is unmatched. A must for anyone looking for extra cash or a full time income. ▲▲▲ http://t.cn/AieX2Loq
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • check out the cool video of me training my dog on this page. ▲▲▲ http://t.cn/Aie4mTQb
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Video concept detection by learning from web images

  1. 1. Video Concept Detection by Learning from Web Images Shiai Zhu, Ting Yao, Chong-Wah Ngo City University of Hong Kong
  2. 2. Why concepts for media remixing? 00:00 00:10 00:20 Time (seconds) Cake is in preparation Someone is talkingSomeone is talking Background music is playing … A example of event recounting
  3. 3. Recounting using 21 concept classifiers kitchen, outdoor/indoor, baseball field, crowd, cake, walking, running, squatting, standing, hand, batting, speech, music, clipping, cheering
  4. 4. Why concept learning is challenging? Requires thousands of concepts for practical applications Collecting training examples is always expensive
  5. 5. Economic solution: Get it free from Internet Thousands of new upload per minute Zoom Wow Usa Texas Critter Creature Olympus Closeup Cat June 2010 Animal pet
  6. 6. Residence Place of Worship Building Country House Temple Church Buildin g House Approach I – Semantic Field (SF)
  7. 7. Residence Place of Worship Building Country House Temple Church Buildin g House Approach II – Semantic Pooling (SP)
  8. 8. Does it work practically? Dancing Dancing Boy TRECVID videos Flickr images Ocean Boy Ocean
  9. 9. Transfer Learning Wenyuan Dai, ICML 2007 (TrAdaBoost) Knowledge  of instance Kate Saenko, ECCV 2010 (shared representation) Feature  representation Jun Yang, ACM MM 2007 (Adaptive‐SVM) Parameter (Model) Yu‐Gang Jiang, ACM MM 2009 (Semantic context transfer) Rational  knowledge Transfer learning
  10. 10. Transfer Learning Adaptive SVM  Model-level learning TrAdaBoost  Instance-level learning Target domain (video) data Source domain (image) data 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.8 0.5 0.5 0.5 0.3 0.3 0.3
  11. 11. Painful Experience on TRECVID 64 runs of other SIN systems TRECVID training data alone Cross domain learning Web image alone Baseline detector ASVM-SF TradaBoost-SF SP SF Negative transfer! Negative transfer happens when knowledge  transfer has a negative impact on target domain Dataset Training set Testing set # evaluated concepts # positive instances TRECVID 2011 266,474 137,327 50/346 1800
  12. 12. Positive or negative transfer?  Number of training examples?  Type of a concept?  People, object, scene, event  Change of data distribution? 2/4 1/13 1/14 2/19 < 500 positive examples 500~1000 1001~2000 >2000 Percentage of improved concepts versus number of positive training examples
  13. 13. A case study on cross-domain learning Target domain (Web videos) – TRECVID 2012 dataset Source domain (Web images) – Semantic Field  1000 positive examples per concept – Semantic Pooling  SF + additional 1000 examples per concept  16,367 of concepts + 0.7 million images for pooling Dataset Training set Testing set # evaluated concepts # positive instances TRECVID 2012 400,289 145,634 46/346 1200
  14. 14. SIFTfeaturespace Basic Framework
  15. 15. Number of positive examples? A-SVM-SF: Semantic Field + A-SVM A-SVM-SP: Semantic Pooling + A-SVM Baseline: learnt using TRECVID training example MinfAP Number of positive instances Baseline A-SVM-SF A-SVM-SP -transfer +transfer  Pooling is a practical strategy to diversify the coverage of training examples  22/46 concepts improve if each concept only has 100 positive examples
  16. 16. Type of Concept?MinfAP Number of positive instances MinfAP Number of positive instances MinfAP Number of positive instances MinfAP Number of positive instances Scene  (15) Object (10)  People  (12) Event (8)  Probably not a good idea to use images for learning event
  17. 17. Change in Data Distribution? Maximum Mean Discrepancy (MMD) 23 concepts with lower mismatch 23 concepts with higher mismatch
  18. 18. ForestForest TRECVID Flickr ComputerComputer SingingSingingMeetingMeeting TRECVID Flickr KitchenKitchen MotorcycleMotorcycle ThrowingThrowing TRECVID Flickr StadiumStadium ChairChair < 10 = 50 > 100 break even point
  19. 19. Average is average MMD Break-evenpoint 50 100 Boat_Ship Glasses Singing AirplaneBaby Male_Person Airplane_Flying Instrumental_Musician OceansForest Man_Wearing_A_Suit Bridges Military_Airplane Fields Stadium Landscape SkierPress_Conference Nighttime Teenagers Highway Walking_RunningLakes Bicycling Computers Roadway_Junction Apartments Clearing Girl Civilian_Person KitchenMotorcycle Meeting Female_Person Government‐Leader Sitting_Down Hill Boy SoldiersChair Basketball Throwing OfficeGeorge_Bush Scene_Text Greeting difficulty difficulty lower mismatch higher mismatch
  20. 20. Question? • Using Web images to learn concept classifiers for video (TRECVID) domain • When positive examples in target domain < 100 • Event might be difficult to transfer • Data distribution can be a cue to predict the difficulty • Pooling strategy has a better chance to survive positive transfer • Feasibility of transfer learning? Key ideas Messages

×