Video search by deep-learning

Video search by deep-learning

• 1.
  Video Search by Deep Learning Cees Snoek
• 2.
  2 Which one isthe plane?
• 3.
  3 Which one isthe plane?
• 4.
  4 Which one isthe bird?
• 5.
  5 Which one isthe bird?
• 6.
  6 Which one isthe Kentucky Warbler?
• 7.
  7 Which one isthe Kentucky Warbler?
• 8.
  8 How difficult isthe problem? Human vision consumes 50% brain power… Van Essen, Science 1992
• 9.
  9 Video recognition ina nutshell Visualization by Jasper Schulte
• 10.
  10 NIST TRECVID Benchmark Promoteprogress in video retrieval research Big data, standardized tasks, independent evaluation and open innovation International video search competition http://trecvid.nist.gov/
• 11.
  11 Concept detection task http://trecvid.nist.gov/ Aircraft Beach Mountain People marching Police/Security Flower
• 12.
  12 From University-lab tospin-off and your mobile phone • = 1000+ others * = UvA / Euvision / Qualcomm Universities win Start-ups win Snoek et al., TRECVID 2004-2015
• 13.
  13 Latest jump dueto deep learning 2006 2009 2015 Meanaverageprecision Progress in video recognition
• 14.
  14 The more featuresthe better Typical shallow learning architecture e.g. SIFT dense sampling Local Feature Extraction Feature Pooling Feature Encoding Classification avg/sum pooling max pooling BoW Sparse coding Fisher VLAD Linear / Non-linear SVM
• 15.
  15 The deeper thebetter Typical deep learning architecture Layer6 Loss Layer7 Max pool. 2 224 224 3×3 4,096 4,096 Dropout Dropout 3×33×35×511×11 Convolution Non-linearity Pooling Krizhevsky et al., NIPS 2012
• 16.
  16 Video search demo’s Socialmedia Forensics Cultural heritage
• 17.
  17 Tomorrow: The Internetof things that video
• 18.
  18 Need to understandwhat is happening where and when?
• 19.
  19 Examples Shaking handsKissing
• 20.
  20 Goal: obtain thered tube around the action Jain et al., IJCV 2017
• 21.
  21 Method: Super-voxel segmentationof the video Jain et al., IJCV 2017
• 22.
  22 Group voxels togenerate action proposals Jain et al., IJCV 2017 Unsupervised and class-agnostic
• 23.
  23 Example proposals
• 24.
  24 Encode video proposalsas 15,000 object scores Jain et al., CVPR 2015 Layer6 Loss Layer7 Max pool. 2 3×3 4,096 4,096 Dropout Dropout 3×33×35×511×11
• 25.
  25 Actions have objectpreference, relation is generic TypingPlaying Cello Bodyweight squats Jain et al., CVPR 2015
• 26.
  26 We consider threeobject encodings − Whole video − Outside of tube only − Inside of tube only Where do objects aid actions the most?
• 27.
  27 Objects aid mostclose to the action 0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 90.00 100.00 Whole video Outside tube Inside tube Jain et al., CVPR 2015
• 28.
  28 Simple convex combinationof known classifiers Objects2action: Translate objects to an action Object representationTest video Object/action affinities where s() = word2vec Mikolov et al., NIPS 2013 Jain et al., ICCV 2015
• 29.
  29 Objects2action localizes actionswithout examples Retrieval results from action query only Jain et al., ICCV15 Prediction Ground truth
• 30.
  30 So far wehave considered video search from text only, what about text search from video? That is: given a video, can we find the best matching sentence? Matching sentences to videos
• 31.
  31 Word2VisualVec: Predicting thevisual representation of text Training time Dong et al., ArXive17
• 32.
  32 Word2VisualVec: Predicting thevisual representation of text Testing time Dong et al., ArXive17
• 33.
  33 Results Dong et al.,ArXive17
• 34.
  34 ‘Arithmetic’ with visualand textual query
• 35.
  35 Video search bydeep learning is powerful, even without examples Field is progressing rapidly Precise spatiotemporal video understanding is next Conclusion www.ceessnoek.info