Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Video Object Linguistic Grounding

180 views

Published on

The goal of this work is segmenting on a video sequence the objects which are mentioned in a linguistic description of the scene. We have adapted an existing deep neural network that achieves state of the art performance in semi-supervised video object segmentation, to add a linguistic branch that would generate an attention map over the video frames, making the segmentation of the objects temporally consistent along the sequence.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Video Object Linguistic Grounding

  1. 1. Video Object Linguistic GroundingWorkshop on Multimodal Understanding & Learning for Embodied Applications (MULEA) Nice, France 25 October 2019 Carles Ventura Alba M. Herrera Xavier Giro-i-Nieto
  2. 2. 2 HCI with Natural Language
  3. 3. #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019.
  4. 4. 4 From Masks to Referring Expressions Model time Model One-shot RVOS [*] Model time Model Referring expression “the woman” [*] #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019.
  5. 5. 5 Related Work Khoreva, A., Rohrbach, A., & Schiele, B. Video object segmentation with language referring expressions. ACCV 2018.
  6. 6. 6 Image Segmentation with Refers #MAttNet Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., & Berg, T. L. . Mattnet: Modular attention network for referring expression comprehension. CVPR 2018
  7. 7. 7 Proposed Approach: MAttNet + RVOS
  8. 8. 8 Qualitative Results FRAME 0 FRAME 2 FRAME 4 REFERRING EXPRESSIONS: "a brown deer on the left" "a brown deer on the right with branched horns"
  9. 9. 9 Quantitative Results
  10. 10. Merci beaucoup xavier.giro@upc.edu @DocXavi Xavier Giro-i-Nieto

×