Recommended
More Related Content
What's hot
What's hot (9)
Similar to [論文読み]Interpretable Coun.ng for Visual Ques.on Answering
Similar to [論文読み]Interpretable Coun.ng for Visual Ques.on Answering (20)
Recently uploaded
Recently uploaded (20)
[論文読み]Interpretable Coun.ng for Visual Ques.on Answering
- 1. Interpretable Coun.ng for Visual Ques.on Answering 3 d-hacks 1
- 2. • • VQA • VQA • Dataset • Model • Object detection • Language • Counting • Experiment & Results • 2
- 3. • (VQA) SOTA • interpretable Model Model SOTA Our approach 2 2 3
- 4. • - Alexander Tro,, Caming Xiong & Richard Socher - Salesforce Research • ICLR 2018 • OpenReview 6,7,7 4
- 5. Visual Question Answering; VQA • • CVPR2016 Competition 5
- 6. 1: • • 2015 CVPRW, Learning to count with deep object features • CNN • • 2016 ECCV, Towards perspective-free object counting with deep learning • … 6
- 7. 2: Visual Question Answering • 11% ( : ) • : • Simple Baseline for Visual Ques7on Answering 2015 • Exploring Models and Data for Image Ques7on Answering. In NIPS, 2015a. 7
- 8. 3: A$en'on • ( UpDown ) • (Attention ) • Multimodal compact bilinear pooling for visual question answering and visual grounding. EMNLP, 2016 • Bottom-Up and Top-Down Attention for Image Captioning and VQA. arXiv, 2017. 8
- 9. 4: VQA • VQA • → • • • It Takes Two to Tango: Towards Theory of AI’s Mind, 2017 • AI AI ( ) 9 AI
- 10. Dataset 10
- 11. • VQA • VQA2.0 Visual Genome / HowMany-QA 11
- 12. VQA2.0 / Visual Genome • VQA2.0 (1.0 2015) • Image + (Ques4on + Answer) • 265,016 • 3 /1 • 10 +3 /1 • Visual Genome • Image + • • 108,077 • 5.4 Million Region Descrip4ons • 1.7 Million Visual Ques4on Answers • 3.8 Million Object Instances • 2.8 Million ALributes • 2.3 Million Rela4onships 12
- 13. HowMany-QA • 0 20 • [“how many”, “number of”, “amount of”, or “count of”] • [ “number of the”] reject • ID 13 Our Dataset
- 14. Model 14
- 15. Input: Input: Output: • 3 15 Caption Grounding
- 16. • → • • : Bounding box • : ( ) • Faster RCNN* 16
- 17. Faster RCNN 17 • • End to End • makora
- 18. 18 makora
- 19. 19 makora
- 20. 20 makora
- 21. 21 makora
- 22. 22 makora
- 23. • → • ( ) • LSTM* • ( ) • x: T: s: 23
- 24. LSTM( ) • RNN ( ) • / 24
- 25. LSTM( ) 25 0 ~ 1
- 26. LSTM( ) 26 0 ~ 1
- 27. LSTM( ) 27 0 ~ 1
- 28. function • • • Gated Tanh Units (GTU)* Gated Tanh Units : Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional Image Generation with PixelCNN Decoders. In NIPS, 2016 29 0 10.02 5 2 1
- 29. • • • : bounding box • : • • • (Interpretable Rainforced Learning Counting) 30
- 31. So#Count • (0 1 ) • → 32 0.02 5 2 1 0.50 0.99 0.88 0.73
- 32. UpDown (Attention ) • Attention (UpDown, 2017) • → 33 0.02 5 2 1 0.50* 0.99* 0.88* 0.73* p_i …
- 33. IRLC ←Our Model • → • • Object • → +=1 • 34 → →
- 34. 35
- 35. 36
- 36. IRLC • K • R • 37 : http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf acAon
- 37. : • H: • 38 →
- 38. & 39
- 39. 3 40
- 40. • accuracy • • ( ) • RMSE • • ( ) 41 10 3
- 41. • / • 5 • 6 : !? 42
- 42. • IRLC • Accuracy IRLC RMSE IRLC Softcount • UpDown Softcount accuracy bad bad 43
- 43. • VQA + VG = HowManyQA • Coun1ng HowMany QA 44
- 46. Grounding • Microsoft COCO: Common Objects in Context: • 80 ( ) • 80 47 Category: train Category: line GloVe: SVD Word2Vec
- 47. Grounding Quality • 0.5=<IoU COCO / Background • IoU(Intersection over union)… boundingBox • : m i 48 m i
- 48. • m (ex. q=“car” How many “cars” are there ) • : So9Count or IRLC Grounding Quality 49 m i i ( )
- 49. Grounding • Grounding IRLC • ( ) ( ) Grounding 50
- 50. • 2 2 51
- 51. ICLR2018 VQA Coun0ng • Learning to Count Objects in Natural Images for Visual Question Answering Yan Zhang, Jonathon Hare, Adam Prügel-Bennett • Open review 6,4,6 52
- 52. • VQA • (So'count UpDown) Grounding →IRLC • Grounding • VQA2.0 VG HowMany-QA (VQA2.0 Accuracy RMSE ) 53