Download as PDF, PPTX
![Open-ended
Visual Question-Answering
[thesis][web][code]
Issey Masuda Mora Santiago Pascual de la PuenteXavier Giró i Nieto](https://image.slidesharecdn.com/tfgisseymasudaoralpresentation-160715105427/85/Open-ended-Visual-Question-Answering-1-320.jpg)
Uploaded byUniversitat Politècnica de Catalunya
Open-ended Visual Question-Answering
The document presents a roadmap for open-ended visual question-answering (VQA), discussing its introduction, methodology, results, and future work. It emphasizes the use of deep learning techniques such as convolutional neural networks and long short-term memory networks to predict answers based on both visual and textual representations. The findings reveal a competitive performance of the proposed VQA models in comparison to existing baselines, with specific focus on generating answers rather than merely classifying them.
More Related Content
![Prepositions of place [โหมดความเข้ากันได้]](https://cdn.slidesharecdn.com/ss_thumbnails/prepositionsofplace-111004011102-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
![Prepositions powerpoint[1]](https://cdn.slidesharecdn.com/ss_thumbnails/prepositionspowerpoint1-120209065321-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Open-ended Visual Question-Answering
More from Universitat Politècnica de Catalunya
Open-ended Visual Question-Answering
- 1. Open-ended Visual Question-Answering [thesis][web][code] Issey MasudaMora Santiago Pascual de la PuenteXavier Giró i Nieto
- 2. Roadmap Introduction Related Work Methodology ResultsConclusions Future work 2
- 3. Introduction Related Work Methodology ResultsConclusions Future Work Introduction 3
- 4. Visual Question-Answering Antol, S.,Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., & Parikh, D. (2015). Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2425-2433). 4
- 5. Predict the answerof a given question related to an image 5
- 6. Visual Question-Answering: Types 6 Realimages Abstract scenes Multi-Choice Open-ended Q: Does it appear to be rainy? A: no Q: What is just under the tree? A: a ball Q: How many slices of pizza are there? A: 1, 2, 3, 4 Q: What is for desert? A: cake, ice cream, cheesecake, pie
- 7. Example 7 Question: What isbobbing in the water other than the boats? Answer: buoys
- 8.
- 9. Motivation: AI research ●Multidisciplinary tasks ● Models able to perform more complex activities ● Different sub-problems tackled at once 9 Computer Vision Knowledge Representation and Reasoning Natural Language Processing
- 10. Introduction Related Work Methodology ResultsConclusions Future Work Related Work 10
- 11.
- 12. VQA: Common approach 12 Visual representation Textual representation PredictanswerMerge Question What object is flying? Answer Kite CNN Word/sentence embedding + LSTM
- 13. Tools: Convolutional NeuralNetworks (CNN) 13 Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105). AlexNet
- 14. Tools: Word andSentence embeddings 14 Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. InAdvances in neural information processing systems (pp. 3111-3119). Experiments from: Socher et. al. (2013b) and Collbert et. al. (2011) King Man- Woman+ Queen=
- 15. Tools: Long Short-TermMemory networks (LSTM) 15 Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
- 16. Introduction Related Work Methodology ResultsConclusions Future Work Methodology 16
- 17.
- 18. Extending text-based QAfor VQA 18 Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
- 19. Substitute VGG-16 withKCNN 19 Liu, Z. (2015). Kernelized Deep Convolutional Neural Network for Describing Complex Images. arXiv preprint arXiv: 1509.04581.
- 20. Sentence embedding andimage projection 20 Image Question Answer
- 21. Introduction Related Work Methodology ResultsConclusions Future Work Results 21
- 22. VQA Dataset: RealImages, Open-ended questions 22 Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., & Parikh, D. (2015). Vqa: Visual question answering. CVPR 2015. 1 (image) x 3 (questions) x 10 (answers)
- 23. Evaluation 23 Metric: Script: ● Charactersto lowercase ● Remove periods (unless decimal periods) ● Number words to digits ● Remove articles ● Add apostrophe to contractions ● Replace punctuation with space
- 24.
- 25. 53.62%CVPR2016 VQA Challenge RealImages Open-ended, test-standard dataset partition 25
- 26. Results in detail 26 VALIDATIONSET TEST SET Model Yes/No Number Other Overall Yes/No Number Other Overall Model 1 71.82 23.79 27.99 43.87 71.62 28.76 29.32 46.70 Model 3 75.02 28.60 29.30 46.32 - - - - Model 2 75.62 31.81 28.11 46.36 - - - - Model 5 78.15 32.79 33.91 50.32 78.15 36.20 35.26 53.03 Model 4 78.73 32.82 35.5 51.34 78.02 35.68 36.54 53.62
- 27. Results in context 27 100%0% Humans 83.30% UCBerkeley & Sony 66.47% Baseline LSTM&CNN 54.06% Baseline Nearest neighbor 42.85% Baseline Prior per question type 37.47% Baseline All yes 29.88% Ours 53.62%
- 28. Comparison with thebaseline Our model ● Single word answer ● Generate answers 28 Baseline ● Multi word answers (hardcoded) ● Classify over the 1000 most common answers
- 29.
- 30.
- 31.
- 32. Research contribution: Extendedabstract 32 VQA workshop, CVPR 2016
- 33. Research controbution: Extendedabstract - Poster 33
- 34. … ticket toLas Vegas 34
- 35. 35Presenting our posterand extended abstract at CVPR 2016, Las Vegas, USA
- 36. VQA Challenge statistics:Answering method 36
- 37. Introduction Related Work Methodology ResultsConclusions Future Work Conclusions 37
- 38. Conclusion 38 ✓ Present toVQA Challenge, CVPR 2016 Goals accomplished ✓ First GPI project using text processing techniques ✓ Create a scalable VQA model ✓ Build a modular and reusable software package ✓ Extended abstract accepted to VQA workshop CVPR 2016
- 39. Conclusion Personal overview ● Submissionto VQA Challenge ● VQA, hot topic at CVPR 2016 ● Model designed to generate answers instead of classifying them ● Question-Answer pair generation proposal 39
- 40. Introduction Related Work Methodology ResultsConclusions Future Work Future Work 40
- 41. Future work 41 ● Decoderfor multiple word answers ● Character embedding ● Attention mechanisms ● Question-Answer pairs generation Next steps
- 42. Automatic Question-Answer PairsGeneration 42
- 43. Thank You! 43 Do youhave any question?
- 44. Project resource links ●Thesis: https://imatge.upc.edu/web/sites/default/files/pub/xMasuda- Mora_0.pdf ● Web page: http://imatge-upc.github.io/vqa-2016-cvprw/ ● Source code: https://github.com/imatge-upc/vqa-2016-cvprw 44
- 45. Motivation: First stepstowards QA Generation 45 AI System Question What is the man doing? Answer Surf
- 46. VQA: Counterexample 46 Dynamic ParameterPrediction Network (DPPnet) Noh, H., Seo, P. H., & Han, B. Image question answering using convolutional neural network with dynamic parameter prediction. CVPR 2016
- 47.
- 48.
- 49.
- 50.
- 51. VQA Challenge statistics:Image modelling 51
- 52. VQA Challenge statistics:Question modelling 52