splu2020 presentation slide

1. An Element-wise Visual-Enhanced
BiLSTM-CRF Model for Location
Name Recognition
Takuya Komada, Takashi Inui
Department of Computer Science
University of Tsukuba
1
komada@mibel.cs.tsukubai.ac.jp

2. Background
l Location Name
l Location information is one of the essential
components for some NLP applications.
l E.g. location name disambiguation , mapping of
location names to geographic locations.
l Named Entity Recognition (NER)
l Detect entities and classify each entity in pre-
defined types. e.g., LOC, PER, ORG.
l For example, Leading to [Tokyo LOC].
komada@mibel.cs.tsukubai.ac.jp
2

3. Background
l Multimodal NER
l Deep learning models using visual information.
l Extract named entities from image attached
social media posts. (Twitter, SnapChat, etc)
l Related works
l Moon, Lu, Zhang Proposed a neural NER model
using images attached to document.
l Only use image attached to the document.
komada@mibel.cs.tsukubai.ac.jp
3

4. Background
l Effectiveness of Images
l Visual information can explain word meanings.
l Skyscrapers in Fig. 1
l Townscapes surrounded by mountains and rivers in
Fig. 2
l Shenzhen and Dubai have the same NE aspect and
have similarities in their images
komada@mibel.cs.tsukubai.ac.jp
4

5. Research Objective
l Image data corresponding to each word
would provide rich information of word
meanings.
l Propose a method that utilizes images more
effectively.
l image data are obtained for each word.
l Introduce a Gate mechanism
l Control the extent to which the visual feature.
komada@mibel.cs.tsukubai.ac.jp
5

6. Proposed Model
l (Baseline) Character-based BiLSTM-CRF Model
komada@mibel.cs.tsukubai.ac.jp
6

7. Proposed Model
komada@mibel.cs.tsukubai.ac.jp
7
l Obtain Image Embeddings

8. Proposed Model
komada@mibel.cs.tsukubai.ac.jp
8
l Image Retrieval

9. Proposed Model
komada@mibel.cs.tsukubai.ac.jp
9
l Image Retrieval

10. Proposed Model
komada@mibel.cs.tsukubai.ac.jp
10
l Obtaining Visual Embeddings

11. Proposed Model
komada@mibel.cs.tsukubai.ac.jp
11
l Visual (Simple) Model

12. Proposed Model
komada@mibel.cs.tsukubai.ac.jp
12
l Visual (Gate) Model

13. Experiments
l Dataset
l Extended Named Entity corpus
l Images
l Google Images with photo option
l Queries are all nouns in documents.
l Obtain top 15 images
komada@mibel.cs.tsukubai.ac.jp
13

14. Experiments
l Settings
l Baseline: No use of visual features.
l Visual (Simple): Uses the element-wise visual
features.
l Visual (Gate): Uses element-wise visual features
with the Gate mechanism.
komada@mibel.cs.tsukubai.ac.jp
14

15. Experimental Results
l Results
l Both models using element-wise visual features
outperformed the baseline model.
komada@mibel.cs.tsukubai.ac.jp
15

16. Experimental Results
l Examples
l (ex.1-E) Signed by 117 countries and two regions at the
Final Protocol and Convention Signing Conference in
[Jamaica Country] in December 1982.
l (ex.2-E) Finally, tomorrow is the last day of our stay in
France, except for the day we leave. We re going to
[Avignon City].
komada@mibel.cs.tsukubai.ac.jp
16

17. Experimental Results
l Unseen words
l precision values improved most significantly
(Seen: +5.08, Unseen: +7.07).
l improve the performance of true-negatives.
komada@mibel.cs.tsukubai.ac.jp
17

18. Conclusions
l Propose an element-wise visual-enhanced NER
model
l Element-wise visual feature
l Image retrieval
l Gate mechanism
l Achieved a higher F1-value performance than the
baseline model
l Future research
l Investigate of effectiveness in other NE classes.
l Improve our model by conducting elaborate query
investigations that are motivated by the error analysis.
l Attempt queries with nouns and adjectives/verbs.
komada@mibel.cs.tsukubai.ac.jp
18