Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Shuangyong Song, Qingliang Miao and Yao Meng | Linking Images to Semantic Knowledge Base with User-generated Tags

243 views

Published on

http://2016.semantics.cc/shuangyong-song

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Shuangyong Song, Qingliang Miao and Yao Meng | Linking Images to Semantic Knowledge Base with User-generated Tags

  1. 1. 関係者外秘関係者外秘 Shuangyong Song Fujitsu R&D Center Co., Ltd. Linking Images to Semantic Knowledge Base with User-generated Tags Copyright 2013 Fujitsu R&D Center Co.,LTD
  2. 2. 関係者外秘関係者外秘 Outlines  Motivation  Method  Experimental results  Conclusion 1Copyright 2010 FUJITSU LIMITED
  3. 3. 関係者外秘関係者外秘 Outlines  Motivation  Method  Experimental results  Conclusion 2Copyright 2010 FUJITSU LIMITED
  4. 4. 関係者外秘関係者外秘 Motivation  Images account for an important part of Multimedia Linked Open Data. (LOD4ALL)  1. Wide semantic annotation of images with image understanding is difficult and fallible; 2. semantic annotation with manual labelling is time-consuming.  Therefore, automatic semantic annotation of images with using other kind of crowdsourced ‘manual labelling’ is considered. 3Copyright 2010 FUJITSU LIMITED Tags Semantic KnowledgeImages medium
  5. 5. 関係者外秘関係者外秘 Outlines  Motivation  Method  Experimental results  Conclusion 4Copyright 2010 FUJITSU LIMITED
  6. 6. 関係者外秘関係者外秘 Method  Module 1. Linking tags to knowledge  Module 2. Detecting semantic relations between tags  Module 3. Extending knowledge linked tags  Module 4. Linking images to knowledge Copyright 2013 Fujitsu R&D Center Co.,LTD5 [M1] [M2] [M3] [M4] Semantic Knowledge Semantic Knowledge Semantic Knowledge Semantic Knowledge Semantic Knowledge Images Tags Topics
  7. 7. 関係者外秘関係者外秘 Method – Offline Part  Module 1. Linking tags to knowledge  Tag ‘Somophyllin-DF’: search it in dbo or dbr, and detect the unambiguous matched ontology as ‘dbr:Somophyllin-DF’,  ‘Apple’, we may not be able to link it with ‘dbr:Apple_Inc.’ or ‘dbr:Apple_III’ or some other ontologies which related to ‘Apple’. 6Copyright 2010 FUJITSU LIMITED 1 1 ( , )* ( , ) ( , ) ( , , ) ( ) a u u uU U i k k ja u a u u i j i k j u k k k C t t C t t R t t R t t t F t    
  8. 8. 関係者外秘関係者外秘 Method – Offline Part  Module 2. Detecting semantic relations between tags 7Copyright 2010 FUJITSU LIMITED T DNd θ(d) α z wϕ(z) β Image data Tags Topic (vector) Tags  Topical vectors For example, tags ‘programmer’ and ‘ilife’ are detected as related tags. Image-tag matrix is sparse and difficult to be well analyzed, topical information can help to discover the implicit semantic relations between tags.
  9. 9. 関係者外秘関係者外秘 Method – Offline Part  Module 3. Extending knowledge linked tags  The rules are as below: (very strict for keep precision)  1) If t1 and t2 have ‘inclusion relation’, such as ‘motor’ and ‘motorcycle’, and cosine- similarity value between their topical vectors is bigger than σ, we judge them as similar tags;  2) If ‘Levenshtein distance’ value between t1 and t2 is equal to or smaller than a threshold β, which is a positive integer, and cosine-similarity value between their topical vectors is bigger than σ, we judge them as similar tags;  3) If t1 and t2 don’t have above relations, we judge them as dissimilar tags.  If t1 and t2 are judged as similar tags, and just one of them has semantic link to knowledge base, we link the other one to the same knowledge with a probability, of which the value equals to C(t1,t2), which is the cosine- similarity value between t1 and t2. 8Copyright 2010 FUJITSU LIMITED
  10. 10. 関係者外秘関係者外秘 Method – Online Part  Module 4. Linking images to knowledge 9Copyright 2010 FUJITSU LIMITED  Define the linking types <subject, predicate, object>  We use 77 million triples collected from our semantic database ‘LOD4ALL’ as criteria for predicate selection. We create links between images and objects with unambiguous predicate. For example, we use predicate ‘dbo:locationCountry’ to create links to objects such as ‘China’ or ‘Denmark’.  We use ‘rdfs:seeAlso’ to be predicate for ambiguous triples.  In particular, we also detect some tag-combined knowledge for expanding the links’ range. For example, if an image has both ‘DigitalCamera’ and ‘Conon’ as its tags, we will check if there is ‘DigitalCamera_Conon’ or ‘Conon_ DigitalCamera’ in dbo or dbr, and link this image to the detected knowledge.
  11. 11. 関係者外秘関係者外秘 Outlines  Motivation  Method  Experimental results  Conclusion 10Copyright 2010 FUJITSU LIMITED
  12. 12. 関係者外秘関係者外秘 Experimental results  1. Experimental Datasets  2. Methods for Comparison  3. Evaluation Metrics & Test Data Preparation  4. Evaluation Results 11Copyright 2010 FUJITSU LIMITED
  13. 13. 関係者外秘関係者外秘 Experimental results  1. Experimental Datasets  mirFlickr 1 , 690,649 images and 659,227 tags.  all the 7,401 ontologies in dbo and 7,925,232 resources in dbr were used as semantic KB.  After removing 41,127 images without any tag and 392 attached with ‘stopword’ tags, we had 649,130 images left, and those images and their tags were used to train the topic model with an empirical K value of 200. 12Copyright 2010 FUJITSU LIMITED 1 Huiskes, M. J., Lew, M. S. The MIR Flickr Retrieval Evaluation. In: Proceedings of the 1st ACM SIGMM International Conference on Multimedia Information Retrieval, 2008, pp.39-43.
  14. 14. 関係者外秘関係者外秘 Experimental results  1. Experimental Datasets  2. Methods for Comparison (our model: LDA based) 13Copyright 2010 FUJITSU LIMITED Word2vec based model (W2V) Word2vec is a recently popular method for getting distributed representation of words, of which the output form is similar to LDA. However, we suppose that W2V is unfit for the image- knowledge linking task, since W2V does well in detecting context similarity, such as the similarity between ‘Paris’ and ‘Beijing’. We will evaluate it in the following part. Co-occurrence based model (Co-occur) Co-occur is a very simple yet effective method for detecting relationship between words, we also take it as a baseline on the tag similarity calculation subtask instead of LDA method in subsection 2.3, and evaluate it in in the following part. Compared to our model, Co-occur is a weaker tool on detecting latent semantic information of tags.
  15. 15. 関係者外秘関係者外秘 Experimental results  1. Experimental Datasets  2. Methods for Comparison  3. Evaluation Metrics & Test Data Preparation 14Copyright 2010 FUJITSU LIMITED To evaluate the effects of different d when β = 2, we manually label 200 images. We roughly set the recall as 1.0 when d = 1 considering we are unable to collect all possible related knowledge to an image. After getting both precision and recall with different d, we also use F1-value as evaluation criterion to evaluate d. precision ; recall ; F1-value.
  16. 16. 関係者外秘関係者外秘 Experimental results  1. Experimental Datasets  2. Methods for Comparison  3. Evaluation Metrics & Test Data Preparation  4. Evaluation Results 15Copyright 2010 FUJITSU LIMITED car girl flower bird dog sport W2V 0.217 0.531 0.233 0.389 0.325 0.442 Co-occur 0.683 0.724 0.631 0.557 0.626 0.702 Our model 0.881 0.767 0.854 0.871 0.873 0.801 Result comparison with F1-value
  17. 17. 関係者外秘関係者外秘 Experimental results - examples Copyright 2013 Fujitsu R&D Center Co.,LTD16 schema:Brand dbr:Citroen_XM dbr:French rdfs:seeAlso schema:inLanguagedbr:Citroen dbr:Car schema:category schema:location dbr:Mallorca dbr:Palma schema:location schema:location dbr:Baleares dbr:Sineu schema:location dbr:Geotagged schema:indication
  18. 18. 関係者外秘関係者外秘 Outlines  Motivation  Method  Experimental results  Conclusion 17Copyright 2010 FUJITSU LIMITED
  19. 19. 関係者外秘関係者外秘 Conclusion Summary: In this paper, we address the problem of automatically linking images to semantic KB by using images’ tagging information. Our framework lifted the F1-value of images’ semantic link creation by 62.43%. Future works: However, this paper is only a preliminary work, and named entity disambiguation and named entity normalization will be considered in our next step work. Besides, predicate discriminate should be performed by considering adjacent tags as context. In addition, we will try to create ground-truth datasets and try some classification models on this task. 18Copyright 2010 FUJITSU LIMITED
  20. 20. 関係者外秘関係者外秘 Thank you! Copyright 2013 Fujitsu R&D Center Co.,LTD19
  21. 21. 関係者外秘関係者外秘 20 Copyright 2013 Fujitsu R&D Center Co.,LTD

×