Recommended
More Related Content
What's hot
What's hot (19)
Similar to Tag-based Approaches to Sharing Background Information regarding Social Problems towards Facilitating Public Collaboration
Similar to Tag-based Approaches to Sharing Background Information regarding Social Problems towards Facilitating Public Collaboration (20)
More from siramatu-lab
More from siramatu-lab (20)
Recently uploaded
Recently uploaded (20)
Tag-based Approaches to Sharing Background Information regarding Social Problems towards Facilitating Public Collaboration
- 1. Tag-based Approaches to Sharing Background Information regarding Social Problems towards Facilitating Public Collaboration Masaru Watanabe, Shun Shiramatsu, Yasuaki Goto Nagoya Institute of Technology 1
- 2. Outline 1. Background and Goal 2. Automatic annotation 1. Generate tags 1. Filtering 2. SVM 2. Annotate tags 1. TF-IDF 2. Paragraph Vector 3. Prototype of API 3. Systems for sharing collaborative activities 4. Conclusion 2
- 3. Background CivicTech : Citizens and IT engineers cooperate to solve social problems Hackathons are frequently held. When participants discuss the solutions of social problems, they need to share background regarding the problems 3 Goal： Sharing background information about social problems
- 4. Our Approaches 4 1. Automatic annotation to web articles with social problem tags If articles have tags of social problems, these articles can be found easily as background information of the problems. 2. Systems for sharing collaborative activities By making the activities in the organization open data, citizen collaboration across the organization is promoted. Goal： Sharing background information about social problems
- 5. Outline 1. Background and Goal 2. Automatic annotation 1. Generate tags 1. Filtering 2. SVM 2. Annotate tags 1. TF-IDF 2. Paragraph Vector 3. Prototype of API 3. Systems for sharing collaborative activities 4. Conclusion 5
- 6. Tag-based Search 6 Disaster Global Warming Hunger Click Articles about "Global Warming" Global WarmingDiscussion about solutions of global warming
- 7. Knowledge Connector Site that can share works such as ideas, applications, datas Tag-based search is supported 7 Users often forget to annotate or do not understand the necessity of annotation Orthographical variants of tags http://idea.linkdata.org/
- 8. Our Solution 8 Users often forget to annotate or do not understand the necessity of annotation Orthographic variants of tags Automatic annotation with social problem tags Automatic generation of a tagset in advance
- 11. Generate Tags Requirements for generating tags Hierarchical structure Because exploratory browsing of related problems promotes understanding of background information Sufficient amount of social problem tags 11 "Social problem" category of DBpedia Japanese DBpedia Japanese： well-known linked open dataset that is converted from Wikipedia. Some articles in the category are unrelated to "social problem"
- 12. Filtering to exclude inappropriate tags 12 Extract page title from "Social Problem" Category and its sub categories (within n hierarchical levels). Filtering noisy resources by tracing other particular categories.
- 13. Categories used for filtering Filter A Stub Category, Computer Science, Judgment, Work, Social Movement Organization, People, Biology Field， Criminal Studies, Crime type, Peace Studies, Logic 13 Filter B almost the same as Filter A, except "Biological Field" is excluded.
- 14. Evaluation method (Filtering) Recall Six participants selected 102 pages that relate social problem from Japanese Wikipedia Precision Select 100 tags randomly from the tag list 14 Calculated the percentage of these items that were included in the tag list Ask 25 participants to evaluate whether these data were social problems on a five-point scale. Calculate the percentage of regarded tags. (more than three on the scale were regarded)
- 15. Evaluation (Tag Generation by Filtering) 15 The method with Filter B and 2 hierarchical levels has best balance. recall : 43% precition : 49%
- 16. Filter based on SVM Dataset Pages belonging to a lower category within three hierarchical levels of "Category: Social problem" Feature vector used a. category page that can reach within 5 hierarchical levels from any one of the acquired pages, the occurrence frequency is 9 or more b. Total of distributed representation vectors of words (word2vec) included in each page title c. Distributed representation vector of the full text of each page(doc2vec) d. Mixing a. and c. 16
- 17. Evaluation method(SVM) 10-fold cross validation test Both positive and negative examples used 120 cases Use the results that obtained when evaluate the precision of filtering method 17 Recall percentage of examples categorized into the positive class among the positive examples Precision percentage of the positive examples among examples categorized into the positive class
- 18. Evaluation (Tag Generation by SVM) 18 Filtering methods : recall is 43% precision is 49% 79.2 68.8 90.8 50.7 73.6
- 20. Annotate Tags Calculate Cos similarity between target article and all Wikipedia articles with title of tag name. When the similarity is equal to or higher than the threshold, the title is set as the tag to be attached. Two methods are used for vector generation. 1. TF-IDF 2. Paragraph Vector 20
- 21. Evaluation method (Annotate) Measure Cos similarity with each method for 10 articles on social problems collected in advance. 21 Evaluate the validity of the tags in seven-point scale by showing to 25 participants up to ten tags which annotated to article and three randomly extracted tags. Calculate correlation coefficient and accuracy based on evaluation.
- 22. Evaluation (Tag Annotated by TF-IDF) 22 correlation coefficient : 0.732 accuracy rate at threshold 0.2 : 0.812 Tags with similarity of 0.2 or more : 37/85
- 23. Example of false recognition: Evaluation value by system differs from the evaluation value by human In the article of Hunger, "Food crisis" Human : 7 (very high) System : 0.154 (low) In the article of Bullying "Social isolation" Human : 5 (high) System : 0.152 (low) Similarity assessment by related terms could not be considered. 23Note : These tags are translated from Japanese.
- 24. Evaluation (Tag Annotated by Paragraph Vector) 24 correlation coefficient : 0.346 accuracy rate at threshold 0.35 : 0.824 Tags with similarity of 0.35 or more ：8/102
- 26. Prototype of API 26 Input : http://foo-bar.net/tag-recom/[Target page URL] Output： Note : These tags are translated from Japanese.
- 27. Outline 1. Background and Goal 2. Automatic annotation 1. Generate tags 1. Filtering 2. SVM 2. Annotate tags 1. TF-IDF 2. Paragraph Vector 3. Prototype of API 3. Systems for sharing collaborative activities 4. Conclusion 27
- 28. Knowledge Connector (Repeated) Site that can share works such as ideas, applications, data 28 We aim to solve these problems by developing MissionForest Users often forget to annotate or do not understand the necessity of annotation Orthographic variants of tags Lack of a task management function
- 29. MissionForest 29 Web system for sharing social activities and research activities. Managing tasks in a tree structure like Work Breakdown Structure. Activity data is published as linked open data.
- 30. Benefits of linked open data 30 You can discovery information about social problem from tags.
- 31. Future work for MissionForest 31 • Annotate each task with social problem tags that can be used for exploratory browsing of social activities browsing other organization's solution is helpful for discussing about own problems Environmental destruction Global warming
- 32. Outline 1. Background and Goal 2. Automatic annotation 1. Generate tags 1. Filtering 2. SVM 2. Annotate tags 1. TF-IDF 2. Paragraph Vector 3. Prototype of API 3. Systems for sharing collaborative activities 4. Conclusion 32
- 33. Conclusion Automatic Annotation • Filtering method based SVM can generate sufficient tag set from DBpedia Japanese. • TF-IDF method can tag articles with reasonable precision. Systems for sharing collaborative activities • We are developing MissionForest for connect collaboration within the university laboratory and cross-organization collaboration 33
Editor's Notes
- Thank you chair. I'll talk about "Tag-based Approaces to Sharing Background Information regarding Social Problems towards Facilitating Public Collaboration."
- This is a brief outline of our presentation. Firstly, Background and Goal; secondly, Automatic annotation; thirdly, Systems for sharing collaborative activities; finnaly, Conclude our presentation.
- In recent years, Civictech are getting active. Civictech refers to activities to solve social problems by collaboration between citizens and IT engineers. Many Civictech Hackathon is being held. In Civictech, participants discuss the solutions of social problems. In order to do that, it is necessary to share background knowledge about the problem. Therefore, we set our goal as "Sharing background information about social problems."
- We chose two approaches to achieve our goal. The first approache is to annotate social problem tags to web articles automatically. If articles have tags of social problems, these articles can be found easily as background information of the problems. The second approache is to develop "Systems for sharing collaborative activities." By making the activities in the organization open data, citizen collaboration across the organization is promoted.
- Next, I'll talk about Automatic annotation.
- If tags are attached to articles on social problems, you can easily investigate about social problems. This is an example of using tags when discussing "global warming". In this discussion, let's assume that a article with the tag "global warming" is shared. By clicking the tag "global warming" you can get a list of articles on other "global warming".
- An example of actually using such a tag-based search is the site called Knoledge Connector. Knoledge Connector is the site that can share works such as ideas, applications and datas. In Knoledge Connector, tags are annotated by users. Therefore, some problems arise from it. For example, users often forget to annotate the tags, some users do not understand the necessity of annotation, or orthographical variants are occur.
- These are our solution for the problems. The solution to forget to annotate tags is to annotate tags automatically. The solution to orthographic variants is to generate a tag set automatically.
- This is architecture of our automatic tag annotation systems. First, we generate tags and use the results to annotate them to the article of social problem on the web.
- First, we would talk about how to generate tag set automatically.
- The tag here has a purpose of exploratory browsing of related problems promotes understanding of background information. Therefore, there should be a hierarchical structure between the tags. Furthermore, it is also important to have sufficient amount of social problem tags. In this research, we selected "social problem" category of Japanese version DBpedia as the source of tag extraction. DBpedia is a project to extract information from Wikipedia and publish it as linked open data. We tried to extract tag candidates from DBpedia. However, if extracted as it is, some articles in the category are unrelated to "social problem."
- In order to solve this problem, we decided to filter by what parent category a page has. First, we obtain pages belonging to the social problem category and its lower category, and extract page title as tag name. Next, remove the page belonging to other specific category and its lower category from the list of extracted pages.
- In this research, two kinds of filters are prepared. Filter A is a filter designed based on common points found that can be judged by observing as having no relation to social problems in non filtered tag list. Filter B is a filter which does not use the category "Biological Field" from filter A. It was not only filtering non related tags but also filtering many related tags.
- In the evaluation of filtering, the recall and precision were calculated. This is how calculate recall. First, six participants selected 102 pages that relate social problem from Japanese Wikipedia. Then, we calculated the percentage of these items that were included in the tag list. This is how calculate precision. First, select 100 tags randomly from the tag list. Then, we asked 25 participants to evaluate whether these data were social problems on a five-point scale. Finaly, we calculated the percentage of the tags that have more than three on the scale.
- This is evaluation result. Precision is dropped considerably when it reaches 3 levels regardless of use or not use filter. It shows that elements not related to social problems are explodingly increasing in the 3 levels or later. In addition, we got the result that the recall of 1 level without filter, it means the list get from pages belonging directly to the social problem category, is quite low. We should also review the point of using only DBpedia for tag generation.
- Therefore, we omit something that is not a social problem from the list by binary classification using support vector machines. We used wikipedia pages belonging to a lower category within 3 levels of "Category: Social problem" for dataset. Four kinds of feature vectors are prepared as support vector machine input vectors. The first one is a corpus of the category page that can reach within 5 levels from any one of the acquired pages. Among them, we decided to use those with an appearance frequency of 9 or more in the whole. The number of hierarchies from the page to the category page is taken as a value. However, if you can not reach within 5 levels, enter 6. The second one is a corpus of the whole wikipedia sentence, and the total value of the distributed representation vectors of words constituting each page title is taken as a vector. The third one is a corpus of the whole wikipedia sentence, and the total value of the distributed representation vectors of words constituting each page article is taken as a vector. The fourth one is combination of the first and third vectors.
- Based on the above vectors, we performed a 10-fold cross validation test on the support vector machine to calculate the recall and precision. We used the questionnaire results gathered when measuring the precision of filtering as the data for the positive examples and the negative examples used for the cross validation test. We calculate recall and precision. In this evaluation, recall means how much positive examples are included among examples of categorized into the positive class, and precision means how much examples of categorized into the positive class are included among positive examples.
- This is evaluation result. In the method using word vectors, recall is high and precision is low. This seems to be because most elements were judged to be positive. Except for recall of word vectors, the method using only category pages shows better performance than others. So, when using support vector machine, it is effective to use only category information. Because the denominator of the ratio is different from the evaluation form of the filterling methods, the ratio values cannot be simply compared. But this results indicate great implovement from the filtering methods.
- Next, We'll talk about automatic tag annotation.
- For automatic assignment, use Wikipedia's article with the title with the same name as tag candidate. We calculate the Cos similarity of the article to be annotated and the Wikipedia article with the same name as the tag candidate, and use tags with a degree of similarity equal to or higher than the threshold as the given tag. In creating a vector for calculating Cos similarity, we used two methods, TF-IDF and paragraph vector.
- In the evaluation of automatic annotation, we calculated correlation coefficient and accuracy. We gave ten articles to the tag annotation system created by each method and calculated Cos similarity. Tags whose calculated Cos similarity was within the top 10 and equal to or more than the threshold and 3 tags selected at random were shown to 25 participants and evaluated in seven-point scale. Then calculate correlation coefficient and accuracy based on evaluation.
- This is evaluation result. The correlation coefficient shows a strong correlation. The system evaluation value of the element which question evaluation value is 7 are greatly dispersed. We think that this is due to the characteristics of TF-IDF which can not handle related words. When the threshold value was set to 0.2, accuracy rate and the number of tags given is sufficient. We think this is useful to support semi-automatic annotation at actual use.
- These are examples where the system evaluation value differs from the questionnaire evaluation value. In the article on hunger, the tag "food crisis" was judged appropriate tag by human, and inappropriate tag by system. Also in the article on bullying, the tag "social isolation" was judged appropriate tag by human, and inappropriate tag by system. When I looked at these articles, there was no mentioned about these tags in the articles itself. It seems that the questionnaire evaluation value got higher due to the high relevance of words themselves "hunger" and "food crisis," "bullying" and "social isolation". We think that these problems can be solved by introducing a method that can handle related words.
- This is evaluation of Paragraph Vector method. The correlation coefficient shows a weak correlation. In this research, we used only 102 Wikipedia articles that is tag candidates for corpus of paragraph vector. We think that correlation was not achieved because the number of documents for corpus was insufficient. Since the paragraph vector algorithm considers word order, there is a possibility that the stylistic difference may be affected. When the threshold value was set to 0.35, accuracy rate is sufficient but number of tags given is not enough. These results indicated that TF-IDF is superior for actual use than Paragraph Vector.
- We provided a system for automatic generation and automatic annotation of tags as an API.
- By passing the URL containing articles on social problem in the API, the tags that the system judged when given to the article are returned together with the similarity. Currently JSON in the format shown in the figure is output.
- Now let's we talked about "Systems for sharing collaborative activities."
- An example of "a system that shares collaborative work" is the Knowledge Connector mentioned. However, the Knowledge Connector has the problem of not having the task management function in addition to the above problem. We aim to solve these problems by developing MissionForest.
- This is the User Interface of MissionForest. MissionForest is a web system for sharing social activities and research activities. This system can managing tasks in a tree structure like Work Breakdown Structure and published activity data as linked open data.
- Linked open data is open data that expresses each data and the relationship between the data by URI.By publishing the data in this format, the user can effectively utilize the related information based on the connection between the data.If there is the above tags as one of the connections between the data, it will lead to the discovery of articles and data of social problems related to a certain mission and task.
- As a future work, We are thinking of incorporating the tag system above. Browsing other organization's solution is helpful for discussing about own problems. So, We think that annotate each task with social problem tags that can be used for exploratory browsing of social activities.
- Conclusion. First, We talked about automatic tag annotation . Filtering method based SVM can generate sufficient tag set from DBpedia Japanese. TF-IDF method can tag articles with reasonable precision. Next, We talked about systems for sharing collaborative activities. We have developing MissionForest for connect collaboration within the university laboratory and cross-organization collaboration. That's all thank you.
- An example of a tag that could not be filtered by Filter B. A typical example of abusive intergenerational chaining called Goller family has been judged as a social problem by the system, and participants are judged to have no relation with social problems. Definitions of key words to be used as social problem tags should be considered. Key words in many fields such as myths and feelings remained. Eleven categories are already selected for filtering already. The method of adding new categories to make it possible to filter these tags is not realistic.