TS3-6: Atsuya Sakai from Nagoya Institute of Technology
•Download as PPTX, PDF•
0 likes•14 views
Please listen to the presentation, read detailed slides and return to first post to make your comments below the corresponding paper author's post. Session Chair: Ahmed Moustafa Session Theme: Machine Learning and Creativity Paper No: 22 Session and Talk No: TS3-6 Type: Full Co-authors: Atsuya Sakai and Takayuki Ito Title: Deep Bidirectional Transformers for Online Discussion Understanding
![Background
・ Crowd discussion support system : D-Agree[1]
・discussion platform for efficient consensus building and opinion integration
on the Web
・It would be impossible for humans to keep an eye on the flow of
large-scale discussions
→ Consider the development of an automated facilitation agent
・In order for the facilitator to facilitate the discussion properly, we
need to extract the structure of the discussion
[1] Takayuki Ito, Shota Suzuki, Naoko Yamaguchi, Tomohiro Nishida, Kentaro Hiraishi, and Kai
Yoshino. D-agree: Crowd discussion support system based on automated facilitation agent. In
Proceedings of 35th AAAI conference, volume 2020, 2020. demonstration paper.
2](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
Similar to TS3-6: Atsuya Sakai from Nagoya Institute of Technology
Similar to TS3-6: Atsuya Sakai from Nagoya Institute of Technology (20)
More from Jawad Haqbeen
More from Jawad Haqbeen (20)
Recently uploaded
Recently uploaded (20)
TS3-6: Atsuya Sakai from Nagoya Institute of Technology
- 1. Deep Bidirectional Transformers for Online Discussion Understanding The 15th International Conference on Knowledge, Information and Creativity Support Systems (KICSS 2020) Nagoya Institute of Technology, Japan Atsuya Sakai and Takayuki Ito
- 2. Background ・ Crowd discussion support system : D-Agree[1] ・discussion platform for efficient consensus building and opinion integration on the Web ・It would be impossible for humans to keep an eye on the flow of large-scale discussions → Consider the development of an automated facilitation agent ・In order for the facilitator to facilitate the discussion properly, we need to extract the structure of the discussion [1] Takayuki Ito, Shota Suzuki, Naoko Yamaguchi, Tomohiro Nishida, Kentaro Hiraishi, and Kai Yoshino. D-agree: Crowd discussion support system based on automated facilitation agent. In Proceedings of 35th AAAI conference, volume 2020, 2020. demonstration paper. 2
- 3. Research Problem ・The model size for extraction becomes large Demerit ・Unable to learn in local environment ・It takes time to learn ・After implementation, it costs a lot to maintain the server → Need to reduce model size 3
- 4. Goal Opinions are classified into the following 5 types ・Issue ：Question whether to agree or disagree ・Positive_Claim ：Claim in agree position ・Negative_Claim ：Claim in disagree position ・Positive_Premise ：Reason of agree ・Negative_Premise ：Reason of disagree →・ What type is opinion classified into? (node classification)[2] ・ Which opinion corresponds to which opinion? (link prediction)[2] We aim to extract discussion structure and reduce model size. [2] Shota SUZUKI, Naoko YAMAGUCHI, Tomohiro NISHIDA, Ahmed MOUSTAFA, Daichi SHIBATA, Kai YOSHINO, Kentaro HIRAISHI, and Takayuki ITO. Extraction of online discussion structures for automated facilitation agent. Proceedings of the Annual Conference of JSAI, JSAI2019:2F1E302– 2F1E302, 2019. [3] Christian Stab and Iryna Gurevych. Parsing argumentation structures in persuasive essays. Computational Linguistics, Vol. 43, No. 3, pp. 619–659, 2017. 4 Figure1 : discussion structure[3]
- 5. CLS token Overview 5 Figure2 : input and output examples in BERT[4] [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2019. Figure3 : Architecture of BERT
- 6. Node Classification by CLS token BERT sentence predicted label [CLS] I am against the restart of Hamaoka nuclear power plant [SEP] [CLS] vector contains information on the whole sentence as used for classification[4]. In this research, the model learn and predict only [CLS] vector. 6 [CLS] 0.2 -0.3 … 0.6 I -0.6 -0.1 … 0.5 : plant 0.4 0.2 … 0.9 [SEP] -0.2 0.6 … 0.5 Dense [CLS] 0.2 -0.3 … 0.6 [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2019.
- 7. Link Prediction by CLS token Dense predicted link predicted vector Calculate cosine similarity 7 ・Unable to learn in local environment ・It takes time to learn ・After implementation, it costs a lot to maintain the server Solve these problems by reducing the input dimension [CLS] 0.2 -0.3 … 0.6 I -0.6 -0.1 … 0.5 : plant 0.4 0.2 … 0.9 [SEP] -0.2 0.6 … 0.5 sentence BERT [CLS] I am against the restart of Hamaoka nuclear power plant [SEP]
- 8. Experimental Settings ・Create data based on discussions held at D-Agree ・Experiment with cross-validation ・Node classification is evaluated by precision, recall and F1 ・Link prediction is evaluated by precision and model size 8
- 9. Result of Node Classification precision recall F1 Issue 0.93 0.84 0.88 Positive_Claim 0.79 0.83 0.81 Negative_Claim 0.75 0.78 0.77 Positive_Premise 0.61 0.59 0.60 Negative_Premise 0.62 0.60 0.61 Table1 : Result of node classification ・The F1 of Issue and Positive_Claim are good. ・The F1 of Negative_Claim is relatively good. ・The F1 of Positive_Premise and Negative_Premise are not good. 9
- 10. Result of Link Prediction The number of parameters model size Word2Vec+Dense 100,250,800 401[MB] fastText+BiLSTM[2] 59,204,736 237[MB] BERT(only[CLS])+Dense 2,128,360 8[MB] Table2 : Result of link prediction Table3 : Model size comparison ・The precision of Negative_Claim is good. ・The precision of Positive_Claim is relatively good. ・The precision of Positive_Premise and Negative_Premise are not good. ・We could make the model much smaller. 10 precision Positive_Claim 0.76 Negative_Claim 0.82 Positive_Premise 0.63 Negative_Premise 0.54 [2] Shota SUZUKI, Naoko YAMAGUCHI, Tomohiro NISHIDA, Ahmed MOUSTAFA, Daichi SHIBATA, Kai YOSHINO, Kentaro HIRAISHI, and Takayuki ITO. Extraction of online discussion structures for automated facilitation agent. Proceedings of the Annual Conference of JSAI, JSAI2019:2F1E302–2F1E302, 2019.
- 11. Discussions of Node Classification 11 ・The F1 of Issue and Positive_Claim is good. → Issue is a question form and has a characteristic grammar → Positive_Claim is almost the same way of saying ・The F1 of Negative_Claim is relatively good. → Negative_Claim is similar in wording ・The F1 of Positive_Premise and Negative_Premise is not good. → These opinions are diverse
- 12. Discussions of Link Prediction 12 ・The precision of Negative_Claim is good. → The number of issues that are candidates is small ・The precision of Positive_Claim is relatively good. → The number of issues that are candidates is small ・The precision of Positive_Premise and Negative_Premise is not good. → There are many candidates for link destinations
- 13. Structure Extraction in Actual Discussion 13 Do you think that only the citizens of Shizuoka should be included in the scope of the prefectural vote for the restart of the Hamaoka nuclear power plant? Or do you think there is another suitable range? ｜ └ Because the Hamaoka nuclear power plant is close to the east- west transportation infrastructure such as the Tomei Expressway and the Shinkansen, there is a possibility that it will become a traffic block in the event of a nuclear accident. I think that not only the problem of Shizuoka Prefecture, but also it should be expanded to the local governments along Tokaido. ｜ └ What are your thoughts on this issue? Please post your opinion with the reason. (by Agent) ｜ └ I agree very much. However, if your wish is to compress the nuclear fuel that Japan has, to make it into a safe material, to make effective use of it, and to restart operations related to research and experiments, please agree and take risks. I would like to move to. ｜ └ What do you think about this opinion? (by Agent) ｜ └ The area on the east side of Aichi is closer to the Hamaoka Nuclear Power Plant than the eastern part of Shizuoka, and I think it should not be targeted to Shizuoka residents only. I feel that the farther away I am from other people. If you live in a short distance, the damage in the event of an accident will be greater, so I would like you to respect the will of local residents near the nuclear power plant. Do you think that only the citizens of Shizuoka should be included in the scope of the prefectural vote for the restart of the Hamaoka nuclear power plant? Or do you think there is another suitable range? : Issue ｜ ├ I think that not only the problem of Shizuoka Prefecture, but also it should be ｜ expanded to the local governments along Tokaido. : Negative_Claim ｜ ├ I agree very much. : Positive_Claim ｜｜ ｜└ However, if your wish is to compress the nuclear fuel that Japan has, to ｜ make it into a safe material, to make effective use of it, and to restart ｜ operations related to research and experiments, please agree and take ｜ risks. I would like to move to. : Positive_Premise ｜ └ The area on the east side of Aichi is closer to the Hamaoka Nuclear Power Plant than the eastern part of Shizuoka, and I think it should not be targeted to Shizuoka residents only. : Negative_Claim ｜ └ If you live in a short distance, the damage in the event of an accident will be greater, so I would like you to respect the will of local residents near the nuclear power plant. : Negative_Premise ｜ └ I feel that the farther away I am from other people. : Negative_Premise ｜ └ Because the Hamaoka nuclear power plant is close to the east-west transportation infrastructure such as the Tomei Expressway and the Shinkansen, there is a possibility that it will become a traffic block in the event of a nuclear accident. : Negative_Premise Actual discussion Extracted discussion structure
- 14. Conclusion and Future Work Goal ・We aim to extract discussion structure by node classification and link prediction and reduce model size. Result of node classification ・The F1 of Issue and Positive_Claim is good. Result of link prediction ・The precision of Negative_Claim is good. ・We could make the model much smaller. Future work ・We will improve the annotations and collect and create additional data to improve the accuracy of node classification and link prediction. 14
Editor's Notes
- Today, I would like to talk about “Deep Bidirectional Transformers for Online Discussion Understanding”.
- In recent years, large-scale discussion support system on the Web have been developed and attracted attention. D-Agree that is discussion platform for efficient consensus building and opinion integration on the Web is one of them. … large-scale discussions. Therefore, consider …. ….
- But, the model size …. The disadvantages are as follows: …. Specifically, ・ There is a possibility that you will not be able to read and apply the papers of this research and previous research ・ It takes time for learning, and it takes days for cross-validation. ・ If you try to rent a server with GPU, it costs tens of thousands of dollars a month. Therefore, need to ….
- In order to structure the discussion, opinions are …. Issue means that question whether …. Positive_Claim means that …. …. And figure 1 shows discussion structure we adopt. Issue node can link to any node. Positive_Claim and Negative_Claim node can link to Issue node. Positive_Premise and Negative_Premise node can link to same position Claim node or Premise node. この構造を採用した理由といたしましては，これから先，技術がより進歩していく上で， マルチエージェントシステムによって賛成と反対の合意点を探すことができるのではないかと考えたためです． We aim to extract discussion structure by node classification and link prediction and reduce model size
- Figure 2 and 3 show the structure of BERT . A CLS token is a meaningless token at the beginning of a sentence. BERT is a bi-directional transformers, therefore, all word information is propagated to the CLS token. Therefore, the CLS token contains information for the entire sentence. In BERT, Next Sentence Prediction (NSP) pre-training to determine whether two sentences are related is learned with CLS tokens. In this research, we adopted the method of using only CLS tokens.
- The slide shows proposed node classification method. First, add CLS token called classification token at the beginning of the sentence and SEP token called separator token at the end of the sentence. Second, input CLS token, sentence, and SEP token to BERT. Third, input the vector of only CLS tokens in the obtained distributed representation to Dense Then, Dense outputs predicted label. CLS token‘s vector contains information ….
- The slide shows proposed link prediction method. First, add CLS token and SEP token in the same way as node classification.. Second, input CLS token, sentence, and SEP token to BERT. Third, input the vector of only CLS tokens to Dense. Then, Dense outputs predicted vector. Finally, calculate the cosine similarity between the vector of actually posted opinion and the predicted vector, And extend the link to the one with the highest similarity.
- In the experiment, we use created data based on discussions held at D-Agree. Experiment with cross-validation. Node classification is evaluated by precision, recall and F1. Link prediction is evaluated by precision and model size.
- Table 1 shows result of node classification. In the table 1, The F1 of Issue and Positive_Claim are good because these F1 are higher than 80%. The F1 of Negative_Claim is relatively good because this F1 is close 80%. The F1 of Positive_Premise and Negative_Premise are not good.
- Table 2 shows result of link prediction. In the table 2, The precision of Negative_Claim is good because these precision are higher than 80%. The precision of Positive_Claim is relatively good because this precision is close 80%. The precision of Positive_Premise and Negative_Premise are not good. Table 3 shows model size comparison. In the table 3, We could make the model much smaller than previous research.
- The F1 of Issue … is good because Issue is a … grammar and Positive_Claim is …. The F1 … good because Negative_Claim is … wording. The F1 of Positive_Premise … not good because these opinions … diverse.
- ・The precision of Negative_Claim is good because the number of issues that are candidates is small ・The precision of Positive_Claim is relatively good for the same reason as Negative_Claim ・The precision of Positive_Premise and Negative_Premise is not good because there are many candidates for link destinations
- The slide shows structure extraction in actual discussion. The text on the left is the actual discussion and extracted discussion structure is on the right. They discuss about “only the citizens … prefectural vote”. In the actual discussion, user post “there is a possibility … a nuclear accident. it should be … along Tokaido.”. Then, Automated facilitation agent facilitate the discussion, other user post “I agree very much.”. Then, Automated facilitation agent facilitate the discussion, other user post “I think it should … residents only. If you live in … be greater.”. In extracted discussion structure, “only the citizens … prefectural vote” is classified Issue, and they discuss about it. Therefore this is correct. Next node, “it should be … along Tokaido” is disagree position to Issue, and this is classified Nagative_Claim. Therefore, this is correct. Next, “I agree very much” is agree position to Issue, and this is classified Positive_Claim. Next, “I think it should … residents only” is classified Negative_Claim, and this is disagree position to Issue. And, “If you live in … be greater” is reason of previous Negative_Claim, and this is classified Negative_Premise. Therefore, this is correct. From the above, our approach is efficient for extraction of discussion structure.
- conclusion That is all. Thank you for listening.