for AgentCrowd2020

1. Deep Bidirectional Transformers for
Online Discussion Understanding
The 2nd International Workshop on Agent-based Crowd
Decision-making (AgentCrowd2020) in conjunction with AAMAS2020
Nagoya Institute of Technology, Japan
Atsuya Sakai, Ahmed Moustafa, and Takayuki Ito

2. Background
・ Crowd discussion support system : D-Agree[1]
・discussion platform for efficient consensus building and opinion integration
on the Web
・It would be impossible for humans to keep an eye on the flow of
large-scale discussions
→ Consider the development of an automated facilitation agent
・In order for the facilitator to facilitate the discussion properly, we
need to extract the structure of the discussion
[1] T Ito, D Shibata, S Suzuki, N Yamaguchi, T Nishida, K Hiraishi, and K Yoshino. Agent
that facilitates crowd discussion. 7th ACM Collective Intelligence, pp. 13‒14, 2019.
2

3. Research Problem
・The model size for extraction becomes large
Demerit
・Unable to learn in local environment
・It takes time to learn
・After implementation, it costs a lot to maintain the server
→ Need to reduce model size
3

4. Goal
Opinions are classified into the following 5 types
・Issue ：Question whether to agree or
disagree
・Positive_Claim ：Claim in agree position
・Negative_Claim ：Claim in disagree position
・Positive_Premise ：Reason of agree
・Negative_Premise ：Reason of disagree
→・ What type is opinion classified into?
(node classification)[2]
・ Which opinion corresponds to which opinion?
(link prediction)[2]
We aim to extract discussion structure and reduce model size.
[2] Shota SUZUKI, Naoko YAMAGUCHI, Tomohiro NISHIDA, Ahmed MOUSTAFA, Daichi SHIBATA, Kai YOSHINO, Kentaro HIRAISHI,
and Takayuki ITO. Extraction of online discussion structures for automated facilitation agent. Proceedings of the Annual Conference
of JSAI, JSAI2019:2F1E302‒2F1E302, 2019.
[3] Christian Stab and Iryna Gurevych. Parsing argumentation structures in persuasive essays. Computational Linguistics, Vol. 43, No.
3, pp. 619‒659, 2017.
4
Figure1 : discussion structure[3]

5. CLS token Overview
5
Figure2 : input and output examples in BERT[4]
[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2019.
Figure3 : Architecture of BERT

6. Node Classification by CLS token
BERT
sentence
predicted
label
[CLS] I am against the restart of
Hamaoka nuclear power plant [SEP]
[CLS] vector contains information on the whole
sentence as used for classification[4].
In this research, the model learn and predict
only [CLS] vector.
6
[CLS] 0.2 -0.3 … 0.6
I -0.6 -0.1 … 0.5
:
plant 0.4 0.2 … 0.9
[SEP] -0.2 0.6 … 0.5
Dense
[CLS] 0.2 -0.3 … 0.6
[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2019.

7. Link Prediction by CLS token
Dense
predicted link
predicted
vector
Calculate cosine similarity
7
・Unable to learn in local environment
・It takes time to learn
・After implementation, it costs a lot to maintain
the server
Solve these problems by reducing the input
dimension
[CLS] 0.2 -0.3 … 0.6
I -0.6 -0.1 … 0.5
:
plant 0.4 0.2 … 0.9
[SEP] -0.2 0.6 … 0.5
sentence
BERT
[CLS] I am against the restart of
Hamaoka nuclear power plant [SEP]

8. Experimental Settings
・Create data based on discussions held at D-Agree
・Experiment with cross-validation
・Node classification is evaluated by precision, recall and F1
・Link prediction is evaluated by precision and model size
8

9. Result of Node Classification
precision recall F1
Issue 0.93 0.84 0.88
Positive_Claim 0.79 0.83 0.81
Negative_Claim 0.75 0.78 0.77
Positive_Premise 0.61 0.59 0.60
Negative_Premise 0.62 0.60 0.61
Table1 : Result of node classification
・The F1 of Issue and Positive_Claim
are good.
・The F1 of Negative_Claim is
relatively good.
・The F1 of Positive_Premise and
Negative_Premise are not good.
9

10. Result of Link Prediction
The number
of parameters
model size
Word2Vec+Dense 100,250,800 401[MB]
fastText+BiLSTM[2] 59,204,736 237[MB]
BERT(only[CLS])+Dense 2,128,360 8[MB]
Table2 : Result of link prediction
Table3 : Model size comparison
・The precision of Negative_Claim is good.
・The precision of Positive_Claim is relatively good.
・The precision of Positive_Premise and
Negative_Premise are not good.
・We could make the model
much smaller.
10
precision
Positive_Claim 0.76
Negative_Claim 0.82
Positive_Premise 0.63
Negative_Premise 0.54
[2] Shota SUZUKI, Naoko YAMAGUCHI, Tomohiro NISHIDA, Ahmed MOUSTAFA, Daichi SHIBATA, Kai YOSHINO, Kentaro HIRAISHI, and Takayuki ITO.
Extraction of online discussion structures for automated facilitation agent. Proceedings of the Annual Conference of JSAI, JSAI2019:2F1E302‒
2F1E302, 2019.

11. Discussions of Node Classification
11
・The F1 of Issue and Positive_Claim is good.
→ Issue is a question form and has a characteristic grammar
→ Positive_Claim is almost the same way of saying
・The F1 of Negative_Claim is relatively good.
→ Negative_Claim is similar in wording
・The F1 of Positive_Premise and Negative_Premise is not good.
→ These opinions are diverse

12. Discussions of Link Prediction
12
・The precision of Negative_Claim is good.
→ The number of issues that are candidates is small
・The precision of Positive_Claim is relatively good.
→ The number of issues that are candidates is small
・The precision of Positive_Premise and Negative_Premise is not
good.
→ There are many candidates for link destinations

13. Structure Extraction in Actual Discussion
13
Do you think that only the citizens of Shizuoka should be included
in the scope of the prefectural vote for the restart of the Hamaoka
nuclear power plant? Or do you think there is another suitable range?
｜
└ Because the Hamaoka nuclear power plant is close to the east-
west transportation infrastructure such as the Tomei Expressway
and the Shinkansen, there is a possibility that it will become a
traffic block in the event of a nuclear accident. I think that not
only the problem of Shizuoka Prefecture, but also it should be
expanded to the local governments along Tokaido.
｜
└ What are your thoughts on this issue? Please post your opinion
with the reason. (by Agent)
｜
└ I agree very much. However, if your wish is to compress the
nuclear fuel that Japan has, to make it into a safe material, to
make effective use of it, and to restart operations related to
research and experiments, please agree and take risks. I would
like to move to.
｜
└ What do you think about this opinion? (by Agent)
｜
└ The area on the east side of Aichi is closer to the Hamaoka
Nuclear Power Plant than the eastern part of Shizuoka,
and I think it should not be targeted to Shizuoka
residents only. I feel that the farther away I am from other
people. If you live in a short distance, the damage in the
event of an accident will be greater, so I would like you
to respect the will of local residents near the nuclear
power plant.
Do you think that only the citizens of Shizuoka should be included in the scope
of the prefectural vote for the restart of the Hamaoka nuclear power plant? Or do
you think there is another suitable range? : Issue
｜
├ I think that not only the problem of Shizuoka Prefecture, but also it should be
｜ expanded to the local governments along Tokaido. : Negative_Claim
｜
├ I agree very much. : Positive_Claim
｜｜
｜└ However, if your wish is to compress the nuclear fuel that Japan has, to
｜ make it into a safe material, to make effective use of it, and to restart
｜ operations related to research and experiments, please agree and take
｜ risks. I would like to move to. : Positive_Premise
｜
└ The area on the east side of Aichi is closer to the Hamaoka Nuclear Power
Plant than the eastern part of Shizuoka, and I think it should not be targeted
to Shizuoka residents only. : Negative_Claim
｜
└ If you live in a short distance, the damage in the event of an accident will
be greater, so I would like you to respect the will of local residents near the
nuclear power plant. : Negative_Premise
｜
└ I feel that the farther away I am from other people. : Negative_Premise
｜
└ Because the Hamaoka nuclear power plant is close to the east-west
transportation infrastructure such as the Tomei Expressway and the
Shinkansen, there is a possibility that it will become a traffic block in
the event of a nuclear accident. : Negative_Premise
Actual discussion Extracted discussion structure

14. Conclusion and Future Work
Goal
・We aim to extract discussion structure by node classification and link
prediction and reduce model size.
Result of node classification
・The F1 of Issue and Positive_Claim is good.
Result of link prediction
・The precision of Negative_Claim is good.
・We could make the model much smaller.
Future work
・We will improve the annotations and collect and create additional data to
improve the accuracy of node classification and link prediction.
14