By Carolina Scarton. Presentation at the Truth and Trust Online Conference (TTO 2021). Link: https://truthandtrustonline.com/wp-content/uploads/2021/10/TTO2021_paper_31.pdf

1. Conference for Truth and Trust Online 2021
Virtual event, 8th October 2021
Cross-Lingual Rumour Stance Classification:
a First Study with BERT and Machine Translation
Carolina Scarton, Yue Li
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 1

2. Rumour Stance Classification
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297
● Four-class classification problem
(SDQC)
○ What is being said about an online
rumour?
● Stance of replies can help in
predicting veracity → specially
denies (Zubiaga et al., 2016)
2

3. Rumour Stance Classification
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 3

4. Rumour Stance Classification
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 4

5. Rumour Stance Classification
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 5

6. Cross-Lingual Rumour Stance Classification
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 6
➢ Most of previous work focuses on English only
➢ Lack of datasets for other languages
➢ Multilingual/cross-lingual NLP approaches can help

7. Datasets
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 7
➢ English (RumourEval 2017 - Derczynski et al., 2017)
• Training: 4,238 source-reply tweets (8 events)
• Test: 1,049 source-reply tweets (10 events)

8. Datasets
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 8
➢ English (RumourEval 2017 - Derczynski et al., 2017)
• Training: 4,238 source-reply tweets (8 events)
• Test: 1,049 source-reply tweets (10 events)
➢ Danish (DAST - Lillie et al., 2019)
• Training: 2,233 source-reply Reddit pairs (8 events)
• Test: 774 source-reply Reddit pairs (3 events)

9. Datasets
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 9
➢ English (RumourEval 2017 - Derczynski et al., 2017)
• Training: 4,238 source-reply tweets (8 events)
• Test: 1,049 source-reply tweets (10 events)
➢ Danish (DAST - Lillie et al., 2019)
• Training: 2,233 source-reply Reddit pairs (8 events)
• Test: 774 source-reply Reddit pairs (3 events)
➢ Russian (Lozhnikov et al., 2018)
• Training: 691 source-reply pairs (15 topics)
• Test: 267 source-reply pairs (3 topics)

10. Datasets
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 10
➢ English (RumourEval 2017 - Derczynski et al., 2017)
• Training: 4,238 source-reply tweets (8 events)
• Test: 1,049 source-reply tweets (10 events)
➢ Danish (DAST - Lillie et al., 2019)
• Training: 2,233 source-reply Reddit pairs (8 events)
• Test: 774 source-reply Reddit pairs (3 events)
➢ Russian (Lozhnikov et al., 2018)
• Training: 691 source-reply pairs (15 topics)
• Test: 267 source-reply pairs (3 topics)
➢ German (Zubiaga et al., 2016)
• 282 source-reply tweets (3 events)

11. Models + Evaluation
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 11
➢ Monolingual / Zero-shot / Few-shot / Multilingual:
• Multilingual BERT + threshold moving (Li and Scarton, 2020)

12. Models + Evaluation
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 12
➢ Monolingual / Zero-shot / Few-shot / Multilingual:
• Multilingual BERT + threshold moving (Li and Scarton, 2020)
➢ Monolingual:
• Machine Translation (from English into Danish/Russian/German)

13. Models + Evaluation
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 13
➢ Monolingual / Zero-shot / Few-shot / Multilingual:
• Multilingual BERT + threshold moving (Li and Scarton, 2020)
➢ Monolingual:
• Machine Translation (from English into Danish/Russian/German)
➢ Zero-shot:
• Machine Translation (from Danish/Russian/German to English)

14. Models + Evaluation
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 14
➢ Monolingual / Zero-shot / Few-shot / Multilingual:
• Multilingual BERT + threshold moving (Li and Scarton, 2020)
➢ Monolingual:
• Machine Translation (from English into Danish/Russian/German)
➢ Zero-shot:
• Machine Translation (from Danish/Russian/German to English)
➢ Evaluation: macro-F1 / GMR / wF2 (Scarton et al., 2020)

15. Monolingual models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 15
BERT trained with monolingual data English training data into Danish/Russian

16. Monolingual models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 16
BERT trained with monolingual data English training data into Danish/Russian

17. Zero-shot models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 17
BERT trained with monolingual data Danish/Russian/German data translated into English

18. Zero-shot models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 18
BERT trained with monolingual data Danish/Russian/German data translated into English

19. Zero-shot models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 19
BERT trained with monolingual data Danish/Russian/German data translated into English

20. Few-shot models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 20

21. Few-shot models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 21
Monolingual models Zero-shot models

22. Few-shot models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 22
Monolingual models Zero-shot models

23. Few-shot models
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 23
Monolingual models Zero-shot models

24. Multilingual model
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 24
Multilingual Monolingual

25. Multilingual model
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 25
Multilingual Few-shot

26. Multilingual model
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 26
Multilingual Few-shot

27. Multilingual model
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 27
Multilingual Zero-shot

28. Conclusions
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 28
➢ Multilingual BERT approaches can help with the task
➢ Machine-translation-based approaches can achieve comparable results
➢ Although state-of-the-art is not achieved (macro-F1: 0.865(RU);
0.421(DA)) → zero-shot/few-shot learning show promising results
➢ Full multilingual model → improved the results for English and Russian

29. Thank you for your attention!
29
www.weverify.eu
@WeVerify
Disclaimer: This presentation reflects the views only of the author(s), and the Commission cannot be held responsible for any use, which
may be made of the information contained therein.
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297
Carolina Scarton
c.scarton@sheffield.ac.uk
@carolscarton
Yue Li
yli381@sheffield.ac.uk

30. References
The WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297 30
● Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz
Zubiaga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for
rumours. In SemEval-2017, pages 69–76, Vancouver, Canada.
● Yue Li and Carolina Scarton. 2020. Revisiting rumour stance classification: Dealing with imbalanced
data. In 3rd RDSM, pages 38–44, Barcelona, Spain (Online).
● Anders Edelbo Lillie, Emil Refsgaard Middelboe, and Leon Derczynski. 2019. Joint rumour stance and
veracity prediction. In 22nd NoDaLiDa, pages 208–221, Turku, Finland.
● Nikita Lozhnikov, Derczynski Leon, and Mazzara Manuel. 2018. Stance Prediction for Russian: Data
and Analysis. In 6th International Conference in Software Engineering for Defence Applications, pages
176–186, Roma, Italy.
● Carolina Scarton, Diego Silva, and Kalina Bontcheva. 2020b. Measuring what counts: The case of
rumour stance classification. In 1st AACL and the 10th IJCNLP, pages 925–932, Suzhou, China.
● Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016.
Analysing how people orient to and spread rumours in social media by looking at conversational
threads. PLOS ONE, 11(3):1–29.