This document summarizes experiments on bilingual semantic text similarity using various word and sentence embedding approaches. It fine-tunes the COMET machine translation evaluation model on 100,000 parallel English-Hindi sentences and compares the results to other sentence encoders like LaBSE, LASER and SBERT using Pearson correlation. LaBSE and SBERT are found to give the best results. Future work could include fine-tuning models on larger data to potentially achieve better performance.

1. Bilingual Semantic Text Similarity
MT Evaluation
(Bilingual Evaluation)
Submitted By
Binod Kumar Mishra, Pursotam Kumar, Vaibhavi Kailash Kothadi
Mentor by
Ms. Ananya Mukherjee, Dr. Manish Shrivastava
11th Advanced Summer School on NLP
(IASNLP-2022)
IIIT Hyderabad

2. Contents
• Introduction
• Problem Statement
• Data
• Experiments
• Result & Analysis
• Conclusion & Future Work

3. 1. Introduction
• Bilingual Semantic Text similarity [1]
• Measures the equivalence of meanings between two languages.
• With the bilingual distributed representation of words we try to capture semantic meaning.
• It can be divided into three phases, Preprocessing, Similarity Measure, and Score estimation
• It can be used in various NLP applications like Question-Answer, Machine Translation, and Information Extraction.
• Word level embedding and Sentence level embedding
• Word Level Embedding: Word2Vec, Glove, Fasttext, Elmo, BERT [2-8]
• Sentence Level Embedding: SBERT, LaBSE, LASER [9-11]
• Fine Tune COMET machine translation evaluation model[12-13]
• Pearson correlation using Normalized score and other sentence embedding approaches

4. 2. Problem Statement
• Word level embedding vs Sentence level embedding
• Pre-trained COMET model vs Fine-tune COMET model
• Compare various sentence encoders using Pearson Correlation

5. 3. Data
• We have collected dataset from [14] containing 1,00,000 parallel English and translated Hindi sentences to
fine-tune COMET machine translation model .

6. 4. Experiment
Fig 1: Fine-tune COMET machine translation evaluation model

7. 5. Result & Analysis

8. 5. Result & Analysis (Cont.)

9. 5. Result & Analysis (Cont.)

10. 6. Conclusion & Future Work
In this report, we focus on three important things
• Embedding approach
• Fine tune COMET machine translation evaluation model
• Compare Correlation Pearson value between LaBSE, LASER, SBERT, and COMET. And concluded that LaBSE and SBERT give the
best result.
• The future work is that if we Fine-tune models with huge data then it may be possible that we get a better
result.

11. References
1. Shajalal, M. and M. Aono, Semantic textual similarity between sentences using bilingual word semantics. Progress in Artificial Intelligence, 2019. 8(2): p. 263-
272.
2. Mikolov, T., et al., Efficient estimation of word representations in vector space. 2013.
3. Mikolov, T., et al. Distributed representations of words and phrases and their compositionality. in Advances in neural information processing systems. 2013.
4. Pennington, J., R. Socher, and C.D. Manning. Glove: Global vectors for word representation. in Proceedings of the 2014 conference on empirical methods in
natural language processing (EMNLP). 2014.
5. Joulin, A., et al., Fasttext. zip: Compressing text classification models. 2016.
6. Bojanowski, P., et al., Enriching word vectors with subword information. 2017. 5: p. 135-146.
7. Ulčar, M. and M.J.a.p.a. Robnik-Šikonja, High quality ELMo embeddings for seven less-resourced languages. 2019.
8. Devlin, J., et al., Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
9. Reimers, N. and I.J.a.p.a. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks. 2019.
10. Feng, F., et al., Language-agnostic bert sentence embedding. 2020.
11. Artetxe, M. and H.J.T.o.t.A.f.C.L. Schwenk, Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. 2019. 7: p. 597-610.
12. Rei, R., et al., COMET: A neural framework for MT evaluation. 2020.
13. Agirre, E., et al. SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability. 2015. Denver, Colorado: Association for
Computational Linguistics.
14. Kunchukuttan, A., et al., Ai4bharat-indicnlp corpus: Monolingual corpora and word embeddings for indic languages. 2020.

12. THANK YOU