This document discusses quality estimation of machine translation using the QuEst++ framework. It summarizes that QuEst++ can predict the quality of unseen machine translated text using only the source and target texts without references, extracting features to build models that estimate metrics like post-editing effort and time from limited labeled training data. The framework extracts features at the word, sentence and document level from the source and target texts and information from the machine translation system, then trains models using those features to predict quality scores for new translations.

1. Machine Translation
Quality Estimation with
QuEst++
Carolina Scarton – c.scarton@sheffield.ac.uk
University of Sheffield

2. Quality Estimation of Machine Translation

Predict quality of unseen data

3. Quality Estimation of Machine Translation

Predict quality of unseen data

Only few labelled data points for training

4. Quality Estimation of Machine Translation

Predict quality of unseen data

Only few labelled data points for training

Only uses information from source and target → no references!

5. Quality Estimation of Machine Translation

Predict quality of unseen data

Only few labelled data points for training

Only uses information from source and target → no references!

Reduce post-editing and revision time → minimize costs!

Spotting errors

Estimate post-editing effort and time

6. QuEst++

Framework for QE

Word-level, sentence-level and document-level

7. QuEst++

Framework for QE

Word-level, sentence-level and document-level

Feature extraction module →features from source and target

plus information from MT system

8. QuEst++

Framework for QE

Word-level, sentence-level and document-level

Feature extraction module →features from source and target

plus information from MT system

Machine learning module → use the features for building a QE
model

QE model → can predict the quality of unseen data

9. Target documents
(training)
Source documents
(training)
Training scenario – annotated data
Example: post-editing effort (1 to 5) – training data (with annotation)
Source MT Score
Barack Obama becomes the fourth American
president to receive the Nobel Peace Prize
Barack Obama se convierte en el cuarto
presidente estadounidense para recibir el
Premio Nobel de la Paz
4.5
The presidential couple then has a meeting
scheduled with King Harald V and Queen
Sonja of Norway.
La pareja presidencial entonces tiene una
reunión programada con el Rey Harald V y
Reina Sonja de Noruega.
4.0
Although, at the cost of the state falling
deeper into debt – next year the treasury
won't just be 163 billion short, but even more.
Aunque, a costa del estado cayendo más –
el año que viene el Tesoro no sólo se 163
millones de corto, pero aún más.
2.0
Transformer worth tens of millions of crowns
burns in Louny region
La pena de transformadores decenas de
millones de coronas Louny quemaduras en
la región
1.5

10. Target documents
(training)
Source documents
(training)
MT system
Training scenario – annotated data
External resources/tools:
- SRILM
- GIZA++ tables
- TreeTagger
- StanfordParser
Information from the MT
system that translated the
documents (if available)
Feature extractor

11. Training scenario – annotated data
Black-box:
- number of tokens
- number of punctuation
- LM perplexity
- n-gram counts
- POS counts
- syntactic tree
- lexical cohesion
Glass-box:
- n-best list information
Target documents
(training)
Feature extractor
Source documents
(training)
MT system
Features for QE

12. Quality labels

Post-editing effort

Post-editing time

HTER

BLEU

...
QE model training
Training scenario – annotated data
ML algorithms:
- SVC
- SVR
- CRF
Target documents
(training)
Feature extractor
Source documents
(training)
MT system
Features for QE

13. QE model
Training scenario – annotated data
Quality labels

Post-editing effort

Post-editing time

HTER

BLEU

...
QE model training
Target documents
(training)
Feature extractor
Source documents
(training)
MT system
Features for QE

14. Target documents
(unseen data)
Source documents
(unseen data)
Predicting labels for unseen data
Source MT Score
Mass Slaughter on a Personal Level El sacrificio masivo a nivel personal
?
People begin to ask why their leaders are
making them fight.
La gente empiece a preguntar por qué sus
líderes están haciendo ellos lucha. ?
As the community affairs officers moved into
the park in their light-blue windbreakers,
many protesters simply gathered their
belongings and left.
Asuntos como la Comunidad oficiales
movido en el parque en su luz-azul
windbreakers, muchos manifestantes
simplemente se reunieron sus pertenencias
y de la izquierda.
?
Some stories are about honor and bravery. Algunas historias son de honor y valentía.
?
Example: post-editing effort (1 to 5) – unseen data

15. Target documents
(unseen data)
Feature extractor
Source documents
(unseen data)
Predicting labels for unseen data
MT system

16. Predicting labels for unseen data
Features are the
same as the ones
extracted at training
time
Target documents
(unseen data)
Feature extractor
Source documents
(unseen data)
MT system
Features for QE

17. QE model
Predicting labels for unseen data
Can predict labels for
the new data → ML
magic!
Target documents
(unseen data)
Feature extractor
Source documents
(unseen data)
MT system
Features for QE

18. Predicting labels for unseen data
Source MT Score
Mass Slaughter on a Personal Level El sacrificio masivo a nivel personal
4.5
People begin to ask why their leaders are
making them fight.
La gente empiece a preguntar por qué sus
líderes están haciendo ellos lucha. 3.0
As the community affairs officers moved into
the park in their light-blue windbreakers,
many protesters simply gathered their
belongings and left.
Asuntos como la Comunidad oficiales
movido en el parque en su luz-azul
windbreakers, muchos manifestantes
simplemente se reunieron sus pertenencias
y de la izquierda.
1.8
Some stories are about honor and bravery. Algunas historias son de honor y valentía.
4.5
Example: post-editing effort (1 to 5) – unseen data → predictions!
Predicted
scores

19. Thank you!
QuEst++ download: http://www.quest.dcs.shef.ac.uk/