1) ReVal is a machine translation evaluation metric based on recurrent neural networks that uses an LSTM or Tree-LSTM to learn vector representations of sentences. It then calculates the similarity between the source and translated sentences using KL divergence. 2) The researchers evaluated ReVal on the WMT-13 and WMT-14 translation tasks, and it achieved high correlation with human judgments, outperforming other automatic metrics like BLEU, METEOR, and TERp. 3) ReVal was able to accurately rank translation systems and had a Kendall's Tau score over 0.7 when compared to human rankings, demonstrating it is a simple and effective automatic evaluation metric for machine translation.

1. ReVal: A Simple and Effec2ve Machine Transla2on
Evalua2on Metric Based on Recurrent Neural Networks
2018/01/10 B4 Hiroki Shimanaka
Rohit Gupta, Constan/n Ora ̆san, Josef van Genabith
EMNLP 2015

2. Abstract & Introduction (1)
l
l RNN LSTM
l 2
1

3. Abstract & Introhuction (2)
l
WMT-13 SICK
l
WMTl -14
2

4. 3
lBLEU [Papineni et al., 2002], NIST [Doddington, 2002]
l
lMETEOR [Denkowski and Lavie, 2014], TERp[Snover et al., 2008]
lWordNet
lDISKOTK-PARTY-TUNED [Joty et al., 2014]
l WMT-14 metric shared
task
lLAYERED [Gautam and Bhat- tacharyya, 2014]
l WMT-14 metric shared
task 2
Related Work

5. 4
WMTl -14 metric shared task
l (WMT SICK )
LSTM
Approach

6. 5
l LSTM
l LSTM
Tree-LSTM
l LSTM !" #" $
Tree-LSTM
$
LSTMs and Tree-LSTMs

7. 6
l
ℎ"#$, ℎ&"' ∶ LSTM
̂*+ :
,- ∶
./
∶ [1 2 … 4]
Evaluation Metric (1)

8. 7
! " ∶ KL
$ : 2
Evalua&on Metric (2)

9. 8
l
(RR)
WMTl -13 RR 5 1 5
1
l RR
Computing Similarity Scores from WMT Rankings (1)

10. 9
Computing Similarity Scores from WMT Rankings (2)

11. 10
Experiment (DataSet)
L: WMT-13 4
LNF: WMT-13 4
L+Sick: L Sick 4,500
Sick 500
XL+Sick: L+Sick WMT-13

12. 11
Experiment (Model)
l GloVe [Pen- nington et al., 2014]
l LSTM Tree-LSTM
lLSTM
lBatch size : 25
lTrain rate : 0.05
lregularization strength : 0.0001
lHidden dimension : 50 or 100
lMemory dimension : 150 or 300
lTrain epochs: 10
lSystem-level scores are computed by ag-gregating and normalising segment-
level scores.

13. 12
Result (1)

14. 13
Result (2)

15. 14
Analysis
Systeml -level segment-level Kendall
Tau
l :
01 :
01 (0.10), 04 (0.71), 02 (0.72), 03 (0.73)
02 :
02 (0.71), 01 (0.72), 03 (0.73), 04 (0.74)
:
01 (0.10), 02 (0.71), 03 (0.72), 04 (0.73)
l 01
Kendall Tau 02
02
l

16. 15
Conclusions
lGupta Tree-LSTM
ReVal
lReVal