The document discusses problems with using maximum likelihood estimation (MLE) for training neural encoder-decoder models for grammatical error correction (GEC). Specifically, MLE focuses on word-level accuracy and the model cannot recover if it predicts an incorrect word. To address this, the author proposes using a neural reinforcement learning (NRL) approach where the model directly optimizes expected reward on the training data rather than word accuracy. The NRL model updates its parameters based on rewards from predicted outputs rather than references. Evaluation shows the NRL model outperforms MLE and other baselines on automated and human evaluations.

1. 2017/12/04
M2 Masahiro Kaneko

2. Problems of MLE
• To train the neural encoder-decoder models, maximum
likelihood estimation (MLE) has been used, where the
objective is to maximize the likelihood of the parameters for
a given training data in Grammatical error correction (GEC)
• The MLE objective is based on word-level accuracy against
the reference, and the model is not exposed to the predicted
output during training
• This becomes problematic, because once the model fails to
predict a correct word, it falls off the right track and does not
come back to it easily

3. Solution
• To address the issues, we employ a neural encoder-decoder
GEC model with a reinforcement learning approach in which
we directly optimize the model toward our final objective
• The objective of the neural reinforcement learning model
(NRL) is to maximize the expected reward on the training
data.
• The model updates the parameters through back-
propagation according to the reward from predicted outputs.

4. Algorithm

5. Maximum Likelihood Estimation (MLE)
• MLE is a standard optimization method for encoder-decoder
models
• Once the model fails to predict a correct word at test time, it
falls off the right track and does not come back to it easily
• Furthermore, in most sentence generation tasks, the MLE
objective does not necessarily correlate with our final
evaluation metrics

6. Neural Reinforcement Learning
• In reinforcement learning, agents aim to maximize expected
rewards by taking actions and updating the policy under a
given state.
• The key difference from MLE is that the reward is not
restricted to token-level accuracy.
S(x): Sampling function

7. Data
• Preprocessing
• We exclude some unreasonable edits (comments by editors, incomplete
sentences such as URLs, etc.) using regular expressions and setting a
maximum token edit distance within 50% of the original length
• We also ignore sentences that are longer than 50 tokens or sentences
where more than 5% of tokens are out-of-vocabulary
• Spelling errors are corrected in preprocessing with Enchant
• Train data: NUCLE 21k, FCE 32k, Lang-8 667k (for MLE and NRL)
• Dev data: JFLEG 754
• Test data: JFLEG 747
• JFLEG has four fluency-oriented references per sentence

8. Hyperparameters
• Both models
• Embedding size: 512
• Hidden size: 1,000
• Gradients are clipped: 1
• beam size: 5
• dropout probability: 0.2
• Vocab size: 35k
• MLE
• Optimizer: ADAM
• NRL
• Optimizer: SGD
• The sentence GLEU score is used as the reward
• Sample size: 20

9. Evaluation
• Automated metric: GLEU
• Human evaluation:
• we run a human evaluation using Amazon Mechanical Turk (MTurk)
• We randomly select 200 sentences each from the dev and test set
• For each sentence, two turkers are repeatedly asked to rank five
systems randomly selected from all eight: the four baseline models,
MLE, NRL, one randomly selected human correction, and the
original sentence
• We infer the evaluation scores by efficiently comparing pairwise
rankings with the TrueSkill algorithm

10. Baselines

11. Result

12. Analysis