- The document presents a Parallel Iterative Edit (PIE) model for local sequence transduction tasks like grammatical error correction. - The PIE model achieves accuracy competitive with encoder-decoder models by predicting edits instead of tokens, iteratively refining predictions, and factorizing logits over edits and tokens to leverage pre-trained language models. - Experiments show the PIE model provides a 5-15x speed improvement over encoder-decoder models for grammatical error correction while maintaining comparable accuracy.

1. マスター タイトルの書式設定
1
Parallel Iterative Edit Models for
Local Sequence Transduction
A b h i j e e t A w a s t h i , S u n i t a S a r a w a g i , R a s n a G o ya l ,
S a b ya s a c h i G h o s h , V i h a r i P i r a t l a
I n t r o d u c e r : H i r o k i H o m m a , i n K o m a c h i ’s l a b .
J a n u a r y 2 7 t h , 2 0 2 0

2. マスター タイトルの書式設定
2
Abstract
概 要
2

3. マスター タイトルの書式設定
3
• They present a Parallel Iterative Edit (PIE) model
for the problem of local sequence transduction
arising in tasks like Grammatical error correction (GEC).
• not encoder-decoder (ED) model
• It achieves accuracy competitive with the ED model for four
reasons:
1. predicting edits instead of tokens
2. labeling sequences instead of generating sequences
3. iteratively refining predictions to capture dependencies
4. factorizing logits over edits and their token argument to harness pretrained language
models like BERT
• They have experimented with tasks that span GEC, OCR fixes, and spelling corrections.
• The PIE model has proven to be an accurate and very fast alternative to local sequence
transformation.
3
Abstract

4. マスター タイトルの書式設定
4 4
Abstract

5. マスター タイトルの書式設定
5 5
Abstract

6. マスター タイトルの書式設定
6
Method
提 案 手 法
6

7. マスター タイトルの書式設定
7
Overview of the PIE model
7
Method
(C) copy 𝑥𝑖
(A) append a q-gram 𝑤 ∈ Σ 𝑎
(D) delete
(R) replace 𝑥𝑖 with q-gram 𝑤 ∈ Σ 𝑎
(T𝑘) word-inflection

8. マスター タイトルの書式設定
8
The Seq2Edits Function
8
Method

9. マスター タイトルの書式設定
9
• The Parallel Edit Prediction Model
9
Method

10. マスター タイトルの書式設定
10
Experiments
実 験
10

11. マスター タイトルの書式設定
11
Grammatical error correction
settings 1
• train: Lang-8 + NUCLE + FCE corpus
• 1.2 million sentence pairs in English
• validation: CoNLL-13 test set
1. initializing: BERT-LARGE model
2. synthetic training: One-Billion-word corpus (for 2 epochs)
3. fine-tuning: real GEC training corpus (for 2 epochs)
• batch size: 64, learning rate: 2e-5
11
Experiments

12. マスター タイトルの書式設定
12
Artificial Error Generation
12
Experiments

13. マスター タイトルの書式設定
13
Grammatical error correction
settings 2 “edit space”
• copy
• delete
• 1000 appends
• 1000 replaces
• 29 transformations and their inverse
• evaluation
• MaxMatch (M2) scores (F0.5) : CONLL-2014-test
• GLEU+ scores: JEFLEG corpus
13
Experiments
punctuations, articles, pronouns,
prepositions, conjunctions, verbs
add suffix s, d, es, ing, ed
replace suffix s to ing, d to s etc.

14. マスター タイトルの書式設定
14
Suffix transformations
14
Experiments

15. マスター タイトルの書式設定
15
Grammatical error correction
result
15
Experiments
ED
ensemble
decoding

16. マスター タイトルの書式設定
16
Grammatical error correction
Running Time Comparison
16
Experiments
PIE models to be
considerably faster than
ED models

17. マスター タイトルの書式設定
17
Grammatical error correction
Impact of Iterative Refinement
17
Experiments
test set: 1312 sentences
The average number of refinement rounds per
example was 2.7. In contrast, a sequential
model on this dataset would require 23.2 steps.

18. マスター タイトルの書式設定
18
Grammatical error correction
Ablation study on the PIE Architecture
18
Experiments

19. マスター タイトルの書式設定
19
More Sequence Transduction Tasks
• Spell Correction
• Correcting OCR errors
19
Experiments

20. マスター タイトルの書式設定
20
Conclusion
結 論
20

21. マスター タイトルの書式設定
21
• They presented a parallel iterative edit (PIE)
model for local sequence transduction with a
focus on the GEC task.
• Compared to the popular encoder-decoder models that
perform sequential decoding, parallel decoding in the PIE
model yields a factor of 5 to 15 reduction in decoding time.
• The PIE model employs a number of ideas to match the accuracy of
sequential models in spite of parallel decoding:
• it predicts in-place edits using a carefully designed edit space, iteratively refines its own
predictions, and effectively reuses state-of-the-art pre-trained bi-directional language models.
• In the future, they plan to apply the PIE model to more ambitious transduction tasks
like translation.
21
Conclusion