This document discusses challenges in natural language generation (NLG) evaluation and corpus development. It presents referenceless quality estimation as a way to evaluate NLG systems without references that better correlates with human ratings. It also describes how crowdsourcing pictorial meaning representations can elicit more natural language and ensure high quality corpora. Finally, it introduces a new end-to-end NLG challenge corpus in the restaurant domain that provides a large set of training data and evaluation benchmarks.

1. Data-Driven Natural Language
Generation:
Paving the Road to Success
Jekaterina Novikova, Ondřej Dušek, Verena Rieser
NLP Lab, Heriot-Watt University

2. Motivation
Two major problems in NLG:
• The lack of reliable evaluation metrics
– Word-overlap metrics do not correlate with
human ratings
– E2E NLG challenge
• The lack of high-quality in-domain corpora
– Framework for developing high-quality
corpora

3. The Task of NLG
inform(name:The Eagle, type:restaurant, food:British)
The Eagle is a restaurant serving British food.
The Eagle is a restaurant serving British food.
Reference
NLGsystem
output
MR
Comparison
to evaluate
quality
of the output
alignment

4. Evaluation Metrics for NLG
• Up to 60% of NLG research in 2012-2015 relies
on automatic metrics (Gkatzia and Mahamood, 2015)
• Do they correlate with human preferences? No*
*J.Novikova, O.Dušek, A. Cercas-Curry and V. Rieser (to appear): Why We Need New
Evaluation Metrics for NLG. In Proceedings of EMNLP 2017
Automatic evaluationmetrics
Word-overlap (used frequently) Grammar-based (not used frequently)
BLEU, NIST, TER, ROUGE, Lepor,
CIDER, METEOR
Readability (+ cpw, wps, sps, spw,
len, pol, ppw)
Grammaticality (prs, msp)

5. Referenceless Quality Estimation
• NL reference is treated as a gold standard,
correct and complete. These assumptions are
often invalid.
• Referenceless approach*:
– Based on RNN
– Only MR matters
– Increases correlation
*O.Dušek,J.Novikova and V.Rieser(to appear). Referenceless Quality Estimationfor Natural
Language Generation.1st Workshopon Learningto GenerateNaturalLanguage,LGNL2017
Setup Pearson
Ref-less
metric
0.259 –
0.373
BLEU 0.074
METEOR 0.095
ROUGE-L 0.079
CIDER 0.061

6. Corpus Development
Main challenges in crowdsourcing corpora:
• Need of varied and natural language
– Compare textual and pictorial MRs
• Ensure high quality
– Automatic quality control measures
– Evaluate lexical richness and syntactic
complexity

7. Meaning Representation
name [Loch Fyne],
eatType[restaurant],
food[Japanese],
price[cheap],
kid-friendly[yes]
Serving low cost Japanese
style cuisine, Loch
Fyne caters for everyone,
includingfamilies with
small children.
Loch Fyne is a kid-
friendly restaurant
serving cheap
Japanese food.
Road to Success
• Utterances elicited by pictorial MRs are more
natural, more informative and better phrased*
*J.Novikova, O.Lemon, V.Rieser (2016). Crowd-
Sourcing NLG Data: Pictures Elicit Better Data, in
Proceedings of INLG'16

8. New Challenge for NLG
E2E NLG Challenge with the new corpus in the
restaurant domain
http://bit.ly/e2e-nlg
Benefits and challenges*:
• Non-aligned training data
• Size: 50k pairs of MR+ref, multi-sentenced, 8.1
ref/MR, lexically rich, syntactically varied
• Baseline provided
• Generating requires content selection
*J.Novikova, O.Dušekand V.Rieser(2017).The E2E Dataset:New Challenges ForEnd-to-End
Generation.18th AnnualSIGdial Meetingon Discourseand DialogueSIGDIAL 2017

9. Thank you!
Contact me:
Personal site
LinkedIn profile
http://jeknov.tumblr.com
www.linkedin.com/in/jnovikova