Adam Bittlingmayer is a technical co-founder at Signal N (http://signaln.com), where he works on sentence-level quality estimation of machine translation. More about our meetup: http://www.meetup.com/de-DE/Berlin-Language-Technology/events/228344365/

1. Predicting
machine translation quality

2. I am @bittlingmayer.
My company is @SignalNLabs
interests: translation quality, translation crowdsourcing,
transliteration, browser translation integrations, topic
classification, automatic source-side correction
previously @Google, @Adobe, @Cerner
Ciao!

3. Today’s topics
◉ Why translation quality?
◉ What is the problem?
◉ Our data model
◉ Our learning infra

4. Quality estimation?
sentence-level quality
good machine translation vs bad
1

5. Quality evaluation?
corpus-level quality given reference translations
machine translation vs human translation
2

6. Why quality?
Why is predicting quality useful?

7. Machine translation
should not be a gamble.

8. $4.50
1M chars by machine
Optimisation Function
$10000
1M chars at 5¢/word by human

9. Perfect Prediction == Perfect Translation
translator
predictor
reward [scores, rankings]
state
action [translations]
Reinforcement Learning

10. What’s the problem?
Is it really harder than self-driving cars?

11. Language is hard.

12. Context.

13. Data are dirty.

14. Bridging.

16. Payoff
What is solvable?
Effort
bad input
50% of errors
context/customisation
like a human
like Search, FB, Maps...source-side ambiguity
ideally interactive
bad output

17. What is quality?
Can we quantify the quality of a translation?

18. Accuracy
What is sentence-level quality?
Fluency
Low Quality
Good Enough
Misleading
Human Quality

19. Recall vs Precision vs Accuracy
actual bad
predicted bad

20. Trivial 90% Accuracy Example
actual bad
predicted bad: 100%

21. How does quality vary?
to English to top languages to other
from English
from top languages
from other

22. How does quality vary?
Wikipedia
news
dialogues, film subtitles, Coursera, Medium
“everyday” reviews, customer service
your children’s WhatsApp messages
my WhatsApp messages

23. Other concepts of quality?

24. How do we solve it?
With data and features

25. What is our data model?
source target score
en-zh Hello 您好 1.0
en-zh The car is driving. The car is driving. 0.0
en-ru The car is driving. Автомобиль вождения. 0.3
... ... ... ...

26. What is our data model?
source target src_length_bytes ... trg_spam_prob score
en-zh Hello 您好 5 ... 0.5 1.0
en-zh The car is driving. The car is driving. 19 ... 0.2 0.0
en-ru The car is driving. Автомобиль вождения. 19 ... 0.1 0.3
... ... ... ... ... ... ...

27. 10-1000 features
signals engineered by us
1000-10M rows
sentences* hand-scored by linguists
language-agnostic
Language is just another feature.

28. Human scores
Evaluate many translations by hand

29. Human Evaluation Score Types
Labels
good/bad
multilabels
word-level labels
Ranking
rank multiple systems
Post-Edit
to comprehensible
to human quality

30. Human Evaluation Score Types
Labels
good/bad
0.0-1.0
multilabels
word-level labels
Ranking
rank multiple systems
Post-Edit
to comprehensible
to human quality
requires smaller dataset and budget

31. $0.001 / row @ 5x redundancy$

32. QuEst baseline features
quest.dcs.shef.ac.uk/quest_files/features_blackbox_baseline_17

33. number of tokens in the source sentence
number of tokens in the target sentence
average source token length
LM probability of source sentence
LM probability of target sentence
number of occurrences of the target word within the target hypothesis (averaged for all words in the hypothesis -
type/token ratio)
average number of translations per source word in the sentence (as given by IBM 1 table thresholded such that prob
(t|s) > 0.2)
average number of translations per source word in the sentence (as given by IBM 1 table thresholded such that prob
(t|s) > 0.01) weighted by the inverse frequency of each word in the source corpus
percentage of unigrams in quartile 1 of frequency (lower frequency words) in a corpus of the source language (SMT
training corpus)
percentage of unigrams in quartile 4 of frequency (higher frequency words) in a corpus of the source language
percentage of bigrams in quartile 1 of frequency of source words in a corpus of the source language
percentage of bigrams in quartile 4 of frequency of source words in a corpus of the source language
percentage of trigrams in quartile 1 of frequency of source words in a corpus of the source language
percentage of trigrams in quartile 4 of frequency of source words in a corpus of the source language
percentage of unigrams in the source sentence seen in a corpus (SMT training corpus)
number of punctuation marks in the source sentence
number of punctuation marks in the target sentence

34. number of tokens
length
LM probability
number of occurrences of the target word within the target hypothesis
average number of translations per source word in the sentence
…
percentage of unigrams in quartile 1 of frequency (lower frequency words)
…
percentage of unigrams in quartile n of frequency (higher frequency words)
…
percentage of trigrams in quartile 1 of frequency of source words
…
percentage of trigrams in quartile n of frequency of source words
number of punctuation marks

35. bad input signals

36. vot tak narod ho4et napisat'
Возможно, вы имели в виду: вот так народ хочет написать

37. human vot tak narod ho4et napisat' vot tak narod ho4et napisat'
search вот так народ хочет написать That's how people want to write
translation Вот так народ хочет написать. So people want to write.

38. bad output signals

39. ambiguity signals

41. translation signals

42. Google Microsoft Wiktionary ...
Merry Christmas Krismasi! Krismasi Njema! heri ya Krismasi
Krismasi njema
...
eat apples kula mapera kula apples ∅ ...

43. lexical signals
sygnały leksykalne

44. char signals
sygnały znaków

45. syntactic signals

46. parse tree to sequence conversion

47. sequence to sequence learning

48. cross-lingual signals

49. outside signals

50. context/customisation
signals

51. Other signals?

52. 50-99+% accuracy
Depends on the benchmark! ;-)
1000-10M rows
10-1000 features

53. Data augmentation?

54. Can we use parallel corpora?
target
Onartutako gertaerak
Aholkuak eta iradokizunak
Etorkizuneko egitasmoei buruz galdetzea
onespena eskatzea
Laguntza eskatzea
Jende galdetzea itxaron
Norbait iritzia eskatzea
Etorkizunari Garrantzia
emanez informazio saihestea
Bad pertsona
…
…
...
Aditu batek ingelesez izatea
Being Lucky
zaharra izatea
pobrea izatea
ari irekietan
aberatsa izatea
Ziur izatea / zenbait
ari kezkaturik
Aspergarria!
Your Mind aldatzeak
Pertsonak txaloak Up
source
받아 들여지는 사실
조언 및 제안
향후 계획에 대해 물어
승인 요청
도움을 요청
사람을 요구하는 대기
누군가의 의견을 물어
미래에 대한 태도
제공 정보 방지
나쁜 사람들
…
…
...
영어 전문가 인
존재 럭키
오래 되
가난
안심되는
부자가되는
확인 인 / 특정
걱정되는
지루한!
당신의 마음을 변경
사람을 응원합니다
score
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
…
…
...
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0

55. What is our learning infra?
H2O.ai deeplearning

56. Do we need
deep learning?

57. Why doesn’t
deep learning
work for
translation?

58. Want to learn more?
The real experts
◉ Dr. Lucia Specia
◉ quest.dcs.shef.ac.uk
◉ statmt.org/wmt15/quality-estimation-task.html
ACL 2016 will be held in Berlin in August.
Reading

59. Any questions ?
You can find me at
◉ @bittlingmayer
◉ adam@signaln.com
Thanks!