Adam Mathias - Learning for sequences

1. learning for sequences
motivation, fundamentals and application
For DataFest Tbilisi 2018

2. language and its structure
applying machine learning
to sequences
goo.gl/RTJnWp
seq2seq lab:
translation + transliteration
Fairseq + Sockeye
deeplanguageclass.github.io/lab/fairseq-transliteration/
deepchar.github.io

3. Know word2vec?

4. Used word2vec?

5. Know seq2seq?

6. Used seq2seq?

7. Know Georgian?

8. Use Georgian?

9. The more interesting the
message,
the less autocorrect helps you.

10. What is language?
Are natural languages like programming

11. Language input/output can be text,
images or audio.
It is has structure, but different than the
structure of programming languages.

12. What is the structure of language?
Sequence of sequences?

13. layers of language
text/image/audio
tokens
morphology
syntax
semantics
dialogue

14. structures of language
Zipfian
token layer
tree-like
syntactic layer
unstructured
discourse layer

15. sequences
fundamentals
What are properties of natural
language text input and output?

16. expressive
You cannot express everything in words.
But there are infinite things that you can express in words.

17. infinite

18. გაკონტრრევოლუციონერებულებისათვისაც

19. This is a sentence about a sentence about a sente
Georgian is not that special.

20. recursive
en.wikipedia.org/wiki/Agglutination
en.wikipedia.org/wiki/Compound_(linguistics)
en.wikipedia.org/wiki/Phrase_structure_grammar

21. 26 x 2 chars
To express most everything the Western world has thought since the Romans.

22. 95 ^ 180
Most of the 9.8...e+355 permutations are invalid.

23. sparsity
www.tensorflow.org/tutorials/representation/word2vec

24. “ “The Pope’s Baby Steps
on Syria”

25. “ “Boy paralyzed after
tumor fights back to
gain black belt”

26. ambiguity
web.stanford.edu/class/cs224n/lectures/lecture1.pdf

27. “ Владимир эту водку
не п…
José es ingrat...

28. long-distance
en.wikipedia.org/wiki/Discontinuity_(linguistics)

29. “ More people have been
to Georgia than I have.

30. “ More people have been
to Georgia than I have.

31. strict
languagelog.ldc.upenn.edu/nll/?p=39477

32. “ “Nice truck attack video”

33. “ “No forgetto!”

34. evolution

35. “ Tehre was a sutdy by
Mcirosoft yares ago...

36. “ Ca n yo u rea d th is?

37. “ Ca n yo u rea d th is?
M@ch1nes C@n’t.

38. “ ուատ աբաուտ զիս...

39. “ ուատ աբաուտ զիս...
ор зис...

40. “ ուատ աբաուտ զիս...
ор зис...
ბატ ნოტ ზის

41. noisy + dirty
www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/
docs.microsoft.com/en-us/typography/develop/word-recognition

42. “ wreck a nice beach

43. “ wreck a nice beach
recognize speech

44. arbitrary
en.wikipedia.org/wiki/Double_articulation

45. “ The car crashed because
it...

46. “ The car crashed because
it...

47. “ The car crashed because
it had old tyres.

48. “ The car crashed because
it had old tyres.
it == car

49. “ The car crashed because
it was rainy.

50. “ The car crashed because
it was rainy.
it == ?

51. “ The car crashed because
it was rainy.
it == ?

52. hard
en.wikipedia.org/wiki/Winograd_Schema_Challenge

53. Humans are good at this.

54. Humans are good at this.
Even baby humans are good at this.

55. representation
s
for text sequences
How can we represent text
sequences numerically?

56. word representations
What are the properties? What are the limitations?

57. “ He was hit by a bat.
Where are the fans?

58. “ bridge to new york
new bridge to york

59. word representations
sub-word
character-level n-grams
supra-word
word-level n-grams

60. applications
text input/output
Which tasks can we frame as
sequence tasks?

61. sequence input
unsupervised word representations,
text classification...

62. sequence output
article generation (e.g. financial
news)

63. sequence input + output
translation, transliteration, spelling +
grammar correction, dialogue
systems (e.g. Q&A, "chat bots"),
summarisation, style transfer

64. mixed medium
image captioning
(e.g. CLEVR, @picdescbot)

65. beyond language
DNA sequences

66. evolution of machine translation
How did statistical machine translation work?

67. 5 minute break
coffee + questions
next: machine translation

68. evolution of machine translation
How did statistical machine translation work?
2000-2015

69. evolution of machine translation
How do we scale statistical machine translation?

70. evolution of machine translation
How do we scale statistical machine translation?
~10

71. evolution of machine translation
How do we scale statistical machine translation?
(~10) * 100+^2 > 100000

72. evolution of machine translation
Can we do it end-to-end?

73. evolution of machine translation
Can we do it end-to-end?
deep learning

74. seq2seq

75. INPUT : OUTPUT
NUMBERS AUDIO TEXT TEXT+
NUMBERS regression
LABEL classification
TEXT sequence

76. For which tasks can we use seq2seq?

77. seq2seq tasks + applications

78. How do we take as a sequence as input?

79. How do we generate a sequence?

80. Generating a sequence
ENCODER + DECODER

81. github.com/google/seq2seq

82. To generate a
sequence
Other ways?

83. What if tokens are not 1:1?

84. How do we represent each token context?

85. Token context
ATTENTION

87. What if a token depends on what comes after it?

88. Token context
BI-DIRECTIONAL

90. seq2seq lab
The tasks of translation and transliteration tasks contain essentially all the challenges
of natural language understanding (NLU) -- ambiguity, agreement, evolution, mixed
language... -- which can require full AI.

91. approach
1
Find a dataset or generate the dataset
from a monolingual corpus
WMT, Wikipedia
2
Find a seq2seq implementation that is
easy to use and productive-strength
Fairseq from Facebook AI Research
3
Convert from word-level to char-level
4
Choose parameters, train, test and iterate

92. So what can I pip install?

93. FAIRSEQ on PyTorch from FAIR
SOCKEYE on MXNet from AWS Labs
...

94. How many hours will it take to train?

95. on my machine
YOU DON’T WANT TO KNOW.

96. deeplanguageclass.github.io/lab/fairseq-transliteration/
deepchar.github.io

97. Why translit?

98. Why translit?
scale and value insight interpretability CPUablility

99. Transliteration
didi madloba
genacvale
vai vai
vay vay
xachapuri
khachapuri
tbiliso
...
დიდი მადლობა
გენაცვალე
ვაი-ვაი
ვაი-ვაი
ხაჭაპური
ხაჭაპური
თბილისო
…

100. Data generation
დიდი მადლობა
გენაცვალე
ვაი-ვაი
ხაჭაპური
თბილისო
…

101. Data generation
დიდი მადლობა
გენაცვალე
ვაი-ვაი
ხაჭაპური
თბილისო
…
didi madloba
genacvale
vai vai
xachapuri
tbiliso
...

102. Transfer learning
didi madloba
genacvale
vai vai
xachapuri
tbiliso
...
დიდი მადლობა
გენაცვალე
ვაი-ვაი
ხაჭაპური
თბილისო
…

103. Transfer learning
didi madloba
genacvale
vai vai
xachapuri
tbiliso
...
vindousit
youtube.com
DataFesti
...
დიდი მადლობა
გენაცვალე
ვაი-ვაი
ხაჭაპური
თბილისო
…
ვინდოუსით
youtube.com
დატაფესტზე
…

104. Transfer learning
didi madloba
genacvale
vai vai
xachapuri
tbiliso
...
vindousit
me amas youtube.com-ze vuqure
DataFesti
...
Thank you!
This is not a test.
History of Post-Hardcore Dubstep
დიდი მადლობა
გენაცვალე
ვაი-ვაი
ხაჭაპური
თბილისო
…
ვინდოუსით
მე ამას youtube.com-ზე ვუყურე.
დატაფესტზე
…
Thank you!
This is not a test.
History of Post-Hardcore Dubstep

105. Can we auto-generate it?

106. Transfer learning
didi madloba
…
...
Thank you!
This is not a test.
History of Post-Hardcore Dubstep
...
中國正在接管
…
დიდი მადლობა
…
…
Thank you!
This is not a test.
History of Post-Hardcore Dubstep
...
中國正在接管
...

107. Transfer learning
didi madloba
…
...
Thank you!
This is not a test.
History of Post-Hardcore Dubstep
...
中國正在接管
...
Eto escho nichego!
…
დიდი მადლობა
…
…
Thank you!
This is not a test.
History of Post-Hardcore Dubstep
...
中國正在接管
…
Это еще ничего!
...

108. Transfer learning
didi madloba
…
...
Thank you!
This is not a test.
History of Post-Hardcore Dubstep
...
中國正在接管
...
Eto escho nichego!
…
диди мадлоба
генацвале
დიდი მადლობა
…
…
Thank you!
This is not a test.
History of Post-Hardcore Dubstep
...
中國正在接管
…
Это еще ничего!
...
დიდი მადლობა
გენაცვალე

109. Transfer learning
wo zai youtube.com shang kanguo ta.

110. What is our result?

111. Transfer learning
wo zai youtube.com shang kanguo ta. 我在youtube.com上看過它。

112. Transfer learning
Robust with less data
More deployable

113. evaluation
How do we evaluate
sequence output?

114. “ This is not a test. =>
Ეს არ არის ტესტია.
ეს არ არის გამოცდა.
ეს არ გამოცდა.
ეს არ არის ტესტი

115. BLEU
en.wikipedia.org/wiki/BLEU

116. SQuAD
rajpurkar.github.io/SQuAD-explorer/

117. evaluation
Has neural machine translation
solved everything?

118. evaluation
Has neural machine translation
solved everything?
aiindex.org

119. “
How are you?

120. “ Mr. Smith!
How are you?

121. “
ის არის საუკეთესო
მოტოციკლიტი
თბილისში.

122. “ გაიცანით დედაჩემი:
ის არის საუკეთესო
მოტოციკლიტი
თბილისში.

123. ambiguity
web.stanford.edu/class/cs224n/lectures/lecture1.pdf

124. looking forward
Translation is framed only as a
line-by-line problem.

125. noisy + dirty
www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/
docs.microsoft.com/en-us/typography/develop/word-recognition

126. evolution

127. looking forward
A model can be only as good as
the data.

128. strict
http://languagelog.ldc.upenn.edu/nll/?p=39477

129. long-distance
dependencies

130. looking forward
Generating sequences is risky.

131. The more interesting the
message,
the less autocorrect helps you.

132. looking forward
Do we know when we’re wrong?

133. looking forward
Do we know when we’re wrong?
@modelfront

134. slides: goo.gl/RTJnWp
Questions?
reddit.com/r/machinetranslation
github.com/deeplanguageclass issues
github.com/deepchar issues
@bittlingmayer | adam@bittlingmayer.org