This document discusses using recurrent neural networks (RNNs) to generate text, specifically novels. It provides an overview of RNNs and how they were trained on small datasets of text to generate new text in the style of the source material. The results showed that after millions of iterations, the RNN-generated text began to resemble the source text in terms of vocabulary, grammar and flow, but was not identical. Other examples discussed include using RNNs trained on Wikipedia text, the Bible, and Obama's speeches.

1. Writing Novels with Deep
Learning
or
LSTMText Generation
Berkay Dinçer

2. Machıne learnıng 101
•Supervised vs Unsupervised
•Traın and test
•Classification vs Regression

3. Perceptron

4. Perceptron

5. Can a perceptron learn?

6. Can a perceptron learn?

7. NEURAL NETWORKS

8. NEURAL NETWORKS
• MNIST – 60k training, 10k testing.
• 28x28 images of handwritten digits
• http://yann.lecun.com/exdb/mnist/

9. NEURAL NETWORKS

10. Recurrent NEURAL NETWORKS
• Ability to use the previous state as input
• Used when the output is somehow related with the previous input

11. How to wrıte novels wıth thıs?
• Represent characters as vectors
• Given a set of vectors guess the next most probable vector
• Convert vector to corresponding character
• Append the character
• Repeat

13. Training
• 1.3 MB of Text (very small dataset)
• Sequence sıze: 100
• 56 distınct characters
• 1.358.037 sequences ın the novel
• 2 days of traınıng ~ 43 hours
• 56-256-256-56 RNN
• Randomly selected seed

14. Fırst hours
city against
those who were cut to pieces. one prisoner there was, he said, who had
been discharged thdaeh en d t etennha u n oak hntna oh nte ontwmin d
adeeohpeyal e lednhg laen ra idn nto te i h u g ae h de an d s ih n he e cv lveen
ttse oe u u ete hge de eke d sea lehts dcu sde t dnune un s eseehe hao7 nnd e
nwwne uo e otr er.ielsa vroih s dne ltn d d heotkte nue tt etep e t iiu hrn lueou
k ehenotnttm u a dcd he t
ie s epshnln ba , ate o e,oe pi saedi g o ne ie eomo qe nteiatpet d dle de f,h
teoddi uenahemern lreo hdl ehl aontehnptye” e mahs e epodtleteten ne e ttelas
n d aatnc est tea oy eoien dn ii ,s weonss n a nhlee oa oh oesme h n h ue c
egainntete d oi er tatne n o ddnleooh e tr enda ettddoiremg d n amwei ont
kewa p ws, te e rhcssr sr e g tdyets m u htnoas o dd tigucse ledopho n et raa
ide u emeoent,kl o tet h o ptd
eod eenh ht t po h e sin mg dyee a s e. dktdnhdett epl , sdhoineho ake

15. After 2.7M ıteratıons
ep it,” said mr. lorry, shaking his head; for he gained in firmness as he saw the
doctor disquieted. the grass were old, my mother at window was towered by the
consciousness with the sight of the door. time to the destination and hours were
here-twents, that the sight from dead of a matter with england.
“what is lifely of danger
of him, i think he understood it. what is to come from the window?”
“nothing.”
seems lasted about a second in the window, and the sense, he dared you eggraven-
-or sagles.

16. Training
• 245 KB of Text (even smaller dataset)
• Sequence sıze: 100
• 66 distınct characters
• 219.569 total sequences
• 66-128-128-66 RNN
• Randomly selected seed

17. vsalam almıyordu bu hazin hali önce
ah, ey zavallı cami, seni böyle görünce
dertli bir çocuk gibi insan
yaşayacaksın gene de aynadan başka kuram,
insanları da memleketim,
muhabber bir hayretle bahsedilir gibi.
halkın oğlum.
beyaz kuşların gözleri üstünde yıldızların sesinin
mutluluk gelip geçti
başlı bir saç başımın elin filâna değecek bu kurşun alava,

18. OTHER RNNS - Train on Wikipedia

19. OTHER RNNS –BIBLE RNN
• 5:11 Therefore encourage one another and build each other up, just as in fact you are doing

20. OTHER RNNS –BIBLE RNN
• 5:11 Therefore encourage one another and build each other up, just as in fact you are doing –
real Bible

21. OTHER RNNS –BIBLE RNN
• 5:11 Therefore encourage one another and build each other up, just as in fact you are doing –
real Bible
• 24:11 Thus saith the LORD of hosts; Ask now this stones are for the righteous and the
children of Israel.

22. OTHER RNNS –BIBLE RNN
• 5:11 Therefore encourage one another and build each other up, just as in fact you are doing –
real Bible
• 24:11 Thus saith the LORD of hosts; Ask now this stones are for the righteous and the
children of Israel – BIBLE-RNN

23. OTHER RNNS –BIBLE RNN
• 5:11 Therefore encourage one another and build each other up, just as in fact you are doing –
real Bible
• 24:11 Thus saith the LORD of hosts; Ask now this stones are for the righteous and the
children of Israel – BIBLE-RNN
• 17:13 And it shall come to pass, that the LORD thy God shall bring thee up out of the land of
Egypt, and thou shalt be clean.

24. OTHER RNNS –BIBLE RNN
• 5:11 Therefore encourage one another and build each other up, just as in fact you are doing –
real Bible
• 24:11 Thus saith the LORD of hosts; Ask now this stones are for the righteous and the
children of Israel – BIBLE-RNN
• 17:13 And it shall come to pass, that the LORD thy God shall bring thee up out of the land of
Egypt, and thou shalt be clean. – BIBLE-RNN

25. OTHER RNNS –BIBLE RNN
• 5:11 Therefore encourage one another and build each other up, just as in fact you are doing –
real Bible
• 24:11 Thus saith the LORD of hosts; Ask now this stones are for the righteous and the
children of Israel – BIBLE-RNN
• 17:13 And it shall come to pass, that the LORD thy God shall bring thee up out of the land of
Egypt, and thou shalt be clean. – BIBLE-RNN
• 18:10 The name of the LORD is a fortified tower; the righteous run to it and are safe.

26. OTHER RNNS –BIBLE RNN
• 5:11 Therefore encourage one another and build each other up, just as in fact you are doing –
real Bible
• 24:11 Thus saith the LORD of hosts; Ask now this stones are for the righteous and the
children of Israel – BIBLE-RNN
• 17:13 And it shall come to pass, that the LORD thy God shall bring thee up out of the land of
Egypt, and thou shalt be clean. – BIBLE-RNN
• 18:10 The name of the LORD is a fortified tower; the righteous run to it and are safe. – REAL
BIBLE

27. OTHER RNNs
• Obama-RNN: https://medium.com/@samim/obama-rnn-machine-
generated-political-speeches-c8abd18a2ea0
• Bible-RNN: https://twitter.com/RNN_Bible
Code
• https://github.com/berkay-dincer/RNN-TextGeneration