Recurrent Neural Networks (RNNs) can be used to generate text that will look like its original training data. There are many articles out there that show hilarious end-results of such adventures, but start-from-scratch walkthroughs that show the raw code, like this one, are hard to come by. This presentation will demonstrate what you need to join in, grab your own data set, process it, train it, and sample it. Training the data on a CPU can take hours, but in this session you can learn how training on a GPU with hardware acceleration takes only seconds. Come away from this session with your own datasets to “randomly” generate new bodies of text!

1. A Whirlwind Tour of
Recurrent Neural Networks
Sarah Sexton
Microsoft, Chicago
Software Engineer
@Saelia

2. Love building new things...?
@Saelia

3. ...but hate thinking of a new name?
@Saelia

4. Thinking of a video game genre is easy...

5. Run and train RNN
To Run
To pull or run Sarah’s pre-trained Docker snapshot to avoid waiting 8 hours, type:
docker pull saelia/rnn-js
To Train
The way to actually make the RNN generate new Shakespeare text is with the data
sampling script:
th sample.lua -gpu -1 -checkpoint cv/checkpoint_12900.t7 -length 150 -temperature .7
GPU: Setting the flag gpu to -1 tells the code to train using CPU; otherwise it defaults to
GPU 0.
Checkpoints: While the model is training, it will periodically write checkpoint files to
the cv folder. The frequency with which these checkpoints are written is controlled by the
number of iterations, specified with the eval_val_every option. (E.g., if this is 1, then a
checkpoint is written every iteration.)
Length: An important flag is -length. 100 would generate a body of text 100 characters in
length. The default is 2000.
Temperature: An important parameter you may want to play with is -temperature, which
takes a number in range (0 to 1, 0 not included), default = 1. Lower temperature will cause
the model to make more “likely” but more boring and conservative predictions. Higher
temperatures cause the model to take more chances and increase diversity of results, but
at a cost of more mistakes.
@Saelia

6. @Saelia

7. Learn:
• Docker experience
• RNN knowledge
• Great names by AI
@Saelia
X1
X2
Y
A1
A2
A3
A4
B1
B2
B3
B4
Input layer Output layerHidden layers

8. Superheroes Designed by Neural Network
Speet Stank
Red Fart
Mister Man
Rad Food
Sapgirl
Woop
Ann Man
Boomss
Boark II
Supperman
Superbore
Slonk
Lid Man
Green Hooter II
Starm Surper
Shartar
Goons
Nana
Rider Farm
Captain In
Redink
Wolver Man
Wizler
http://aiweirdness.com/post/140829108357/superheroes-designed-by-neural-network @Saelia

9. Quincelax
• Abilities: Sturdy, Secene Grace
• Hidden ability: Tunged Leus
Tortabool
• Ability: Healy Stream
Strangy
• Abilities: Wharmwbra, Darp
• Hidden ability: Magic Guard
Stangute
• Ability: Banger
• Hidden Ability: Drang
Tyrnakine
• Ability: Beak Eye
Minma
• Abilities: Buttery armor, Shell Armor
• Hidden ability: Weak armor
Pokémon Generated by Neural Network
http://aiweirdness.com/post/147834883707/pokemon-generated-by-neural-network @Saelia

10. Recipes at your own risk!
http://aiweirdness.com/post/163878889437/try-these-neural-network-generated-
recipes-at-your

11. Craft beer names by RNN
IPAs
• Dang River
• Yamquak
• Bigly Bomb Session IPA
• Binglezard Flack
• Earth 2 Sanebus
• Tower Of Ergelon
• Juicy Dripple IPA
• Wicked Geee
• Yampy
• Widee Banger Fripper IPA
Strong Pale Ales
• The Great Rebelgion
• Thick Back
• The Fraggerbar
• Dankering
• Third Maus
• Sip’s The Stunks Belgian
• Slambertangeriss
• Devil’s Chard
• Spore Of Gold
• The Oldumbrett’s Ring
• Gunder Of Traz
• Cherry Boof Cornester
• Humple Bobstore Barrel Aged
Amber Ales
• Snarging Red
• Warmel Halce’s Comp Ale
• Fire Pipe
• Blangelfest
• Stoodemfest
• Ole Blood Whisk
• Frog Trail Ale
• Ricias Donkey Brain
• Sacky Rover
• Gate Rooster
• Cramberhand
• O’Brien Irish Red
• River Smush Hoppy Amber Ale
• Rivernillion Amber
• Special North Imperial Red
• Ambre O’Woo’s Omella
Imperial Red Ale
Stouts
• The Moon
• The Bopberry Stout
• Cherry Coconut Mint Chocolate
Stout
• Black Morning
• Sir Coffee
• Shock State
• Take Bean
• Single Horde
• Whata Stout
• Shany Lace
• Barrel Aged Chocolate Milksmoke
• Shump
http://aiweirdness.com/post/163753995072/craft-beer-names-invented-by-neural-network@Saelia

12. Harry Potter and the difference between word-level and character-le
vel RNN
http://aiweirdness.com/post/164291045392/harry-potter-and-the-word-level-recurrent-neural

13. A character-by-character, or “char” model takes one text file as
input, and trains an RNN to predict the next character in a sequence.
The RNN can then be used to generate text character by character
that will look like the original training data.

14. New paint colors invented by neural network
http://aiweirdness.com/post/160985569682/paint-colors-designed-by-neural-network-part-2

15. • Star Trek:
The Next Generation

16. • Doctor Who and
the Daleks!

17. • Baseball players

18. • a fake lightning talk
(generated from
existing TED talks)

19. • The temperature flag makes the most difference. (Expects a number between 0 and 1.)
• Changes the novelty and noise is the system,
• Creates dramatically different output.
• Lower temperatures (e.g. 0.2) makes the RNN more confident, but more conservative
• It generates less noise, but less novel results.
• Using -temperature 0.2 gives clear English, but includes a lot of repeated words.
• Higher temperature makes more interesting/novel output, but more nonsense, misspelled words
• Everything is a trade-off.
• Experiment with all settings.
Temperature

20. Temperature 0.3 and 0.4

21. Temperature 0.5 and 0.6

22. • There are lots of things that affect how well the algorithm does. Temperature adjusts:
• whether the RNN always picks the most likely next character as it’s generating text,
or whether it will go with something farther down the list.
• Setting the temperature higher or lower can make the algorithm produce a much better output.
Temperature 0.7 (my favorite)

23. Deep Learning Virtual Machine on Azure
@Saelia

24. Deep Learning Virtual Machine on Azure
@Saelia

25. Deep Learning Virtual Machine on Azure
@Saelia

26. Deep Learning Virtual Machine on Azure: Price Calculator
@Saelia

27. Deep Learning Virtual Machine on Azu
re

28. • Commands to give you permission to write in Azure VM:
• sudo chown –R username: /dsvm/tools/torch
• sudo chmod –R u+w /dsvm/tools/torch/
• (replace “username” with your own username)
Deep Learning Virtual Machine on Azure
@Saelia

29. Deep Learning Virtual Machine on Azure
@Saelia

30. Create your free account today!
$200 credit
to explore
services for
30 days
12 months Always free
of popular
free
services
25+ services
aka.ms/MCTAzure
@Saelia

31. Complete our survey for a chance to win
a GoPro Hero 6!
aka.ms/MCT18
@Saelia

32. Need Resources? Check out Microsoft Docs!
Home of Microsoft Technical Documentation, API references, code
examples, quickstarts, and tutorials for developers and IT professionals
aka.ms/MCTDocs
.NET ASP.NET
SQL Enterprise Mobility
+Security
Dynamics 365 Azure Bot Service
System Center Microsoft Education
@Saelia

33. @Saelia
Thank you!
Ask me questions on Twitter: @Saelia
Sarah Sexton
Microsoft, Chicago
Software Engineer

34. @Saelia
Sequence Modelling and NLP With Deep Learning (Keras) Video:
https://www.youtube.com/watch?v=ZmCzrPVzDQI
Documentation Resources on RNNs:
https://github.com/jcjohnson/torch-rnn/blob/master/doc/flags.md#training
http://www.jeffreythompson.org/blog/2016/03/25/torch-rnn-mac-install/
https://github.com/jcjohnson/torch-rnn/issues/24
https://github.com/karpathy/char-rnn
https://github.com/crisbal/docker-torch-rnn
https://github.com/Element-Research/rnn
https://github.com/zer0n/deepframeworks/blob/master/README.md