This document provides an overview of recurrent neural networks and their applications. It discusses how RNNs can remember previous inputs through feedback loops and internal states. Long short-term memory networks are presented as an improvement over standard RNNs in dealing with long-term dependencies. The document also introduces word embeddings to map words to vectors, and transformers which provide an alternative to RNNs using self-attention. Code examples of RNNs in TensorFlow 2.0 are also shown.

1. Recurrent Neural Networks
via TensorFlow 2.0

2. ORGANIZATION
PLATINUM SPONSORS
Thank you!
COLLABORATORS

5. We all need a friend

6. Woebot
2017

7. ELIZA
1966

8. @ematde
ematallanas@plainconcepts.com
Eduardo Matallanas
AI Lead – Plain Concepts UK
@PabloDoval
Pablo Doval
Data Pontifex – Plain Concepts UK
palvarez@plainconcepts.com

9. Things you will learn
today
• Handling of sequences in Neural Networks
• RNN in Natural Language Processing Tasks
• Basics of TF2
• Show me the code!!
ATTENTION: Some changes ahead!

10. Things you will not learn
today
• What is a…
• Neural network
• Autoencoder
• …
• What is TensorFlow
• How to code in Python
• Star Trek lore

11. How can we interact
with our bots?

12. Language is a process of free creation; its laws and
principles are fixed; but the manner in which the
principles of generation are used is free and
infinitely varied. Even the interpretation and used
of words involves a process of free creation.
Noam Chomsky

13. You're just not thinking fourth dimensionally!

14. Time

15. Dynamic Structures
• Consider time as a variable
• Use energy functions to describe information inside the
network
• First time using own generated information → feedback
loop

16. Recurrent Neural Networks
• Dynamic properties → feedback loops
• Short term memory
• Adaptive behaviour
• Differential equation system
• Applications
• Signal processing
• Temporal series forecasting

17. How can we remember things?
Use an internal state

18. Internal State
Better if we unrolled the structure
Xt
ht
X0
h0
X1
h1
X2
h2
Xt
ht
=

19. X0
h0
hasta
X1
h1
que
X1
h1
se
X1
h1
seque
X1
h1
el
X1
h1
malecón
ThoghtVector
X1
h1
Until
X1
h1
the

20. Far, far ahead in the future
Problem with long term dependencies

21. LSTM to the rescue!
• LSTMs are explicitly designed to avoid the long-term
dependency problem
• Remembering as default behavior.
• How? → 3 in 1 operation
Forget Update Output

22. Talk to the recurrence

23. Does this make sense?
𝑓 𝑋 = 𝑋2 𝑓 "ℎ𝑒𝑦 𝑤ℎ𝑎𝑡′𝑠 𝑢𝑝? " = ?
Of course it doesn’t…

24. From Words To vectors

25. Word embeddings
• Words or phrases are mapped to vectors of real numbers
• Relationships of words
• Their representation is learned by the usage of words

26. It’s alive!!

27. hasta
que
se
seque
el
malecón
0.128 0.291 -0.711 0.889
0.462 -0.012 0.119 0.505
0.664 -0.008 0.386 0.464
-0.662 0.937 -0.335 -0.084
0.911 0.007 0.380 0.109
0.707 -0.111 0.844 0.117

28. V2.0

29. Previously on TFv1.0
• Elements:
• Variables
• Inputs → Placeholders
• Operations
• Execution → Session

30. Some code about it
import tensorflow as tf
b = tf.Variable(tf.zeros((100,)))
W = tf.Variable(tf.random_uniform((784, 100), -1, 1))
x = tf.placeholder(tf.float32, (None, 784))
h_i = tf.nn.relu(tf.matmul(x, W) + b)
sess = tf.Session()
sess.run(tf.initialize_all_variables())
sess.run(h_i, {x: np.random.random(64, 784)})

31. TF 2.0: Eager Execution
import tensorflow as tf
import numpy as np
b = np.zeros(100,)
W = np.random_uniform((784, 100), -1, 1)
x = np.random.random(64, 784)
h_i = tf.nn.relu(tf.matmul(x, W) + b)

32. TF2.0 – They did it!!!
• Eager Execution as default
• No need to add tf.enable_eager_execution()
• Blocks or Functions can be defined to be executed in
graph mode
• Less conventions, more object-oriented and pythonic
design
• Variable_scope removed
• And yes Keras is inside
• Clean up in libraries and contrib

33. Show me
the code

34. So LSTMs are here to stay,
right?
Ermmm…
• Unable to parallelize within training
examples
• Large memory constraints impact the
parallelization across training examples

35. Transformers

36. Transformer
Jun 2017
• Google Research:
• “Attention is all you need” [Arxiv: 1706.03762]
• Vs. RNNs:
• Order-of-magnitude improvement on training time
• Vs. Convolutional Models:
• Complexity grows with distance/lenght on
convolutional models

38. Transformer
Jun 2017
Hasta que se seque el malecón.
Until the beach goes …
dry

39. Self-Attention

41. One more thing …

42. Learning paradigm
Up until now we had translations, but what can we do if we don’t have it
How about to learn new languages?
What???

43. Universal translator

44. Reality
No labelled language
Learn extracting relationships from text
Emmbedding structures are similar
Language Agnostics Models → Yes!!
Cross-lingual embeddings

45. Beyond words…
How about machines that make machines?
Encoder Decoderen en Encoder Decoderes es
Encoder Decoderen es Encoder Decoderes en

47. Huge models
• GPT2 – OpenAI project
• Training on larger datasets: books, webs…
• increases: reading comprehension, translation,
summarization and QA
The bigger they are, the harder they fall

49. Help me!!
Decision Making Support
HHRR and Hiring Processes
Grading Test
Law, Regulation and Compliance
Contract Analysis

50. Help me!!
Harder, Better, Faster, Stronger
Large and Multiple Documents
Multi-hop Reasoning
Contextualized Information in Dialogues

51. The Endgame
Towards a Generalized AI

52. Questions & Answers

53. Thanks and …
See you soon!
Thanks also to the sponsors.
Without whom this would not have been posible.

54. And a very, very
special thanks!