The document introduces Cristiano De Nobili, who has a PhD in Statistical Physics and works as a Deep Learning Scientist. It then provides an overview of using very deep convolutional neural networks (VDCNNs) to perform text classification from scratch without any prior knowledge about syntax, semantics, or words. VDCNNs operate directly on character-level representations of text and can learn hierarchical representations through multiple convolutional layers to understand text despite being locally-connected. In conclusion, character-level VDCNNs are able to understand text from only its lowest-level atomic representation without any previous linguistic knowledge.

1. Who am I?
• Theoretical Particle Physicist (UniPd)
• Ph.D. in Statistical Physics (SISSA, Trieste)
• Master in High Performance Computing (ICTP)
• Private Pilot Licence
@denocris Cristiano-De-Nobili
Cristiano De Nobili, Ph.D.
Deep Learning Scientist @ Harman Samsung AI
Machine Learning Instructor @ Deep Leaning Italia
c.denobili@deeplearningitalia.com
✈

2. TEXT UNDERSTANDING FROM SCRATCH
We are going to show you a new and non-standard
way to perform text classification.
[1] Text Understanding from Scratch - X. Zhang, Y. LeCun
[2] Character-level CNNs for Text Classification - X. Zhang, J. Zhao, Y. LeCun
[3] Very Deep CNNs for Text Classification - A. Conneau, H. Schwenk, Y. LeCun, L. Barrault
Intro to DL for Text. Char-level set up. Temporal CNN.
Preliminary results using char-level VDCNNs
Comparison with FastText
Cristiano
Daniele
Elena

3. There is a belief in DL…
Computer
Vision
NLP

4. H[i, j] =
X
k,l
W[i, j, k, l] X[k, l]
From FC to CNNS
(thanks to symmetries)

5. H[i, j] =
X
a,b
K[i, j, a, b] X[i + a, j + b]
From FC to CNNS
(thanks to symmetries)

6. From FC to CNNS
(thanks to symmetries)
H[i, j] =
X
a,b
K[a, b] X[i + a, j + b]Translational Invariance:
H[i, j] =
X
a,b
K[i, j, a, b] X[i + a, j + b]
K[i, j, a, b] = K[a, b]
H[i, j] =
X
a,b=
K[a, b] X[i + a, j + b]
Fully Connected…
Locality:
K[a, b] = 0
(outside some range)

7. From FC to CNNS
(thanks to symmetries)
H[i, j] =
X
a,b
K[a, b] X[i + a, j + b]Translational Invariance:
H[i, j] =
X
a,b
K[i, j, a, b] X[i + a, j + b]
K[i, j, a, b] = K[a, b]
H[i, j] =
X
a,b=
K[a, b] X[i + a, j + b]
Fully Connected…
Locality:
K[a, b] = 0
(outside some range) It’s a convolution!!

8. Recurrent Neural Networks
Siamo fatti di
sorrisi e silenzi,
sorrisi e silenzi.
I primi vincono,
i secondi passano.

9. Recurrent Neural Networks
Siamo fatti di
sorrisi e silenzi,
sorrisi e silenzi.
I primi vincono,
i secondi passano.
short-range and long-range dependencies!

10. Recurrent Neural Networks
“La principale funzione della memoria è dimenticare”
Ciao Giulia, come stai?
Giulia, ti vedo triste. Come mai?

11. TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
TOKENIZATION
yesterday,
night,
we,
were,
…
POS,
LEMMATIZATION,
NAMED ENTITIES,
…
WORD
EMBEDDING
Neural
Networks
STANDARD TEXT PIPELINE

12. TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
TOKENIZATION
yesterday,
night,
we,
were,
…
RNN
(LSTM, GRU,..)
STANDARD TEXT PIPELINE
99%
WORD
EMBEDDING
POS,
LEMMATIZATION,
NAMED ENTITIES,
…

13. TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
TOKENIZATION
yesterday,
night,
we,
were,
…
RNN
(LSTM, GRU,..)
STANDARD TEXT PIPELINE
99%
WORD
EMBEDDING
NON
POS,
LEMMATIZATION,
NAMED ENTITIES,
…

14. TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
TOKENIZATION
yesterday,
night,
we,
were,
…
STANDARD TEXT PIPELINE
WORD
EMBEDDING CNNs
CNNs can be applied to WE without any
knowledge on syntactic or semantic structures
NON
POS,
LEMMATIZATION,
NAMED ENTITIES,
…

15. TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
STANDARD TEXT PIPELINE
WORD
EMBEDDING CNNs
CNNs can be applied to WE without any
knowledge on syntactic or semantic structures
NON
CHAR as TOKEN
y, e, s, t, e, r, d, a, y…
POS,
LEMMATIZATION,
NAMED ENTITIES,
…

16. POS,
LEMMATIZATION,
NAMED ENTITIES,
…
TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
STANDARD TEXT PIPELINE
WORD
EMBEDDING CNNs
CNNs can be applied to WE without any
knowledge on syntactic or semantic structures
NON
CHAR as TOKEN
y, e, s, t, e, r, d, a, y…

17. TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
POS,
LEMMATIZATION,
NAME ENTITIES,
…
STANDARD TEXT PIPELINE
WORD
EMBEDDING CNNs
CNNs can be applied to WE without any
knowledge on syntactic or semantic structures
NON
CHAR as TOKEN
y, e, s, t, e, r, d, a, y…

18. STANDARD TEXT PIPELINENON
TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
VDCNNsCHAR as TOKEN
y, e, s, t, e, r, d, a, y…
VDCNNs not only do not require
any knowledge on syntactic or semantic structures,
but they also not require
any knowledge on words (WE).

19. TEXT UNDERSTANDING FROM SCRATCH
No knowledge of semantics, syntactics
&
No knowledge of words
No previous knowledge is needed!
TEXT
Yesterday
night we were
out at
Terrazza
Martini with
some friends
and…
VDCNNsCHAR as TOKEN
y, e, s, t, e, r, d, a, y…

20. How CNNs work with Text?
h
o
w
c
embedding-dim
1d (or temporal) convolution
2-window kernel
4-window kernel

21. How CNNs work with Text?
output of the 1d convolution

22. How CNNs work with Text?

23. Conclusion
Character-level VDCNNS are able with no previous knowledge to
understand from scratch a given text. This architecture operates
at the lowest atomic representation of text.
Despite convolutional layers are by construction local, deep
CNNs are anyway able to learn high-level hierarchical
representation of a text.
@denocris Cristiano-De-Nobilic.denobili@deeplearningitalia.com

24. @denocris Cristiano-De-Nobili
Thanks!!!
Siamo fatti di
sorrisi e silenzi,
sorrisi e silenzi.
I primi vincono,
i secondi passano.