This document outlines topics related to computational linguistics and neural networks, including: 1) It discusses machine learning concepts like training data, models, and feedback in machine learning. 2) It then covers neural networks, including how artificial neurons work and how they can be used for tasks like binary classification. 3) The document concludes by discussing how neural language models like word2vec represent words as vectors in a semantic space to model relationships between words.

1. Computa(onal
Linguis(cs
Week
5
Neural
Networks
and
Neural
Language
Models
By
Mark
Chang

2. Outlines
• Machine
Learning
• Neural
Networks
• Training
Neural
Networks
• Vector
Space
of
Seman(cs
• Neural
Language
Models
(word2vec)

3. Machine
Learning

4. Machine
Learning
Training
Data
Machine
Learning
Model
Output
Answer
Error
FeedBack
Machine
Learning
Model
Tes(ng
Data
AJer
Training
Output

5. Machine
Learning
Training
Data
X
,
Y
x(i),
y(i)
Model
h
Parameter
w
Output
h(X)
Answer
Y
Cost
Func(on
E(h(X),Y)
Feedback
X
Y

6. Logis(c
Regression

7. Training
Data
X
Y
-­‐0.47241379 0
-­‐0.35344828 0
-­‐0.30148276 0
0.33448276 1
0.35344828 1
0.37241379 1
0.39137931 1
0.41034483 1
0.44931034 1
0.49827586 1
0.51724138 1
…. ….

8. Model
Sigmoid
func(on
h(x) =
1
1 + e (w0+w1x)
w0 + w1x < 0
h(x) ⇡ 0
w0 + w1x > 0
h(x) ⇡ 1

9. Cost
Func(on
• Cross
Entropy
E(h(X), Y ) =
1
m
(
mX
i
y(i)
log(h(x(i)
)) + (1 y(i)
)log(1 h(x(i)
)))
y(i)
= 1
E(h(x(i)
), y(i)
) = log(h(x(i)
))
h(x(i)
) ⇡ 0 ) E(h(x(i)
), y(i)
) ⇡ 1
h(x(i)
) ⇡ 1 ) E(h(x(i)
), y(i)
) ⇡ 0
y(i)
= 0
E(h(x(i)
), y(i)
) = log(1 h(x(i)
))
h(x(i)
) ⇡ 0 ) E(h(x(i)
), y(i)
) ⇡ 0
h(x(i)
) ⇡ 1 ) E(h(x(i)
), y(i)
) ⇡ 1

10. Cost
Func(on
• Cross
Entropy
E(h(X), Y ) =
1
m
(
mX
i
y(i)
log(h(x(i)
)) + (1 y(i)
)log(1 h(x(i)
)))
h(x(i)
) ⇡ 0 and y(i)
= 0 ) E(h(X), Y ) ⇡ 0
h(x(i)
) ⇡ 1 and y(i)
= 1 ) E(h(X), Y ) ⇡ 0
h(x(i)
) ⇡ 0 and y(i)
= 1 ) E(h(X), Y ) ⇡ 1
h(x(i)
) ⇡ 1 and y(i)
= 0 ) E(h(X), Y ) ⇡ 1

11.
w1
w0
Feedback
• Gradient
Descent:
w0 w0–⌘
@E(h(X), Y )
@w0
w1 w1–⌘
@E(h(X), Y )
@w1
(
@E(h(X), Y )
@w0
,
@E(h(X), Y )
@w1
)

12. Feedback

13. Neural
Networks

14. Neurons
&
Ac(on
Poten(al
h`p://humanphisiology.wikispaces.com/file/view/neuron.png/
216460814/neuron.png
h`p://upload.wikimedia.org/wikipedia/commons/thumb/
4/4a/Ac(on_poten(al.svg/1037px-­‐
Ac(on_poten(al.svg.png

15. Synapse
h`p://www.quia.com/files/quia/users/lmcgee/Systems/endocrine-­‐nervous/
synapse.gif

16. Ar(ficial
Neurons
n
W1
W2
x1
x2
b
Wb
y
nin = w1x1 + w2x2 + wb
nout =
1
1 + e nin
nin
nout
y =
1
1 + e (w1x1+w2x2+wb)

17. nout = 1
nout = 0.5
nout = 0(0,0)
x2
x1
Ar(ficial
Neurons
nin = w1x1 + w2x2 + wb
nout =
1
1 + e nin
nin = w1x1 + w2x2 + wb
nout =
1
1 + e nin
w1x1 + w2x2 + wb = 0
w1x1 + w2x2 + wb > 0
w1x1 + w2x2 + wb < 0
1
0

18. Binary
Classifica(on：AND
Gate
x1
x2
y
0
0
0
0
1
0
1
0
0
1
1
1
(0,0)
(0,1)
(1,1)
(1,0)
0
1
n
20
20
b
-­‐30
y
x1
x2
y =
1
1 + e (20x1+20x2 30)
20x1 + 20x2 30 = 0

19. Binary
Classifica(on：OR
Gate
x1
x2
y
0
0
0
0
1
1
1
0
1
1
1
1
y =
1
1 + e (20x1+20x2 10)
(0,0)
(0,1)
(1,1)
(1,0)
0
1
n
20
20
b
-­‐10
y
x1
x2
20x1 + 20x2 10 = 0

20. XOR
Gate
?
(0,0)
(0,1)
(1,1)
(1,0)
0
0
1
x1
x2
y
0
0
0
0
1
1
1
0
1
1
1
0

21. Binary
Classifica(on：XOR
Gate
n
-­‐20
20
b
-­‐10
y
(0,0)
(0,1)
(1,1)
(1,0)
0
1
(0,0)
(0,1)
(1,1)
(1,0)
1
0
(0,0)
(0,1)
(1,1)
(1,0)
0
0
1
n1
20
20
b
-­‐30
x1
x2
n2
20
20
b
-­‐10
x1
x2
x1
x2
n1
n2
y
0
0
0
0
0
0
1
0
1
1
1
0
0
1
1
1
1
1
1
0

22. Neural
Networks
x
y
n11
n12
n21
n22
W12,y
W12,x
b
W11,y
W11,b
W12,b
b
W11,x
W21,11
W22,12
W21,12
W22,11
W21,b
W22,b
z1
z2
Input
Layer
Hidden
Layer
Output
Layer

23. Visual
Pathway
http://www.nature.com/neuro/journal/v8/n8/images/nn0805-975-F1.jpg

24. Training
Neural
Networks

25. Training
Neural
Networks
Training
Data
Neural
Networks
Output
Answer
Ini(aliza(on
Forward
Propaga(on
Error
Func(on
Backward
Propaga(on

26. Ini(aliza(on
• Randomly
sampling
W
from
–N
~
N
x
y
n11
n12
n21
n22
W12,y
W12,x
b
W11,y
W11,b
W12,b
b
W11,x
W21,11
W22,12
W21,12
W22,11
W21,b
W22,b
z1
z2

27. Forward
Propaga(on

28. Error
Func(on
J = (z1log(n21(out)) + (1 z1)log(1 n21(out)))
(z2log(n22(out)) + (1 z2)log(1 n22(out)))
n21
n22
z1
z2
nout ⇡ 0 and z = 0 ) J ⇡ 0
nout ⇡ 1 and z = 1 ) J ⇡ 0
nout ⇡ 0 and z = 1 ) J ⇡ 1
nout ⇡ 1 and z = 0 ) J ⇡ 1

29.
w1
w0
Gradient
Descent
w21,11 w21,11 ⌘
@J
@w21,11
w21,12 w21,12 ⌘
@J
@w21,12
w21,b w21,b ⌘
@J
@w21,b
w22,11 w21,11 ⌘
@J
@w22,11
w22,12 w21,12 ⌘
@J
@w22,12
w22,b w21,b ⌘
@J
@w22,b
w11,x w11,x ⌘
@J
@w11,x
w11,y w11,y ⌘
@J
@w11,y
w11,b w11,b ⌘
@J
@w11,b
w12,x w12,x ⌘
@J
@w12,x
w12,y w12,y ⌘
@J
@w12,y
w12,b w12,b ⌘
@J
@w12,b
(–
@J
@w0
, –
@J
@w1
)

30. Backward
Propaga(on
http://cpmarkchang.logdown.com/posts/277349-neural-network-backward-propagation

31. Vector
Space
of
Seman(cs

32. Distribu(on
Seman(cs
• The
meaning
of
a
word
can
be
inferred
from
its
context.
The meanings of dog
and cat are similar.
The
dog
run.
A
cat
run.
A
dog
sleep.
The
cat
sleep.
A
dog
bark.
The
cat
meows.

33. Seman(c
Vectors
The
dog
run.
A
cat
run.
A
dog
sleep.
The
cat
sleep.
A
dog
bark.
The
cat
meows.
the
a
run
sleep
bark
meow
dog
1
2
2
2
1
0
cat
2
1
2
2
0
1

34. Seman(c
Vectors
dog
(1,
2,...,
xn)
cat
(2,
1,...,
xn)
Car
(0,
0,...,
xn)

35. Cosine
Similarity
• Cosine
Similarity
between
A
&
B
is:
A · B
|A||B|
dog
(a1,a2,...,an)
cat
(b1,b2,...,bn)
Cosine
similarity
between
dog
&
cat
is:
a1b1 + a2b2 + ... + anbn
p
a2
1 + a2
2 + ... + a2
n
p
b2
1 + b2
2 + ... + b2
n

36. Opera(on
of
Vectors
Woman
+
King
-­‐
Man
=
Queen
Woman
Queen
Man
King
King
-­‐
Man
King
-­‐
Man

37. Neural
Language
Models
(word2vec)

38. Dimension
is
too
LARGE
(x1=the,
x2
=a,...,
xn)
dog
Dimension
of
seman(c
vectors
is
equal
to
the
size
of
vocabulary.
x1
x2
x3
x4
xn
...

39. Compressed
Vectors
dog
One-­‐Hot
Encoding
Neural
Network
Compressed
Vector
1.2
0.7
0.5
1
0
0
0

40. One-­‐Hot
Encoding
dog
cat
run
fly
１
０
０
０
０
１
０
０
０
０
１
０
０
０
０
１

41. Ini(alize
Weights
dog
cat
run
fly
dog
cat
run
fly
W =
2
6
6
4
w11 w12 w13
w21 w22 w23
w31 w32 w33
w31 w32 w43
3
7
7
5V =
2
6
6
4
v11 v12 v13
v21 v22 v23
v31 v32 v33
v31 v32 v43
3
7
7
5

42. Compressed
Vectors
１
０
０
０
dog
High
dimension
Low
dimension
v11
v12
v13
v11
v12
v13
v11
v12
v13
１
０
０
０

43. Compressed
Vectors
dog
cat
run
fly
v11
v12
v13
v21
v22
v23
v31
v32
v33
v41
v42
v43
dog
cat
run
fly

44. Context
Word
dog
１
０
０
０
v11
v12
v13
v11
v12
v13
run
０
０
１
０
w31
w32
w33
dog
cat
run
fly
dog
cat
run
fly
1
1 + e V1W3
⇡ 1
V1 · W3 = v11w31 + v12w32 + v13w33

45. Context
Word
cat
１
０
０
０
v11
v12
v13
v21
v22
v23
run
０
０
１
０
w31
w32
w33
dog
cat
run
fly
V2 · W3 = v21w31 + v22w32 + v23w33
dog
cat
run
fly
1
1 + e V2W3
⇡ 1

46. Non-­‐context
Word
dog
１
０
０
０
v11
v12
v13
v11
v12
v13
fly
０
０
１
０
w41
w42
w43
V1 · W4 = v11w41 + v12w42 + v13w43
1
1 + e V1W4
⇡ 0
dog
cat
run
fly
dog
cat
run
fly

47. Non-­‐context
Word
cat
１
０
０
０
v11
v12
v13
v21
v22
v23
w41
w42
w43
０
０
１
０
V2 · W4 = v21w41 + v22w42 + v23w43
dog
cat
run
fly
dog
cat
run
fly
fly
1
1 + e V2W4
⇡ 0

48. Result
dog
cat
run
fly
v11
v12
v13
v21
v22
v23
v31
v32
v33
v41
v42
v43
dog
cat
run
fly

49. Further
Reading
• Logis(c
Regression
3D
– h`p://cpmarkchang.logdown.com/posts/189069-­‐logis(-­‐regression-­‐model
• OverFimng
and
Regulariza(on
– h`p://cpmarkchang.logdown.com/posts/193261-­‐machine-­‐learning-­‐overfimng-­‐and-­‐regulariza(on
• Model
Selec(on
– h`p://cpmarkchang.logdown.com/posts/193914-­‐machine-­‐learning-­‐model-­‐selec(on
• Neural
Network
Back
Propaga(on
– h`p://cpmarkchang.logdown.com/posts/277349-­‐neural-­‐network-­‐backward-­‐propaga(on

50. Further
Reading
• Neural
Probabilis(c
Language
Model：
– h`p://cpmarkchang.logdown.com/posts/255785-­‐neural-­‐
network-­‐neural-­‐probabilis(c-­‐language-­‐model
– h`p://cpmarkchang.logdown.com/posts/276263-­‐-­‐
hierarchical-­‐probabilis(c-­‐neural-­‐networks-­‐neural-­‐network-­‐
language-­‐model
• Word2vec
– h`p://arxiv.org/pdf/1301.3781.pdf
– h`p://papers.nips.cc/paper/5021-­‐distributed-­‐
representa(ons-­‐of-­‐words-­‐and-­‐phrases-­‐and-­‐their-­‐
composi(onality.pdf
– h`p://www-­‐personal.umich.edu/~ronxin/pdf/w2vexp.pdf