・本資料は下記イベントにおける第三者による論文紹介用資料です。間違い等のご指摘は竹中までご連絡お願いいたします。 ・イベント情報 https://nlpaper-challenge.connpass.com/event/152676/ ACL網羅的サーベイ報告会, 2019/11/02 ・紹介論文 Ethayarajh et al. "Towards Understanding Linear Word Analogies", ACL2019

1. Towards Understanding
Linear Word Analogies
ACL LT
, 2019/11/02,

2. • Towards Understanding Linear Word Analogies
• Kawin Ethayarajh, David Duvenaud, Graeme Hirst
• University of Toronto
• ACL 2019
•
• SGNS GloVe
(1) (2)
csPMI
2

3. • SGNS GloVe
•
⃗king − ⃗man + ⃗woman ≈ ⃗queen
3

4. •
• Latent Variable Model [Arora+,2016]
•
• [Mimno
and Thompson, 2017]
• Paraphrase Model [Gittens+, 2017]
•
• Zipf’s
4
https://ja.wikipedia.org/wiki/

5. •
5

6. • SGNS PMI [Levy & Goldberg 2014]
• SGNS
PMI
• NS
•
• k>0 SGNS negative sample
• GloVe log
⃗x
⃗y c
6
⟨ ⃗x , ⃗y c⟩ = log Xx,y − bx − by
⟨ ⃗x , ⃗y c⟩ = PMI(x, y) − log k

7. Co-occurrence Shifted PMI Theorem
•
• S (x, y) csPMI(x, y)
• S (x, y), (a, b) csPMI(a, x) = csPMI(b, y)
• a, b, x, y
• king - man + woman = queen
• csPMI(king, queen) = csPMI(man, woman)
• csPMI(man, king) = csPMI(woman, queen)
• man, woman, king, queen
7
x a b y
csPMI(x, y) = PMI(x, y) + log p(x, y)

8. • [Mikolov+2013]
•
•
8
https://github.com/tmikolov/word2vec/blob/master/questions-words.txt
capital-world

9. csPMI
9
⃗wc = λ ⃗w
p(w) ∝ p(w, w)
α ∈ ℝ−
γ′, λ ∈ ℝ, α ∈ ℝ−

10. csPMI
•
•
• null csPMI
• csPMI
• ” ” cf:IDF
• : (x,y) =(the, apple) z = “the apple”
• p(the) > p(apple)
• csMPI(“the apple”, “the”) < csPMI(“the apple”, “apple”)
⃗z = ⃗x + ⃗y
∅ ⃗0
csPMI(x, z) = csPMI(∅, y)
⇒ csPMI(x, z) = log p(y) + δ
⃗z ⃗x ⃗y p(y)
p(x) > p(y) ⟺ csPMI(z, y) > csPMI(z, x)
10
δ

11. csPMI
•
• -csPMI
• [Pennungton+’14][Arora+’16]
• (16) (2) 2
11
(csPMI(x, y) ∈ {−∞,0])
p(w|king)
p(w|queen)
≈
p(w|king)
p(w|queen)
(w ∈ Vocab) ⃗king − ⃗man + ⃗woman ≈ ⃗queen

12. •
• csPMI
• csPMI
• csPMI
12

13. csPMI
13

14. csPMI
• -csPMI >0.5
•
•
csPMI
14

15. csPMI
• csPMI
→ csPMI
•
• Fig2
• (x,y)
• OK
• csPMI
• csPMI
cf. Table1 currency
εx,y = Mx,y − < ⃗x , ⃗y c >
15
⟨ ⃗x , ⃗y c⟩ = PMI(x, y) − log k
⟨ ⃗x , ⃗y c⟩ ≈ PMI(x, y) − log k
⟨ ⃗x , ⃗y c⟩ ≠ PMI(x, y) − log k

16. • csPMI
• csPMI=
• csPMI
16