This document summarizes a seminar on unsupervised deep learning in natural language processing given by Amir Hadifar. The seminar covered topics such as the advantages of learned features from deep learning models over manually designed features, recent progress and applications of combining deep learning and NLP, including word representation models like Word2Vec. It also discussed future work, such as GloVe and models that learn multi-word representations. The seminar evaluated word embedding models on similarity tasks and corpora from different languages.

1. ‫طبیعی‬ ‫زبانهای‬ ‫پردازش‬ ‫ماهیانه‬ ‫سمینارهای‬(‫ممتازی‬ ‫دکتر‬ ‫سرکار‬)
‫دهنده‬ ‫ارائه‬:‫هادیفر‬ ‫امیر‬
= [_ _ _ _ _ _]
= [_ _ _ _ _ _]
= [_ _ _ _ _ _]
= [_ _ _ _ _ _]
= [_ _ _ _ _ _]
‫یادگیری‬
‫عمیق‬
‫در‬
‫پردازش‬
‫زبانهای‬‫طبیعی‬

2. ‫مقدمه‬
•Unsupervised (Self taught) Deep Learning (DL) in NLP
•‫باشد‬ ‫داشته‬ ‫بیشتری‬ ‫داده‬ ‫که‬ ‫است‬ ‫برنده‬ ‫کسی‬...
•‫ماشین‬ ‫یادگیری‬ ‫با‬ ‫آن‬ ‫تفاوت‬
40/2

3. ‫مقدمه‬
•‫چراااا؟‬
•‫شده‬ ‫طراحی‬ ‫دستی‬ ‫ویژگیهای‬(manually designed feature)
•‫زمانگیر‬ ،‫بایاس‬ ،‫ناکامل‬
•‫شده‬ ‫یادگیری‬ ‫ویژگیهای‬(learned feature)
•‫آسان‬ ‫تغییر‬ ‫و‬ ‫سریع‬ ‫یادگیری‬
•‫از‬2006‫شدند‬ ‫سنتی‬ ‫روشهای‬ ‫زدن‬ ‫کنار‬ ‫به‬ ‫شروع‬ ‫عمیق‬ ‫یادگیری‬ ‫بر‬ ‫مبتنی‬ ‫روشهای‬
•‫جدیدتر‬ ‫های‬ ‫ایده‬ ‫و‬ ‫الگوریتم‬ ،‫سریعتر‬ ‫ماشینهای‬ ،‫بیشتر‬ ‫های‬ ‫داده‬
•‫کارایی‬(performance)
•‫در‬ ‫ابتدا‬image‫و‬speech‫در‬ ‫سپس‬ ‫و‬NLP
[Socher, 2016]
40/3

4. ‫مقدمه‬
•‫چراااا؟‬
•“The next step for Deep Learning is natural language understanding…”
•Yann LeCun (Facebook AI Director, Professor in university of New-York)
•“I think that the most exciting areas over the next five years will be really
understanding text and videos…”
•Geoffry Hinton (Google researcher, Professor in university of Toronto)
•Yoshua Bengio
•Michael Jordan
[D.Manning, 2015]
40/4

5. ‫مقدمه‬
•‫دیپ‬+‫پی‬ ‫ال‬ ‫ان‬!
•‫ایده‬ ‫ترکیب‬‫های‬NLP‫و‬ ‫با‬DL‫مسائل‬ ‫حل‬ ‫برای‬
•‫در‬ ‫اخیر‬ ‫های‬ ‫پیشرفت‬NLP
•‫سطح‬ ‫در‬:Speech, Morphology, Syntax, Semantic
•‫اپلیکشن‬ ‫در‬:MT, Sentiment analysis and QA
40/5

6. ‫مقدمه‬
•‫سطح‬ ‫در‬Phoneme
•‫واج‬ ‫که‬ ‫میشود‬ ‫داده‬ ‫آموزش‬ ‫نحوی‬ ‫به‬(‫لغت‬ ‫مستقیم‬ ‫طور‬ ‫به‬ ‫یا‬)‫آوایی‬ ‫های‬ ‫ویژگی‬ ‫از‬ ‫را‬(featuressound)
‫پیش‬‫کند‬ ‫بینی‬
[Hank et.al, 2013]
40/6

7. ‫مقدمه‬
•‫سطح‬ ‫در‬Morpheme
•‫هر‬Morpheme‫بردار‬ ‫یک‬ ‫با‬
•‫می‬ ‫ترکیب‬ ‫را‬ ‫بردار‬ ‫دو‬ ‫عصبی‬ ‫شبکه‬‫کند‬
[Thang et.al, 2013]
40/7

8. ‫مقدمه‬
•‫سطح‬ ‫در‬Semantic‫و‬Syntax
[Socher et.al 2011; Bowman et.al 2015]
40/8

9. ‫مقدمه‬
•‫اپلیکشن‬ ‫برخی‬‫ها‬
[research.google.com; Wu et.al, 2016]
40/9

10. ‫ادامه‬ ‫در‬...
•‫مقدمه‬
•NNLM
•Word2vec
•‫بعدی‬ ‫کارهای‬
•‫ارزیابی‬
40/10

11. ‫عصبی‬ ‫شبکه‬
40/11

12. ‫عصبی‬ ‫شبکه‬
40/11

13. 𝑤1[_ _ _ _ _ _ _ ]
𝑤2[_ _ _ _ _ _ _ ]
𝑤3[_ _ _ _ _ _ _ ]
…..
𝑤 𝑚[_ _ _ _ _ _ _]
𝑤1[_ _ _ _ _ _ _ ]
𝑤2[_ _ _ _ _ _ _ ]
𝑤3[_ _ _ _ _ _ _ ]
…..
𝑤 𝑚[_ _ _ _ _ _ _]
40/12

14. 𝑤1[_ _ _ _ _ _ _ ]
𝑤2[_ _ _ _ _ _ _ ]
𝑤3[_ _ _ _ _ _ _ ]
…..
𝑤 𝑚[_ _ _ _ _ _ _]
𝑤1[_ _ _ _ _ _ _ ]
𝑤2[_ _ _ _ _ _ _ ]
𝑤3[_ _ _ _ _ _ _ ]
…..
𝑤 𝑚[_ _ _ _ _ _ _]
40/12

15. ‫زبانی‬ ‫مدل‬
P (Dog | I saw a…) // general
P (Dog| a) // bigram
P (Dog| a) = #count(an Dog) /
# count(an)
‫مشکالت؟؟؟‬
1) The cat is walking in the bedroom.
2) A dog is running in a room.
40/13

16. NNLM
[Bengio et.al 2003]
40/14

17. NNLM
[Bengio et.al 2003]
𝑤1[_ _ _ _ _ _ _ _ _ _ _]
𝑤2[_ _ _ _ _ _ _ _ _ _ _]
𝑤3[_ _ _ _ _ _ _ _ _ _ _]
…..
𝑤 𝑉[_ _ _ _ _ _ _ _ _ _ _]
m = 100
40/14

18. NNLM
[Bengio et.al 2003]
𝑤1[_ _ _ _ _ _ _ _ _ _ _]
𝑤2[_ _ _ _ _ _ _ _ _ _ _]
𝑤3[_ _ _ _ _ _ _ _ _ _ _]
…..
𝑤 𝑉[_ _ _ _ _ _ _ _ _ _ _]
m = 100
Evaluation on Brown
(Perplexity):
n-gram (KN smoothing) : 321
NNLM : 276
n-gram + NNLM : 252
40/14

19. Word Representation
•‫بازنمایی‬(representation)‫واقعی‬ ‫دنیای‬ ‫های‬ ‫اپلیکشن‬ ‫در‬ ‫کارها‬ ‫مهمترین‬ ‫از‬ ‫یکی‬ ‫متن‬
•‫توصیه‬ ،‫جستجو‬ ‫موتورهای‬‫رتبه‬ ،‫فیلترینگ‬ ‫اسپم‬ ،‫گرها‬‫بندی‬...
•‫محلی‬ ‫بازنمایی‬(Local Representation)
•N-grams
•1-of-N coding
•Bag of words
40/15

20. Word Representation
•‫بازنمایی‬(representation)‫واقعی‬ ‫دنیای‬ ‫های‬ ‫اپلیکشن‬ ‫در‬ ‫کارها‬ ‫مهمترین‬ ‫از‬ ‫یکی‬ ‫متن‬
•‫توصیه‬ ،‫جستجو‬ ‫موتورهای‬‫رتبه‬ ،‫فیلترینگ‬ ‫اسپم‬ ،‫گرها‬‫بندی‬...
•‫محلی‬ ‫بازنمایی‬(Local Representation)
•N-grams
•1-of-N coding
•Bag of words
•‫پیوسته‬ ‫بازنمایی‬(Continuous Representation)
•LSA
•LDA
•Distributed Representations
40/15

21. Word2vec
a [0.9 0.2 -0.3 …]
cat [0.1 -0.4 0.5 …]
dog [0.1 -0.4 0.4 …]
…
zoo [0.3 -0.8 0.9 …]
This is cat.
cat is pet.
Human love cats
…
40/16

22. Word2vec
[Mikolov et.al 2013]
40/17

23. Word2vec
[Mikolov et.al 2013]
𝑤1[_ _ _ _ _ _ _ _ _ _ _]
𝑤2[_ _ _ _ _ _ _ _ _ _ _]
𝑤3[_ _ _ _ _ _ _ _ _ _ _]
…..
𝑤 𝑉[_ _ _ _ _ _ _ _ _ _ _]
h = 100
40/17

24. Word2vec
𝑤1[_ _ _ _ _ _ _ _ _ _ _]
𝑤2[_ _ _ _ _ _ _ _ _ _ _]
𝑤3[_ _ _ _ _ _ _ _ _ _ _]
…..
𝑤 𝑉[_ _ _ _ _ _ _ _ _ _ _]
h = 300
40/18

25. fox
brown
brownfox
40/19

26. Word2Vec
[Mikolov et.al 2013b]
40/20

27. Word2Vec
[Mikolov et.al 2013b]
•Softmax-based
•Hierarchical Softmax
40/20

28. Word2Vec
[Mikolov et.al 2013b]
•Softmax-based
•Hierarchical Softmax
40/20

29. Word2Vec
[Mikolov et.al 2013b]
•Softmax-based
•Hierarchical Softmax
•Sampling-based
•Negative Sampling
40/20

30. Distributed Representation
•Doc2vec
•Tweet2vec
•Lda2vec
•Sentence2vec
•BioVec, Gene2Vec,
•Node2vec
•Author2vec
•Paper2vec
•…
[https://github.com/MaxwellRebo/awesome-2vec]
40/21

31. Distributed Representation
40/22

32. 40/23

33. + pregnant
40/24

34. ‫بعدی‬ ‫کارهای‬
[Pennington et.al 2014]
•Glove
•Word2vec version of Stanford
•‫بهتر؟‬ ‫کارایی‬
40/25

35. ‫بعدی‬ ‫کارهای‬
[Le & Mikolov 2014]
40/26

36. ‫بعدی‬ ‫کارهای‬
[Le & Mikolov 2014]
table
tab + abl + ble + table
40/27

37. ‫بعدی‬ ‫کارهای‬
[Levy and Goldberg, 2014]
40/28

38. ‫بعدی‬ ‫کارهای‬
[Levy and Goldberg, 2014]
40/28

39. ‫بعدی‬ ‫کارهای‬
[Levy and Goldberg, 2014]
40/29

40. ‫بعدی‬ ‫کارهای‬
[Levy and Goldberg, 2014]
40/30

41. ‫بعدی‬ ‫کارهای‬
40/31

42. ‫بعدی‬ ‫کارهای‬
[Kiros et.al 2015]
40/32

43. ‫بعدی‬ ‫کارهای‬
[Levy & Goldberg 2014b]
40/33

44. ‫بعدی‬ ‫کارهای‬
[Levy & Goldberg 2014b]
40/33

45. ‫ارزیابی‬
[Mikolov et.al 2013a]
40/34

46. ‫ارزیابی‬
[Mikolov et.al 2013a]
40/35

47. Corpus
Spearman's
correlation
Twitter 0.4499
Wikipedia 0.5347
Hamshahri 0.5444
irBlogs 0.5119
Twitter + Wikipedia 0.5514
Twitter + Hamshahri 0.5602
Wikipedia + Hamshahri 0.5677
Wikipedia + irBlogs 0.5123
Hamshahri + irBlogs 0.5385
Twitter + Wikipedia + Hamshahri 0.5665
Wikipedia + Hamshahri + irBlogs 0.5692
All corpora 0.5463
Word Most similar words in Wikipedia Most similar words in irBlogs
Astin
(‫آستین‬)
powers (‫پاورز‬), Dallas (‫داالس‬), Amarillo
(‫آماریلو‬), Houston (‫هوستون‬)
sleeve (‫استین‬), sleeves (‫استینهای‬), an
sleeve (‫استینی‬), his/her sleeves
(‫آستینهایش‬)
Kelk
(‫کلک‬)
opal (‫نگین‬), spite (‫لج‬), dandelion
(‫قاصدک‬), Chista (‫چیستا‬), hair (‫زلف‬)
and trick (‫وکلک‬), bag (‫جوال‬), knack
(‫حقه‬), deception (‫نیرنگ‬)
Ootoo
(‫اتو‬)
Otto (‫اتو‬), Preminger (‫پرمنیجر‬), Ludwig
(‫لودویگ‬), Rodolph (‫رودولف‬), August
(‫آوگوست‬)
cloth-iron (‫اطو‬), hair-dryer (‫سشوار‬),
a cloth-iron (‫اتوی‬), ironing (‫اتوکشی‬),
spray (‫اسپری‬)
Quebec
(‫کبک‬)
Riviere (‫ریوییر‬), Missisquoi (‫میسیسکوا‬),
Laurentides (‫لورانتید‬), Chaudiere (‫شودییر‬),
Yamaska (‫یاماسکا‬)
Pheasant (‫قرقاول‬), a Perdicinae
(‫کبکی‬), Ammoperdix griseogularis
(‫تیهو‬), Sandgrouse (‫باقرقره‬)
40/36

48. Analogy type /Corpus Twitter Wikipedia Hamshahri irBlogs
Capital common country
30.5%
(83/272)
83.5%
(386/462)
71.6%
(331/462)
40.8%
(98/240)
Capital world
20.4%
(81/398)
66.6%
(1429/2147)
64.9%
(1002/1544)
26.8%
(107/400)
Currency
0.4%
(1/240)
9.5%
(40/420)
13.6%
(57/420)
4.2%
(10/240)
City in state
6.7%
(1/15)
22.5%
(235/1045)
7.5%
(12/160)
0%
(0/6)
Family
49.4%
(89/180)
29.4%
(70/238)
52.4%
(109/208)
66.1%
(119/180)
Gram2 opposite
39%
(82/210)
13.2%
(24/182)
29.2%
(135/462)
34.2%
(82/240)
Gram3 comparative
61%
(255/418)
41.9%
(211/504)
65%
(455/700)
69.4%
(562/810)
Gram4 superlative
63.7%
(172/270)
34.4%
(93/270)
52.4%
(198/378)
47%
(143/304)
Gram6 nationality adjective
83%
(1046/1260)
86.6%
(1351/1560)
86.3%
(1214/1406)
66.9%
(796/1190)
Gram7 past tense
65.1%
(729/1120)
44.9%
(417/928)
40.7%
(456/1121)
56.4%
(711/1260)
Gram8 plural
47.3%
(86/182)
25%
(60/240)
42.6%
(162/380)
45.2%
(209/462)
Accuracy on semantic
23%
(255/1105)
50%
(2160/4312)
54%
(1511/2794)
31.3%
(334/1066)
Accuracy on syntactic
68.4%
(2370/3460)
58.5%
(2156/3684)
58.9%
(2620/4447)
58.6%
(2503/4266)
Overall accuracy
57.5%
(2625/4565)
54%
(4316/7996)
57.1%
(4131/7241)
53.2%
(2837/5332)
Capital: Iran Tehran Iraq Baghdad
Family: She He Aunt Uncle
Currency: Rial Iran America Dollar
40/37

49. ‫آخر‬ ‫نکته‬...
•‫ورد‬2‫کرد‬ ‫استفاده‬ ‫آن‬ ‫از‬ ‫نمیتوان‬ ‫هرجایی‬ ‫ولی‬ ‫بود‬ ‫ساده‬ ‫چون‬ ‫بود‬ ‫موفق‬ ‫وک‬
•‫کنید‬ ‫استفاده‬ ‫بازگشتی‬ ‫های‬ ‫شبکه‬ ‫از‬ ‫جمالت‬ ‫کردن‬ ‫مدل‬ ‫برای‬
•‫نمیکند‬ ‫عمل‬ ‫خوب‬ ‫زیاد‬ ‫بردارها‬ ‫کردن‬ ‫جمع‬
•‫باشید‬ ‫مدل‬ ‫پارامترهای‬ ‫مراقب‬
40/38

50. ‫منابع‬
[1] Y. LeCun, Y. Bengio, and G. Hinton “Deep learning.” Nature, 2015.
[2] R. Socher “Deep learning for Natural Language Processing.”. 2016. Presentation.
[3] C. D. Manning “Last Words.” ACL, 2015.
[4] L. Hank, E. McDermott, and A. Senior. "Large scale deep neural network acoustic modeling
with semi-supervised training data for YouTube video transcription." ASRU, 2013.
[5] L. Thang, R. Socher, and C. D. Manning. "Better word representations with recursive neural
networks for morphology." CoNLL, 2013.
[6] S. Bowman, C. Potts, and C. D. Manning. "Recursive neural networks can learn logical
semantics." arXiv, 2014.
[7] O. Levy, Y. Goldberg, “Neural word embedding as implicit Matrix factorization”, NIPS, 2014
40/39

51. ‫منابع‬
[8] T. Mikolov, et al. "Efficient estimation of word representations in vector space." arXiv, 2013.
[9] T. Mikolov, et al. "Distributed representations of words and phrases and their
compositionality." NIPS, 2013.
[10] J. Pennington, et al. "Glove: Global Vectors for Word Representation." EMNLP, 2014.
[11] R. Kiros, et al. "Skip-thought vectors." NIPS, 2015.
[12] Y. Wu, et.al, “Google ’ s Neural Machine Translation System : Bridging the Gap between
Human and Machine Translation,” arXiv, 2016.
[13] Q. Lee, T.Mikolov, et.al, “Distributed Representations of Sentences and Documents,” arXiv,
2014.
[14] O. Levy, Y. Goldberg “Dependency-based word embeddings" ACL, 2014.
40/40