In this invited talk, presented at the Dialogue'2018 conference, I argue for the usefulness of graph representations for NLP in the deep learning era. In the lecture, it is described how to extract symbolic linguistic structures, such as word senses and semantic frames in an unsupervised way from text corpora using graph-based algorithms and distributional semantics.

1. Alexander Panchenko
From unsupervised induction of
linguistic structures from text
towards applications in deep
learning

2. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 2/98
In close collaboration with …

3. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 3/98
Andrei Kutuzov
Eugen Ruppert
Fide Marten
Nikolay Arefyev
Steffen Remus
Martin Riedl
Hubert Naets
Maria Pelevina
Anastasiya Lopukhina
Konstantin Lopukhin
In collaboration with …

4. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 4/98
Overview

5. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98
Inducing word sense representations:
word sense embeddings via retrofitting
[Pelevina et al., 2016, Remus & Biemann, 2018];
inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a,
Ustalov et al., 2018b]
inducing semantic classes [Panchenko et al., 2018b]
Overview
Overview

6. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98
Inducing word sense representations:
word sense embeddings via retrofitting
[Pelevina et al., 2016, Remus & Biemann, 2018];
inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a,
Ustalov et al., 2018b]
inducing semantic classes [Panchenko et al., 2018b]
Making induced senses interpretable
[Panchenko et al., 2017b, Panchenko et al., 2017c]
Overview
Overview

7. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 5/98
Inducing word sense representations:
word sense embeddings via retrofitting
[Pelevina et al., 2016, Remus & Biemann, 2018];
inducing synsets [Ustalov et al., 2017b, Ustalov et al., 2017a,
Ustalov et al., 2018b]
inducing semantic classes [Panchenko et al., 2018b]
Making induced senses interpretable
[Panchenko et al., 2017b, Panchenko et al., 2017c]
Linking induced word senses to lexical
resources [Panchenko, 2016, Faralli et al., 2016,
Panchenko et al., 2017a, Biemann et al., 2018]
Overview
Overview

8. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98
A shared task on word sense induction
[Panchenko et al., 2018a, Arefyev et al., 2018]
Overview
Overview

9. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98
A shared task on word sense induction
[Panchenko et al., 2018a, Arefyev et al., 2018]
Inducing semantic frames [Ustalov et al., 2018a]
Inducing FrameNet-like structures;
…using multi-way clustering.
Overview
Overview

10. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 6/98
A shared task on word sense induction
[Panchenko et al., 2018a, Arefyev et al., 2018]
Inducing semantic frames [Ustalov et al., 2018a]
Inducing FrameNet-like structures;
…using multi-way clustering.
Learning graph/network embeddings [ongoing joint work
with Andrei Kutuzov and Chris Biemann]
How to represent induced networks/graphs?
… so that they can be used in deep learning architectures.
…effectively and efficiently.
Overview
Overview

11. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 7/98
Inducing word sense representations

12. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 8/98
Inducing word sense representations
Word vs sense embeddings

13. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 9/98
Inducing word sense representations
Word vs sense embeddings

14. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 10/98
Inducing word sense representations
Related work

15. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 11/98
AutoExtend [Rothe & Schütze, 2015]
* image is reproduced from the original paper
Inducing word sense representations
Related work: knowledge-based

16. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98
Adagram [Bartunov et al., 2016]
Multiple vector representations θ for each word:
Inducing word sense representations
Related work: knowledge-free

17. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98
Adagram [Bartunov et al., 2016]
Multiple vector representations θ for each word:
p(Y, Z, β|X, α, θ) =
V∏
w=1
∞∏
k=1
p(βwk|α)
N∏
i=1
[p(zi|xi, β)
C∏
j=1
p(yij|zi, xi, θ
zi – a hidden variable: a sense index of word xi in context C;
α – a meta-parameter controlling number of senses.
Inducing word sense representations
Related work: knowledge-free

18. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 12/98
Adagram [Bartunov et al., 2016]
Multiple vector representations θ for each word:
p(Y, Z, β|X, α, θ) =
V∏
w=1
∞∏
k=1
p(βwk|α)
N∏
i=1
[p(zi|xi, β)
C∏
j=1
p(yij|zi, xi, θ
zi – a hidden variable: a sense index of word xi in context C;
α – a meta-parameter controlling number of senses.
See also: [Neelakantan et al., 2014] and [Li and Jurafsky, 2015]
Inducing word sense representations
Related work: knowledge-free

19. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 13/98
Word sense induction (WSI) based on graph clustering:
[Lin, 1998]
[Pantel and Lin, 2002]
[Widdows and Dorow, 2002]
Chinese Whispers [Biemann, 2006]
[Hope and Keller, 2013]
Inducing word sense representations
Related work: word sense induction

20. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 14/98
* source of the image: http://ic.pics.livejournal.com/blagin_anton/33716210/2701748/2701748_800.jpg
Inducing word sense representations
Related work: Chinese Whispers#1

21. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 15/98
Inducing word sense representations
Related work: Chinese Whispers#2

22. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 16/98
Inducing word sense representations
Related work: Chinese Whispers#2

23. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 17/98
Inducing word sense representations
Related work: Chinese Whispers#2

24. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 18/98
RepL4NLP@ACL’16 [Pelevina et al., 2016], LREC’18 [Remus & Biemann, 2018]
Prior methods:
Induce inventory by clustering of word instances
Use existing sense inventories
Our method:
Input: word embeddings
Output: word sense embeddings
Word sense induction by clustering of word ego-networks
Inducing word sense representations
Sense embeddings using retrofitting

25. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 19/98
From word embeddings to sense embeddings
Calculate Word
Similarity Graph
Learning Word Vectors
Word Sense Induction
Text Corpus
Word Vectors
Word Similarity Graph
Pooling of Word Vectors
Sense Inventory
Sense Vectors
1 2
4 3
Inducing word sense representations
Sense embeddings using retrofitting

26. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 20/98
Word sense induction using ego-network clustering
Inducing word sense representations
Sense embeddings using retrofitting

27. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/98
Neighbours of Word and Sense Vectors
Vector Nearest Neighbors
table tray, bottom, diagram, bucket, brackets, stack, basket,
list, parenthesis, cup, saucer, pile, playfield, bracket,
pot, drop-down, cue, plate
Inducing word sense representations
Sense embeddings using retrofitting

28. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/98
Neighbours of Word and Sense Vectors
Vector Nearest Neighbors
table tray, bottom, diagram, bucket, brackets, stack, basket,
list, parenthesis, cup, saucer, pile, playfield, bracket,
pot, drop-down, cue, plate
table#0 leftmost#0, column#1, tableau#1, indent#1, bracket#3,
pointer#0, footer#1, cursor#1, diagram#0, grid#0
table#1 pile#1, stool#1, tray#0, basket#0, bowl#1, bucket#0,
box#0, cage#0, saucer#3, mirror#1, pan#1, lid#0
Inducing word sense representations
Sense embeddings using retrofitting

29. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 22/98
Word and sense embeddings
of words iron and vitamin.
LREC’18 [Remus & Biemann, 2018]
Inducing word sense representations
Sense embeddings using retrofitting

30. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 23/98
Word Sense Disambiguation
1 Context extraction: use context words around the target
word
2 Context filtering: based on context word’s relevance for
disambiguation
3 Sense choice in context: maximise similarity between a
context vector and a sense vector
Inducing word sense representations
Sense embeddings using retrofitting

31. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 24/98
Inducing word sense representations
Sense embeddings using retrofitting

32. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 25/98
Inducing word sense representations
Sense embeddings using retrofitting

33. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 26/98
Inducing word sense representations
Sense embeddings using retrofitting

34. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 27/98
Inducing word sense representations
Sense embeddings using retrofitting

35. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 28/98
Unsupervised WSD SemEval’13, ReprL4NLP [Pelevina et al., 2016]:
Model Jacc. Tau WNDCG F.NMI F.B-Cubed
AI-KU (add1000) 0.176 0.609 0.205 0.033 0.317
AI-KU 0.176 0.619 0.393 0.066 0.382
AI-KU (remove5-add1000) 0.228 0.654 0.330 0.040 0.463
Unimelb (5p) 0.198 0.623 0.374 0.056 0.475
Unimelb (50k) 0.198 0.633 0.384 0.060 0.494
UoS (#WN senses) 0.171 0.600 0.298 0.046 0.186
UoS (top-3) 0.220 0.637 0.370 0.044 0.451
La Sapienza (1) 0.131 0.544 0.332 – –
La Sapienza (2) 0.131 0.535 0.394 – –
AdaGram, α = 0.05, 100 dim 0.274 0.644 0.318 0.058 0.470
w2v 0.197 0.615 0.291 0.011 0.615
w2v (nouns) 0.179 0.626 0.304 0.011 0.623
JBT 0.205 0.624 0.291 0.017 0.598
JBT (nouns) 0.198 0.643 0.310 0.031 0.595
TWSI (nouns) 0.215 0.651 0.318 0.030 0.573
Inducing word sense representations
Sense embeddings using retrofitting

36. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 29/98
Semantic relatedness, LREC’2018 [Remus & Biemann, 2018]:
autoextend
adagram
SGNS
.
glove
.
sympat
.
LSAbow
.
LSAhal
.
paragramSL
.
SimLex999 0.45 0.29 0.44 0.37 0.54 0.30 0.27 0.68
MEN 0.72 0.67 0.77 0.73 0.53 0.67 0.71 0.77
SimVerb 0.43 0.27 0.36 0.23 0.37 0.15 0.19 0.53
WordSim353 0.58 0.61 0.70 0.61 0.47 0.67 0.59 0.72
SimLex999-N 0.44 0.33 0.45 0.39 0.48 0.32 0.34 0.68
MEN-N 0.72 0.68 0.77 ____ 0.76 ____ 0.57 ____ 0.71 ____ 0.73 ____ 0.78 ____
Inducing word sense representations
Sense embeddings using retrofitting

37. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 30/98
Unsupervised WSD SemEval’13, ReprL4NLP [Pelevina et al., 2016]:
comparable to SOTA, incl. sense embeddings.
Semantic relatedness, LREC’2018 [Remus & Biemann, 2018]:
autoextend
adagram
SGNS
SGNS+senses
glove
glove+senses
sympat
sympat+senses
LSAbow
LSAbow+senses
LSAhal
LSAhal+senses
paragramSL
paragramSL+senses
SimLex999 0.45 0.29 0.44 0.46 0.37 0.41 0.54 0.55 0.30 0.39 0.27 0.38 0.68 0.64
MEN 0.72 0.67 0.77 0.78 0.73 0.77 0.53 0.68 0.67 0.70 0.71 0.74 0.77 0.80
SimVerb 0.43 0.27 0.36 0.39 0.23 0.30 0.37 0.45 0.15 0.22 0.19 0.28 0.53 0.53
WordSim353 0.58 0.61 0.70 0.69 0.61 0.65 0.47 0.62 0.67 0.66 0.59 0.63 0.72 0.73
SimLex999-N 0.44 0.33 0.45 0.50 0.39 0.47 0.48 0.55 0.32 0.46 0.34 0.44 0.68 0.66
MEN-N 0.72 0.68 0.77 0.79 0.76 0.80 0.57 0.74 0.71 0.73 0.73 0.76 0.78 0.81
Inducing word sense representations
Sense embeddings using retrofitting

38. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 31/98
ACL’17 [Ustalov et al., 2017b]
Examples of extracted synsets:
Size Synset
2 {decimal point, dot}
3 {gullet, throat, food pipe}
4 {microwave meal, ready meal, TV dinner, frozen dinner}
5 {objective case, accusative case, oblique case, object
case, accusative}
6 {radiotheater, dramatizedaudiobook, audiotheater, ra-
dio play, radio drama, audio play}
Inducing word sense representations
Synset induction

39. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 32/98
Outline of the ’Watset’ method:
Background Corpus
Synonymy Dictionary
Learning
Word
Embeddings
Graph
Construction
Synsets
Word
Similarities
Ambiguous
Weighted Graph
Local
Clustering:
Word Sense Induction
Global
Clustering:
Synset Induction
Sense Inventory
Disambiguation
of
Neighbors
Disambiguated
Weighted Graph
Local-Global
Fuzzy
Graph
Clustering
Inducing word sense representations
Synset induction

40. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 33/98
Inducing word sense representations
Synset induction

41. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 34/98
Inducing word sense representations
Synset induction

42. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 35/98
Inducing word sense representations
Synset induction

43. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 36/98
CW MCL MaxMax ECO CPM Watset
0.0
0.1
0.2
0.3
WordNet (English)
F−score
CW MCL MaxMax ECO CPM Watset
0.0
0.1
0.2
0.3
BabelNet (English)
F−score
Inducing word sense representations
Synset induction

44. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 36/98
CW MCL MaxMax ECO CPM Watset
0.0
0.1
0.2
0.3
WordNet (English)
F−score
CW MCL MaxMax ECO CPM Watset
0.0
0.1
0.2
0.3
BabelNet (English)
F−score
CW MCL MaxMax ECO CPM Watset
0.00
0.05
0.10
0.15
0.20
RuWordNet (Russian)
F−score
CW MCL MaxMax ECO CPM Watset
0.0
0.1
0.2
0.3
0.4
YARN (Russian)
F−score
Inducing word sense representations
Synset induction

45. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 37/98
Word Sense Local Sense Cluster: Related Senses Hypernyms
mango#0 peach#1, grape#0, plum#0, apple#0, apricot#0,
watermelon#1, banana#1, coconut#0, pear#0,
fig#0, melon#0, mangosteen#0, …
fruit#0, food#0, …
apple#0 mango#0, pineapple#0, banana#1, melon#0,
grape#0, peach#1, watermelon#1, apricot#0,
cranberry#0, pumpkin#0, mangosteen#0, …
fruit#0, crop#0, …
Java#1 C#4, Python#3, Apache#3, Ruby#6, Flash#1,
C++#0, SQL#0, ASP#2, Visual Basic#1, CSS#0,
Delphi#2, MySQL#0, Excel#0, Pascal#0, …
programming
language#3, lan-
guage#0, …
Python#3 PHP#0, Pascal#0, Java#1, SQL#0, Visual Ba-
sic#1, C++#0, JavaScript#0, Apache#3, Haskell#5,
.NET#1, C#4, SQL Server#0, …
language#0, tech-
nology#0, …
Inducing word sense representations
Sample of induced sense inventory

46. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 38/98
ID Global Sense Cluster: Semantic Class Hypernyms
1 peach#1, banana#1, pineapple#0, berry#0, black-
berry#0, grapefruit#0, strawberry#0, blueberry#0,
mango#0, grape#0, melon#0, orange#0, pear#0,
plum#0, raspberry#0, watermelon#0, apple#0, apri-
cot#0, watermelon#0, pumpkin#0, berry#0, man-
gosteen#0, …
vegetable#0, fruit#0,
crop#0, ingredi-
ent#0, food#0, ·
2 C#4, Basic#2, Haskell#5, Flash#1, Java#1, Pas-
cal#0, Ruby#6, PHP#0, Ada#1, Oracle#3, Python#3,
Apache#3, Visual Basic#1, ASP#2, Delphi#2, SQL
Server#0, CSS#0, AJAX#0, JavaScript#0, SQL
Server#0, Apache#3, Delphi#2, Haskell#5, .NET#1,
CSS#0, …
programming lan-
guage#3, technol-
ogy#0, language#0,
format#2, app#0
Inducing word sense representations
Sample of induced semantic classes

47. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 39/98
Text
Corpus
Representing
Senses
with
Ego
Networks
Semantic
Classes
Word
Sense
Induction
from
Text
Corpus
Sense
Graph
Construction
Clustering
of
Word
Senes
Labeling
Sense
Clusters
with
Hypernyms
Induced Word Senses Sense Ego-Networks Global Sense Graph
s
Noisy
Hypernyms
Cleansed
Hypernyms
Induction
of
Semantic
Classes
Global Sense Clusters
Inducing word sense representations
Induction of semantic classes

48. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 40/98
Filtering noisy hypernyms with semantic classes
LREC’18 [Panchenko et al., 2018b]:
fruit#1
food#0
apple#2 mango#0 pear#0
Hypernyms,
Sense Cluster,
mangosteen#0
city#2
Removed
Wrong
Added
Missing
Inducing word sense representations
Induction of sense semantic classes

49. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 41/98
http://panchenko.me/data/joint/nodes20000-layers7
Inducing word sense representations
Global sense clustering

50. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 42/98
Inducing word sense representations
Global sense clustering

51. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 43/98
Filtering of a noisy hypernymy database with semantic classes.
LREC’18 [Panchenko et al., 2018b]
Precision Recall F-score
Original Hypernyms (Seitner et al., 2016) 0.475 0.546 0.508
Semantic Classes (coarse-grained) 0.541 0.679 0.602
Inducing word sense representations
Induction of sense semantic classes

52. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 44/98
Making induced senses interpretable

53. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 45/98
Knowledge-based sense representations are interpretable
Making induced senses interpretable
Making induced senses interpretable

54. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 46/98
Most knowledge-free sense representations are uninterpretable
Making induced senses interpretable
Making induced senses interpretable

55. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 47/98
Making induced senses interpretable
Making induced senses interpretable

56. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 48/98
Hypernymy prediction in context. EMNLP’17 [Panchenko et al., 2017b]
Making induced senses interpretable
Making induced senses interpretable

57. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 49/98
11.702 sentences, 863 words with avg.polysemy of 3.1.
WSD Model Accuracy
Inventory Features Hypers HyperHypers
Word Senses Random 0.257 0.610
Word Senses MFS 0.292 0.682
Word Senses Cluster Words 0.291 0.650
Word Senses Context Words 0.308 0.686
Making induced senses interpretable
Making induced senses interpretable

58. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 49/98
11.702 sentences, 863 words with avg.polysemy of 3.1.
WSD Model Accuracy
Inventory Features Hypers HyperHypers
Word Senses Random 0.257 0.610
Word Senses MFS 0.292 0.682
Word Senses Cluster Words 0.291 0.650
Word Senses Context Words 0.308 0.686
Super Senses Random 0.001 0.001
Super Senses MFS 0.001 0.001
Super Senses Cluster Words 0.174 0.365
Super Senses Context Words 0.086 0.188
Making induced senses interpretable
Making induced senses interpretable

59. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 50/98
Linking induced senses to resources

60. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 51/98
Text Corpus
Linking
Induced
Senses
to
Senses
of
the
LR
Induce
a
Graph
of
Sem.
Related
Words
Enriched
Lexical Resource
Graph of Related Words
Word
Sense
Induction
Labeling
Senses
with
Hypernyms
Disambiguation
of
Neighbours
Typing
of
the
Unmapped
Induced
Senses
Word Sense Inventory Labeled Word Senses
PCZ
Construction
of
Proto-Conceptualization (PCZ)
Linking
Proto-Conceptualization
to
Lexical
Resource
Part. Linked Senses to the LR
Lexical Resource (LR):
WordNet, BabelNet, ...
Construction
of
sense
feature
representations
Graph of Related Senses
LREC’16 [Panchenko, 2016], ISWC’16 [Faralli et al., 2016],
SENSE@EACL’17 [Panchenko et al., 2017a],
NLE’18 [Biemann et al., 2018]
Linking induced senses to resources
Linking induced senses to resources

61. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 52/98
Word AdaGram BabelNet AdaGram BoW BabelNet BoW
python 2 bn:01713224n perl, php, java, smalltalk, ruby,
lua, tcl, scripting, javascript,
bindings, binding, programming,
coldfusion, actionscript, net, . . .
language, programming, python-
ista, python programming,
python3, python2, level, com-
puter, pythonistas, python3000,
python 1 bn:01157670n monty, circus, spamalot, python,
magoo, muppet, snoopy, fea-
turette, disney, tunes, tune, clas-
sic, shorts, short, apocalypse, . . .
monty, comedy, monty python,
british, monte, monte python,
troupe, pythonesque, foot, artist,
record, surreal, terry, . . .
python 3 bn:00046456n spectacled, unicornis, snake, gi-
ant, caiman, leopard, squirrel,
crocodile, horned, cat, mole, ele-
phant, opossum, pheasant, . . .
molurus, indian, boa, tigris,
tiger python, rock, tiger, indian
python, reptile, python molurus,
indian rock python, coluber, . . .
python 4 bn:01157670n circus, fly, flying, dusk, lizard,
moth, unicorn, puff, adder, vul-
ture, tyrannosaurus, zephyr, bad-
ger, . . .
monty, comedy, monty python,
british, monte, monte python,
troupe, pythonesque, foot, artist,
record, surreal, terry, . . .
python 1 bn:00473212n monty, circus, spamalot, python,
magoo, muppet, snoopy, fea-
turette, disney, tunes, tune, clas-
sic, shorts, short, apocalypse, . . .
pictures, monty, python monty
pictures, limited, company,
python pictures limited, king-
dom, picture, serve, director,
. . .
python 1 bn:03489893n monty, circus, spamalot, python,
magoo, muppet, snoopy, fea-
turette, disney, tunes, tune, clas-
sic, shorts, short, apocalypse, . . .
film, horror, movie, clabaugh,
richard, monster, century, direct,
snake, python movie, television,
giant, natural, language, for-tv,
. . .
Linking induced senses to resources
Linking induced senses to resources

62. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 53/98
Model Representation of the Sense ”disk (medium)”
WordNet memory, device, floppy, disk, hard, disk, disk, computer, science, computing, diskette, fixed, disk,
floppy, magnetic, disc, magnetic, disk, hard, disc, storage, device
WordNet + Linked recorder, disk, floppy, console, diskette, handset, desktop, iPhone, iPod, HDTV, kit, RAM, Discs, Blu-
ray, computer, GB, microchip, site, cartridge, printer, tv, VCR, Disc, player, LCD, software, component,
camcorder, cellphone, card, monitor, display, burner, Web, stereo, internet, model, iTunes, turntable,
chip, cable, camera, iphone, notebook, device, server, surface, wafer, page, drive, laptop, screen,
pc, television, hardware, YouTube, dvr, DVD, product, folder, VCR, radio, phone, circuitry, partition,
megabyte, peripheral, format, machine, tuner, website, merchandise, equipment, gb, discs, MP3,
hard-drive, piece, video, storage device, memory device, microphone, hd, EP, content, soundtrack,
webcam, system, blade, graphic, microprocessor, collection, document, programming, battery, key-
board, HD, handheld, CDs, reel, web, material, hard-disk, ep, chart, debut, configuration, recording,
album, broadcast, download, fixed disk, planet, pda, microfilm, iPod, videotape, text, cylinder, cpu,
canvas, label, sampler, workstation, electrode, magnetic disc, catheter, magnetic disk, Video, mo-
bile, cd, song, modem, mouse, tube, set, ipad, signal, substrate, vinyl, music, clip, pad, audio, com-
pilation, memory, message, reissue, ram, CD, subsystem, hdd, touchscreen, electronics, demo, shell,
sensor, file, shelf, processor, cassette, extra, mainframe, motherboard, floppy disk, lp, tape, version,
kilobyte, pacemaker, browser, Playstation, pager, module, cache, DVD, movie, Windows, cd-rom, e-
book, valve, directory, harddrive, smartphone, audiotape, technology, hard disk, show, computing,
computer science, Blu-Ray, blu-ray, HDD, HD-DVD, scanner, hard disc, gadget, booklet, copier, play-
back, TiVo, controller, filter, DVDs, gigabyte, paper, mp3, CPU, dvd-r, pipe, cd-r, playlist, slot, VHS, film,
videocassette, interface, adapter, database, manual, book, channel, changer, storage
Linking induced senses to resources
Linking induced senses to resources

63. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 54/98
Evaluation of linking accuracy:
Linking induced senses to resources
Linking induced senses to resources

64. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 55/98
Evaluation of enriched representations based on WSD:
Linking induced senses to resources
Linking induced senses to resources

65. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 56/98
Shared task on word sense induction

66. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 57/98
An ACL SIGSLAV sponsored shared task on word sense
induction (WSI) for the Russian language.
More details: https://russe.nlpub.org/2018/wsi
Shared task on word sense induction
A shared task on WSI

67. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98
Target word, e.g. “bank”.
Shared task on word sense induction
A lexical sample WSI task

68. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98
Target word, e.g. “bank”.
Contexts where the word occurs, e.g.:
“river bank is a slope beside a body of water”
“bank is a financial institution that accepts deposits”
“Oh, the bank was robbed. They took about a million dollars.”
“bank of Elbe is a good and popular hangout spot complete
with good food and fun”
Shared task on word sense induction
A lexical sample WSI task

69. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/98
Target word, e.g. “bank”.
Contexts where the word occurs, e.g.:
“river bank is a slope beside a body of water”
“bank is a financial institution that accepts deposits”
“Oh, the bank was robbed. They took about a million dollars.”
“bank of Elbe is a good and popular hangout spot complete
with good food and fun”
You need to group the contexts by senses:
“river bank is a slope beside a body of water”
“bank of Elbe is a good and popular hangout spot complete
with good food and fun”
“bank is a financial institution that accepts deposits”
“Oh, the bank was robbed. They took about a million dollars.”
Shared task on word sense induction
A lexical sample WSI task

70. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 59/98
Shared task on word sense induction
Dataset based on Wikipedia

71. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 60/98
Shared task on word sense induction
Dataset based on RNC

72. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 61/98
Shared task on word sense induction
Dataset based on dictionary glosses

73. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 62/98
Shared task on word sense induction
A sample from the wiki-wiki dataset

74. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 63/98
Shared task on word sense induction
A sample from the wiki-wiki dataset

75. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 64/98
Shared task on word sense induction
A sample from the wiki-wiki dataset

76. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 65/98
Shared task on word sense induction
A sample from the bts-rnc dataset

77. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 66/98
Shared task on word sense induction
A sample from the active-dict dataset

78. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 67/98
1 Get the neighbors of a target word, e.g. “bank”:
1 lender
2 river
3 citybank
4 slope
5 …
2 Get similar to “bank” and dissimilar to “lender”:
1 river
2 slope
3 land
4 …
3 Compute distances to “lender” and “river”.
Shared task on word sense induction
jamsic: sense induction

79. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 68/98
Induction of semantic frames

80. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 69/98
Induction of semantic frames
FrameNet: frame “Kidnapping”

81. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 70/98
ACL’2018 [Ustalov et al., 2018a]
Example of a LU tricluster corresponding to the “Kidnapping”
frame from FrameNet.
FrameNet Role Lexical Units (LU)
Perpetrator Subject kidnapper, alien, militant
FEE Verb snatch, kidnap, abduct
Victim Object son, people, soldier, child
Induction of semantic frames
Frame induction as a triclustering

82. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 71/98
Induction of semantic frames
SVO triple elements

83. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 72/98
Officer|chair|Committee
officer|head|team
mayor|lead|city
officer|lead|company
Mayor|lead|city
boss|lead|company
chairman|lead|company
director|lead|department
chief|lead|department
president|lead|government
president|lead|state
director|lead|company
president|lead|department
officer|chair|committee
Chief|lead|department
chairman|lead|committee
Director|lead|Department
Director|lead|department
Director|lead|agency
Director|lead|company
minister|lead|team
Director|head|team
director|head|team
Chairman|lead|company
Chairman|lead|Committee
President|lead|company
Director|chair|Committee
President|lead|party
President|head|teamleader|head|team
Director|chair|committee
director|chair|committee
Director|head|Department
president|head|team
director|head|department
director|head|agency
director|head|committee
Chairman|run|committee
Chairman|chair|Committee
Chairman|chair|committee
President|chair|Committee
President|chair|committee
Governor|lead|state
chairman|head|committee
chairman|run|committee
president|chair|committee
president|head|committee
president|chair|Committee
Minister|chair|committee
representative|chair|committee
representative|head|committee
General|command|department
General|command|Department
General|head|Department
General|head|department
officer|head|department
minister|head|department
leader|head|agency
leader|head|party
leader|head|committee
leader|head|department
minister|head|committee
King|run|company
leader|head|government
Minister|head|government
president|head|government
Officer|chair|Committee
officer|head|team
mayor|lead|city
officer|lead|company
Mayor|lead|city
boss|lead|company
chairman|lead|company
director|lead|department
chief|lead|department
president|lead|government
president|lead|state
director|lead|company
president|lead|department
officer|chair|committee
Chief|lead|department
chairman|lead|committee
Director|lead|Department
Director|lead|department
Director|lead|agency
Director|lead|company
minister|lead|team
Director|head|team
director|head|team
Chairman|lead|company
Chairman|lead|Committee
President|lead|company
Director|chair|Committee
President|lead|party
President|head|teamleader|head|team
Director|chair|committee
director|chair|committee
Director|head|Department
president|head|team
director|head|department
director|head|agency
director|head|committee
Chairman|run|committee
Chairman|chair|Committee
Chairman|chair|committee
President|chair|Committee
President|chair|committee
Governor|lead|state
chairman|head|committee
chairman|run|committee
president|chair|committee
president|head|committee
president|chair|Committee
Minister|chair|committee
representative|chair|committee
representative|head|committee
General|command|department
General|command|Department
General|head|Department
General|head|department
officer|head|department
minister|head|department
leader|head|agency
leader|head|party
leader|head|committee
leader|head|department
minister|head|committee
King|run|company
leader|head|government
Minister|head|government
president|head|government
Induction of semantic frames
An SVO triple graph

84. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98
Input: an embedding model v ∈ V → ⃗v ∈ Rd,
a set of SVO triples T ⊆ V 3,
the number of nearest neighbors k ∈ N,
a graph clustering algorithm Cluster.
Induction of semantic frames
Triframes frame induction

85. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98
Input: an embedding model v ∈ V → ⃗v ∈ Rd,
a set of SVO triples T ⊆ V 3,
the number of nearest neighbors k ∈ N,
a graph clustering algorithm Cluster.
Output: a set of triframes F.
Induction of semantic frames
Triframes frame induction

86. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/98
Input: an embedding model v ∈ V → ⃗v ∈ Rd,
a set of SVO triples T ⊆ V 3,
the number of nearest neighbors k ∈ N,
a graph clustering algorithm Cluster.
Output: a set of triframes F.
S ← {t→ ⃗t ∈ R3d : t ∈ T}
E ← {(t, t′) ∈ T2 : t′ ∈ NNS
k (⃗t), t ̸= t′}
F ← ∅
for all C ∈ Cluster(T, E) do
fs ← {s ∈ V : (s, v, o) ∈ C}
fv ← {v ∈ V : (s, v, o) ∈ C}
fo ← {o ∈ V : (s, v, o) ∈ C}
F ← F ∪ {(fs, fv, fo)}
return F
Induction of semantic frames
Triframes frame induction

87. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 74/98
Frame # 848
Subjects: Company, firm, company
Verbs: buy, supply, discharge, purchase, expect
Objects: book, supply, house, land, share, company,
grain, which, item, product, ticket, work,
this, equipment, House, it, film, water,
something, she, what, service, plant, time
Induction of semantic frames
Example of an extracted frame

88. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 75/98
Frame # 849
Subjects: student, scientist, we, pupil, member,
company, man, nobody, you, they, US,
group, it, people, Man, user, he
Verbs: do, test, perform, execute, conduct
Objects: experiment, test
Induction of semantic frames
Example of an extracted frame

89. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 76/98
Frame # 3207
Subjects: people, we, they, you
Verbs: feel, seek, look, search
Objects: housing, inspiration, gold, witness, part-
ner, accommodation, Partner
Induction of semantic frames
Example of an extracted frame

90. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 77/98
Dataset # instances # unique # clusters
FrameNet Triples 99,744 94,170 383
Poly. Verb Classes 246 110 62
Induction of semantic frames
Evaluation datasets

91. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 78/98
Dataset # instances # unique # clusters
FrameNet Triples 99,744 94,170 383
Poly. Verb Classes 246 110 62
Induction of semantic frames
Evaluation settings

92. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 78/98
Dataset # instances # unique # clusters
FrameNet Triples 99,744 94,170 383
Poly. Verb Classes 246 110 62
Quality Measures:
nmPU: normalized modified purity,
niPU: normalized inverse purity.
Induction of semantic frames
Evaluation settings

93. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 79/98
F1-scores for verbs, subjects, objects, frames
Induction of semantic frames
Results: comparison to state-of-art

94. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 80/98
Graph embeddings

95. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 81/98
Graph embeddings
Text: sparse symbolic representation

96. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 81/98
Image source:
https://www.tensorflow.org/tutorials/word2vec
Graph embeddings
Text: sparse symbolic representation

97. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 82/98
Graph embeddings
Graph: sparse symbolic representation

98. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 83/98
From a survey on graph embeddings [Hamilton et al., 2017]:
Graph embeddings
Embedding graph into a vector space

99. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 84/98
From a survey on graph embeddings [Hamilton et al., 2017]:
Graph embeddings
Learning with an “autoencoder”

100. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 85/98
From a survey on graph embeddings [Hamilton et al., 2017]:
Graph embeddings
Some established approaches

101. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 86/98
An submitted joint work with Andrei Kutuzov and Chris Biemann:
Given a tree (V, E)
Leackock-Chodorow (LCH) similarity measure:
sim(vi, vj) = − log
shortest_path_distance(vi, vj)
2h
Graph embeddings
Graph embeddings using similarities

102. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 86/98
An submitted joint work with Andrei Kutuzov and Chris Biemann:
Given a tree (V, E)
Leackock-Chodorow (LCH) similarity measure:
sim(vi, vj) = − log
shortest_path_distance(vi, vj)
2h
Jiang-Conrath (JCN) similarity measure:
sim(vi, vj) = 2
ln Plcs(vi, vj)
ln P(vi) + ln P(vj)
Graph embeddings
Graph embeddings using similarities

103. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 87/98
path2vec: Approximating Structural Node Similarities with Node
Embeddings:
J =
1
|T|
∑
(vi,vj)∈T
(vi · vj − sim(vi, vj))2
,
where:
sim(vi, vj) - the value of a ‘gold’ similarity measure between
a pair of nodes vi and vj;
vi - an embeddings of node;
T - training batch.
Graph embeddings
Graph embeddings using similarities

104. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 88/98
Computation of 82,115 pairwise similarities:
Model Running time
LCH in NLTK 30 sec.
JCN in NLTK 6.7 sec.
FSE embeddings 0.713 sec.
path2vec and other float vectors 0.007 sec.
Graph embeddings
Speedup: graph vs embeddings

105. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 89/98
Spearman correlation scores with WordNet similarities on
SimLex999 noun pairs:
Selection of synsets
Model JCN-SemCor JCN-Brown LCH
WordNet 1.0 1.0 1.0
Node2vec 0.655 0.671 0.724
Deepwalk 0.775 0.774 0.868
FSE 0.830 0.820 0.900
path2vec 0.917 0.914 0.934
Graph embeddings
Results: goodness of fit

106. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 90/98
Spearman correlations with human SimLex999 noun similarities:
Model Correlation
Raw WordNet JCN-SemCor 0.487
Raw WordNet JCN-Brown 0.495
Raw WordNet LCH 0.513
node2vec [Grover & Leskovec, 2016] 0.450
Deepwalk [Perozzi et al., 2014] 0.533
FSE [Subercaze et al., 2015] 0.556
path2vec JCN-SemCor 0.526
path2vec JCN-Brown 0.487
path2vec LCH 0.522
Graph embeddings
Results: SimLex999 dataset

107. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 91/98
JCN (left) and LCH (right):
Graph embeddings
Results: SimLex999 dataset

108. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 92/98
WSD: each column lists all the possible synsets for the
corresponding word.
Graph embeddings
Results: word sense disambiguation

109. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 93/98
F1 scores on Senseval-2 word sense disambiguation task:
Model F-measure
WordNet JCN-SemCor 0.620
WordNet JCN-Brown 0.561
WordNet LCH 0.547
node2vec [Grover & Leskovec, 2016] 0.501
Deepwalk [Perozzi et al., 2014] 0.528
FSE [Subercaze et al., 2015] 0.536
path2vec
Batch size: 20 50 100
JCN-SemCor 0.543 0.543 0.535
JCN-Brown 0.538 0.515 0.542
LCH 0.540 0.535 0.536
Graph embeddings
Results: word sense disambiguation

110. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 94/98
Conclusion

111. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 95/98
Conclusion
Vectors + Graphs = ♡

112. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98
We can induce word senses, synsets, semantic classes, and
semantic frames in a knowledge-free way using graph
clustering and distributional models.
Conclusion
Take home messages

113. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98
We can induce word senses, synsets, semantic classes, and
semantic frames in a knowledge-free way using graph
clustering and distributional models.
We can make the induced word senses interpretable in a
knowledge-free way with hypernyms, images, definitions.
Conclusion
Take home messages

114. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98
We can induce word senses, synsets, semantic classes, and
semantic frames in a knowledge-free way using graph
clustering and distributional models.
We can make the induced word senses interpretable in a
knowledge-free way with hypernyms, images, definitions.
We can link induced senses to lexical resources to
improve performance of WSD;
enrich lexical resources with emerging senses.
Conclusion
Take home messages

115. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 96/98
We can induce word senses, synsets, semantic classes, and
semantic frames in a knowledge-free way using graph
clustering and distributional models.
We can make the induced word senses interpretable in a
knowledge-free way with hypernyms, images, definitions.
We can link induced senses to lexical resources to
improve performance of WSD;
enrich lexical resources with emerging senses.
We can represent language graphs using graph embeddings
in deep neural models.
Conclusion
Take home messages

116. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 97/98
A special issue on informing neural architectures for NLP
with linguistic and background knowledge.
… with Ivan Vulić and Simone Paolo Ponzetto.
goo.gl/A76NGX
Conclusion
Natural Language Engineering journal

117. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98
Conclusion
Thank you! Questions?

118. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98
Arefyev, N., Ermolaev, P., & Panchenko, A. (2018).
How much does a word weigh? weighting word embeddings
for word sense induction.
arXiv preprint arXiv:1805.09209.
Bartunov, S., Kondrashkin, D., Osokin, A., & Vetrov, D. (2016).
Breaking sticks and ambiguities with adaptive skip-gram.
In Artificial Intelligence and Statistics (pp. 130–138).
Biemann, C., Faralli, S., Panchenko, A., & Ponzetto, S. P. (2018).
A framework for enriching lexical semantic resources with
distributional semantics.
In Journal of Natural Language Engineering (pp. 56–64).:
Cambridge Press.
Faralli, S., Panchenko, A., Biemann, C., & Ponzetto, S. P. (2016).
Linked disambiguated distributional semantic networks.
In International Semantic Web Conference (pp. 56–64).:
Springer.
Grover, A. & Leskovec, J. (2016).

119. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98
Node2vec: Scalable feature learning for networks.
In Proceedings of the 22nd ACM SIGKDD international
conference on Knowledge discovery and data mining (pp.
855–864).: ACM.
Hamilton, W. L., Ying, R., & Leskovec, J. (2017).
Representation learning on graphs: Methods and
applications.
IEEE Data Engineering Bulletin, September 2017.
Panchenko, A. (2016).
Best of both worlds: Making word sense embeddings
interpretable.
In LREC.
Panchenko, A., Faralli, S., Ponzetto, S. P., & Biemann, C.
(2017a).
Using linked disambiguated distributional networks for word
sense disambiguation.

120. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98
In Proceedings of the 1st Workshop on Sense, Concept and
Entity Representations and their Applications (pp. 72–78).
Valencia, Spain: Association for Computational Linguistics.
Panchenko, A., Lopukhina, A., Ustalov, D., Lopukhin, K.,
Arefyev, N., Leontyev, A., & Loukachevitch, N. (2018a).
Russe’2018: A shared task on word sense induction for the
russian language.
arXiv preprint arXiv:1803.05795.
Panchenko, A., Marten, F., Ruppert, E., Faralli, S., Ustalov, D.,
Ponzetto, S. P., & Biemann, C. (2017b).
Unsupervised, knowledge-free, and interpretable word sense
disambiguation.
In Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations (pp.
91–96). Copenhagen, Denmark: Association for
Computational Linguistics.
Panchenko, A., Ruppert, E., Faralli, S., Ponzetto, S. P., &
Biemann, C. (2017c).

121. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98
Unsupervised does not mean uninterpretable: The case for
word sense induction and disambiguation.
In Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics: Volume 1,
Long Papers (pp. 86–98). Valencia, Spain: Association for
Computational Linguistics.
Panchenko, A., Ustalov, D., Faralli, S., Ponzetto, S. P., &
Biemann, C. (2018b).
Improving hypernymy extraction with distributional
semantic classes.
In Proceedings of the LREC 2018 Miyazaki, Japan: European
Language Resources Association.
Pelevina, M., Arefiev, N., Biemann, C., & Panchenko, A. (2016).
Making sense of word embeddings.
In Proceedings of the 1st Workshop on Representation
Learning for NLP (pp. 174–183). Berlin, Germany: Association
for Computational Linguistics.
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014).

122. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98
Deepwalk: Online learning of social representations.
In Proceedings of the 20th ACM SIGKDD international
conference on Knowledge discovery and data mining (pp.
701–710).: ACM.
Remus, S. & Biemann, C. (2018).
Retrofittingword representations for unsupervised sense
aware word similarities.
In Proceedings of the LREC 2018 Miyazaki, Japan: European
Language Resources Association.
Rothe, S. & Schütze, H. (2015).
Autoextend: Extending word embeddings to embeddings for
synsets and lexemes.
In Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume 1: Long
Papers) (pp. 1793–1803). Beijing, China: Association for
Computational Linguistics.
Subercaze, J., Gravier, C., & Laforest, F. (2015).

123. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98
On metric embedding for boosting semantic similarity
computations.
In Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume 2: Short
Papers) (pp. 8–14).: Association for Computational
Linguistics.
Ustalov, D., Chernoskutov, M., Biemann, C., & Panchenko, A.
(2017a).
Fighting with the sparsity of synonymy dictionaries for
automatic synset induction.
In International Conference on Analysis of Images, Social
Networks and Texts (pp. 94–105).: Springer.
Ustalov, D., Panchenko, A., & Biemann, C. (2017b).
Watset: Automatic induction of synsets from a graph of
synonyms.
In Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers) (pp.

124. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 98/98
1579–1590). Vancouver, Canada: Association for
Computational Linguistics.
Ustalov, D., Panchenko, A., Kutuzov, A., Biemann, C., &
Ponzetto, S. P. (2018a).
Unsupervised semantic frame induction using triclustering.
arXiv preprint arXiv:1805.04715.
Ustalov, D., Teslenko, D., Panchenko, A., Chernoskutov, M., &
Biemann, C. (2018b).
Word sense disambiguation based on automatically induced
synsets.
In LREC 2018, 11th International Conference on Language
Resources and Evaluation : 7-12 May 2018, Miyazaki (Japan)
(pp. tba). Paris: European Language Resources Association,
ELRA-ELDA.
Accepted for publication.