This document proposes a method for multi-label, multi-class text classification using polylingual embeddings. It generates document embeddings in different languages using pooling methods and learns cross-language embeddings with an autoencoder. Experimental results on a dataset with 12,670 instances across 100 classes show that distributed representations perform better with limited labeled data compared to bag-of-words models. Neighborhood-based classifiers like k-NN outperform SVMs on the polylingual embeddings, likely due to their semantic nature. The authors conclude more work is needed on composition functions for word representations and efficiently combining them with bag-of-words models.

1. Multi-label,Multi-classClassificationUsingPolylingual
Embeddings
Georgios.Balikas@imag.fr and Massih-Reza.Amini@imag.fr
1. MOTIVATION
Less than 50% of the current Internet content is written in English. There
are a lot of high quality resources for English.
Can we transfer knowledge between different languages? How can we
profitably exploit the multilingual content?
Text classification, like summarization, are applications that would benefit
by such approaches.
Figure 1: Multilinguality.
2. OUR REPRESENTATION LEARNING APPROACH
Figure 2: The generation of polylingual document embeddings starting from the given languages.
Generate document embeddings in each language (English, French, ...) using average pooling methods or paragraph vectors.
Learn language-independent embeddings for each document using the denoising auto-encoder.
Evaluate the learning methods on those polylingual representations learned on the auto-encoder’s hidden layer.
3. THE EXPERIMENTAL FRAMEWORK
0.1 0.3 0.5 0.7 0.9
Proportion of the training set
0.4
0.5
0.6
F1measure
cbow
SVMPE
k-NNPE
SVMBoW
0.1 0.3 0.5 0.7 0.9
Proportion of the training set
0.4
0.5
0.6
F1measure
skip-gram
SVMPE
k-NNPE
SVMBoW
Figure 3: Polylingual embeddings Vs bag-of-words representations. Com-
plete dataset: 12,670 instances (100 classes).
cbow
dim. k-NNDR SVMDR k-NNPE SVMPE
50 39.19 37.20 39.58 32.84
100 40.20 40.01 43.53 37.54
200 40.48 43.41 45.86 42.50
300 40.42 44.25 46.33 43.38
DBOWpv
50 24.45 25.06 30.26 24.08
100 31.20 28.53 34.63 26.88
200 27.73 29.80 36.02 30.80
300 27.79 29.92 38.71 30.82
SVMBoW 36.03
Table 1: A detailed analysis of the performance gains.
Interestingly, distributed representations of documents perform better when few labeled data are available.
k-NN outperforms SVMs probably due to the semantic nature of such representations.



Take home message!
Need for composition functions that retain more information when combining word representations.
How can we efficiently combine such representations with bag-of-word representations?
REFERENCES
[1] Mikolov, Tomas, et al. "Efficient estimation of word representations in vector space." arXiv:1301.3781 (2013).
[2] Le, Quoc V., and Tomas Mikolov. "Distributed representations of sentences and documents." arXiv:1405.4053
(2014).
ACKNOWLEDGEMENTS
This work is partially supported by the CIFRE N
28/2015 and by the LabEx PERSYVAL Lab ANR-11-
LABX-0025.