NS-CUK Seminar: H.E.Lee, Review on "PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks", KDD 2015

1. Hyo Eun Lee
Network Science Lab
Dept. of Biotechnology
The Catholic University of Korea
E-mail: gydnsml@gmail.com
2023.08.14

2. 1
 Problem definition
 Predictive text embedding
• Bipartite Network Embedding
• Heterogeneous Text Network Embedding
• Text Embedding
 Experiments
 Discussion and conclusion

3. 2
1. Problem definition
Definition
• Definition 1. (Word-Word Network)
• Capture co-occurrence information in unlabeled local contexts
𝐺𝑊𝑊 = (𝑉 , 𝐸𝑊𝑊)
• Traditional word embedding approaches such as skipgrams
• Definition 2. (Word-Document Network) Word-document
• Capture connections between words and documents in a corpus
𝐺𝑊𝐷 = (𝑉 ∪ 𝐷 , 𝐸𝑊𝐷)
• Definition 3. (Word-Label Network) Word-label
𝐺𝑊𝑙 = 𝑉 ∪ 𝐿 , 𝐸𝑊𝑙
𝑤𝑖𝑗 = 𝑛𝑑𝑙

4. 3
1. Problem definition
Definition
• Definition 4. (Heterogeneous Text Network) The heterogeneous text network
• Represents a combination of defined networks
• Captures co-occurrences at multiple levels and includes both labeled and unlabeled data
• Definition 5. (Predictive Text Embedding)
• The resulting low-dimensional embeddings are powerful for certain tasks

5. 4
2. Predictive text embedding
Bipartite Network Embedding
• LINE model was introduced for large-scale information embedding, but weights for different types of
edges cannot be compared
• Therefore, we propose an applied method that applies quadratic proximity between nodes
• 𝐺 = (𝑉𝐴 ∪ 𝑉𝐵, 𝐸)

6. 5
2. Predictive text embedding
Bipartite Network Embedding
• Optimization of the objective function using stochastic gradient descent.
• Using edge sampling and negative sampling techniques.
• Edge sampling method to obtain binary edges e with probability proportional to their weights at each
step and negative samples from the noise distribution p.
• After learning all the embeddings, we can define the objective function

7. 6
Heterogeneous Text Network Embedding
• There are three different networks shared by the word vertices
2. Predictive text embedding

8. 7
Heterogeneous Text Network Embedding
• Training unlabeled and labeled together
2. Predictive text embedding

9. 8
Heterogeneous Text Network Embedding
• Train with unlabeled data and refine using labeled
2. Predictive text embedding

10. 9
Text Embedding
• After training the vector representation, it can be averaged to obtain a representation of all the text.
• Learn by minimizing a loss function, specified as the Euclidean distance between embeddings, using a
gradient descent algorithm.
2. Predictive text embedding

11. 10
3. Experiments

12. 11
3. Experiments

13. 12
3. Experiments

14. 13
3. Experiments

15. 14
3. Experiments

16. 15
3. Experiments

17. 16
3. Experiments

18. 17
3. Experiments

19. 18
4. Discussion and conclusion
Discussion and conclusion
• Unsupervised learning uses either local context-level or document-level word co-occurrences, with
document-level co-occurrences being more useful for long documents and local context-level
being more useful for short documents.
• PTE joint training on both labeled and unlabeled data, and outperforms CNNs with more labeled
data.
• PTE needs improvement, such as taking into account the order of words.