LINE: Large-scale Information Network Embedding

1. LINE: Large-scale Information
Network Embedding
Tien-Bach-Thanh Do
Network Science Lab
Dept. of Artificial Intelligence
The Catholic University of Korea
E-mail: xxx@catholic.ac.kr
2024/01/02
Jian Tang et al.
WWW 2015 - Proceedings of the 24th International Conference on World
Wide Web (2015)

2. 2
Introduction
● LINE is a novel network embedding method
● It is suitable for arbitrary types of information networks: undirected, directed, and/or weighted

3. 3
Motivation
● Challenges in working with large-scale graphs
● Need for methods preserving both local and global structures

4. 4
Overview
● LINE (Large-scale Information Network Embeddings) is a network embedding algorithm designed to learn
distributed representations (embeddings) of nodes in a graph. The primary objective of LINE is to capture
the structural information of a network in a continuous vector space. It specifically aims to preserve both
first-order and second-order proximity structures within the graph.
● First-order proximity refers to the immediate neighborhood relationships between nodes. In the context of
LINE, preserving first-order proximity means that nodes that are close in the original graph should also
have similar vector representations in the embedding space.
● Second-order proximity considers the structural equivalence between nodes based on their neighborhood
connectivity. If two nodes have similar neighbors, they are considered structurally equivalent. LINE aims
to preserve second-order proximity by ensuring that nodes with similar neighborhood structures have
similar embeddings.

5. 5
Objectives
● The method optimizes a carefully designed objective function
● It preserves both local and global network structures

6. 6
Efficiency
● The algorithm is very efficient
● It is able to learn the embedding of a network with millions of nodes and billions of edges

7. 7
Methodology
Optimization Process using Stochastic Gradient Descent (SGD)
● Initialization: Initialize the embeddings of nodes randomly in a continuous vector space.
● Stochastic Gradient Descent:
○ For each edge (u, v) in the graph:
■ For first-order proximity:
● Sample a context node c from the neighbors of the target node u.
● Update the embedding of u to minimize the negative log probability of c given u.
■ For second-order proximity:
● Sample a common neighbor c from the neighbors of both u and v.
● Update the embeddings of both u and v to minimize the negative log probability of
observing each other's context nodes given the common neighbor c.
● Objective Function Minimization: The optimization process involves adjusting the embeddings to
minimize the negative log likelihood of the observed context nodes given the target nodes for both first-
order and second-order proximity. This is typically done using stochastic gradient descent, which
incrementally updates the embeddings based on small batches of sampled edges.
● Convergence: Iterate through the stochastic gradient descent updates until the objective functions
converge or reach a predefined stopping criterion

8. 8
Advantages
● Effective capture of both local and global structures
● Scalability for large-scale networks

9. 10
Comparison
DeepWalk Node2Vec LINE
Objective Generate node embeddings by
preserving local proximity in the
graph
Extend DeepWalk by introducing a
biased random walk strategy to
capture both local and global
structures
Optimize a likelihood function to
preserve first-order and second-
order proximity in the network
Random Walk Strategy Unbiased random walks Biased random walks with
parameters controlling exploration
and exploitation
Utilizes first-order and second-
order proximity through a network-
wide objective
Proximity Metrics Typically uses random walk
embeddings based on Skip-gram
model (Word2Vec)
Combines Skip-gram and
Continuous Bag of Words (CBOW)
models to capture local and global
structures
Directly optimizes first-order and
second-order proximity measures
Scalability Efficient for large-scale graphs Can be less efficient for large-scale
graphs due to the biased random
walks strategy
Designed for scalability and is
suitable for large networks
Exploration-Exploitation No explicit control over exploration
and exploitation
Parameters control the balance
between exploration and
exploitation during random walks
Utilizes a unified objective to
implicitly balance exploration and
exploitation
Network Structure Primarily captures local structures Captures both local and global
structures
Tries to capture both first-order and
second-order proximity structures

10. 12
Applications
● Useful in many tasks such as visualization, node classification, and link prediction
● Empirical experiments prove the effectiveness of the LINE on a variety of real-world information
networks, including language networks, social networks, and citation networks

11. 13
Challenges
● Scalability
○ LINE is designed for scalability, but extremely large networks may still pose computational challenges
● Sensitivity to hyperparameters
○ LINE involves parameters like the number of dimensions and the number of negative samples, which can
impact results
○ Careful parameter tuning is crucial. Automated methods or sensitivity analyses can be employed
● Handling dynamic networks
○ LINE is primarily designed for static networks, and may not fully capture the dynamics in evolving networks
○ Extensions or combination with dynamic network embedding methods may be explored for more accurate
representations
● Lack of edge weight consideration
○ LINE does not explicitly consider edge weights, potentially limiting its applicability to weighted networks
○ Future research might explore adaptations or extensions of LINE to incorporate edge weights more
effectively
● Limited interpretability
○ The resulting embeddings may lack interpretability, making it challenging to understand the meaning of
individual dimensions
○ Incorporating domain-specific information or post-processing techniques for interpretability can be
explored

12. 14
Experiment

13. 15
Experiment
Language Network
GF: graph factorization, applies to undirected networks only
LINE-SGD(1st): first-order proximity, applies to undirected networks only
LINE-SGD(2nd): second-order proximity, applies to both undirected and directed graphs
LINE(1st): LINE model optimized through edge-sampling,applies to undirected networks only
LINE (2nd): LINE model optimized through edge-sampling,applies to undirected and directed graphs

14. 16
Experiment
Language Network
LINE (1st+2nd): Concatenate the vector representations learned by LINE(1st) and LINE(2nd) into longer
vector, apply in supervised tasks only

15. 17
Experiment
Social Network

16. 18
Experiment
Social Network

17. 19
Experiment
DBLP Network

18. 20
Experiment
DBLP Network

19. 21
Implement

20. 22
Conclusion
● LINE is powerful tool for embedding large-scale information networks
● It preserves important network structures and scales to networks with millions of vertices and billions of
edges