1. Assiut University | Faculty of computers and information 1
Algorithms for Data Science
Graph Neural Network
Chapter 2
Prepared by
Amjad Dife
2023 / 2024
GNN 2023 / 2024
2. outline
Assiut University | Faculty of computers and information 2
1. Motivation of graph representation learning.
2. Book big picture
3. Chapter 2: Comprehensive overview of a large number of graph representation learning
methods.
▪ The traditional graph representation-learning.
▪ Modern graph representation learning
▪ Graph neural networks.
GNN 2023 / 2024
3. Remember from the Previous Lecture
Assiut University | Faculty of computers and information 3
“The goal of representation learning
is
to extract sufficient but minimal information from data.”
GNN 2023 / 2024
4. Assiut University | Faculty of computers and information 4
Motivation
Representation Learning is the core of Machine Learning and Deep Learning
GNN 2023 / 2024
5. Assiut University | Faculty of computers and information 5
Computer Vision perspective
GNN 2023 / 2024
6. Motivation | What is Representation Learning [1]
Assiut University | Faculty of computers and information 6
GNN 2023 / 2024
Representation 1 Representation 2
➢ It is a Cat ! ➢ It is a Brown Cat that is
napping next to a jar
on a sunny afternoon.
➢ Is this a Cat or a Dog?
➢ Is this cat mine?
➢ What is this cat doing?
➢ Is the Cat safe?
7. Motivation | How the Computer Represents the image
Assiut University | Faculty of computers and information 7
GNN 2023 / 2024
➢ Is this a Cat or a Dog ?
➢ Is this cat mine?
➢ What is this cat doing?
➢ Is the Cat safe?
Computer Representation
8. Assiut University | Faculty of computers and information 8
NLP perspective
GNN 2023 / 2024
9. Motivation | How the Computer Represents the Text
Assiut University | Faculty of computers and information 9
GNN 2023 / 2024
Image source: Stanford NLP Course
How do we represent the meaning of a word?
10. Assiut University | Faculty of computers and information 10
So, We need to derive features first from the row data.
So, we need Representation Learning. . .
GNN 2023 / 2024
11. Remember from the Previous Lecture
Assiut University | Faculty of computers and information 11
“The goal of representation learning
is
to extract sufficient but minimal information from data.”
GNN 2023 / 2024
14. Motivation
Assiut University | Faculty of computers and information 14
GNN 2023 / 2024
How do we represent the meaning of a word?
Image source: Stanford NLP Course
15. Motivation
Assiut University | Faculty of computers and information 15
GNN 2023 / 2024
Representation
Representation
Representation
Representation
Image source: https://medium.com/@hari4om/word-embedding-d816f643140
How do we represent the meaning of a word?
16. Motivation
Assiut University | Faculty of computers and information 16
GNN 2023 / 2024
Image source: https://medium.com/@hari4om/word-embedding-d816f643140
How do we represent the meaning of a word?
17. Assiut University | Faculty of computers and information 17
The Big Picture
Of
Graph Neural Networks Book
GNN 2023 / 2024
18. Assiut University | Faculty of computers and information 18
GNN 2023 / 2024
GNN
Foundations
▪ Why Graphs
▪ A Brief History of GNNs
▪ GNNs: Foundations and Models
GNNs
Frontiers
▪ Graph Matching
▪ Graph Structure Learning
▪ Other Frontiers in GNNs
GNNs
Applications
▪ Recommendation
▪ Other Applications With GNNs
19. Assiut University | Faculty of computers and information 19
GNN 2023 / 2024
GNN
Foundations
▪ Why Graphs
▪ A Brief History of GNNs
▪ GNNs: Foundations and Models
20. What and why Graphs [2,3]
Assiut University | Faculty of computers and information 20
➢ A Graph is a type of data structure that contains nodes and edges. [3]
➢ A node can be a person, place, or thing, and the edges define the relationship
between nodes.
➢ The edges can be directed and undirected based on directional dependencies.
➢ Graphs are a general Language for describing and modeling complex systems.
GNN 2023 / 2024
21. Introduction: Graph Representation Learning
Assiut University | Faculty of computers and information 21
▪ Many complex systems take the form of graphs, such as social networks, biological
networks, and information networks.
▪ Why?
▪ The pattern discovery, analysis, and prediction, can be conducted efficiently in both
time and space.
▪ Graph Representation
▪ Traditionally, we usually represent a graph as G = (V, E ), where V is a node set and E is
an edge set.
▪ For large graphs, such as those with billions of nodes, the traditional graph
representation poses several challenges to graph processing and analysis.
GNN 2023 / 2024
22. Assiut University | Faculty of computers and information 22
Graph-structured data are ubiquitous (present or found everywhere )
GNN 2023 / 2024
23. Image as Graph
Assiut University | Faculty of computers and information 23
GNN 2023 / 2024
Image source:
A Gentle Introduction to Graph Neural Networks (distill.pub)
24. Text as Graph
Assiut University | Faculty of computers and information 24
GNN 2023 / 2024
Image source:
A Gentle Introduction to Graph Neural Networks (distill.pub)
25. Graph-structured data are ubiquitous (present or found everywhere )
Assiut University | Faculty of computers and information 25
GNN 2023 / 2024
Image source:
A Gentle Introduction to Graph Neural Networks (distill.pub)
26. Assiut University | Faculty of computers and information 26
GNN 2023 / 2024
Graph Representation Learning
▪ The main goal of graph representation learning is to obtain node embeddings that can
be used for various downstream tasks, such as node classification, node clustering,
graph visualization, and link prediction.
27. Assiut University | Faculty of computers and information 27
GNN 2023 / 2024
GNN
Foundations
▪ Why Graphs
▪ A Brief History of GNNs
▪ GNNs: Foundations and Models
28. GNNs: A Brief History [2]
Assiut University | Faculty of computers and information 28
GNN 2023 / 2024
29. Assiut University | Faculty of computers and information 29
Graph Machine Learning: Recent Trending
GNN 2023 / 2024
30. Assiut University | Faculty of computers and information 30
GNN 2023 / 2024
GNN
Foundations
▪ Why Graphs
▪ A Brief History of GNNs
▪ GNNs: Foundations and Models
31. Types of Graph Neural Networks Tasks
Assiut University | Faculty of computers and information 31
GNN 2023 / 2024
32. Machine ( Deep ) Learning with Graphs
Assiut University | Faculty of computers and information 32
GNN 2023 / 2024
33. Assiut University | Faculty of computers and information 33
Graph Representation Learning
Chapter 2
GNN 2023 / 2024
34. Assiut University | Faculty of computers and information 34
GNN 2023 / 2024
1 Motivation Graph Representation
2
Traditional graph Embedding
3
Modern graph Embedding
4
Graph neural networks
Graph Representation Learning Methods
35. Motivation for graph Representation
Assiut University | Faculty of computers and information 35
GNN 2023 / 2024
▪ Graph data is often sophisticated and thus is challenging to deal with.
➢ How to represent graphs concisely so that advanced analytic tasks, such as pattern discovery, analysis, and
prediction, can be conducted efficiently in both time and space.
36. Graph Representation Learning Methods | 1. Motivation to graph Representation
Assiut University | Faculty of computers and information 36
GNN 2023 / 2024
▪ Traditionally, we usually represent a graph as G = (V ,E )
➢ V : is a node set.
➢ E : is an edge set.
▪ For large graphs, The traditional graph representation poses several challenges to graph processing and
analysis.
37. Graph Representation Learning Methods | 1. Motivation to graph Representation
Assiut University | Faculty of computers and information 37
GNN 2023 / 2024
High computational
complexity
➢ To compute a distance, by finding
the shortest path or average path
length , using the traditional
graph representation, Such
methods result in high
computational complexity that
prevents them from being
applicable to large-scale real
world graphs.
Low parallelizability
➢ The bottleneck is that nodes in a
graph are coupled to each other
explicitly reflected by E.
➢ Thus, distributing different nodes
in different shards or servers often
causes demandingly high
communication cost among
servers, and holds back speed-up
ratio.
Inapplicability of machine
learning methods
➢ most of the off-the-shelf machine
learning methods may not be
applicable . Since it assume that
data samples can be represented by
independent vectors in a vector
space, while the samples in graph
data (i.e., the nodes) are dependent
to each other to some degree.
38. Graph Representation Learning Methods | 1. Motivation to graph Representation
Assiut University | Faculty of computers and information 38
GNN 2023 / 2024
▪ To tackle these challenges, substantial effort has been committed to develop novel graph representation
learning, i.e., learning the dense and continuous low-dimensional vector representations for nodes, so
that the noise or redundant information can be reduced and the intrinsic structure information can be
preserved.
▪ In the learned representation space,
▪ the relationships among the nodes, which were originally represented by edges or other high-order
topological measures in graphs, are captured by the distances between nodes in the vector space,
▪ and the structural characteristics of a node are encoded into its representation vector.
39. Two goals for graph representation learning
Assiut University | Faculty of computers and information 39
GNN 2023 / 2024
What are the main Goals for The new representation space So it well supporting graph analysis tasks
1
➢ First, the original graph can be reconstructed
from the learned representation space.
➢ It requires that, if there is an edge or
relationship between two nodes, then the
distance of these two nodes in the
representation space should be relatively small.
2
➢ Second, the learned representation space can
effectively support graph inference.
➢ such as predicting unseen links, identifying
important nodes, and inferring node labels.
40. Two goals for graph representation learning
Assiut University | Faculty of computers and information 40
GNN 2023 / 2024
▪ Note That : a representation space with only the goal of graph reconstruction is not sufficient for graph
inference.
▪ After the representation is obtained, downstream tasks such as node classification , node clustering , graph
visualization and link prediction can be dealt with based on these representations.
▪ There are three main categories of graph representation learning methods:
1. Traditional graph embedding
2. Modern graph embedding
3. Graph neural networks
41. Traditional Graph Embedding
Assiut University | Faculty of computers and information 41
GNN 2023 / 2024
▪ Traditional graph embedding methods were originally studied as dimension-reduction
techniques.
▪ The objective functions of traditional graph embedding methods mainly target the goal
of graph reconstruction. (The first Goal)
43. Traditional Graph Embedding
Assiut University | Faculty of computers and information 43
GNN 2023 / 2024
▪ You can think of Linear dimensionality reduction as a projection of your dataset down
into Linear sub-scales.
Image source: https://builtin.com/data-science/step-step-explanation-principal-component-analysis
Linear dimensionality reduction Non-Linear dimensionality reduction
44. Traditional Graph Embedding
Assiut University | Faculty of computers and information 44
GNN 2023 / 2024
▪ First construct a neighborhood graph G using connectivity algorithms such as K nearest neighbors (KNN).
▪ Then based on G, the shortest path between different data can be computed.
▪ Consequently, for all the N data entries in the data set, we have the matrix of graph distances.
▪ Finally, the classical multidimensional scaling (MDS) method is applied to the matrix to obtain the
coordinate vectors.
▪ The representations learned by Isomap approximately preserve the geodesic distances of the entry pairs in
the low-dimensional space.
▪ The key problem of Isomap is its high complexity due to the computing of pair-wise shortest paths.
46. Traditional Graph Embedding
Assiut University | Faculty of computers and information 46
GNN 2023 / 2024
▪ Locally linear embedding (LLE) is proposed to eliminate the need to estimate the pairwise
distances between widely separated entries.
▪ Laplacian eigenmaps (LE).
▪ Furthermore, the locality preserving projection (LPP), a linear approximation of the
nonlinear LE, is proposed
➢ Note that: traditional graph embedding mostly works on graphs constructed from
feature-represented data sets, where the proximity among nodes encoded by the edge
weights is well-defined in the original feature space.
47. Modern Graph Embedding
Assiut University | Faculty of computers and information 47
GNN 2023 / 2024
▪ modern graph embedding mostly works on naturally formed networks, such as social
networks, biology networks, and e-commerce networks.
▪ In those networks, the proximities among nodes are not explicitly or directly defined.
▪ For example, an edge between two nodes usually just implies there is a relationship
between them, but cannot indicate the specific proximity.
▪ Also, even if there is no edge between two nodes, we cannot say the proximity between
these two nodes is zero.
48. Modern Graph Embedding
Assiut University | Faculty of computers and information 48
GNN 2023 / 2024
▪ In Modern Graph Embedding Techniques, the definition of node proximities depends on
specific analytic tasks and application scenarios.
▪ Therefore, modern graph embedding usually incorporates rich information, such as
network structures, properties, side information, and advanced information.
▪ Note that: Modern graph embedding targets both goals of graph representation.
49. Modern Graph Embedding
Assiut University | Faculty of computers and information 49
GNN 2023 / 2024
▪ Modern graph embedding considers much richer information in a graph.
▪ According to the types of information that are preserved in graph representation learning,
the existing methods can be categorized into three categories
1
Graph structures and properties preserving graph embedding
2
Graph representation learning with side information
3
Advanced Information-preserving graph embedding
50. Modern Graph Embedding
Assiut University | Faculty of computers and information 50
GNN 2023 / 2024
Graph structures and properties preserving graph embedding
➢ Many network embedding methods aim to preserve the local structure of a node,
including neighborhood structure, high-order proximity as well as community
structure.
51. Modern Graph Embedding
Assiut University | Faculty of computers and information 51
GNN 2023 / 2024
Graph representation learning with side information
➢ Besides graph structures, side information is another important information source
for graph representation learning.
➢ Side information in the context of graph representation learning can be divided into
two categories: node content and types of nodes and edges
52. Modern Graph Embedding
Assiut University | Faculty of computers and information 52
GNN 2023 / 2024
Advanced Information-preserving graph embedding
➢ The advanced information refers to the supervised or pseudo-supervised information
in a specific task.
➢ The advanced information-preserving network embedding usually consists of two
parts:
1. One is to preserve the network structure to learn the representations of nodes.
2. The other is to establish the connection between the representations of nodes
and the target task.
54. References
Assiut University | Faculty of computers and information 54
[1] https://youtu.be/W7ZM2U2ds5M
[2] https://youtu.be/EJQnvefwHes
[3] A Comprehensive Introduction to Graph Neural Networks (GNNs) | DataCamp
[4] A Gentle Introduction to Graph Neural Networks (distill.pub)
[5]
GNN 2023 / 2024
55. Assiut University | Faculty of computers and information 55
Thank You
GNN 2023 / 2024
