Fragment-based Pretraining and Finetuning on Molecular Graphs

1. Van Thuy Hoang
Network Science Lab
Dept. of Artificial Intelligence
The Catholic University of Korea
E-mail: hoangvanthuy90@gmail.com
240302
Kha-Dinh Luong et.al., NIPS23

2. 2
Graph Convolutional Networks (GCNs)
 Generate node embeddings based on local network neighborhoods
 Nodes have embeddings at each layer, repeating combine messages
from their neighbor using neural networks

3. 3
Higher-order Graph Neural Networks
 Higher-order Graph Neural Networks | Semantic Scholar

4. 4
Higher-order structural arrangements
 Motif-based or fragment-based pretraining is a new direction that
potentially overcomes these problems.
 Existing fragment-based methods use either suboptimal
fragmentation or fragmentation embeddings.
 GROVER predicts fragments from node and graph embeddings,
however, their fragments are k-hop subgraphs that cannot
account for chemically meaningful subgraphs with varying sizes
and structures.

5. 5
Molecule Fragmentation
 Fragment-based contrastive pretraining framework
Principal Subgraph Mining
Molecule generation by principal subgraph mining and assembling. NIPS, 2022.

6. 6
Principal Subgraph Extraction
 Given a graph G = (V, E), a subgraph of G is defined as S
 Fragment extraction on {C=CC=C, CC=CC, C=CCC}.
(a) Initialize vocabulary with atoms.
(b) Fragment CC is the most frequent and added to the vocabulary. All
CC are merged and highlighted in red.
(c) Fragment C=CC is the most frequent and added to the vocabulary.
All C=CC are merged and highlighted in green (molecules 1 and 3).
After 2 iterations the vocabulary is {C, CC, C=CC}.

7. 7
Fragment-based Contrastive Pretraining
 To obtain the collective embedding of atom nodes corresponding to
a fragment, we define a function FRAGPOOL(·) that combines node
embeddings
 FRAGPOOL: average function in the experiments

8. 8
contrastive learning objective
 We minimize the contrastive learning objective based on the InfoNCE
loss

9. 9
Fragment-based predictive pretraining (task 1)
 A multi-label prediction task that outputs a vocabulary-size binary
vector indicating which fragments exist in the molecular graph.
 Thanks to the optimized fragmentation procedure that we use, the
output dimension is compact without extremely rare classes or
fragments, resulting in more robust learning.

10. 10
Fragment Graph Structure Prediction (task 2)
 predict the structural backbones of fragment graphs.
 The number of classes is the number of unique structural backbones.
Essentially, a backbone is a fragment graph with no node or edge
attributes.
 With predictive objective of each task.

11. 11
Experimental Settings
 Dataset:
 a processed subset containing 456K molecules from the ChEMBL
database
 A fragment vocabulary of size 800 is extracted
 Models : 5-layer Graph Isomorphism Network (GIN)

12. 12
On binary molecular property prediction
 Test ROC-AUC on binary molecular property prediction benchmarks
using different pretraining strategies in GraphFP
 C, P, and F indicate contrastive pretraining, predictive pretraining, and
inclusion of fragment encoders in downstream prediction

13. 13
On Long-range Chemical Benchmarks
 Performances on PEPTIDE-FUNC (graph classification) and PEPTIDE-
STRUCT (graph regression).
 These tasks require capturing long-range interactions within large
peptide molecules.

14. 14
On vocabulary of various sizes
 Downstream performances with GINs pretrained on vocabulary of
various sizes.

15. 15
Conclusions and Future Work
 contrastive and predictive learning strategies for pretraining GNNs
based on graph fragmentation
 pretrain two separate encoders for molecular graphs and fragment
graphs, thus capturing structural information at different resolutions.
 When benchmarked on chemical and long-range peptide datasets,
The method achieves competitive or better results compared to
existing methods.
 pretraining via larger datasets, more extensive featurizations, better
fragmentations, and more optimal representations.

17. Van Thuy Hoang
Network Science Lab
Dept. of Artificial Intelligence
The Catholic University of Korea
E-mail: hoangvanthuy90@gmail.com
240302
Namkyeong Lee et.al., ICLR2023

18. 18
BACKGROUND
 Molecular Relational Learning
 Learning the interaction behavior between a pair of molecules
 Examples
 Predicting optical properties when a Chromophore and Solvent
react
 Predicting solubility when a solute and solvent react
 Predicting side effects when taking two types of drugs
simultaneously

19. 19
Functional Group
 Specific atomic groups that play an important role in determining
the chemical reactivity of organic compounds
 Compounds with the same functional group generally have similar
properties and undergo similar chemical reactions
 Hence, it is important to consider functional group for molecular
relational learning

20. 20
Functional Group
 Specific atomic groups that play an important role in determining
the chemical reactivity of organic compounds
 Compounds with the same functional group generally have similar
properties and undergo similar chemical reactions
 Molecule can be represented as a graph Functional group can be
represented as a subgraph

21. 21
INFORMATION BOTTLENECK
 A theoretical approach to trade-off between information compression
and preservation

22. 22
Information Bottleneck Graph
 Subgraph that maximally preserves the property of the original
graph
 Motif in ordinary graphs
 Functional group in molecules

23. 23
Extract a subgraph in terms of nodes
 Inject noise into node embeddings to perform graph compression

24. 24
Conditional Graph Information Bottleneck
 Consider Graph 2 (Solvent) when detecting the important subgraph
from Graph 1 (Solute)
Graph Information Bottleneck
Conditional Graph Information Bottleneck

25. 25
CONDITIONAL GRAPH INFORMATION BOTTLENECK
 Overall procedure
 Decompose the conditional MI based on
the chain rule of MI, and then derive the
upper bound of the decomposed terms

26. 26
A
 A

27. 27
EXPERIMENTS
 Chromophore
 dataset Absorption max,
Emission max, Lifetime
 Solvation Free Energy
dataset:
 MNSol
 FreeSolv
 CompSol
 Abraham – CombiSolv
 Drug-Drug Interaction
dataset
 ZhangDDI
 ChChMiner

28. 28
MAIN TABLE
 Observations
Outperforms baselines on both Molecular Interaction / Drug-Drug
Interaction tasks

29. 29
SENSITIVITY ANALYSIS
 β =1.0:
 CGIB focuses on compression e.g., CGIB focuses an aromatic ring,
which is not relevant to chemical reactions
 β = 0.01:
 CGIB focuses on prediction e.g., CGIB focuses on external part,
which generally more relevant to chemical reactions

30. 30
QUALITATIVE ANALYSIS
 Observations:
 (a) Chromophore
 interact with ordinary solvents
 Focus on external parts à Aligns with domain knowledge
 (b) Chromophore interact with liquid oxygen solvents : Focus on
all parts à Aligns with domain knowledge

31. 31
CONCLUSION
 Proposed a method for tackling relation learning tasks, which are
crucial for scientific discovery
 Based on Conditional Information Bottleneck
 It is crucial to consider Graph 2 (Solvent) when detecting the
important subgraph from Graph 1 (Chromophore)