Translating data to predictive models

1. Translating data to
predictive models
Akos Tarcsay

2. Machine learning life-cycle

3. Trainer Engine

4. From data to prediction
Data ingestion Preprocessing Modelling
Features Models
Review Prediction
Model repository

5. Chemaxon
Descriptor
generation
Chemaxon
Standardizer
Persistence and
search
System overview
DB layer
PostgreSQL
Statistical
evaluation
ML library
(SMILE)
Conformal
prediction wrapper
Service layer
Programmatic
access
REST interface
“Comp Chem”
Trainer GUI
“Med Chem”
Prediction GUI

6. How to reduce noise?
Preprocessing

7. Effect of standardization
- Simple descriptors (Mw, fsp3,
HBDA, etc. )
Imipramine pamoate Furan-2-ol
- Phys-chem (logD, pKa)
- Molecular graph, Fingerprints
Salts, solvates Tautomerism
“Overall and despite our efforts to use open software wherever possible, we find that
ChemAxon Tautomers node outperforms the other approaches we tested.”
https://jcheminf.biomedcentral.com/articles/10.1186/s13321-022-00606-7

8. Small molecule retention time (SMRT) dataset: Tautomerization
https:/
/www.nature.com/articles/s41467-019-13680-7

9. SMRT Tautomer example, edge case
Training set: single tautomer random 7k cases
Test set: tautomerization affected 252 cases
Run with and without tautomerization

10. SMRT Tautomer effect

11. SMRT results
https:/
/www.nature.com/articles/s41467-019-13680-7
ChemAxon Trainer Engine Original publication
15k test compounds, R2
=0.9

12. Prediction power?
ChEMBL Benchmark

13. Activity dataset: the ‘ChEMBL bioactivity benchmark set’
Data source: Journal of Cheminformatics, 9, 45 (2017) by Eelke B. Lenselink, Niels
ten Dijke, Brandon Bongers, George Papadatos, Herman W. T. van Vlijmen, Wojtek
Kowalczyk, Adriaan P. IJzerman, Gerard J. P. van Westen
- ChEMBL database (version 20)
- Activities were selected that met the following criteria:
- at least 30 compounds tested per protein and from at least 2 separate publications
- assay confidence score of 9
- ‘single protein’ target type
- assigned pCHEMBL value
- no flags on potential duplicate or data validity comment
- originating from scientific literature
- data points with activity comments ‘not active’, ‘inactive’, ‘inconclusive’, and ‘undetermined’ were
removed
- MED value was chosen
https:/
/jcheminf.biomedcentral.com/articles/10.1186/s13321-017-0232-0

14. Application Study on ChEMBL
- Data points in range: 500-4703 (med:776)
- 161 ChEMBL targets, pAct
- Sorted by Document Year, last 30 points
reserved as External set: Ext
Last 30 Ext

15. Application Study on ChEMBL
- Data points in range: 500-4703 (med:776)
- 161 ChEMBL targets, pAct
- Sorted by Document Year, last 30 points reserved
as External set: Ext
- 10-90% test-training set split: Test
- ~160k total training size
- ~18k total test size
- 20 different descriptor configurations
- Random Forest
Rnd 90% Train
Last 30 Ext
Rnd 10% Test
...

16. Prediction power?
ChEMBL Regression

17. Benchmark on 161 ChEMBL targets, best single configuration
161 ChEMBL Targets

18. Analysis of the best models per target
Pearson R2
Ext Test Ext Test
Avg 0.672 0.824 0.306 0.679
Median 0.722 0.833 0.385 0.697

19. Prediction power?
ChEMBL Classification
FP
TP
TN
FN

20. Is it a hard task? Original results random split
https:/
/jcheminf.biomedcentral.com/articles/10.1186/s13321-017-0232-0
“The best method overall is the DNN_MC with an MCC of 0.57 (±0.07)”

21. Is it a hard task? Original results temporal split
https:/
/jcheminf.biomedcentral.com/articles/10.1186/s13321-017-0232-0

22. Classification case: class balance

23. Random Forest results
RF
MCC Ext Test
Avg 0.310 0.541
Median 0.282 0.589

24. Is it a hard task? Base model: MPNN
https:/
/keras.io/examples/graph/mpnn-molecular-graphs/
atom_featurizer = AtomFeaturizer(
allowable_sets={
"symbol": {"B", "Br", "C", "Ca", "Cl", "F", "H", "I", "N", "Na", "O", "P", "S"},
"n_valence": {0, 1, 2, 3, 4, 5, 6},
"n_hydrogens": {0, 1, 2, 3, 4},
"hybridization": {"s", "sp", "sp2", "sp3"},
}
)
bond_featurizer = BondFeaturizer(
allowable_sets={
"bond_type": {"single", "double", "triple", "aromatic"},
"conjugated": {True, False},
}
)

25. MCC reverse cumulative histogram
Random split Test case
RF MPNN
MCC Ext Test Ext Test
Avg 0.310 0.541 0.207 0.471
Median 0.282 0.589 0.178 0.493

26. Binned MCC evaluation random selected Test case
MCC<0.45: Low
0.45<MCC<0.65: Medium
0.65<MCC: High

27. Binned MCC evaluation random selected Test case
MCC<0.45: Low
0.45<MCC<0.65: Medium
0.65<MCC: High

28. MCC reverse cumulative histogram
Temporal split Ext case
RF MPNN
MCC Ext Test Ext Test
Avg 0.310 0.541 0.207 0.471
Median 0.282 0.589 0.178 0.493

29. Binned MCC evaluation temporal split Ext case
MCC<0.45: Low
0.45<MCC<0.65: Medium
0.65<MCC: High

30. Binned MCC evaluation temporal split Ext case
MCC<0.45: Low
0.45<MCC<0.65: Medium
0.65<MCC: High

31. How reliable?
Confidence
?

32. Conformal prediction
Proper
Training Set
Model Error model
Calibration
set
Error Prediction
Training Set
P(80%)
calibration
factor (ɑ)
https:/
/www.jmlr.org/papers/volume9/shafer08a/shafer08a.pdf
https:/
/pubs.acs.org/doi/10.1021/ci5001168

33. Conformal prediction
Proper
Training Set
Model Error model
Calibration
set
Error Prediction
Training Set
P(80%)
calibration
factor (ɑ)
Test: 14233 / 17661 80.6% within the error bound
Ext: 3344 / 4890 68.4% within the error bound

34. Feasible?
Performance

35. AVG prediction time per set
Descriptor: ECFP6_1024_MACCS_PHYSCHEM
Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz
~50 ms/ cpd
72k cpd /h

36. Drill down to the lowest performance set

37. Drill down to the lowest performance set
TC_key: CHEMBL247 - CHEMBL2373969
BIOACT_PCHEMBL_VALUE: 7.54
TGT_CHEMBL_ID: CHEMBL247
TGT_ORGANISM: Human immunodeficiency virus 1

38. Feature engineering is the
process of using domain
knowledge to extract
features from raw data.

39. Combined descriptor set
∑ 20 *161 =3220 descriptor sets

40. RF Importance (ECFP4_CHEMTERM:All)
Sum of node
impurity decreases
for each individual
variable over trees.
Importance based
rank is evaluated

41. RF Importance (ECFP4_CHEMTERM:All)
50% related to
protonation or
partitioning

42. https:/
/www.nature.com/articles/s41598-019-55886-1
Protonation de-tour: Fentanyl

43. https:/
/www.nature.com/articles/s41598-019-55886-1
Protonation de-tour: Fentanyl

44. Protonation de-tour: Fentanyl F-derivative: FF3
https:/
/www.nature.com/articles/s41598-019-55886-1
No difference at pH 6.5
10x MOR IC50
difference at pH 7.4

45. Statistical Assessment of the Modeling of Proteins and Ligands,
SAMPL6 challenge logP
https:/
/chemaxon.com/news/cxn-logp-prediction-sampl-6

46. SAMPL7 pKa
https://link.springer.com/article/10.1007/s10822-021-00397-3

47. SAMPL7 logD
https://link.springer.com/article/10.1007/s10822-021-00397-3

48. Classification use case:
Blood-Brain Barrier
Penetration

49. Blood brain barrier penetration model
MoleculeNet, 10.1039/c7sc02664a

50. Model analysis

51. Rich prediction results

52. Classification use case:
PAMPA Permeability

53. Data preparation
1) Source: PubChem BioAssay AID: 1508612
a) NCATS Parallel Artificial Membrane Permeability Assay (PAMPA) Profiling
b) Observed PAMPA at pH 7.4 (x 10-6 cm/sec)
2) Standardize (strip salts, tautomerize)
3) Mw 0-800 -> 2029 cases
4) Permeability 100 cutoff using “Phenotype” field
0:Low, Medium (646 cases)
1:High (1383 cases)

54. Clustering MACCS similarity matrix tSNE projection

55. Is it a hard task? Base model: MPNN
Epoch 40/40
Train: AUC: 0.7897 MCC: 0.3921
Validation: AUC: 0.5879 MCC: 0.0740
https:/
/keras.io/examples/graph/mpnn-molecular-graphs/
atom_featurizer = AtomFeaturizer(
allowable_sets={
"symbol": {"B", "Br", "C", "Ca", "Cl", "F", "H", "I", "N", "Na",
"O", "P", "S"},
"n_valence": {0, 1, 2, 3, 4, 5, 6},
"n_hydrogens": {0, 1, 2, 3, 4},
"hybridization": {"s", "sp", "sp2", "sp3"},
}
)
bond_featurizer = BondFeaturizer(
allowable_sets={
"bond_type": {"single", "double", "triple", "aromatic"},
"conjugated": {True, False},
}
)

56. MPNN: Epoch 40/40
Train: AUC: 0.7897 MCC: 0.3921
Validation: AUC: 0.5879 MCC: 0.0740

57. Classification use case:
hERG

58. Data source: Scientific Reports vol 9, 12220 (2019)
Keiji Ogura, Tomohiro Sato, Hitomi Yuki, Teruki Honma
- 9 890 hERG inhibitors (IC50 ≤ 10 μM or ≥50% inhibition at 10 μM)
- 281 329 inactive compounds (IC50 > 10 μM or <50% inhibition at 10
μM) according to their standardized chemical structures
- 204k training data points
- 87k test data points
- Descriptor set: ECFP6_2048_MACCS_PHYSCHEM
Classification model
https:/
/www.nature.com/articles/s41598-019-47536-3

59. hERG Classification Model
https:/
/www.nature.com/articles/s41598-019-47536-3
Chemaxon

60. CONVERTING DATA TO
PROJECT TEAM INSIGHTS

61. Machine learning life-cycle

62. Discovery teams
Design Landscape
Design Hub
Series
H1 H2 H3 H4
Discovery hub
connecting chemical
series, data,
predictions and
chemical project
management

63. Discovery teams
Analysis
Design Hub
Series
H1 H2 H3 H4
Biological
measurements

64. Discovery teams
Biological data focus
Design Hub
Series
Trainer GUI
Training /
Analysis
Comp. Chem
Trainer
Engine
H1 H2 H3 H4
Trainer
Engine

65. Discovery teams
Train&Deploy
Production
Models
Design Hub
Services Series
Trainer GUI
Training /
Analysis
Comp. Chem
Trainer
Engine
H1 H2 H3 H4
Trainer
Engine { }
REST
…
API
{ }
REST
…
API

66. Discovery teams
Fill the gap
Production
Models
Design Hub
Services Series
Trainer GUI
Training /
Analysis
Comp. Chem
Trainer
Engine
H1 H2 H3 H4
Trainer
Engine { }
REST
…
API
{ }
REST
…
API

67. Discovery teams
Multi parameter optimization
Production
Models
Design Hub
Services Series
Trainer GUI
Training /
Analysis
Comp. Chem
Trainer
Engine
H1 H2 H3 H4
Trainer
Engine

68. Local and global models

69. Local and global models

70. Translate data to reliable
models
Centralize model
management
Connect project team
members and resources
Track and manage discovery
Design Hub
Lower the barrier to adopt AI models in design
Trainer Engine

71. Take away
- Chemical standardization to reduce noise
- Role of protonation and partitioning descriptors
- Successful model building on large and diverse
set of targets
- ML inference, delivery to medicinal chemists

72. Interested?
atarcsay@chemaxon.com

73. Thank you