EuroQSAR2018, September 19, 2018 Thessaloniki, Greece

1. International Computational Collaborations
to Solve Toxicology Problems
Kamel Mansouri
EuroQSAR
September 19, 2018
Disclaimer: ILS staff provide technical support for NICEATM,
but do not represent NIEHS, NTP, or the official positions of any federal agency.

2. • Projects and background
• International collaborators
• Modeling efforts
– Endpoints
– Available data and preparation for modeling
– Establishing training, evaluation, and prediction sets
– Collaborators contributions
• Consensus modeling
– Combining predictions
– Analysis of the consensus predictions
– Implementation of the consensus models to screen new chemicals
• Conclusions
Outline

3. Collaborative projects
Mansouri et al. (http://ehp.niehs.nih.gov/15-10267/)
Mansouri et al. (Internal review)
CoMPARA
Collaborative Modeling Project for
Androgen Receptor Activity (2017/18)
CATMoS
Collaborative Acute Toxicity Modeling
Suite (2017/18)
Mansouri et al. (in preparation)
Endocrine Disruptor Screening Program
(EDSP)
ICCVAM
NICEATM
Acute Toxicity Workgroup: alternative methods
Interagency Coordinating Committee on the Validation of Alternative Methods
NTP Interagency Center for the Evaluation of Alternative Toxicological Methods

4. ER & AR screening: CERAPP & CoMPARA
• Concern over environmental chemical disruption of
endocrine hormone signaling
• U.S. Congress mandated that the EPA screen chemicals for
their potential to be endocrine disruptors
• This led to the development of the Endocrine Disruptor
Screening Program (EDSP)
• Cost ~$1,000,000 per chemical
• Throughput is ~50 chemicals / year
• Total cost of Tier 1 is billions of dollars and will take 100
years at the current rate

5. Computational Toxicology
Too many chemicals to test with standard
animal-based methods
–Cost (~$1,000,000/chemical), time, animal welfare
–10,000 chemicals to be tested for EDSP
–Fill the data gaps and bridge the lack of knowledge
Alternative

6. International Collaborators
Over 100 scientists from
different locations around
the globe.

7. • Participants from around the globe representing
academia, industry, and government contributed
International Collaboration
Interactive map:
https://batchgeo.com/map/9d3ff810a72
d8a84093c74ab0601f01d

8. Plan of the projects
Steps Tasks
1: Training and prioritization sets
Organizers
- ToxCast assays for training set data
- AUC values and discrete classes for continuous/classification
modeling
- QSAR-ready training set and prioritization set
2: Experimental validation set
Organizers
- Collect and clean experimental data from the literature
- Prepare validation sets for qualitative and quantitative models
3: Modeling & predictions
All participants
- Train/refine the models based on the training set
- Deliver predictions and applicability domains for evaluation
4: Model evaluation
Organizers
- Evaluate the predictions of each model separately
- Assign a score for each model based on the evaluation step
5: Consensus modeling
Organizers
- Use the weighting scheme based on the scores to generate the
consensus
- Use the same validation set to evaluate consensus predictions
6: Manuscript writing
All participants
- Descriptions of modeling approaches for each individual
model
- Input of the participants on the draft of the manuscript

9. Tox21/ToxCast Pathway Models
Judson et al Toxicol. Sci. (2015) 148: 137-154 Kleinstreuer N. C. et al. 2017 30 (4), 946-964.
Tox21/ToxCast ER Pathway Model Tox21/ToxCast AR Pathway Model
18 assays 11 assays

10. ER & AR combined assay data
AUC=0.1
Equivalent to
AC50=100 uM
ER training data AR training data
Active Inactive Total Active Inactive Total
Binding 237 1440 1677 198 1464 1662
Agonist 219 1458 1677 43 1616 1659
Antagonist 41 1636 1677 159 1366 1525
Total 237 1440 1677 198 1648 1688
Judson et al Toxicol. Sci. (2015) 148: 137-154

11. QSAR-ready KNIME workflow
Remove of
duplicates
Normalize of
tautomers
Clean salts and
counterions
Remove inorganics
and mixtures
Final inspection
QSAR-ready
structures
Indigo
Aim of the workflow:
• Combine different procedures and ideas
• Minimize the differences between the structures used for
prediction
• Produce a flexible free and open source workflow to be
shared
Structure standardization procedure
Mansouri et al. (http://ehp.niehs.nih.gov/15-10267/)
Fourches et al. J Chem Inf Model, 2010, 29, 476 – 488
Wedebye et al. Danish EPA Environmental Project No. 1503, 2013

12. Chemicals for Prediction:
Chemical lists of interest to the US EPA
• EDSP Universe (10K)
• Chemicals with known use (40K) (CPCat
& ACToR)
• Canadian Domestic Substances List
(DSL) (23K)
• EPA DSSTox – structures of EPA/FDA
interest (15K)
• ToxCast and Tox21 (In vitro ER data) (8K)
32,464 unique QSAR-ready structures
• CERAPP list: ~32k structures
• EINECS: European INventory
• ~60k structures
• ~55k QSAR-ready structures
• ~38k non overlapping with the
CERAPP list
• ~18k overlap with DSSTox
• ToxCast metabolites: ~6k unique
structures
47,888 + 6592 = 55,450 QSAR ready
structures
CoMPARA Prediction setCERAPP Prediction set

13. Experimental data for evaluation set
a) Tox21: ~8k chemicals
b) FDA EDKB DB: ~8k
chemicals;
c) METI DB: ~2k chemicals
d) ChEMBL DB: ~2k
chemicals
~60k experimental values for ~15k chemicals
Mansouri et al. (2016) EHP 124:1023–1033
CERAPP validation set CoMPARA validation set
Mansouri et al. (in preparation)
1.2 million assay records
2.1 million chemical structures
10 thousand protein targets
~80K experimental values for ~11k chemicals
ER validation data AR validation data
Active Inactive Total Active Inactive Total
Binding 1982 5301 7283 453 3429 3882
Agonist 350 5969 6319 167 4672 4839
Antagonist 284 6255 6539 355 3685 4040
Total 2017 7024 7522 487 4928 5273

14. Received models for both projects
Validation of the models:
CERAPP participants models CoMPARA participants models
Categorical Continuous Total Categorical Continuous Total
Binding 21 3 24 35 5 40
Agonist 11 3 14 21 5 26
Antagonist 8 2 10 22 3 25
Total 40 8 48 78 13 91

15. Coverage of the received models
Distributions of the number of the predicted chemical structures by all binding models.
CERAPP models CoMPARA models

16. Consensus models assessment
Binding Agonist Antagonist
Train Test Train Test Train Test
Sn 0.93 0.58 0.85 0.94 0.67 0.18
Sp 0.97 0.92 0.98 0.94 0.94 0.90
BA 0.95 0.75 0.92 0.94 0.80 0.54
CERAPP CoMPARA
Active Inactive Active Inactive
Binding 4001 28463 8202 40656
Agonist 2475 29989 1764 47094
Antagonist 2793 29671 9899 38959
Total 4001 28463 10623 47613
ToxCast metabolites
Active Inactive
Binding 1609 4983
Agonist 428 6164
Antagonist 1820 4772
Total 1989 6325
Binding Agonist Antagonist
Train Test Train Test Train Test
Sn 0.99 0.69 0.95 0.74 1.00 0.61
Sp 0.91 0.87 0.98 0.97 0.95 0.87
BA 0.95 0.78 0.97 0.86 0.97 0.74
CERAPP consensus
CoMPARA consensus
CoMPARA predictions

17. Goal: help with regulatory purposes

18. • ICCVAM is developing alternative test methods for the EPA’s
six pack tests: Acute oral, dermal, inhalation, eye & skin
irritation and skin sensitization
• Acute Toxicity Workgroup: identifies federal agency
requirements, needs, and decision contexts for using acute
systemic toxicity data
Acute Oral Toxicity: CATMoS

19. Agency-Based Modeling Endpoint Selection
Hazard
Toxic
(>50-5000 mg/kg)
Highly toxic
(≤50 mg/kg)
I (≤ 50 mg/kg)
II (>50 ≤ 500 mg/kg)
III (>500 ≤ 5000 mg/kg)
IV (>5000 mg/kg)Hazard
EPA Categories
Point estimates of
LD50 values
+ Nontoxic (>2000 mg/kg)
I (≤ 5 mg/kg)
II (>5 ≤ 50 mg/kg)
III (>50 ≤ 300 mg/kg)
IV (>300 ≤ 2000 mg/kg)
HazardPacking
Group
GHS Categories
NC (> 2000 mg/kg)
Binary Models
Categorical Models
Continuous Model

20. Require QSAR-ready structure
Desalted, stereochemistry stripped,
tautomers and nitro groups standardized,
valence corrected, structures neutralized
Data Sources and Inventory
Rat Oral Acute Toxicity LD50 Database
Database Resource
Rows of Data
(number of
LD50 values)
Unique
CAS
ECHA (ChemProp) 5,533 2,136
JRC AcutoxBase 637 138
NLM HSDB 3,981 2,205
OECD (eChemPortal) 10,119 2,290
PAI (NICEATM) 364 293
TEST (NLM ChemIDplus) 13,069 12,974
15,688 chemicals
21,200 LD50 values
Rat oral LD50s:
16,297 chemicals total
34,508 LD50 values
Require unique
LD50 values
with mg/kg units
11,992 chemicals
16,209 LD50 values

21. • 11,992 chemicals from the final inventory of chemicals with
QSAR-ready structures having rat oral acute toxicity data
were split into training and test sets:
– 75% training set: 8,994 chemicals
– 25% evaluation set: 2,998 chemicals
• The training set was provided to modelers
– Encouraged to consider different modeling approaches to ensure
optimum predictivity without restrictions or recommendations (QSAR,
machine learning approaches, global/local modeling, etc.)
– Models could include any variety of data inputs (chemical features and
structure classes, physiochemical properties, chemical use categories, etc.)
For Each of the 5 Modeling Endpoints Selected:
Establishing Modeling Dataset

22. • The evaluation set (2,998 chemicals) was hidden
within a larger “prediction set” (48,137 chemicals)
provided to modelers
– Upon submission, data were evaluated:
• Evaluation set predictions were used for quantitative evaluation of
predictivity, goodness of fit, and overall robustness
• QMRF-based documentation provided for the modeling approach
was evaluated for qualitative evaluation to ensure transparency
• Later, predicted values from the prediction set were
used to generate the “consensus predictions”
Evaluation and Prediction Sets
Establishing Modeling Dataset

23. Submitted Models
Consortium Comprised 35 Participants/Groups
• Very Toxic: 32 models
• Non-toxic: 33 models
• EPA categories: 26 models
• GHS categories: 23 models
• LD50: 25 models
Total: 139 models

24. Distribution of the number of models/chemical
Coverage of the models

25. • Models passing qualitative evaluation (requirement for
transparency; description of approach was sufficient)
• Integrating only in-domain predictions across
chemicals in the prediction set (48,137 chemicals) for
each model, respectively
– Categorical models: weighted majority rule
– Continuous model: weighted average
Generation of Consensus Predictions
Kleinstreuer et al., Predictive models for acute oral systemic toxicity: A workshop to
bridge the gap from research to regulation, Computational Toxicology, 2018,
https://doi.org/10.1016/j.comtox.2018.08.002.

26. Consensus Model Statistics
Performance Assessment
Very Toxic Non-Toxic EPA GHS
Train Eval Train Eval Train Eval Train Eval
Sensitivity 0.87 0.67 0.93 0.70 0.73 0.50 0.63 0.45
Specificity 0.94 0.96 0.96 0.88 0.96 0.91 0.91 0.92
Balanced
Accuracy
0.93 0.81 0.94 0.79 0.83 0.71 0.77 0.68
In vivo
Balanced
Accuracy
0.81 0.89 0.82 0.79
The consensus
predictions perform just
as well as replicate in vivo
data do at predicting oral
acute toxicity outcome
LD50 values LD50 values
Train Eval In Vivo
R2 0.84 0.64 0.80
RMSE 0.32 0.51 0.42

27. Distributions of the concordance between models
Consensus concordance

28. Desktop and Online Resources
Access to Consensus Predictions
EPA CompTox dashboard:
https://comptox.epa.gov/dashboard/
Standalone OPERA app:
batch mode for new chemicals
https://github.com/kmansouri/OPERA
(Mansouri et al. J Cheminform (2018).
https://doi.org/10.1186/s13321-018-0263-1)

29. OPERA prediction report on EPA’s CompTox dashboard
Calculation Result
for a chemical Model Performance
with full QMRF
Nearest Neighbors from Training
Set
Mansouri et al. OPERA models
(https://link.springer.com/article/10.1186/s13321-018-0263-1)

30. • Success of the project was due in great part to the use
of the 3C concept as well as up-front and continuous
engagement of regulators in the process
The “3C” Concept at Work!
https://ntp.niehs.nih.gov/go/natl-strategy
Communication Collaboration Commitment

31. Thank you for your attention!
• EPA/ORD, Oak Ridge Institutes through U.S. DoE & EPA.
• The Lush Prize 2017, young researchers, supporting animal free testing.
• ILS/NICEATM under NIEHS contract HHSN273201500010C
Acknowledgements
• EPA/NCCT
• ICCVAM Acute Toxicity Workgroup
• NTP/NICEATM
• ILS
• ScitoVation
• All international collaborators
Funding

32. From QSAR to New Approach Methodologies (NAM)
Critical use of QSAR and other in silico approaches for new applications and regulatory assessment
QSAR 2020
early June 2020
Organizers/sponsorsLocation: RTP, NC, USA
Host: US Environmental Protection Agency
Industry
Academia
Animal
Welfare
Agencies