Michelle Angrish presented on using systematic review methods and tools to evaluate new approach methods (NAMs). She described the systematic review workflow, including problem formulation, literature screening and tagging, data extraction, and interactive displays of evidence. She provided a case study on using these methods to identify potential developmental toxicants from the zebrafish literature, which would then be evaluated in mammalian studies. Machine learning tools could help prioritize large amounts of literature and identify relevant studies for further review. Integrating systematic review data with adverse outcome pathway ontologies may help map review findings.

1. Michelle Angrish, PhD
U.S. EPA
Grounding NAMs Using
Systematic Review Methods and Tools
8/9/2020 APCRA 2019 1
The views and opinions expressed here are mine alone and do not reflect official US Environmental Protection Agency policy.

2. What we are
trying to
accomplish
Explain the concept of SR methods and modules
Describe the systematic review workflow and
customization
Case study: New Approach Methods and SR modules to
identify developmental toxicants
Identify other aspects for consideration:
FAIR Principles and as training sets in artificial intelligence
Improvements and/or problems?
8/9/2020 APCRA 2019 2

3. Systematic
Review:
A structured
and
documented
process for
transparent
literature
review1
“As defined by IOM [Institute
of Medicine], systematic
review ‘is a scientific
investigation that focuses on
a specific question and uses
explicit, pre-specified
scientific methods to
identify, select, assess, and
summarize the findings of
similar but separate
studies.’” [p. 4] (NRC, 2014)
APCRA 2019 3
1 Institute of Medicine. Finding What
works in Health Care: Standards for
Systematic Reviews. p.13-34. The
National Academies Press. Washington,
D.C. 2011

4. 8/9/2020 APCRA 2019 4
Experimental Process and Design in a Literature Based Assessment

5. 8/9/2020 APCRA 2019 5

6. Systematic evidence
map (also called
systematic map or
scoping review) is a
pre-decisional
systematic review
analysis that compiles
and summarizes
evidence but does NOT
reach assessment
conclusions
• summarize evidence base characteristics to…
Summarize
• Allocate resources depending on step-wise scoping,
planning and problem formulation;
Allocate
• Prioritize reference chemicals with a range of
potencies for model and test guideline development;
Prioritize
• Identify data gaps for new test method development.
Identify

7. Evidence map core contents
8/9/2020 APCRA 2019 7

8. 8/9/2020 APCRA 2019 8
Problem Formulation and Scoping
• Core content customizable based on background
and Agency need, exposure context, objectives
and specific aims, key areas of scientific
complexity
Literature Screening and Tagging
• Automated tagging
• Prefilter/Prioritize
• Active screening
Data Extraction
• Chemical form
• Evidence type
• Health outcome and endpoints
Validation
• QC
• Study quality evaluation
Interactive Displays
• Tableau
• HAWC
• Chemistry dashboard
Evidence maps to inform a diverse array of potential literature review products (including toxicity
assessments), as well as other decisions (e.g., prioritization; resource-planning, etc.).
Values
• Reference values
• Other toxicity values
• Screening level/bounding estimates

9. Evidence maps are useful for problem formulation and scoping,
prioritization, identifying need for assessment update, resource
allocation, identifying data gaps

10. Deeper dive
108/9/2020 APCRA 2019
• Data extraction downloadable
• Summary level information available
in the Chemistry Dashboard

11. Semantic Challenge
Natural language – Variation in natural language presents significant identification and
interpretational challenges
• Highly variable
• Synonyms – different words for the same thing
• Homographs/polysemes – different words for the same thing
• Dialects – context matters!
Need for common language
• Thesaurus
• Controlled vocabulary
What is required in addition to controlled vocabularies and thesauri are ontologies.

12. Goal: Use previous knowledge and SR methods to evaluate and develop
AOP networks
and nodes and their relationships
8/9/2020 Theme 2: Literature-based AOP development 12
Evidence MapPrevious knowledge
MIE AO
Natural language
Controlled
Vocabularies
Points of integration
Ontologies
Points of integration
MIE AO

13. Ontologies that describe key events in AOPs
8/9/2020 APCRA 2019 13

14. 8/9/2020 APCRA 2019 14
Problem Formulation and Scoping
• Core content customizable based on background
and Agency need, exposure context, objectives
and specific aims, key areas of scientific
complexity
Literature Screening and Tagging
• Automated tagging
• Prefilter/Prioritize
• Active screening
Data Extraction
• TBD
Validation
• QC
• Study quality evaluation
Interactive Displays
• Tableau
Case Study: identify developmental toxicants using the
zebrafish literature
Values
• NA

15. Goal: identify potential developmental
toxicants using the zebrafish literature
• Use systematic review
methods to identify
potential developmental
toxicants (chemicals) from
the zebrafish literature.
• These chemicals would be
used in a new systematic
review as the “Population”
using mammalian literature
search strings.
8/9/2020 15

16. What is
wrong with a
systematic
review
including
1,434
chemicals?
Finding chemical synonyms,
creating search strings, and
doing a database search for
1,434 chemicals in mammals
is NOT practical.
Leverage Artificial
Intelligence to strategically
approach the problem
considering constraints.
16

17. Workflow:
8/9/2020 17
Prioritize
• Prioritize the literature by health outcome (developmental).
Prioritize
• Tag Literature for the 1,434 chemicals identified from the Zebrafish ZET systematic review.
Prioritize
• Prioritize literature for the 1,434 chemicals
Train
• Introduce a training set from the EBTC mammalian systematic review for 75 chemicals
Priority Rank
• Priority rank the literature and use a test set to identify a threshold
Screen
• Screen prioritized literature using active learning SR Tool

18. Prioritize Literature
8/9/2020 18
Chemical
Health
Outcome
=Developmental
Chemical
tagging tool

19. Tag studies for 1,434
EBTC chemicals
Prioritize by Outcome
Prioritize by Chemical
49,911 studies 13,642 studies
Broad literature search
48,883 studies
1. Upload into
SWIFT-Review
2. Upload training
data and test set
Prioritization
machine assistance
for topic clustering,
chemical tagging,
evidence inventory,
and prioritization
Prioritize Mammalian Embryotoxicity Literature Using
Machine Assistance: SWIFT-Review
8/9/2020 19

20. 1. Seed the model training and
test data to priority rank
2. Use ranking performance (blue line)
and test set (green line) score to
prioritize studies for screening
3. Use lowest ranked “test” study as cut-
off, send studies scoring 0.42 and above
to screen8/9/2020 20

21. Active learning predicted 763 included studies
8/9/2020 21

22. Something to think about:
• What is the schema for mapping systematic
review forms (the screening decisions, tags,
and extracted information) and content in
the AOP database?
APCRA 2019 22

23. Acknowledgements
8/9/2020 APCRA 2019 23
Ingrid Druwe, Ph.D.
Janice Lee, Ph.D.
Kristan Markey, Ph.D.
Kellie Fay, Ph.D.
Michele Taylor, Ph.D.
Kris Thayer, Ph.D.
Sean Watford, Ph.D.
Andrea Kirk, Ph.D.
Paul Whaley, M.Litt. Katia Tsaioun
Sebastian Hoffman
Steve Edwards, Ph.D.