A Talk to the NIAID Data Sciences Seminar in May of 2021 on prioritizing research in response to emergency threats.

1. Nina Fefferman, Ph.D.
Math/EEB/NIMBioS
University of TN, Knoxville
Scientific Triage :
How to make strategic choices about
prioritizing basic science during
developing threats

2. Decisions in the Absence of Information
COVID-19 is the epitome of a “zero-day” threat
Great science started almost immediately
Medical studies – pharmaceutical/np-therapeutics, etiology, etc.
Virological studies – genomes, proteins, phylogenies, etc.
Immunological studies – vaccine development, antibody
responses, etc.
Policy makers needed to make decisions faster
than science could provide answers

3. Which Answers First? From Whom?
COVID-19: Scientists stepped up!
So many people said: “I can refocus on something relevant” and
did it
But almost instantly there was too much information
• Policy makers were confused
• Journals were overwhelmed
• Reporters were swamped
• Public was confused
A lot of insight was brilliant science, but not what we
needed to inform policies
Self-organization isn’t always the answer

4. Formal Risks Assessment as a Tool for
Triage in Coordinated Science
COVID-19 is an example, but the ideas are general
• Pandemics
• Novel Natural and Man-made Disasters
• Potentially harmful new exposures/ environments/
combinations of environments
Note: Triage is critical during sudden threat – it is
NOT a tool for calm, reasoned, long-term
planning (though it can identify which foundational ideas are needed)

5. Formal Risk Assessment for Science
Three main elements critical to any quantitative
risk assessment:
1) Vulnerability
2) Likelihood
3) Severity
These are the factors in planning against threats
We should use them to decide which scientific
questions to tackle first as threats emerge

6. Clarifying Definitions
1) Vulnerability: how defended against the
threat is the target
2) Likelihood: what is the probability that
the threat will occur
3) Severity: how bad is the outcome if the
threat successfully compromises the
target

7. Clarifying Examples
1) Vulnerability 2) Likelihood 3) Severity
Any two, without the third, leads to a potential
misallocation of concern and effort
Case a: Meteor Strike – 1 and 3 are very high,
luckily 2 is small

8. Clarifying Examples
1) Vulnerability 2) Likelihood 3) Severity
Any two, without the third, leads to a potential
misallocation of concern and effort
Case a: Meteor Strike – 1 and 3 are very high,
luckily 2 is small
Case b: Global Pandemic of Sniffles – 1 and 2 are
high, 3 is small

9. Clarifying Examples
1) Vulnerability 2) Likelihood 3) Severity
Any two, without the third, leads to a potential
misallocation of concern and effort
Case a: Meteor Strike – 1 and 3 are very high,
luckily 2 is small
Case b: Global Pandemic of Sniffles – 1 and 2 are
high, 3 is small
Case c: Extinction of all Pollinators – 2 is
moderate, 3 is HUGE, luckily 1 is small
(due to diversity of pollinators)

10. 1) Vulnerability 2) Likelihood 3) Severity
Case d: nuclear strike on NYC – 2 is large, 3 is
HUGE, luckily 1 is small (due to active surveillance and
intervention technology)
Without active surveillance and intervention, all
three are at least large and we would be very
worried!
These Factors Aren’t Immutable

11. Goal of splitting things up this way:
Figure out which of the 3 elements of risk (if
any) can be most effectively reduced and
consider putting our efforts there
Case a: Meteor Strike – 1 and 3 are very high, luckily 2 is small
Case b: Global Pandemic of Sniffles – 1 and 2 are high, 3 is small
Case c: Extinction of all Pollinators – 2 is moderate, 3 is HUGE,
luckily 1 is small
Case d: Nuclear Attack on NYC– 2 is moderate, 3 is HUGE, we
work to make 1 small
1) Vulnerability 2) Likelihood 3) Severity

12. Important note:
Public and Policy makers respond to perception –
sometimes we should educate about which things are
critical threats rather than address benign perceived risks
Case a: Meteor Strike – 1 and 3 are very high, luckily 2 is small
Case b: Global Pandemic of Sniffles – 1 and 2 are high, 3 is small
Case c: Extinction of all Pollinators – 2 is moderate, 3 is HUGE,
luckily 1 is small
Case d: Nuclear Attack on NYC– 2 is moderate, 3 is HUGE, we
work to make 1 small
1) Vulnerability 2) Likelihood 3) Severity

13. Still using COVID-19 as my example
1) Vulnerability 2) Likelihood 3) Severity
Vulnerability – what could COVID-19 cause to fail?
• Healthcare Resource Availability
• Safe Water Availability
• Economies Depending on In-Person Interaction
(e.g. service industries, communal production industries, etc.)
• Function of densely-populated venues of people
(e.g. schools, nursing homes, places of worship, sports arenas,
prisons, etc.)
• etc.
High
Very Low
Variable
High

14. Estimating these probabilities is itself
challenging
1) Vulnerability 2) Likelihood 3) Severity
Likelihood – how likely is it COVID-19 will cause it to fail?
• Healthcare Resource Availability
• Safe Water Availability
• Economies Depending on In-Person Interaction
(e.g. service industries, communal production industries, etc.)
• Function of densely-populated venues of people (e.g.
schools, nursing homes, places of worship, sports arenas, prisons,
etc.)
• etc.
High
Very Low
Moderate /
Variable
High

15. Models are critical tools for estimating
the unknown
1) Vulnerability 2) Likelihood 3) Severity
Severity – how bad will it be for someone/everyone if it fails?
• Healthcare Resource Availability
• Safe Water Availability
• Economies Depending on In-Person Interaction
(e.g. service industries, communal production industries, etc.)
• Function of densely-populated venues of people (e.g.
schools, nursing homes, places of worship, sports arenas, prisons,
etc.)
• etc.
Terrible
It Depends
It Depends
Terrible

16. How to pick a best first problem
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
High High Terrible
Very Low Very Low It Depends
Variable Moderate/Variable It Depends
High High Terrible

17. How to pick a best first problem
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
High High Terrible
Very Low Very Low It Depends
Variable Moderate/Variable It Depends
High High Terrible
It’s not clear yet!

18. What can we affect?
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
High High Terrible
Very Low Very Low It Depends
Variable Moderate/Variable It Depends
High High Terrible

19. What can we affect?
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
High High Terrible
Very Low Very Low It Depends
Variable Moderate/Variable It Depends
High High Terrible
Now we have some targets!

20. Some obvious options:
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
High High Terrible
Very Low Very Low It Depends
Variable Moderate/Variable It Depends
High High Terrible
Increase
capacity/
production
Decrease
density Stimulus and
Protection

21. Let’s focus on one
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
High High Terrible
Very Low Very Low It Depends
Variable Moderate/Variable It Depends
High High Terrible
Decrease
density

22. What are typical dense venues in
the US?
Schools – Became Remote really fast
Nursing homes – Couldn’t do anything, instead focused on
reducing Likelihood by prohibiting visitors
(not great, best we could do)
Places of worship – Shut Down / Became Remote really fast
Sports arenas – Shut Down really fast
Prisons – Oh Dear…

23. Prisons in the United States
0.7% of US CURRENTLY incarcerated
By Delphi234 - Own work, CC BY-SA 4.0,
https://commons.wikimedia.org/w/index.php?curid=41197475
Excludes
Federal
Incarceration

24. How to take first steps when we know
nothing yet? (a brief aside into quantitative models)
Data challenges for zero-day threats are
immense
• How do we know what’s happening?
• How do we estimate things we know happen,
but can’t measure?
• How can we make recommendations without
waiting to see how the threat/the science
plays out?

25. This is where models are most useful!
Usually, sensitivity of models is done
post hoc
For zero-day threats, that is
exactly backwards

26. Steps for building useful models
during emerging threats:
Step 1: Hypothesize your logic
Step 2: Build your model
Step 3: Make best guesses based on bad data
Step 4: Run a sensitivity analysis
Step 5: Can you improve data for the most
sensitive parameters?
Step 6: Make recommendations (with disclaimers)

27. Step 3 Example – COVID-19 Jails: Some of these were
impossible to know without huge amounts of data are
gathered from ‘natural experiments’ over time:
etc.

28. For these, run the model over ranges of options
and move on (i.e. Step 4)
etc.

29. Others are sensitive but can be estimated via
parallel sources:
China is definitely
not like the US
incarcerated
population
Use the best
approximation
available at the
time
(If none exists,
guess)

30. Step 6: Make Recommendations (with
disclaimers)
Our models informed
policies at the local, state,
and national level
Were referenced by NJ
Governor in COVID-19
carceral policy
Informed over 30 court
cases
Produced recommendations
in under a month that I
stand by after a year

31. During early 2020, I used this framework to define
what I would research to help the US COVID-19
response, but it’s MUCH more general than that
Step 1: Hypothesize your logic
Step 2: Build your model
Step 3: Make best guesses based on bad
data
Step 4: Run a sensitivity analysis
Step 5: Can you improve data for the most
sensitive parameters?
Step 6: Make recommendations (with disclaimers)
1) Vulnerability
2) Likelihood
3) Severity

32. How to operationalize this framework:
Triage and Risk Assessment
Research needs
• How bad are the current systems for the particular
threat?
• What are the options for changing them? When?
• What is the sensitivity of the outcome to the changes
we can actually make?
• When do those changes achieve best goals?
• Diminishing returns due to delay are an important
explicit component

33. Revisiting this slide, but now about timing:
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
High High Terrible
Very Low Very Low It Depends
Variable Moderate/Variable It Depends
High High Terrible
Increase
capacity/
production
Decrease
density Stimulus and
Protection
How fast can we do
any of these?
How fast can we
know what to do?

34. Timing is critical in real-time
assessment and action
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
High High Terrible
Very Low Very Low It Depends
Variable Moderate/Variable It Depends
High High Terrible
I assumed we knew how to fill out this table –
this is rarely true

35. Timing is critical in real-time
assessment and action
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
???
High ??? Terrible
??? Very Low It Depends
Variable Moderate/Variable ???
High ??? Terrible
??? ??? ???
I assumed we knew how to fill out this table –
this is rarely true
Now what?

36. Informed Trade-offs
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
How quickly can we know things?
• What we can do
• How well it could work
Increase
capacity/
production
Decrease
density
Stimulus and
Protection
High ??? Terrible
??? Very Low It Depends
Variable Moderate/Variable ???
High ??? Terrible

37. Only chase LENGTHY unknowns if
of potential HUGE impact
Vulnerability Likelihood Severity
Healthcare
Resources
Safe Water
Economy
Dense venues
How quickly can we know things?
• What we can do
• How well it could work
Increase
capacity/
production
Decrease
density
Stimulus and
Protection
High ??? Terrible
??? Very Low It Depends
Variable Moderate/Variable ???
High ??? Terrible

38. Good Threat Management has
Prevention AND Mitigation
Avoiding lengthy research in real-time response
does NOT mean we never try to investigate
things that take a long time
It means we DON’T do it in real-time response
When there is NO THREAT, then we do
research for lengthy understanding (especially
for questions that might reduce risk factors)

39. At the onset of a threat:
• Identify Risk Factors (vulnerability, likelihood, severity)
• Identify what is known and what remains unknown
• Focus FIRST on known or easy-to-learn tasks with
HIGH potential to REDUCE one or more risk factors
• Include delay in gaining understanding in estimate of reduction
Good Threat Management has
Prevention AND Mitigation

40. Boom and Bust Funding Cycles
Undermine this Process
Sudden interest in important applied science that
will take longer than funding-cycle interest
is where NIH in-house research could
change EVERYTHING
Rope in external scientists strategically depending
on where we are in threat cycle
Centralize and Coordinate!!!

41. Independent Parallel Investigation
Isn’t Always Best
Centralize and Coordinate:
Tactics ≠ Strategy
Our current response system is tactical. It should be strategic.
We need infrastructure to identify who will study what, when
• Combine forces
• Complement understanding
Risk frameworks might be a good way to build this
infrastructure for rapid self-organization

42. As you choose your next challenge:
“There is a single light of science, and to
brighten it anywhere is to brighten it
everywhere.” - Asimov
But sometimes it helps faster to brighten it
in certain places before others
If you are debating between two studies,
maybe use risk assessment to help
choose (or call someone like me to talk it through)

43. Quick aside about what I do when
I’m not doing this

44. The Fefferman Lab:
Collaborators (and former students) on Jails : Dr. Eric Lofgren, Dr. Kristian Lum, Aaron
Horowitz, Brooke Madubuonwu, Dr. Kellen Myers
Post docs: Dr. Matthew Hasenjager, Dr. Alex Pritchard, Dr. Kimberlyn Roosa, Dr.
Matthew Silk, Dr. Matthew Young
Grad Students: Kelly Buch, Jeff DeSalu, Md. Belal Hossain, Anna Sisk