High-throughput screening (HTS) is a technique applied in drug discovery and synthetic biology to screen thousands of 'candidates' (e.g. molecules, cells, strains) for expression of interesting traits. In exchange for throughput, we often pay a penalty in terms of sample size and variance. Small sample sizes and highly variable measurements can lower our ability to detect valuable improvements. In this talk, I describe a simulation environment for conducting power analysis and describe special considerations for maintaining high power with a moving target.

1. Erin Shellman
DAML July, 27 2016
Catching the most with
high-throughput screening

3. Zymergen provides
a platform for the
rapid improvement
of microbial strains
through genetic
engineering.

4. http://www.yourgenome.org/facts/what-is-genetic-engineering

5. What if you don’t know which
gene to perturb?

6. Reduce the system to it’s
constituent parts and
experiment on each part, one
at a time, until you’ve described
the causal mechanism.
…then publish a paper.

7. Try thousands of
things and see
what works.

8. High-throughput
screening (HTS) is a
process for evaluating
many simultaneous
hypotheses.

9. Tier 1 Screen
Tier 2 Screen
Tank validation
Screening goals
Minimize false negatives
Minimize false positives
Confirm that our best tier 2
strains perform well in the
tank
Operational
Implications

10. HTS poses two
unique challenges:
1. low sample size
2. high variance

11. Small sample size
This is largely intentional. We
know most things don’t work, so
why waste resources on a
gamble?

12. High variance
Experimental
complexity creates
opportunities for
injection of bias and
unwanted variance.

13. 😓
• Many classical statistical methods
assume normality and common
variance—which we can’t assume.
• We need to be especially thoughtful in
designing our tests.

14. Real, unknown distributions
15% difference in means

15. When we take
measurements, we get
little bits of information
about what these shapes
might be.

16. But we don’t always arrive
at the right answer.
0.86 0.98 1.2
meanbasestrain = 1.01 meancandidate = 0.81
0.55 0.87 1.02
p-value = 0.31

17. 0.86 0.98 1.2
meanbasestrain = 1.01 meancandidate = 0.81
0.55 0.87 1.02
How many measurements
should I take so I can sleep
knowing that I’ve made the
best possible promotion
decisions?
p-value = 0.31

18. It depends.
What is the expected effect size, i.e. how different
do you think the strains are?
5% difference in means 50% difference in means

19. How variable are the measurement values?
It depends.
5% difference in means 50% difference in means

20. Power analysis
• Power analysis is a method for estimating the
sample size required to detect changes at
assumed levels.
• Power is the probability of detecting a difference,
when a difference is present.
• We compute it through simulation.

21. Power is a fixed parameter
The power threshold is set at 0.80,
meaning if we run the same experiment
100 times, we can expect to detect
differences in means at least 80 out of
those 100 times.

22. Simulation study design
t-test sum rank
contamination
no
contamination

23. • Parametric test, i.e.
assumes that the data
are normally distributed
• Sensitive to extreme
values
• Non-parametric test, i.e.
makes no distributional
assumptions
• Less sensitive to extreme
values
t-test sum rank

24. Initialize Strains:
𝝁basestrain, 𝝈basestrain,
𝝁mutant, 𝝈mutant
Initialize Campaign:
basestrain, mutants, N,
contamination rate, test
Simulate Data
Test for differences
in means
X5000
power = # times diff
detected / 5000
N = range(3, 11)
mu_ref = 0.80
sigma_ref = 0.30
mu_range = np.arange(0.80, 1.70, 0.10)
sigma_range = np.arange(0.05, 2, 0.10)
reference.get_observations(3)
Out:array([ 0.96, 1.00, 1.28])
mutant.get_observations(3)
Out: array([ 1.98, 1.60, 1.70])
e.g.
e.g.

29. A candidate strain would
have to show about 40%
improvement to be
detectable with 3 replicates
A candidate strain would
have to show about 15%
improvement to be
detectable with 10 replicates

30. No contamination
5% contamination

32. Results
• The presence of extreme values undermines
our ability to detect differences by effectively
decreasing N.
• We can make progress in the face of
extreme values by using non-parametric tests,
like sum rank, that perform equally well in
ideal conditions and better than the t-test in
typical conditions.

33. Adaptive experimental design?
STRAINPERFORMANCE
PROJECT TIME
1. ZERO TO MILLIGRAMS
Hits are big enough to detect with
low N.
2. MILLIGRAMS TO KILOGRAMS
Hits sizes shrinking and becoming more
variable as low hanging fruits dry up.
3. KILOGRAMS TO COMMODITY
Hit sizes are at their smallest as we
approach the theoretical max.

34. Tier 1 Screen
Tier 2 Screen
Tank validation
Screening goals
Minimize false negatives
Minimize false positives
Confirm that our best tier 2
strains perform well in the
tank
many hypotheses
low N
low promotion threshold
fewer hypotheses
bigger N
higher promotion threshold
Operational
Implications

35. Zero to
Milligrams
Milligrams to
Kilograms
Kilograms to
Commodity
Tier 1
4 replicates
p-value <= 0.10
6 replicates
p-value <= 0.10
8 replicates
p-value <= 0.10
Tier 2
8 replicates
p-value <= 0.05
12 replicates
p-value <= 0.05
16 replicates
p-value <= 0.05
Tank Tank is truth.☝
Hypothetical design

36. Thanks for listening!
Questions?