The document discusses several common pitfalls in preclinical cancer research studies that can lead to inaccurate or irreproducible results. These include: 1) Failing to establish a causal rather than just correlative relationship between a target and cancer phenotype. 2) Lack of robustness, as results are not tested under diverse conditions. 3) Inability to distinguish on-target from off-target effects, particularly for assays measuring decreases in signals. 4) Overreliance on statistical significance without consideration of biological or clinical significance.

1. Ash Jogalekar
July 28, 2017

2.  I am not a cancer biologist, I’m a “biology watcher”.
 Some of you are card-carrying cancer biologists.
 Corollary: You know more cancer biology than I do.
 So feel free to interject, educate, enlighten.
2

3.  Efficacy still leading cause of failure in drug discovery.
 Reproducibility of preclinical cancer studies depressingly low:
 Amgen study: Only 6 out of 53 ‘landmark’ cancer studies
could be replicated.
 Bayer study: Only 25% of preclinical studies had data that
warranted further continuation of projects.
 Bad preclinical data = bad clinical trials. Huge potential
negative impact on patients.
3

4.  Robustness: Ability of a system (eg. aircraft, Internet) to
withstand perturbations, removal of key components.
 Robustness in cancer studies: Data must be gathered
under diverse conditions.
 Diverse cell lines
 Diverse assay and buffer conditions
 Diverse tissue and patient samples.
 Important for scientists to test for and report robustness –
and lack thereof.
 Unfortunately, journals actively discourage publication of
‘conflicting’ data that may not only indicate degree of
robustness but also point to unexpected and novel
findings.
4

5. 5
Sies, H. Nature, 1988, 323, 495

6. 6
Diamond G, 1988 Am J Cardiol 63, 392

7. 7
“Common pitfalls in preclinical cancer target validation”
Kaelin, W. G. Nat. Rev. Cancer, 2017, 17, 441

8.  Can emerge simply by chance. Eg. Identification of patient
subpopulations for ‘precision medicine’: Retrospective
evidence not enough, tests need to be re-run for statistical
significance.
 Hypoxia-Inducible Factor (HIF) and hypoxia. Possibilities:
 HIF upregulation and hypoxia cause tumor progression.
 Tumor progression causes HIF upregulation and hypoxia (reverse
causation).
 Only hypoxia promotes tumor growth. HIF merely biomarker.
 Reality: HIF correlates with proteins causing tumor growth and
regression!
 Distinguishing cause, consequence and irrelevant correlations
key and non-trivial (eg. ß amyloid and Alzheimer’s).
8

9.  Phrases like “linked to”, “associated with”
and “consistent with” maddeningly
confusing. Important to know whether A
necessary or sufficient (or both) for B.
 Eg. BRAF mutations found in both
malignant and benign melanoma. Thus:
 BRAF not sufficient for melanoma maintenance.
 BRAF necessary for melanoma maintenance.
 Distinguishing necessary from sufficient
important in connecting multiple driver
mutations to multidrug therapy.
9
“Common pitfalls in preclinical cancer target validation”
Kaelin, W. G. Nat. Rev. Cancer, 2017, 17, 441

10.  Assays can look for either increase or decrease of particular
signals (eg. protein production, fluorescence, apoptosis).
 Signal in down assays can be swamped by noise.
 More important: Down assays likely to suffer from multiple off-
target causes, can produce results with trivial/uninteresting
explanations.
 Reality: Many assays used by cancer biologists are down assays,
measuring decreases in protein phosphorylation, cell viability
etc.
 Important to know whether results of down assays are real and
on-target/biologically interesting.
10
“Common pitfalls in preclinical cancer target validation”
Kaelin, W. G. Nat. Rev. Cancer, 2017, 17, 441

11. 11

12.  Another graveyard for distinguishing correlation and causation
between target modulation and phenotype.
 Are your down assays producing results because of off-target
effects? Especially true at higher drug concentrations.
 Potential experiments to validate on-target effects:
 Rescue experiments: eg. BCR-ABL variants resistant to imatinib make
CML cells resistant.
 Restoring downstream effector function: Eg. Activation of MEK1 makes
mutant melanoma cells resistant to BRAF inhibitors.
 Establishing correlation between gain of function and opposite
phenotype (Caveat: Overexpressing some targets might cause loss of
general cell function because of overload).
12

13. 13

14.  Positive controls can distinguish between technical assay
failures and real effects of perturbant.
 Negative controls can test for specificity: eg. distinguish
between various perturbants.
 Negative controls particularly valuable in down assays,
especially when tested as minor variations (eg. as
enantiomers).
 When possible, different kinds of controls (genetic, chemical,
antibodies etc.) should be tested for corroborative evidence.
 What’s the null model?:
 Probability you got the result by chance?
 Did you use the right vehicle?
14

15.  Role of target in development not always predictive of role in
adult: eg. Germline knockout of Abl embryonic lethal in mice,
imatinib modulation viable in humans.
 Therapeutic vs toxic response all about right level of target
engagement: Eg. Proteasome inhibitor Bortezomib viable at
concentration that reduces 50-80% of proteasome activity, toxic
at higher concentration.
 Animal models not always predictive of humans: Eg. Development
of imatinib delayed due to liver toxicity seen only in animal
model.
 Ultimately, therapeutic window dictated not by target
requirement in normal cells but by difference between
requirements in normal vs cancer cells.
15

16. 16
“There are known knowns; there are things we know we
know. We also know there are known unknowns; that is to
say we know there are some things we don’t know. But there
are also unknown unknowns – the ones that we don’t know
we don’t know.”
- Donald “Iraq War” Rumsfeld

17. 17

18.  Known knowns: Targets whose necessary causal connections to
disease states have been established through a variety of means.
BCR-ABL, HER2, EGFR, PD-1?
 Note: Truly speaking, known known targets validated only years after
drug hits market, and perhaps never…
 Known unknowns: Targets whose general role is known but whose
effects on particular disease states are unknown, especially when
perturbing other related targets.
 Unknown unknowns: Targets whose existence per se may or may not
be known, but whose role in disease states may not even be
suspected.
 Unknown knowns: Targets whose role in disease states you think
you know, which may be doing something completely different.
18

19.  System rewards positive results, actively discourages negative results.
 “Technological solutionism”: Easy to do experiments, therefore do
experiments without thinking whether they’re asking the right
questions (eg. CRISPR, chemical genetics, HTS).
 “Dataism”: Use data because it’s available, not necessarily because it
means something relevant (eg. ‘targets’ from cheap genomic
sequencing).
 Ignoring statistics: eg. excessive reliance on p values, failure to
distinguish between statistical and real life significance etc.
“The first question that should be asked when assessing the importance of a
paper is whether its findings are likely to withstand the test of time, not
which journal it appeared in…we should lower the bar with respect to the
number of claims required for publication and raise the bar in terms of the
burden of proof required to support those claims” – William Kaelin
19
Morozov, E. “To Save Everything, Click Here: The Fallacy of Technological Solutionism” (2013)
Yaffe, M. “The Scientific Drunk and the Lamppost” Sci. Signal. 2013, 2, 269
Zilliak, S. and McCloskey, D. “The Cult of Statistical Significance” (2008)

20.  Is our target merely correlated with the phenotype or causing it?
 Is the target-phenotype relationship robust, irrespective of effect size? Does it hold up
under multiple conditions, cell types, assays, animal models etc.?
 Is there a clear causal relationship between target and phenotype in our down
assays?
 Are we clearly distinguishing between on-target and off-target effects?
 Can we use up assays instead?
 Is our target necessary, sufficient or both for effecting the phenotype?
 Are we testing enough positive and negative controls? Both genetic and chemical?
 Do our assays/studies have good statistical validation?
 Occam’s Razor: Is there a simpler explanation for the results?
 Psychology: Are we falling for conventional wisdom? Groupthink? Confirmation bias?
20

21. “It ain’t what you don’t know that gets you into
trouble, it’s what you know for sure that just ain’t
so.” – Mark Twain
21

22. “It ain’t what you don’t know that gets you into
trouble, it’s what you know for sure that just ain’t
so.” – Mark Twain (although there’s no evidence that
Twain actually said this)
22