Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Biomarkers for psychological phenotypes?

801 views

Published on

Slides for seminar given at Department of Experimental Psychology, University of Oxford, 25th October 2018

Published in: Science
  • Be the first to comment

  • Be the first to like this

Biomarkers for psychological phenotypes?

  1. 1. Should we stop looking for genetic biomarkers for psychological traits and disorders? Dorothy V. M. Bishop Professor of Developmental Neuropsychology University of Oxford @deevybee
  2. 2. Biomarker • ˈbʌɪəʊmɑːkə/ • noun • noun: biomarker; plural noun: biomarkers • a naturally occurring molecule, gene, or characteristic by which a particular pathological or physiological process, disease, etc. can be identified.
  3. 3. Data from online website NIH Reporter and online Congressional Research Service report Between 2009-2017 over 3 fold increase in funding. Cf. all NIH funding; $30,545m in 2009 $34,301m in 2017 (1.12 fold increase) 0.47% NIH budget 1.64% NIH budget 2009, the promise: ‘Advances in genomics are revolutionizing medicine with discoveries that help elucidate mechanisms and design novel treatments’
  4. 4. ‘Early findings revealed a more complex genetic architecture than was anticipated for most common diseases — complexity that seemed to limit the immediate utility of these findings. As a result, the practice of utilizing the DNA of an individual to predict disease has been judged to provide little to no useful information.’ Torkamani, A., Wineinger, N. E., & Topol, E. J. (2018). The personal and clinical utility of polygenic risk scores. Nature Reviews Genetics, 19, 581-590. A recent expert review: but is this too gloomy?
  5. 5. Important distinction between Rare diseases: Point mutations and Copy Number Variants Common conditions: Allelic variants common in general population
  6. 6. Classic genetics taught using clearcut Mendelian examples • Huntington’s disease – only occurs in those with rare mutation in gene HTT • FOXP2 mutation: associated with severe speech-language disorder • Good progress in finding rare genetic variants that cause severe, rare disorders, using arrays (to detect deletion or duplication of segments of DNA) and exome sequencing (to detect harmful changes to DNA sequence). But each accounts for very small % of cases.
  7. 7. Example of progress: Deciphering Developmental Disorders study https://www.ddduk.org/
  8. 8. Complex multifactorial disorders Aggregate but do not segregate in families – i.e. run in families but you can’t trace effect of gene through the generations according to simple Mendelian rules Thought to be the most usual type of etiology for common disorders e.g. Heart disease, diabetes, asthma, allergies Also dyslexia, ADHD, Developmental Language Disorder Common disease – common variant model Idea that there is continuous distribution of liability for disorder, represents summed effect of numerous very small genetic and environmental effects 8
  9. 9. 9 Association analysis: Compare how different genotypes differ in terms of a quantitative trait or a categorical diagnosis • Early studies looked for impact of genes on phenotypes (common disorders or normal variation) • Very different from FOXP2: ‘Risk’ alleles common in general population and have small effect size
  10. 10. 10 How association analysis works: Compare different genotypes on a quantitative or categorical trait. Example N = 219 N = 175 N = 55 Effect size can be estimated using correlation between N risk alleles and phenotype. Here r ≈ .15 2001
  11. 11. This field affected by 3 factors that generate false positive errors • Publication bias • P-hacking • HARKing
  12. 12. This field affected by 3 factors that generate false positive errors • Publication bias • P-hacking • HARKing But also affected by 3 factors that mean we may miss real effects • Low power • Unreliable measures • Unrepresentative samples
  13. 13. Mainstream genetics is aware of many of these issues and has taken steps to tackle them But ease of obtaining and analysing DNA (spit pots, commercial companies) has led to surge of non-geneticists treating genetic variants as independent variables, without understanding the methodological problems
  14. 14. High profile study reports association; published in ‘high impact’ journal Studies with null results not published Publication More dependent on results than quality of methods Problem: File Drawer effect or ‘Prejudice against the null’ “As it is functioning in at least some areas of behavioral science research, the research-publication system may be regarded as a device for systematically generating and propagating anecdotal information.” Greenwald, 1975 Generally blamed on journal publication policies, though researchers also don’t bother with negative findings. Lure of exciting findings; failure to value negative evidence Publication bias
  15. 15. P-hacking and HARKing 1. P-hacking and HARKing • Usually many possible ways of analyzing data • Have to decide: • Which SNPs (genetic variants) to look at • Which measures of phenotype • Which genetic model (dominant, additive, recessive) • Whether to divide into subgroups Typically researchers report as if decisions made a priori, but no way to know if that is true If many independent decisions made, the N of possible analyses multiplies – i.e. not just additive
  16. 16. 1 contrast Probability of a ‘significant’ p-value < .05 = .05 Study exploring link between COMT and frontal lobe function in schizophrenia https://figshare.com/articles/The_Garden_of_Forking_Paths/2100379 Schizophrenia + control Val/Val, Val/Met, Met/Met
  17. 17. Pick specific measure of phenotype: • 2 contrasts at this level • (NB in reality in this field, typically many more potential measures) • Probability of a ‘significant’ p-value < .05 = .10 Study exploring link between COMT and frontal lobe function Schizophrenia + control Val/Val, Val/Met, Met/Met WCST categories WCST persev. errors
  18. 18. Divide sample into those with and without paranoia Probability of a ‘significant’ p-value < .05 = .19 Study exploring link between COMT and frontal lobe function Schizophrenia + control Val/Val, Val/Met, Met/Met WCST categories WCST persev. errors
  19. 19. Focus just on Females Probability of a ‘significant’ p-value < .05 = .34 Study exploring link between COMT and frontal lobe function Schizophrenia + control Val/Val, Val/Met, Met/Met WCST categories WCST persev. errors
  20. 20. Use model of recessive effect of SNP (R), rather than additive (U) Probability of a ‘significant’ p-value < .05 = .56 Study exploring link between COMT and frontal lobe function Schizophrenia Val/Val, Val/Met, Met/Met WCST categories WCST persev. errors
  21. 21. The biomarker discovery cycle Grey denotes unpublished studies Impression of large body of confirmatory work But few studies actually replicate findings All inconsistent findings have been disregarded
  22. 22. Example of changing the phenotype when the original association doesn’t replicate
  23. 23. Extended sample: Schizophrenic probands (n = 325), their nonpsychotic siblings (n = 359), and normal control subjects (n = 330). Includes many from earlier study. “There was no significant main effect of COMT val108/158met genotype on perseverative errors (t-score) or categories achieved on the WCST.” Means (SDs) not reported for any tasks. Can’t compare with previous. Focuses instead on results from N-back task “In a series of mixed-model ANOVAs in which diagnostic group, COMT genotype, and gender served as main effects, COMT val108/158met genotype had a significant effect on 1-back accuracy, and a near significant effect on 0-back and 2- back accuracy. The val-val group performed more poorly than the val-met and met- met groups, which did not differ from each other.”
  24. 24. So the next set of studies look at N-back task…. but find no effect
  25. 25. A large sample of 402 healthy adults were tested on four working memory tests: Spatial Delayed Response (SDR), Word Serial Position Test (WSPT), N-back, and Letter–Number Sequencing. A subsample (n 246) was tested on the Wisconsin Card Sorting Test (WCST). • No COMT effect on N-back • Trend is in opposite direction to prediction • Focuses on another test, and on result with Wisconsin card-sorting, which only replicates original Egan et al if Met/Val group excluded
  26. 26. High profile study reports association; published in ‘high impact’ journal More studies Different phenotype Different risk allele Just in one subgroup In G x E interaction In G x G interaction Regarded as replications, but shifting goalposts Problems: Focus on confirming rather than disproving hypothesis • P-hacking/data-dredging/incomplete reporting • Hyping of marginal findings • Retrofitting hypothesis to fit the data (HARKing) • Without open data, can’t replicate prior work Problem potentially solved by replication
  27. 27. “Conclusions: Despite initially promising results, the COMT Val158/108Met polymorphism appears to have little if any association with cognitive function. Publication bias may hamper attempts to understand the genetic basis of psychological functions and psychiatric disorders.” Yet citations of the original work continue (now at over 2600), and new studies still attempt to build on this. Eventually, enough studies done to merit a meta-analysis 49 studies
  28. 28. Can anything be done to short-circuit this wasteful cycle?
  29. 29. Mainstream genetics: candidate gene studies largely abandoned in favour of GWAS Genome-wide association study “The scientific breakthrough of 2007” • HapMap: collection of SNPs covering giving dense map of genome • Allows consideration of all genes, rather than just candidates • Can then see whether distribution of probabilities of association is as expected by chance • Some robust associations in field of psychiatry/psychology, e.g. APOE for Alzheimer’s, and genes for smoking/alcohol use • But problem: for many phenotypes of interest (cognitive tests/brain measures) can’t get large samples 29http://genomesunzipped.org/2010/07/how-to-read-a-genome-wide-association-study.php -log(expected p-values) -log(observedp-values) Above line: p-values more extreme than expected Q-Q plot
  30. 30. Candidate gene studies continue with hope we’ll see stronger effects with better phenotypes ”weak and inconsistent effects of genetic variation at the level of human cognition, emotion, and behaviour are much more strongly associated with imaging phenotypes.” Bigos, K.L., Hariri, A. R. & Weinberger, D. R. (2016). Neuroimaging genetics: OUP.
  31. 31. Candidate gene studies continue with hope we’ll see stronger effects with better phenotypes ”weak and inconsistent effects of genetic variation at the level of human cognition, emotion, and behaviour are much more strongly associated with imaging phenotypes.” Bigos, K.L., Hariri, A. R. & Weinberger, D. R. (2016). Neuroimaging genetics: OUP. • So, is it true? • Are we now seeing methodologically strong candidate gene studies?
  32. 32. 32 We took a look at recent studies published in neuroscience journals
  33. 33. Search criteria • Nature Neuroscience • Neuron • Annals of Neurology • Brain • Molecular Psychiatry • Biological Psychiatry • Journal of Neuroscience • Neurology • Journal of Cognitive Neuroscience • Pain • Cerebral Cortex • Neurolmage • genetic OR gene OR allele • association • cognition OR behaviour OR individual differences OR endophenotype • human, not disorders 33
  34. 34. 34 Candidate genes included: 5-HTTLPR; ADCYAP1R1; ADH1B; ANKK1; APOE and related genes; Autosomal catecholamine genes; CACNA1C; COMT; DAT1 and SERT; DISC1, COMT, NRG1, NRG1r, ESR1, BDNF, GAD1, APOE; DRD4; fatty acid amide hydrolase: FAAH; MAOA; NRG3; OXTR; TESC regulating polymorphism; TMEM106B; TREM gene family; WWC1 Phenotypes included: Memory, learning, attention, pain response, mood, personality, age-related cognitive decline, smoking behaviour, response to odours, brain volume, amyloid load, brain response to aversive stimuli, brain activation in memory task
  35. 35. 35 Did studies correct for multiple testing? Total number tests conducted ranged from 2 to 368 Frequent failure to correct for: N subgroups x N genetic models x N variants x N phenotypes in addition to corrections applied in imaging to adjust for N voxels • 8 studies fully corrected for all tests • 9 partially corrected • 13 did not correct Seems to be a common failure to understand multiplicative nature of forking paths.
  36. 36. But studies ALSO ran risk of MISSING true effects 1. Low statistical power (small samples) 2. Samples with restricted range 3. Unreliable measures
  37. 37. Effect size for which there is 80% power, by N 37 80% power to detect r = .2, N = 200 Sample size ranged from 24 to over 4600
  38. 38. 38 Largest reported effect size in 30 studies, by log N 13/30 studies underpowered to detect effect size of r = .2
  39. 39. 39 Use of convenience samples Regression analysis underestimates true effect if sample has restricted range on phenotype under study. Slope of line decreases, correlation declines. Also: proportion with risk genotype decreases Simulated data: True association between genotype and phenotype in population: r = .45 N risk alleles
  40. 40. 40 Psychometric aspects of phenotype measures Recommendation Don’t expect an association with genotype, if the measure doesn’t correlate well with itself on a different occasion! • Most published psychometric tests have information on test-retest reliability. • Not so for measures from experimental tasks, and for measures of brain structure and function. • Where reliability has been evaluated for such measures, it does not inspire confidence. Dubois, J., & Adolphs, R. (2016). Building a science of individual differences from fMRI. Trends in Cognitive Sciences, 20(6), 425-443. doi:10.1016/j.tics.2016.03.014
  41. 41. Future directions • Huge, publicly available samples (e.g. Biobank) now available • Usually large N studies are limited in terms of phenotypic measures. • Biobank sample skewed in socioeconomic background • Small effects -> move from individual SNPs to polygenic risk scores • i.e. a weighted sum of many SNP effects, each with tiny influence • Drawback – hard to understand function of SNPs • Whole exome sequencing – identify variants with known functional significance, so can look at networks • But mutations/CNVs don’t always have expected effect • E.g. may be shared with parent or sib who has no problems • Increasing interest in interactions between mutation and genetic background
  42. 42. Cell 169, ISSUE 7, P1177-1186, June 15, 2017 Cell 173, ISSUE 7, P1573-1580, June 14, 2018 Recommended: two recent reviews with contrasting views on which way we should go
  43. 43. Progress in new techniques needs to be matched by improvements in scientific practices Need to modify researchers’ cognitive biases and change incentives
  44. 44. Recommendation: those still doing candidate gene studies: Pre-register your analysis if you want your results to be believed
  45. 45. Plan study Do study Submit to journal Respond to reviewers Publish paper Plan study Submit to journal Respond to reviewers Do study Publish paper Acceptance! Classic publishing Registered reports Acceptance!
  46. 46. Plan study Do study Submit to journal Respond to reviewers Publish paper Plan study Submit to journal Respond to reviewers Do study Publish paper Acceptance! Classic publishing Registered reports Acceptance!
  47. 47. Registered reports solves issues of: • Publication bias: publication decision made on the basis of quality of introduction/methods, before results are known • Low power: researchers required to have 90% power • P-hacking: analysis plan specified up-front • HARKing: hypotheses specified up-front. Also likely that reviewers will demand evidence that methods are adequately reliable, and sample has appropriate range on phenotype Unanticipated findings can be reported but clearly demarcated as ‘exploratory’
  48. 48. So…… should we stop looking for genetic biomarkers for psychological traits and disorders? Not necessarily, but if we’re going to do this, we need to: • Pre-register studies • Publish null results • Take statistical power seriously (may require collaboration) • Use psychometrically strong measures • Use samples with appropriate range of phenotypes

×