B4 jeanmougin
Upcoming SlideShare
Loading in...5
×
 

B4 jeanmougin

on

  • 283 views

statstics

statstics

Statistics

Views

Total Views
283
Views on SlideShare
283
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

B4 jeanmougin B4 jeanmougin Presentation Transcript

  • Should we abandon the t-test ? A statistical comparison of 8 differential gene expression tests Marine Jeanmougin, Master student INSA-Lyon Mickaël Guedj1,3, Grégory Nuel2,3 SMPGD’091. Ligue Nationale contre le Cancer – Cartes d’Identité des Tumeurs program (CIT)2. Paris Descartes University – MAP5 laboratory3. Statistics for Systems Biology working group
  • Analysis process microarray data Pre-processingExperiment (normalization…) Normalized microarray data Differential analysis Differentially expressed gene lists Post-analysis (classification, multiple- testing, prediction…) 2
  • Hypothesis testingDifferential analysis : comparison of 2 populations (or more) according to a variable of interest (expression level).Statistical hypothesis : assumption about a population parameter : 2 types of statistical hypotheses : H0 : Expression level is the same between the 2 populations H1 : Expression level differs between the 2 populations 3
  • Power H0 accepted H0 rejected H0 TRUE False positif True negatif (type I error) H0 FALSE False negatif True positif (type II error)Type I error rate : α (0.05) Ability of a test to detect a gene as Powerα = 1 – β differentially expressed, given thatType II error rate : β this gene is actually differentially expressed.Compare power according to a same type I error rate 4
  • In literatureNumerous tests dedicated to the differential analysis :Performs differently according to : • Sample size • Data noise • Distribution of expression levelsVarious conditions of applicationA lack of comparison studies in the literature (Jeffery et al. 2006) Choosing one test is difficult 5
  • Variance modelling An essential point: variance modelling«…accurate estimation of variability is difficult. » (RVM, Wright and Simon, 2003)« The importance of variance modelling is now widely known…» (SMVar, Jaffrézic et al, 2007)« Many different sources of variability affect gene expression intensity measurements […]. Not atall are well characterized or even identified» (VarMixt, Delmar et al, 2004) Many approaches : 6
  • Test description2 types of tests : Parametric test : assumptions about probability distribution of data Non-parametric test : free distributionStatistical model for parametric tests : ygcs = µgc + εgcsy gcs : expression level of gene g in condition c for the sample sµ gc : mean effect of gene g in condition cε gcs : residual error assumed independent and normallydistributed : 7 ε ~ N (0 , σ² )
  • Test Variance modeling Reference Package R - Fixed varianceWelch’s T-test - Heterodasticity Welch CIT (internal) - Fixed variance ANOVA - homoscedasticity Fisher CIT (internal) Wilcoxon Non parametric Wilcoxon stats Tusher et al SAM Non parametric 2001 samr Inverse gamma distribution on the Wright & Simon RVM variance (estimated from all the data set) 2003 CIT (internal) Moderate t-test. Usual variance replaced Smyth Limma by a conditional variance. Bayesian 2004 limma approach Delmar et al VARMIXT Gamma mixture model on the variance 2005 varmixt Mixed model (fixed condition effect and Jaffrézic et al SMVAR random gene effect) 2007 SMVar 8
  • Comparison process 9
  • SimulationsParameters (determined with CIT datasets): • sample size • genes number • µ: mean of expression level • σ: standard deviation • π0 and π1: proportions of genes under H0 and H1 samples Simulated under H0 genes Simulated under H1 group A group B Assumption: independence of genes 10
  • SimulationsModel 1: Gaussian model Model 2: Uniform modelYgcs ~ N (µobs , σ²obs) Ygcs ~ U (a,b) H0: µA = µBH0: µA = µB H1: µB = µA + dmH1: µB = µA + dm Model 3: Mixture model on variances Model 4: Small variances model Ygcs ~ N (µobs , σ²group *) Ygcs ~ N (µobs , σ²gv *) H0: µA = µB H0: µA = µB H1: µB = µA + dm H1: µB = µA + dm * Groups of variance are simulated * 1000 genes (10%) are simulated with a small variance and small expression level 11
  • SimulationsModel 1: Gaussian model Model 2: Uniform modelYgcs ~ N (µ , σ²obs) Ygcs ~ U (a,b) H0: µA = µBH0: µA = µB H1: µB = µA + dmH1: µB = µA + dmModel 3: Mixture on variances model Model 4: Small variances modelYgcs ~ N (µ , σ²group *) Ygcs ~ N (µ , σ²gv *)H0: µA = µB H0: µA = µBH1: µB = µA + dm H1: µB = µA + dm* Groups of variance are simulated * 1000 genes (10%) are simulated with a small variance and small expression level 12
  • SimulationsModel 1: Gaussian model Model 2: Uniform modelYgcs ~ N (µ , σ²obs) Ygcs ~ U (a,b) H0: µA = µBH0: µA = µB H1: µB = µA + dmH1: µB = µA + dmModel 3: Mixture on variances model Model 4: Small variances modelYgcs ~ N (µ , σ²group *) Ygcs ~ N (µ , σ²gv *)H0: µA = µB H0: µA = µBH1: µB = µA + dm H1: µB = µA + dm* Groups of variance are simulated * 1000 genes (10%) are simulated with a small variance and small expression level 13
  • SimulationsModel 1: Gaussian model Model 2: Uniform modelYgcs ~ N (µ , σ²obs) Ygcs ~ U (a,b) H0: µA = µBH0: µA = µB H1: µB = µA + dmH1: µB = µA + dmModel 3: Mixture on variances model Model 4: Small variances modelYgcs ~ N (µ , σ²group *) Ygcs ~ N (µ , σ²gv *)H0: µA = µB H0: µA = µBH1: µB = µA + dm H1: µB = µA + dm* Groups of variance are simulated * 1000 genes (10%) are simulated with a small variance and small expression level 14
  • SimulationsModel 1: Gaussian model Model 2: Uniform modelYgcs ~ N (µ , σ²obs) Ygcs ~ U (a,b) H0: µA = µBH0: µA = µB H1: µB = µA + dmH1: µB = µA + dmModel 3: Mixture on variances model Model 4: Small variances modelYgcs ~ N (µ , σ²group *) Ygcs ~ N (µ , σ²gv *)H0: µA = µB H0: µA = µBH1: µB = µA + dm H1: µB = µA + dm* Groups of variance are simulated * 1000 genes (10%) are simulated with a small variance and small expression level 15
  • SimulationsModel 4 : H0 Gene with small variances H1 group A group B 16
  • Results : 1 model (gaussian) st1 – Power according to sample size
  • Results : 1 model (gaussian) st1 – Power according to sample size
  • Results : 1 model (gaussian) st Small sample Large sample Tests sizes sizes1 – Power according to sample size n=10 n=200 T-test - = 3.83-4.61 4.92-5.8 Anova = = 4.46-5.24 4.92-5.8 Wilcoxo - + n 2.78-3.46 5.14-6.04 = = SAM 4.59-5.45 4.96-5.84 + + RVM 5.74-6.68 5.01-5.89 = = Limma 4.64-5.5 4.98-5.86 + + SMVar 7.04-8.08 5.08-5.98 = = VarMixt 4.65-5.51 4.99-5.87
  • Results : 1 model (gaussian) st1 – Power according to sample size Adjusted Type I error rate
  • Results : 1st model (gaussian)1 – Power relative to t-test, according to sample size Loss of power Gain in power t-test 21
  • Results : 1st model (gaussian)1 – Power according to difference mean (dm) Adjusted Type I error rate Loss Gain Loss Gain t-test t-testSame observations
  • Results : 1st model (gaussian)Conclusions : Few power differences Observed differences in small sample sizes partly due to type I error rates Wilcoxon: less powerful Limma and varmixt : similar good results Anova : equivalent to the t-test
  • Results : 2nd model (uniform) 1 – Power according to sample size Adjusted Type I error rate Loss Gain Loss Gain t-test t-testSMVar didn’t converge for this simulation model
  • ResultsConclusions model 2 (uniform): Results are similar to the first 1st model Wilcoxon : no improvement unexpected T-test : no loss of power results are robust to the assumption of Gaussian distributionConclusion models 3 (groups of variance) and 4 (smallvariances): Similar results
  • Random test:Reference test. 10 000 p-values are sampled from an uniform distribution U(0,1). 26
  • Gene lists Sample Project Condition Genes Publication sizes Disease staging Lamant et lymphoid tumors 22 283 37 Gender al, 2007 TP53 mutation Boyault et Liver tumors 22 283 65 Gender al, 2006 Rickman ethead and neck tumors Gender 22 283 81 al, 2008 Soulier et al, leukemia Gender 22 283 104 2006 Bertheau et Response to treatment breast tumors 22 283 500 al, 2007 ESR1 27
  • Gene lists for each dataset 5 datasets FDR: 0.05, 0.1, 0.3,0.5 Sample size: complete dataset, 10, 20 dendrograms g3 0 1011... binary metric g72 1 Ward’s method g110 0 9 gene lists . 0 (8 tests + “rand”) . 1 . . . Principal . PCA Component Analysis CountRand: Random Test (sample 10 000 p-values from an matrixuniform distribution)FDR: False discovery rate (Benjamini and Hochberg) 28
  • Gene lists : results lymphoid tumors / disease stagingleukemia / gender FDR = 0.05 Complete dataset breast tumors / response to treatment
  • Gene lists : results lymphoid tumors / disease staging leukemia / gender FDR = 0.05 Complete datasetT-testSMVarWilcoxon breast tumors / response to treatment
  • Gene lists : results FDR = 0.05 Complete dataset breast tumors / ESR1 lymphoid tumors / genderRVMLimmaAnova head and neckVarmixt tumors / gender
  • Gene lists : results (PCA) FDR = 0.05 Complete dataset RVM T-test Limma SMVarleukemia/gender Anova Wilcoxon Varmixt SAM breast tumors/response
  • Spike-in datasetHuman Genome U133 dataset:
  • Spike-in dataset : results Type I error rates : T-test Anova Wilcoxon SAM RVM Limma SMVar VarmixtType I error rates - = - = + = + = same results as simulations
  • Spike-in dataset : resultsPower according to 13 pairwise comparisons Adjusted Type I error rate Loss Gain Loss Gain
  • Re-sampling approach Differentially Expressed Genes (pv < 1.10-4) Breast tumor 500 samples Non-Differentially Expressed Genes (pv > 0.1) Sampling Reduced dataset (5-5)X 1000 8 tests + « rand* » *rand: Random Test (10 000 p-values sampled from a uniform distribution) Power computation
  • Re-sampling approach: results 37
  • Computation time Computation time (sec)* TEST 22 283 genes / 20 samples 52 188 genes / 200 samples T-test 0.87 5.34 rvm 121.69 585.00 anova 0.91 2.98 Limma 0.95 6.81 Wilcoxon 19.18 107.40 SAM 24.48 415.20 SMVar 0.97 19.72 VarMixt 508.98 1h20* Intel® Core ™ 2 DUO CPU 38
  • SummaryComparison according to 3 criterion families :
  • Summary table Use 40
  • Summary table Use 41
  • Summary table Use 42 42
  • Summary table Use 43 43
  • Conclusions• We propose a comparison process• Type I error rates is an issue that explains some differences in power• Tests cluster: similar gene lists (T-test-SMVar/Varmixt-limma-anova-RVM) 44
  • Conclusions• We propose a comparison process• Type I error rates is an issue that explains some differences in power• Tests cluster: similar gene lists (T-test-SMVar/Varmixt-limma-anova-RVM)• On large sample sizes: the 8 tests are equivalent• On small sample sizes: - weak differences in simulations - more important differences in applications 45
  • Conclusions• We propose a comparison process• Type I error rates is an issue that explains some differences in power• Tests cluster: similar gene lists (T-test-SMVar/Varmixt-limma-anova-RVM)• On large sample sizes: the 8 tests are equivalent• On small sample sizes: - weak differences in simulations - more important differences in applications• 2 tests inadvisable: Wilcoxon (worse) and SAM (unstable power) => NP tests• 2 tests appear more efficient: Limma and VarMixt• Considering an intensive use: Limma 46
  • ThanksTo the whole CIT’s team of the Ligue Nationale Contre le Cancer :To M. Guedj, L. Marisa, AS. Valin, F. Petel, E. Thomas, R. Schiappa,L. Vescovo, A. de Reynies, J. Metral and J. GodetTo G. Nuel, MAP5 laboratory, UMR 8145, Paris V Universityand V. Dumeaux, Tromsø University et Paris V UniversityTo the IRISA of RennesTo the SMPGD’s team who enables me to introduce my work
  • Thank you for your attention !Contact : marine.jeanmougin@gmail.com 48