![Bigfoot: Little
evidence he
exists, but he’d
be large if he
did exist
Pygmy hippo: We know it exists and it’s
small
LARGE EFFECT SIZE,
LOW RELIABILITY
[-.20, 1.80]
SMALL EFFECT SIZE,
HIGH RELIABILITY
[.15, .35]](https://image.slidesharecdn.com/20effectsize-210702024844/75/mixed-effects-models-effect-size-29-2048.jpg?cb=1668560473)
![Effect Size: Standardization
• scale() puts things in terms of z-scores
• New z-scored version of our predictors:
• lifeexpectancy %>% mutate(
YrsEducation.z = scale(YrsEducation)[,1],
IncomeThousands.z = scale(IncomeThousands)[,1]) ->
lifeexpectancy
• # of standard deviations above/below mean
income](https://image.slidesharecdn.com/20effectsize-210702024844/75/mixed-effects-models-effect-size-39-2048.jpg?cb=1668560473)
![Effect Size: Standardization
• scale() puts things in terms of z-scores
• New z-scored version of our predictors:
• lifeexpectancy %>% mutate(
YrsEducation.z = scale(YrsEducation)[,1],
IncomeThousands.z = scale(IncomeThousands)[,1]) ->
lifeexpectancy
• # of standard deviations above/below mean
income
• Then use these in a new model
• model.life <- lmer(Lifespan ~ 1 +
YrsEducation.z + IncomeThousands.z +
(1|Subject), data=lifeexpectancy)](https://image.slidesharecdn.com/20effectsize-210702024844/75/mixed-effects-models-effect-size-40-2048.jpg?cb=1668560473)
Mixed Effects Models - Effect Size
•0 likes•260 views
Report
Share
Download to read offline
Lecture 20 from my mixed-effects modeling course: Cross-lagged designs and effect size
More Related Content
What's hot
What's hot(20)
Similar to Mixed Effects Models - Effect Size
Similar to Mixed Effects Models - Effect Size(20)
More from Scott Fraundorf
More from Scott Fraundorf(13)
Recently uploaded
Recently uploaded(20)
Mixed Effects Models - Effect Size
- 1. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 2. Follow-Up on Autocorrelation • Autocorrelation: An empirical phenomenon in longitudinal data where the value of a variable at time t correlates with its value at time t+1 • Could test this with a Pearson correlation
- 3. Correlation of time t with time t+1 (lag 1) Correlation of time t with time t+2 (lag 2) Correlation of time t with time t+3 (lag 3) • acf.fnc() plot shows pairwise correlations Correlation of time t with itself (always 1)
- 4. Follow-Up on Autocorrelation • Autocorrelation: An empirical phenomenon in longitudinal data where the value of a variable at time t correlates with its value at time t+1 • We can incorporate into a mixed-effects model • Unlike a pairwise correlation, accounts for nested structure and other variables • model.auto <- lmer(WarmthToday ~ 1 + Day + WarmthYesterday + (1 + Day + WarmthYesterday|Couple), data=relationship)
- 5. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 6. Cross-Lagged Models • Our diary study also includes records of emotional support attempts from the partner • Do these cause increased warmth towards the partner?
- 7. Cross-Lagged Models • Our diary study also includes records of emotional support attempts from the partner • Do these cause increased warmth towards the partner? Wait just a darn minute! Correlation does not imply causation! You didn’t experimentally manipulate these support attempts, so you don’t know which caused which! I’VE FINALLY GOT YOU, FRAUNDORF!!
- 8. Cross-Lagged Models • Problem: Relation between support attempts & warmth is ambiguous • What could cause this? • Support attempts could increase warmth towards partner • Warmth towards partner could motivate support attempts • A third variable could explain both Perceived warmth Support attempt TIME t ? Relationship commitment
- 9. Cross-Lagged Models • Problem: Relation between support attempts & warmth is ambiguous • But: Causes precede effects in time • Support attempt on a previous day should influence warmth now Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
- 10. Cross-Lagged Models • Use lags.fnc() to create a SupportYesterday variable • relationship %>% mutate(SupportYesterday= lags.fnc(relationship, time='Day', group='Couple', depvar='PartnerSupport', lag=1)) -> relationship • Then, use that in a model: • model.lagged <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
- 11. Cross-Lagged Models • model.lagged <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
- 12. Cross-Lagged Models • Warmth at t can’t be the cause of support at t-1 • Helps clarify which is the cause and which is the effect Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Duckworth, Tsukayama, & May, 2010
- 13. Cross-Lagged Models • Warmth at t can’t be the cause of support at t-1 • But, warmth at time t-1 could still function as a 3rd variable • Causes support attempts at time t-1 • Leads to greater warmth at time t (autocorrelation) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010 X
- 14. Cross-Lagged Models • To rule this out, we need to include the autocorrelative effect of perceived warmth (our DV) • model.lagged2 <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + WarmthYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010
- 15. Cross-Lagged Models • model.lagged2 <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + WarmthYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010
- 16. Cross-Lagged Models • Now, we are seeing a time-lagged effect of support attempts over and above what can predicted by previous warmth • No way to explain this in a model where the causation only works in reverse • Strong evidence against the reverse direct of causation Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Duckworth, Tsukayama, & May, 2010
- 17. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 18. Establishing Causality • Between-person variation in support attempts predicts within-couple change in warmth Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Duckworth, Tsukayama, & May, 2010
- 19. Establishing Causality • But, there’s still the possibility of a third variable that really drives this between-person difference • e.g., relationship commitment could explain variation in previous support attempts and increase in warmth Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Relationship commitment Duckworth, Tsukayama, & May, 2010
- 20. Establishing Causality • If relationship is driven by an underlying 3rd variable, then warmth & support don’t have a cause/effect relation • Should see the same relation regardless of their order Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Relationship commitment Duckworth, Tsukayama, & May, 2010
- 21. Establishing Causality • To establish causality, show that the direction of the relationship matters • Run the inverse model where support attempts are the DV and previous warmth is the predictor • model.lagged3 <- glmer(PartnerSupport ~ 1 + Day + WarmthYesterday + SupportYesterday + (1|Couple), data=relationship, family=binomial) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Perceived warmth X X
- 22. Establishing Causality • model.lagged3 <- glmer(PartnerSupport ~ 1 + Day + WarmthYesterday + SupportYesterday + (1|Couple), data=relationship, family=binomial) • No significant effects • Earlier support attempts predict later warmth (model.lagged2) • But earlier warmth doesn’t predict later support attempts (model.lagged3) • Evidence for directionality of effect
- 23. Establishing Causality • This kind of evidence is called Granger causality • Still one kind of 3rd variable not ruled out: One with immediate effect on support attempts & a delayed effect on warmth • However, much less likely • So, not quite as good as randomized experiment • But, effective when experimental control not possible (e.g., economics, neuroscience) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X ???
- 24. Establishing Causality • This kind of evidence is called Granger causality • Still one kind of 3rd variable not ruled out: One with immediate effect on support attempts & a delayed effect on warmth • However, much less likely • So, not quite as good as randomized experiment • But, effective when experimental control not possible (e.g., economics, neuroscience) Adapted from Kaminski et al., 2011
- 25. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 26. Effect Size • Revisiting lifexpectancy.csv, let’s run a model predicting Lifespan from fixed effects of YrsEducation and IncomeThousands, and a random intercept of Family • Two variables related to socioeconomic status • Does each matter, when controlling for the other? Which is the most important?
- 27. Effect Size • Revisiting lifexpectancy.csv, let’s run a model predicting Lifespan from fixed effects of YrsEducation and IncomeThousands, and a random intercept of Family • Two variables related to socioeconomic status • Which significantly predict the number of years that people live? • model.life <- lmer(Lifespan ~ 1 + YrsEducation + IncomeThousands + (1|Family), data=lifeexpectancy) They both do! Which is bigger?
- 28. Effect Size • Remember that t statistics and p-values tell us about whether there’s an effect in the population • Is the effect statistically reliable? • A separate question is how big the effect is • Effect size
- 29. Bigfoot: Little evidence he exists, but he’d be large if he did exist Pygmy hippo: We know it exists and it’s small LARGE EFFECT SIZE, LOW RELIABILITY [-.20, 1.80] SMALL EFFECT SIZE, HIGH RELIABILITY [.15, .35]
- 30. • Is bacon really this bad for you?? October 26, 2015
- 31. • Is bacon really this bad for you?? • True that we have as much evidence that bacon causes cancer as smoking causes cancer! • Same level of statistical reliability
- 32. • Is bacon really this bad for you?? • True that we have as much evidence that bacon causes cancer as smoking causes cancer! • Same level of statistical reliability • But, effect size is much smaller for bacon
- 33. Effect Size • Our model results tell us both Parameter estimate tells us about effect size t statistic and p-value tell us about statistical reliability
- 34. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 35. Effect Size: Parameter Estimate • Simplest measure: Parameter estimates • Effect of 1-unit change in predictor on outcome variable • “Each additional $1,000 of annual income predicts another 0.25 years of life” • “Each minute of exercise increases life expectancy by about 7 minutes.” (Moore et al., 2012, PLOS ONE) • “People with a college diploma earn around $24,000 more per year.” (Bureau of Labor Statistics, 2018) • Concrete! Good for “real-world” outcomes
- 36. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 37. Effect Size: Standardization • Which is the bigger effect? • 1 year of education = 0.57 years of life expectancy • $1,000 of annual income = 0.25 years of life expectancy • Problem: These are measured in different, non-comparable units • Years of education vs. (thousands of) US dollars
- 38. Effect Size: Standardization • Which is the bigger effect? • 1 year of education = 0.57 years of life expectancy • $1,000 of annual income = 0.25 years of life expectancy • Problem: These are measured in different, non-comparable units • Years of education vs. (thousands of) US dollars • Convert to z-scores: # of standard deviations from the mean • This scale applies to anything! • Standardized scores
- 39. Effect Size: Standardization • scale() puts things in terms of z-scores • New z-scored version of our predictors: • lifeexpectancy %>% mutate( YrsEducation.z = scale(YrsEducation)[,1], IncomeThousands.z = scale(IncomeThousands)[,1]) -> lifeexpectancy • # of standard deviations above/below mean income
- 40. Effect Size: Standardization • scale() puts things in terms of z-scores • New z-scored version of our predictors: • lifeexpectancy %>% mutate( YrsEducation.z = scale(YrsEducation)[,1], IncomeThousands.z = scale(IncomeThousands)[,1]) -> lifeexpectancy • # of standard deviations above/below mean income • Then use these in a new model • model.life <- lmer(Lifespan ~ 1 + YrsEducation.z + IncomeThousands.z + (1|Subject), data=lifeexpectancy)
- 41. Effect Size: Standardization • New results: • 1 SD increase in education = +2.1 years of life expectancy • 1 SD increase in income = +2.4 years of life expectancy • Income effect is bigger in this dataset
- 42. Effect Size: Standardization • Old results: • New results: No change in statistical reliability Effect size is now estimated differently
- 43. Effect Size: Standardization • Standardized effects make effect sizes more reliant on our data • Effect of 1 SD relative to the mean depends on what the M and SD are! • e.g., Effect of cigarette smoking on life • Smoking rates vary a lot from country to country! • Might get different standardized effects even if unstandardized is the same
- 44. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 45. Overall Variance Explained • How well can we explain this DV? • In the linear model context: • R2= • But in mixed-effect context: Which variance are we talking about? Model-explained variance Model-explained variance + Error variance Level-1 error variance Variance from level-2 clustering Jaeger, Edwards, Das, & Sen, 2017; Rights & Sterba, 2019
- 46. • One R2 we can compute in mixed-effects context: • R2 c= • Obtain with the squared correlation between model-predicted and observed values • cor(fitted(model.life), lifeexpectancy$Lifespan)^2 • Here, 46%! Variance explained by fixed & random effects Conditional R2 All variance Jaeger, Edwards, Das, & Sen, 2017; Rights & Sterba, 2019
- 47. 70 75 80 85 50 60 70 80 90 100 PREDICTED lifespan ACTUAL lifespan • One R2 we can compute in mixed-effects context: • R2 c= Variance explained by fixed & random effects Conditional R2 All variance
- 48. • One R2 we can compute in mixed-effects context: • R2 c= • But, are random effects really “explaining” the variance? Variance explained by fixed & random effects Conditional R2 All variance Some families have longer average lifespans Some subjects have faster RTs than others
- 49. • One R2 we can compute in mixed-effects context: • R2 c= • Summary: • Conditional R2 counts both fixed and random effects as explained variance • Preserves R2 as square of the correlation between observed & predicted data • Evaluates model’s ability to make good predictions • But, may overstate scientific/theoretical explanatory power Variance explained by fixed & random effects Conditional R2 All variance
- 50. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 51. • Another R2 that may be more helpful: • R2 β*= • library(r2glmm) • r2beta(model.life) Variance explained by fixed effects only Marginal R2 All variance Variance explained by all fixed effects combined (14%) Partial R2 for each fixed effect Jaeger, Edwards, Das, & Sen, 2017
- 52. • Another R2 that may be more helpful: • R2 β*= • library(r2glmm) • r2beta(model.life) • r2dt(model1, model2) to test R2 difference between two models Variance explained by fixed effects only Marginal R2 All variance Jaeger, Edwards, Das, & Sen, 2017
- 54. • Another R2 that may be more helpful: • R2 β*= • Summary: • Marginal R2 counts only fixed effects as explained variance • Evaluates model’s scientific/theoretical explanatory ability • Probably more useful in most purposes Variance explained by fixed effects only Marginal R2 All variance
- 55. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
- 56. • Some conventional interpretations of R2 and partial R2: • But, take these with several grains of salt • Cohen (1988) just made them up • Unclear why we care about variance explained (R2) rather than standard deviations (r), in original units • Even small effects can accumulate over time (Funder & Ozer, 2019) Cohen (1988) “Small” .01 “Medium” .06 “Large” .15 Interpreting Effect Size
- 57. • Some conventional interpretations of R2 and partial R2: Cohen (1988) Funder & Ozer (2019) “Small” .01 “Medium” .06 “Large” .15 Interpreting Effect Size .001 .04 .09 .0025 “Very Small” .16 “Very Large”
- 58. • Consider in context of other effect sizes in this domain: • vs: • For interventions: Consider cost, difficulty of implementation, etc. • Aspirin’s effect in reducing heart attacks: r = .03, R2 < .01, but cheap! (Rosenthal, 1990) Our effect: .10 Other effect 1: .20 Other effect 2: .30 Our effect: .10 Other effect 1: .01 Other effect 2: .05 Effect Size: Interpretation
- 59. • For theoretically guided research, compare to predictions of competing theories • The lag effect in memory: • Is this about intervening items or time? Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day POOR recall of RACCOON GOOD recall of RACCOON Effect Size: Interpretation
- 60. Effect Size: Interpretation • Is lag effect about intervening items or time? • Intervening items hypothesis predicts A > B • Time hypothesis predicts B > A • Goal here is to use direction of the effect to adjudicate between competing hypotheses • Not whether the lag effect is “small” or “large” Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day TEST A: Study RACCOON 5 sec. Study WITCH 5 sec. Study RACCOON 5 sec. 10 sec 10 sec 1 day TEST B: