SlideShare a Scribd company logo
1 of 58
Download to read offline
Week 4.1: Model Comparison
! Lab: Interactions Practice
! Model Comparison
! Nested Models
! Hypothesis Testing
! REML vs ML
! Non-Nested Models
! Shrinkage
! The Problem
! Solutions
Interpreting Interactions
• Numerical interaction term tells us how the
interaction works:
• Strengthens individual effects with the same sign
as the interaction
• Weakens individual effects with a different sign as
the interaction
• Or, again, just look at the graph ☺
Interpreting Interactions Practice
• Dependent variable: Classroom learning
• Independent variable 1: Intrinsic motivation
• Learning because you want to learn (intrinsic) vs.
to get a good grade (extrinsic)
• Intrinsic motivation has a + effect on learning
• Independent variable 2: Autonomy language
• “You can…” (vs. “You must…”)
• Also has a + effect on learning
• Motivation x autonomy interaction is +
• Interpretation: Combining intrinsic
motivation and autonomy
language especially benefits
learning
• “Synergistic” interaction
Vansteenkiste
et al., 2004,
JPSP
Interpreting Interactions Practice
• Dependent variable: Satisfaction with a
consumer purchase
• Number of choices: - effect on
satisfaction
• “Maximizing” strategy: - effect on satisfaction
• Trying to find the best option vs. “good enough”
• Choices x maximizing strategy is -
• Interpretation: Having lots
of choices when you’re a
maximizer especially
reduces satisfaction
• Also a synergistic
interaction
(Carrillat, Ladik, & Legoux, 2011; Marketing Letters)
Week 4.1: Model Comparison
! Lab: Interactions Practice
! Model Comparison
! Nested Models
! Hypothesis Testing
! REML vs ML
! Non-Nested Models
! Shrinkage
! The Problem
! Solutions
Model Formulae Practice
• Write the R formula for each model:
• 1) We’re interested in the effects of FamilySES,
PriorNightSleep, and Nutrition on MathTest
Performance, but we don’t expect them to interact
• 2) We factorially manipulated SentenceType (active
or passive) and Plausibility (low or high) in a test
of TextComprehensionAccuracy
Model Formulae Practice
• Write the R formula for each model:
• 1) We’re interested in the effects of FamilySES,
PriorNightSleep, and Nutrition on MathTest
Performance, but we don’t expect them to interact
• MathPerformance ~ 1 + SES + Sleep +
Nutrition
• 2) We factorially manipulated SentenceType (active
or passive) and Plausibility (low or high) in a test
of TextComprehensionAccuracy
• ComprehensionAccuracy ~ 1 + SentenceType +
Plausibility + SentenceType:Plausibility
or
ComprehensionAccuracy ~ 1 +
SentenceType*Plausibility
Interpreting Interactions Practice
• Second language proficiency: + effect on
translation accuracy
• Word frequency: + effect on accuracy
• Frequency x proficiency interaction is -
• Interpretation: Proficiency matters less when translating
high frequency words
• Or: Difference between high & low proficiency words gets
smaller if you have high proficiency
• “Antagonistic” interaction. Combining the effects reduces or
reverses the individual effects.
(e.g., Diependaele, Lemhöfer,
Brysbaert, 2012, QJEP)
Interpreting Interactions Practice
• Retrieval practice: + effect on long-term
learning
• Working memory span: + effect on learning
• Retrieval practice x WM span interaction is -
(Agarwal et al., 2016)
• Interpretation: Retrieval practice is especially
beneficial for people with low working memory.
• Or: Low WM confers less of a disadvantage if you
do retrieval practice
Interpreting Interactions Practice
• Affectionate touch: + effect on feeling of
relationship security
• Avoidant attachment style: - effect on security
• Touch x avoidant attachment interaction is -
• Interpretation: Affectionate touch enhances
relationship security less for people with
an avoidant attachment style
(Jakubiak & Feeney, SPPS, 2016)
Interpreting Interactions Practice
• Age: - effect on picture memory
• Older adults have poorer memory
• Emotional valence: - effect on accuracy
• Positive pictures are not remembered as well
compared to negative pictures
• Age x Valence interaction is +
• Interpretation: Age declines are smaller for positive pictures
• Or: Disadvantage of positive pictures is not as strong for
older adults
(e.g., Mather & Carstensen, 2005, TiCS)
Week 4.1: Model Comparison
! Lab: Interactions Practice
! Model Comparison
! Nested Models
! Hypothesis Testing
! REML vs ML
! Non-Nested Models
! Shrinkage
! The Problem
! Solutions
Model Comparison
• Sometimes, we may have more than 1 model
that we could consider applying to the data
• 2 or more competing theoretical models
• e.g., critical period in language acquisition
No critical period (Vanhove, 2013)
Critical period hypothesis
(Hartshorne et al., 2020)
1 + AgeOfAcquisition
1 + AgeOfAcquisition*CriticalPeriod
Model Comparison
• Sometimes, we may have more than 1 model
that we could consider applying to the data
• 2 or more competing theoretical models
• Exploratory analysis where we don’t yet know
which model would be appropriate
Dataset
! Social support & health (e.g., Cohen & Wills, 1985)
! lifeexpectancy.csv:
! Longitudinal study of 1000 subjects – some
siblings from same family, so 517 total families
! Perceived social support (z-scored)
! Lifespan
! And several control variables
Nested Models
! Three possible models of life expectancy:
! Amount of weekly exercise
! Amount of weekly exercise & perceived social
support
! Amount of weekly exercise, perceived social
support, years of education, conscientiousness,
yearly income, and number of vowels in your last
name
! These are nested models—each one can be
formed by subtracting variables from the one
below it (“nested inside it”)
Nested Models
Nested Models
! Three possible models of life expectancy:
! Amount of weekly exercise
! Amount of weekly exercise & perceived social
support
! Amount of weekly exercise, perceived social
support, years of education, conscientiousness,
yearly income, and number of vowels in your last
name
! Which set of information would give us the
most accurate fitted() values?
Nested Models
! Three possible models of life expectancy:
! Amount of weekly exercise
! Amount of weekly exercise & perceived social
support
! Amount of weekly exercise, perceived social
support, years of education, conscientiousness,
yearly income, and number of vowels in your last
name
• The “biggest” nested model will always provide
predictions that are at least as good
• Adding info can only explain more of the variance
Nested Models
• The “biggest” nested model will always provide
predictions that are at least as good
• Adding info can only explain more of the variance
• Might not be much better (“number of vowels”
effect zero or close to zero) but can’t be worse
Slope of regression
line relating last
name vowels to life
expectancy is near 0
But that merely fails
to improve
predictions; doesn’t
hurt them
Week 4.1: Model Comparison
! Lab: Interactions Practice
! Model Comparison
! Nested Models
! Hypothesis Testing
! REML vs ML
! Non-Nested Models
! Shrinkage
! The Problem
! Solutions
Hypothesis Testing
! Let’s think about our first two models:
! Comparing these two statistical models closely
relates to our research question: Which theoretical
model best explains data?
! The theoretical model where social support does affect life
expectancy
! The model where social support doesn’t affect life
expectancy
E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport
model1
E(Yi(j)) = γ00 + γ10HrsExercise
model2
Hypothesis Testing
! Let’s think about our first two models:
! What are some possible values of γ20 (the
SocSupport effect) in model1?
! 3.83
! -1.04
! 0 – there is no social support effect
E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport
model1
E(Yi(j)) = γ00 + γ10HrsExercise
model2
! Let’s think about our first two models:
! What happens when γ20 is equal to 0?
! Anything multiplied by 0 is 0, so SocSupport just
drops out of the equation
! Becomes the same thing as model2
E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport
Hypothesis Testing
0
E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport
model1
E(Yi(j)) = γ00 + γ10HrsExercise
model2
Hypothesis Testing
! Let’s think about our first two models:
! model2 is just a special case of model1
! The version of model1 where γ20 happens to be 0
! One of many possible versions of model1
! Why we say model2 is “nested” in model1
E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport
E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport
model1
E(Yi(j)) = γ00 + γ10HrsExercise
model2
0
Hypothesis Testing
! Let’s think about our first two models:
! This also helps show why model1 always
fits as well as model2 or better
! model1 can account for the case where γ20 = 0
! But it can also account for many other cases, too
E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport
E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport
model1
E(Yi(j)) = γ00 + γ10HrsExercise
model2
0
Likelihood Ratio Test
! We can compare nested models (only) using
the likelihood-ratio test
! Remember that likelihood is what we search for in
fitting an individual model (find the values with the
highest likelihood)
! First, fit each of the models to be compared
! model1 <- lmer(Lifespan ~ 1 +
HrsExercise + SocSupport + (1|Family),
data=lifeexpectancy)
! model2 <- lmer(Lifespan ~ 1 +
HrsExercise + (1|Family),
data=lifeexpectancy)
Likelihood Ratio Test
• Then, compare them with anova():
• anova(model1, model2)
• Order doesn’t matter
• Differences in (log) likelihoods are
distributed as a chi-square
• d.f. = # of variables added or removed
• Here, χ2
(1) = 8.67, p = .003
Log likelihood will also be somewhat higher (better) for the
complex model … but is it SIGNIFICANTLY better?
We’ll discuss what
this means in a
moment (don’t
worry; it’s what we
want)
Likelihood Ratio Test
• t-test and LR test are very similar!
• t-test: Tests whether an effect differs from 0,
based on this model
• Likelihood ratio: Compare to a model where the
effect actually IS constrained to be 0
• With an infinitely large sample, these two
tests would produce identical conclusions
• With small sample, t-test is less likely to
detect spurious differences (Luke, 2017)
• But, large differences uncommon
Likelihood Ratio Test
• t-test and LR test are very similar!
• t-test: Tests whether an effect differs from 0,
based on this model
• Likelihood ratio: Compare to a model where the
effect actually IS constrained to be 0
p-value from likelihood
ratio test: .0032
p-value from lmerTest t-
test: .0033
Likelihood Ratio Test
• t-test and LR test are very similar!
• t-test: Tests whether an effect differs from 0,
based on this model
• Likelihood ratio: Compare to a model where the
effect actually IS constrained to be 0
• Guidance:
• LR test is useful for testing groups of variable
• model1 <- lmer(Lifespan ~ 1 + HrsExercise …)
• model3 <- lmer(Lifespan ~ 1 + HrsExercise +
SocSupport + YrsEducation +
Conscientiousness …)
• If testing just one variable at a time, use t-test—
slightly less likely to produce Type I error
Week 4.1: Model Comparison
! Lab: Interactions Practice
! Model Comparison
! Nested Models
! Hypothesis Testing
! REML vs ML
! Non-Nested Models
! Shrinkage
! The Problem
! Solutions
REML vs ML
• Technically, two different algorithms that R can
use “behind the scenes” to get the estimates
# REML: Restricted Maximum Likelihood
• Assumes the fixed effects structure is correct
• Bad for comparing models that differ in fixed effects
# ML: Maximum Likelihood
• OK for comparing models
• But, may underestimate variance of random effects
• Ideal: ML for model comparison, REML for final
results
• lme4 does this automatically for you!
• Defaults to REML. But automatically refits models
with ML when you do likelihood ratio test.
REML vs ML
• The one time you might want to mess with this:
• If you are going to be doing a lot of model
comparisons, can fit the model with ML to begin
with
• model1 <- lmer(DV ~ 1 + Predictors,
data=lifeexpectancy, REML=FALSE)
• Saves refitting for each comparison
• Remember to refit the model with REML=TRUE
for your final results
Week 4.1: Model Comparison
! Lab: Interactions Practice
! Model Comparison
! Nested Models
! Hypothesis Testing
! REML vs ML
! Non-Nested Models
! Shrinkage
! The Problem
! Solutions
Non-Nested Models
• Which of these pairs is not a case of nested
models?
• A
• Accuracy ~ SentenceType + Aphasia +
SentenceType:Aphasia
• Accuracy ~ SentenceType + Aphasia
• B
• MathAchievement ~ SocioeconomicStatus
• MathAchievement ~ TeacherRating + ClassSize
• C
• Recall ~ StudyTime
• Recall ~ StudyTime + StudyStrategy
Non-Nested Models
• Which of these pairs is not a case of nested
models?
• A
• Accuracy ~ SentenceType + Aphasia +
SentenceType:Aphasia
• Accuracy ~ SentenceType + Aphasia
• B
• MathAchievement ~ SocioeconomicStatus
• MathAchievement ~ TeacherRating + ClassSize
• Each of these models has something that the other doesn’t have.
Non-Nested Models
• Models that aren’t nested can’t be tested the
same way
• A non-nested comparison:
• What would support 1st model over 2nd?
• γ20 is significantly greater than 0, but also γ10 is 0
• But remember we can’t test that something is 0 with
frequentist statistics … can’t prove the H0 is true
• Parametric statistics don’t apply here $
E(Yi(j)) = γ00 + γ10YrsEducation + γ20IncomeThousands
E(Yi(j)) = γ00 + γ10YrsEducation + γ20IncomeThousands
0
0
Non-Nested Models: Comparison
• Can be compared with information criteria
• Remember our fitted values from last week?
• fitted(model2)
• What if we replaced all of our observations with
just the fitted (predicted) values?
• We’d be losing some information
• However, if the model predicted the data well, we
would not be losing that much
• Information criteria measure how much information is
lost with the fitted values (so, lower is better)
Non-Nested Models: Comparison
• AIC: An Information Criterion or Akaike’s Information Criterion
• -2(log likelihood) + 2k
• k = # of fixed and random effects in a particular model
• A model with a lower AIC is better
Akaike, 1974
Non-Nested Models: Comparison
• AIC: An Information Criterion or Akaike’s Information Criterion
• -2(log likelihood) + 2k
• k = # of fixed and random effects in a particular model
• A model with a lower AIC is better
• Doesn’t assume any of the models is correct
• Appropriate for correlational / non-experimental data
• BIC: Bayesian Information Criterion
• -2(log likelihood) + log(n)k
• k = # of fixed & random effects, n = num. observations
• A model with a lower BIC is better
• Typically prefers simpler models than AIC
• Assumes that there’s a “true” underlying model in the
set of variables being considered
• Appropriate for experimental data Yang, 2005; Oehlert, 2012
Non-Nested Models: Comparison
• Can also get these from anova(model1, model2)
• Just ignore the chi-square if non-nested models
• AIC and BIC do not have a significance test
associated with them
• The model with the lower AIC/BIC is preferred, but
we don’t know how reliable this preference is
Week 4.1: Model Comparison
! Lab: Interactions Practice
! Model Comparison
! Nested Models
! Hypothesis Testing
! REML vs ML
! Non-Nested Models
! Shrinkage
! The Problem
! Solutions
Shrinkage
• The “Madden curse”…
• Each year, a top NFL football player is picked to
appear on the cover of the Madden NFL video
game
• That player often doesn’t
play as well in the following
season
• Is the cover “cursed”?
Shrinkage
• The “Madden curse”…
• Each year, a top NFL football player is picked to
appear on the cover of the Madden NFL video
game
• That player often doesn’t
play as well in the following
season
• Is the cover “cursed”?
Shrinkage
• What’s needed to be one of the top NFL players
in a season?
• You have to be a good player
• Genuine predictor (signal)
• And, luck on your side
• Random chance or error
• Top-performing player probably
very good and very lucky
• The next season…
• Your skill may persist
• Random chance probably won’t
• Regression to the mean
• Madden video game cover imperfect predicts next
season’s performance because it was partly based
on random error
Shrinkage
• Our estimates (& any choice of variables
based on them) always partially reflect random
chance in the dataset we used to obtain them
• Won’t fit any later data set quite
as well … shrinkage
• Problem when we’re using the
data to decide the model
Shrinkage
• Our estimates (& any choice of variables
based on them) always partially reflect random
chance in the dataset we used to obtain them
• Won’t fit any later data set quite
as well … shrinkage
• “If you use a sample to construct a model, or to
choose a hypothesis to test, you cannot make a
rigorous scientific test of the model or the hypothesis
using that same sample data.”
(Babyak, 2004, p. 414)
Shrinkage—Examples
• Relations that we observe between a predictor
variable and a dependent variable might simply
be capitalizing on random chance
• U.S. government puts out 45,000 economic
statistics each year (Silver, 2012)
• Can we use these to predict whether US economy
will go into recession?
• With 45,000 predictors, we are very likely to find a
spurious relation by chance
• Especially w/ only 15
recessions since
the end of WW II
Shrinkage—Examples
• Relations that we observe between a predictor
variable and a dependent variable might simply
be capitalizing on random chance
• U.S. government puts out 45,000 economic
statistics each year (Silver, 2012)
• Can we use these to predict whether US economy
will go into recession?
• With 45,000 predictors, we are very likely to find a
spurious relation by chance
• Significance tests try to address this … but with
45,000 predictors, we are likely to find significant
effects by chance (5% Type I error rate at ɑ=.05)
Shrinkage—Examples
• Adak Island, Alaska
• Daily temperature here predicts
stock market activity!
• r = -.87 correlation with the price
of a specific group of stocks!
• Completely true—I’m not making this up!
• Problem with this:
• With thousands of weather stations & stocks, easy to find a
strong correlation somewhere, even if it’s just sampling error
• Problem is that this factoid doesn’t reveal all of the other (non-
significant) weather stations & stocks we searched through
• Would only be impressive if this hypothesis continued to be
true on a new set of weather data & stock prices
Vul et al., 2009
Shrinkage—Examples
• “Puzzlingly high correlations” in some fMRI work
• Correlate each voxel in a brain scan with a behavioral
measure (e.g., personality survey)
• Restrict the analysis to voxels where
the correlation is above some threshold
• Compute final correlation in this region
with behavioral measure—very high!
• Problem: Voxels were already chosen based on
those high correlations
• Includes sampling error favoring the correlation but
excludes error that doesn’t
Vul et al., 2009
Week 4.1: Model Comparison
! Lab: Interactions Practice
! Model Comparison
! Nested Models
! Hypothesis Testing
! REML vs ML
! Non-Nested Models
! Shrinkage
! The Problem
! Solutions
Shrinkage—Solutions
• One solution: Select model(s) in advance
(perhaps even pre-registered)
• A theory is valuable for this
• Adak Island example is implausible in part because there’s
no causal reason why an island in Alaska would relate to
stock prices
“Just as you do not need to know exactly how a car engine
works in order to drive safely, you do not need to
understand all the intricacies of the economy to accurately
read those gauges.” – Economic forecasting firm ECRI
(quoted in Silver, 2012)
Shrinkage—Solutions
• One solution: Select model(s) in advance
(perhaps even pre-registered)
• A theory is valuable for this
• Not driven purely by the data or by chance if we have an a
priori reason to favor this variable
“There is really nothing so practical as a good theory.”
-- Social psychologist Kurt Lewin (Lewin’s Maxim)
Shrinkage—Solutions
• One solution: Select model(s) in advance
(perhaps even pre-registered)
• A theory is valuable for this
• Not driven purely by the data or by chance if we have an a
priori reason to favor this variable
• Based on some other measure (e.g., another brain
scan)
Shrinkage—Solutions
• One solution: Select model(s) in advance
(perhaps even pre-registered)
• A theory is valuable for this
• Not driven purely by the data or by chance if we have an a
priori reason to favor this variable
• Based on some other measure (e.g., another brain
scan)
• Based on research design
• For factorial experiments, typical to include all
experimental variables and interactions
• Research design implies you were interested in all of these
Shrinkage—Solutions
• For more exploratory analyses: Show that the
finding replicates
• On a second dataset
• Test a model obtained from one subset of the data
applies to another subset (cross-validation)
• e.g., training and test sets
• A better version: Do this with
many randomly chosen subsets
• Monte Carlo methods
• Reading on Canvas for some
general ways to do this in R

More Related Content

What's hot

Mixed Effects Models - Empirical Logit
Mixed Effects Models - Empirical LogitMixed Effects Models - Empirical Logit
Mixed Effects Models - Empirical LogitScott Fraundorf
 
Mixed Effects Models - Simple and Main Effects
Mixed Effects Models - Simple and Main EffectsMixed Effects Models - Simple and Main Effects
Mixed Effects Models - Simple and Main EffectsScott Fraundorf
 
Mixed Effects Models - Autocorrelation
Mixed Effects Models - AutocorrelationMixed Effects Models - Autocorrelation
Mixed Effects Models - AutocorrelationScott Fraundorf
 
Mixed Effects Models - Crossed Random Effects
Mixed Effects Models - Crossed Random EffectsMixed Effects Models - Crossed Random Effects
Mixed Effects Models - Crossed Random EffectsScott Fraundorf
 
Mixed Effects Models - Orthogonal Contrasts
Mixed Effects Models - Orthogonal ContrastsMixed Effects Models - Orthogonal Contrasts
Mixed Effects Models - Orthogonal ContrastsScott Fraundorf
 
Mixed Effects Models - Logit Models
Mixed Effects Models - Logit ModelsMixed Effects Models - Logit Models
Mixed Effects Models - Logit ModelsScott Fraundorf
 
Mixed Effects Models - Random Slopes
Mixed Effects Models - Random SlopesMixed Effects Models - Random Slopes
Mixed Effects Models - Random SlopesScott Fraundorf
 
Mixed Effects Models - Growth Curve Analysis
Mixed Effects Models - Growth Curve AnalysisMixed Effects Models - Growth Curve Analysis
Mixed Effects Models - Growth Curve AnalysisScott Fraundorf
 
Causal discovery and prediction mechanisms
Causal discovery and prediction mechanismsCausal discovery and prediction mechanisms
Causal discovery and prediction mechanismsShiga University, RIKEN
 
西山計量経済学第8章 制限従属変数モデル
西山計量経済学第8章 制限従属変数モデル西山計量経済学第8章 制限従属変数モデル
西山計量経済学第8章 制限従属変数モデルKatsuya Ito
 
2 5 2.一般化線形モデル色々_ロジスティック回帰
2 5 2.一般化線形モデル色々_ロジスティック回帰2 5 2.一般化線形モデル色々_ロジスティック回帰
2 5 2.一般化線形モデル色々_ロジスティック回帰logics-of-blue
 
Learning Theory 101 ...and Towards Learning the Flat Minima
Learning Theory 101 ...and Towards Learning the Flat MinimaLearning Theory 101 ...and Towards Learning the Flat Minima
Learning Theory 101 ...and Towards Learning the Flat MinimaSangwoo Mo
 
regression and correlation
regression and correlationregression and correlation
regression and correlationPriya Sharma
 
Discovery of Linear Acyclic Models Using Independent Component Analysis
Discovery of Linear Acyclic Models Using Independent Component AnalysisDiscovery of Linear Acyclic Models Using Independent Component Analysis
Discovery of Linear Acyclic Models Using Independent Component AnalysisShiga University, RIKEN
 
lispmeetup#63 Common Lispでゼロから作るDeep Learning
lispmeetup#63 Common Lispでゼロから作るDeep Learninglispmeetup#63 Common Lispでゼロから作るDeep Learning
lispmeetup#63 Common Lispでゼロから作るDeep LearningSatoshi imai
 
数学を数学で数学した人々
数学を数学で数学した人々数学を数学で数学した人々
数学を数学で数学した人々Akira Yamaguchi
 
Applications of regression analysis - Measurement of validity of relationship
Applications of regression analysis - Measurement of validity of relationshipApplications of regression analysis - Measurement of validity of relationship
Applications of regression analysis - Measurement of validity of relationshipRithish Kumar
 
Multiple Linear Regression II and ANOVA I
Multiple Linear Regression II and ANOVA IMultiple Linear Regression II and ANOVA I
Multiple Linear Regression II and ANOVA IJames Neill
 
Rで実験計画法 後編
Rで実験計画法 後編Rで実験計画法 後編
Rで実験計画法 後編itoyan110
 

What's hot (20)

Mixed Effects Models - Empirical Logit
Mixed Effects Models - Empirical LogitMixed Effects Models - Empirical Logit
Mixed Effects Models - Empirical Logit
 
Mixed Effects Models - Simple and Main Effects
Mixed Effects Models - Simple and Main EffectsMixed Effects Models - Simple and Main Effects
Mixed Effects Models - Simple and Main Effects
 
Mixed Effects Models - Autocorrelation
Mixed Effects Models - AutocorrelationMixed Effects Models - Autocorrelation
Mixed Effects Models - Autocorrelation
 
Mixed Effects Models - Crossed Random Effects
Mixed Effects Models - Crossed Random EffectsMixed Effects Models - Crossed Random Effects
Mixed Effects Models - Crossed Random Effects
 
Mixed Effects Models - Orthogonal Contrasts
Mixed Effects Models - Orthogonal ContrastsMixed Effects Models - Orthogonal Contrasts
Mixed Effects Models - Orthogonal Contrasts
 
Mixed Effects Models - Logit Models
Mixed Effects Models - Logit ModelsMixed Effects Models - Logit Models
Mixed Effects Models - Logit Models
 
Mixed Effects Models - Random Slopes
Mixed Effects Models - Random SlopesMixed Effects Models - Random Slopes
Mixed Effects Models - Random Slopes
 
Mixed Effects Models - Growth Curve Analysis
Mixed Effects Models - Growth Curve AnalysisMixed Effects Models - Growth Curve Analysis
Mixed Effects Models - Growth Curve Analysis
 
Causal discovery and prediction mechanisms
Causal discovery and prediction mechanismsCausal discovery and prediction mechanisms
Causal discovery and prediction mechanisms
 
西山計量経済学第8章 制限従属変数モデル
西山計量経済学第8章 制限従属変数モデル西山計量経済学第8章 制限従属変数モデル
西山計量経済学第8章 制限従属変数モデル
 
2 5 2.一般化線形モデル色々_ロジスティック回帰
2 5 2.一般化線形モデル色々_ロジスティック回帰2 5 2.一般化線形モデル色々_ロジスティック回帰
2 5 2.一般化線形モデル色々_ロジスティック回帰
 
Learning Theory 101 ...and Towards Learning the Flat Minima
Learning Theory 101 ...and Towards Learning the Flat MinimaLearning Theory 101 ...and Towards Learning the Flat Minima
Learning Theory 101 ...and Towards Learning the Flat Minima
 
regression and correlation
regression and correlationregression and correlation
regression and correlation
 
Discovery of Linear Acyclic Models Using Independent Component Analysis
Discovery of Linear Acyclic Models Using Independent Component AnalysisDiscovery of Linear Acyclic Models Using Independent Component Analysis
Discovery of Linear Acyclic Models Using Independent Component Analysis
 
lispmeetup#63 Common Lispでゼロから作るDeep Learning
lispmeetup#63 Common Lispでゼロから作るDeep Learninglispmeetup#63 Common Lispでゼロから作るDeep Learning
lispmeetup#63 Common Lispでゼロから作るDeep Learning
 
数学を数学で数学した人々
数学を数学で数学した人々数学を数学で数学した人々
数学を数学で数学した人々
 
Key ideas, terms and concepts in SEM
Key ideas, terms and concepts in SEMKey ideas, terms and concepts in SEM
Key ideas, terms and concepts in SEM
 
Applications of regression analysis - Measurement of validity of relationship
Applications of regression analysis - Measurement of validity of relationshipApplications of regression analysis - Measurement of validity of relationship
Applications of regression analysis - Measurement of validity of relationship
 
Multiple Linear Regression II and ANOVA I
Multiple Linear Regression II and ANOVA IMultiple Linear Regression II and ANOVA I
Multiple Linear Regression II and ANOVA I
 
Rで実験計画法 後編
Rで実験計画法 後編Rで実験計画法 後編
Rで実験計画法 後編
 

Similar to Mixed Effects Models - Model Comparison

35878 Topic Discussion5Number of Pages 1 (Double Spaced).docx
35878 Topic Discussion5Number of Pages 1 (Double Spaced).docx35878 Topic Discussion5Number of Pages 1 (Double Spaced).docx
35878 Topic Discussion5Number of Pages 1 (Double Spaced).docxrhetttrevannion
 
PSY 150 403 Chapter 9 SLIDES
PSY 150 403 Chapter 9 SLIDESPSY 150 403 Chapter 9 SLIDES
PSY 150 403 Chapter 9 SLIDESkimappel
 
M08 BiasVarianceTradeoff
M08 BiasVarianceTradeoffM08 BiasVarianceTradeoff
M08 BiasVarianceTradeoffRaman Kannan
 
NLP_KASHK:Evaluating Language Model
NLP_KASHK:Evaluating Language ModelNLP_KASHK:Evaluating Language Model
NLP_KASHK:Evaluating Language ModelHemantha Kulathilake
 
8 Week Curriculum Map
8 Week Curriculum Map8 Week Curriculum Map
8 Week Curriculum MapSerena Porter
 
Mixed Effects Models - Missing Data
Mixed Effects Models - Missing DataMixed Effects Models - Missing Data
Mixed Effects Models - Missing DataScott Fraundorf
 
Endogeneity and Entrepreneurship Research
Endogeneity and Entrepreneurship ResearchEndogeneity and Entrepreneurship Research
Endogeneity and Entrepreneurship ResearchBrian Anderson
 
Mba724 s2 w1 elements of scientific research
Mba724 s2 w1 elements of scientific researchMba724 s2 w1 elements of scientific research
Mba724 s2 w1 elements of scientific researchRachel Chung
 
1RUNNING HEAD METHODS AND RESULTS1RUNNING HEAD METHODS.docx
1RUNNING HEAD METHODS AND RESULTS1RUNNING HEAD METHODS.docx1RUNNING HEAD METHODS AND RESULTS1RUNNING HEAD METHODS.docx
1RUNNING HEAD METHODS AND RESULTS1RUNNING HEAD METHODS.docxdrennanmicah
 
35881 DiscussionNumber of Pages 1 (Double Spaced)Number o.docx
35881 DiscussionNumber of Pages 1 (Double Spaced)Number o.docx35881 DiscussionNumber of Pages 1 (Double Spaced)Number o.docx
35881 DiscussionNumber of Pages 1 (Double Spaced)Number o.docxrhetttrevannion
 
Statistical Calculations 5Statistical Calculations.docx
Statistical Calculations 5Statistical Calculations.docxStatistical Calculations 5Statistical Calculations.docx
Statistical Calculations 5Statistical Calculations.docxdessiechisomjj4
 
BUS 308 Week 3 Lecture 1 Examining Differences - Continued.docx
BUS 308 Week 3 Lecture 1 Examining Differences - Continued.docxBUS 308 Week 3 Lecture 1 Examining Differences - Continued.docx
BUS 308 Week 3 Lecture 1 Examining Differences - Continued.docxcurwenmichaela
 
Personalized Learning_1.pptx
Personalized Learning_1.pptxPersonalized Learning_1.pptx
Personalized Learning_1.pptxWillSoo1
 
Gender and language (linguistics, social network theory, Twitter!)
Gender and language (linguistics, social network theory, Twitter!)Gender and language (linguistics, social network theory, Twitter!)
Gender and language (linguistics, social network theory, Twitter!)Tyler Schnoebelen
 
Gender, language, and Twitter: Social theory and computational methods
Gender, language, and Twitter: Social theory and computational methodsGender, language, and Twitter: Social theory and computational methods
Gender, language, and Twitter: Social theory and computational methodsIdibon1
 
How Do Coping Strategies Correlate With Job Satisfaction Revised
How Do Coping Strategies Correlate With Job Satisfaction   RevisedHow Do Coping Strategies Correlate With Job Satisfaction   Revised
How Do Coping Strategies Correlate With Job Satisfaction Revisedpaneil
 
BUS308 – Week 1 Lecture 2 Describing Data Expected Out.docx
BUS308 – Week 1 Lecture 2 Describing Data Expected Out.docxBUS308 – Week 1 Lecture 2 Describing Data Expected Out.docx
BUS308 – Week 1 Lecture 2 Describing Data Expected Out.docxcurwenmichaela
 
Running head Organization behaviorOrganization behavior 2.docx
Running head Organization behaviorOrganization behavior 2.docxRunning head Organization behaviorOrganization behavior 2.docx
Running head Organization behaviorOrganization behavior 2.docxtoltonkendal
 
F ProjHOSPITAL INPATIENT P & L20162017Variance Variance Per DC 20.docx
F ProjHOSPITAL INPATIENT P & L20162017Variance Variance Per DC 20.docxF ProjHOSPITAL INPATIENT P & L20162017Variance Variance Per DC 20.docx
F ProjHOSPITAL INPATIENT P & L20162017Variance Variance Per DC 20.docxmecklenburgstrelitzh
 

Similar to Mixed Effects Models - Model Comparison (20)

35878 Topic Discussion5Number of Pages 1 (Double Spaced).docx
35878 Topic Discussion5Number of Pages 1 (Double Spaced).docx35878 Topic Discussion5Number of Pages 1 (Double Spaced).docx
35878 Topic Discussion5Number of Pages 1 (Double Spaced).docx
 
PSY 150 403 Chapter 9 SLIDES
PSY 150 403 Chapter 9 SLIDESPSY 150 403 Chapter 9 SLIDES
PSY 150 403 Chapter 9 SLIDES
 
M08 BiasVarianceTradeoff
M08 BiasVarianceTradeoffM08 BiasVarianceTradeoff
M08 BiasVarianceTradeoff
 
NLP_KASHK:Evaluating Language Model
NLP_KASHK:Evaluating Language ModelNLP_KASHK:Evaluating Language Model
NLP_KASHK:Evaluating Language Model
 
8 Week Curriculum Map
8 Week Curriculum Map8 Week Curriculum Map
8 Week Curriculum Map
 
Mixed Effects Models - Missing Data
Mixed Effects Models - Missing DataMixed Effects Models - Missing Data
Mixed Effects Models - Missing Data
 
Endogeneity and Entrepreneurship Research
Endogeneity and Entrepreneurship ResearchEndogeneity and Entrepreneurship Research
Endogeneity and Entrepreneurship Research
 
Mba724 s2 w1 elements of scientific research
Mba724 s2 w1 elements of scientific researchMba724 s2 w1 elements of scientific research
Mba724 s2 w1 elements of scientific research
 
1RUNNING HEAD METHODS AND RESULTS1RUNNING HEAD METHODS.docx
1RUNNING HEAD METHODS AND RESULTS1RUNNING HEAD METHODS.docx1RUNNING HEAD METHODS AND RESULTS1RUNNING HEAD METHODS.docx
1RUNNING HEAD METHODS AND RESULTS1RUNNING HEAD METHODS.docx
 
35881 DiscussionNumber of Pages 1 (Double Spaced)Number o.docx
35881 DiscussionNumber of Pages 1 (Double Spaced)Number o.docx35881 DiscussionNumber of Pages 1 (Double Spaced)Number o.docx
35881 DiscussionNumber of Pages 1 (Double Spaced)Number o.docx
 
Statistical Calculations 5Statistical Calculations.docx
Statistical Calculations 5Statistical Calculations.docxStatistical Calculations 5Statistical Calculations.docx
Statistical Calculations 5Statistical Calculations.docx
 
BUS 308 Week 3 Lecture 1 Examining Differences - Continued.docx
BUS 308 Week 3 Lecture 1 Examining Differences - Continued.docxBUS 308 Week 3 Lecture 1 Examining Differences - Continued.docx
BUS 308 Week 3 Lecture 1 Examining Differences - Continued.docx
 
Personalized Learning_1.pptx
Personalized Learning_1.pptxPersonalized Learning_1.pptx
Personalized Learning_1.pptx
 
Gender and language (linguistics, social network theory, Twitter!)
Gender and language (linguistics, social network theory, Twitter!)Gender and language (linguistics, social network theory, Twitter!)
Gender and language (linguistics, social network theory, Twitter!)
 
Gender, language, and Twitter: Social theory and computational methods
Gender, language, and Twitter: Social theory and computational methodsGender, language, and Twitter: Social theory and computational methods
Gender, language, and Twitter: Social theory and computational methods
 
Seawell_Exam
Seawell_ExamSeawell_Exam
Seawell_Exam
 
How Do Coping Strategies Correlate With Job Satisfaction Revised
How Do Coping Strategies Correlate With Job Satisfaction   RevisedHow Do Coping Strategies Correlate With Job Satisfaction   Revised
How Do Coping Strategies Correlate With Job Satisfaction Revised
 
BUS308 – Week 1 Lecture 2 Describing Data Expected Out.docx
BUS308 – Week 1 Lecture 2 Describing Data Expected Out.docxBUS308 – Week 1 Lecture 2 Describing Data Expected Out.docx
BUS308 – Week 1 Lecture 2 Describing Data Expected Out.docx
 
Running head Organization behaviorOrganization behavior 2.docx
Running head Organization behaviorOrganization behavior 2.docxRunning head Organization behaviorOrganization behavior 2.docx
Running head Organization behaviorOrganization behavior 2.docx
 
F ProjHOSPITAL INPATIENT P & L20162017Variance Variance Per DC 20.docx
F ProjHOSPITAL INPATIENT P & L20162017Variance Variance Per DC 20.docxF ProjHOSPITAL INPATIENT P & L20162017Variance Variance Per DC 20.docx
F ProjHOSPITAL INPATIENT P & L20162017Variance Variance Per DC 20.docx
 

Recently uploaded

The Ultimate Guide to Social Media Marketing in 2024.pdf
The Ultimate Guide to Social Media Marketing in 2024.pdfThe Ultimate Guide to Social Media Marketing in 2024.pdf
The Ultimate Guide to Social Media Marketing in 2024.pdfdm4ashexcelr
 
factors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptxfactors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptxSanjay Shekar
 
Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...Mark Carrigan
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Denish Jangid
 
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...Nguyen Thanh Tu Collection
 
2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptxmansk2
 
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17Celine George
 
Basic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & EngineeringBasic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & EngineeringDenish Jangid
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdfVikramadityaRaj
 
How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17Celine George
 
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45MysoreMuleSoftMeetup
 
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...Sayali Powar
 
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdfINU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdfbu07226
 
Morse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxMorse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxjmorse8
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽中 央社
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfmstarkes24
 
Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Celine George
 

Recently uploaded (20)

The Ultimate Guide to Social Media Marketing in 2024.pdf
The Ultimate Guide to Social Media Marketing in 2024.pdfThe Ultimate Guide to Social Media Marketing in 2024.pdf
The Ultimate Guide to Social Media Marketing in 2024.pdf
 
factors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptxfactors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptx
 
Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
 
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
 
2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx
 
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
 
Word Stress rules esl .pptx
Word Stress rules esl               .pptxWord Stress rules esl               .pptx
Word Stress rules esl .pptx
 
Basic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & EngineeringBasic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf
 
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
Operations Management - Book1.p  - Dr. Abdulfatah A. SalemOperations Management - Book1.p  - Dr. Abdulfatah A. Salem
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
 
“O BEIJO” EM ARTE .
“O BEIJO” EM ARTE                       .“O BEIJO” EM ARTE                       .
“O BEIJO” EM ARTE .
 
How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17
 
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
 
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
 
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdfINU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
 
Morse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxMorse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptx
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdf
 
Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17
 

Mixed Effects Models - Model Comparison

  • 1. Week 4.1: Model Comparison ! Lab: Interactions Practice ! Model Comparison ! Nested Models ! Hypothesis Testing ! REML vs ML ! Non-Nested Models ! Shrinkage ! The Problem ! Solutions
  • 2. Interpreting Interactions • Numerical interaction term tells us how the interaction works: • Strengthens individual effects with the same sign as the interaction • Weakens individual effects with a different sign as the interaction • Or, again, just look at the graph ☺
  • 3. Interpreting Interactions Practice • Dependent variable: Classroom learning • Independent variable 1: Intrinsic motivation • Learning because you want to learn (intrinsic) vs. to get a good grade (extrinsic) • Intrinsic motivation has a + effect on learning • Independent variable 2: Autonomy language • “You can…” (vs. “You must…”) • Also has a + effect on learning • Motivation x autonomy interaction is + • Interpretation: Combining intrinsic motivation and autonomy language especially benefits learning • “Synergistic” interaction Vansteenkiste et al., 2004, JPSP
  • 4. Interpreting Interactions Practice • Dependent variable: Satisfaction with a consumer purchase • Number of choices: - effect on satisfaction • “Maximizing” strategy: - effect on satisfaction • Trying to find the best option vs. “good enough” • Choices x maximizing strategy is - • Interpretation: Having lots of choices when you’re a maximizer especially reduces satisfaction • Also a synergistic interaction (Carrillat, Ladik, & Legoux, 2011; Marketing Letters)
  • 5. Week 4.1: Model Comparison ! Lab: Interactions Practice ! Model Comparison ! Nested Models ! Hypothesis Testing ! REML vs ML ! Non-Nested Models ! Shrinkage ! The Problem ! Solutions
  • 6. Model Formulae Practice • Write the R formula for each model: • 1) We’re interested in the effects of FamilySES, PriorNightSleep, and Nutrition on MathTest Performance, but we don’t expect them to interact • 2) We factorially manipulated SentenceType (active or passive) and Plausibility (low or high) in a test of TextComprehensionAccuracy
  • 7. Model Formulae Practice • Write the R formula for each model: • 1) We’re interested in the effects of FamilySES, PriorNightSleep, and Nutrition on MathTest Performance, but we don’t expect them to interact • MathPerformance ~ 1 + SES + Sleep + Nutrition • 2) We factorially manipulated SentenceType (active or passive) and Plausibility (low or high) in a test of TextComprehensionAccuracy • ComprehensionAccuracy ~ 1 + SentenceType + Plausibility + SentenceType:Plausibility or ComprehensionAccuracy ~ 1 + SentenceType*Plausibility
  • 8. Interpreting Interactions Practice • Second language proficiency: + effect on translation accuracy • Word frequency: + effect on accuracy • Frequency x proficiency interaction is - • Interpretation: Proficiency matters less when translating high frequency words • Or: Difference between high & low proficiency words gets smaller if you have high proficiency • “Antagonistic” interaction. Combining the effects reduces or reverses the individual effects. (e.g., Diependaele, Lemhöfer, Brysbaert, 2012, QJEP)
  • 9. Interpreting Interactions Practice • Retrieval practice: + effect on long-term learning • Working memory span: + effect on learning • Retrieval practice x WM span interaction is - (Agarwal et al., 2016) • Interpretation: Retrieval practice is especially beneficial for people with low working memory. • Or: Low WM confers less of a disadvantage if you do retrieval practice
  • 10. Interpreting Interactions Practice • Affectionate touch: + effect on feeling of relationship security • Avoidant attachment style: - effect on security • Touch x avoidant attachment interaction is - • Interpretation: Affectionate touch enhances relationship security less for people with an avoidant attachment style (Jakubiak & Feeney, SPPS, 2016)
  • 11. Interpreting Interactions Practice • Age: - effect on picture memory • Older adults have poorer memory • Emotional valence: - effect on accuracy • Positive pictures are not remembered as well compared to negative pictures • Age x Valence interaction is + • Interpretation: Age declines are smaller for positive pictures • Or: Disadvantage of positive pictures is not as strong for older adults (e.g., Mather & Carstensen, 2005, TiCS)
  • 12. Week 4.1: Model Comparison ! Lab: Interactions Practice ! Model Comparison ! Nested Models ! Hypothesis Testing ! REML vs ML ! Non-Nested Models ! Shrinkage ! The Problem ! Solutions
  • 13. Model Comparison • Sometimes, we may have more than 1 model that we could consider applying to the data • 2 or more competing theoretical models • e.g., critical period in language acquisition No critical period (Vanhove, 2013) Critical period hypothesis (Hartshorne et al., 2020) 1 + AgeOfAcquisition 1 + AgeOfAcquisition*CriticalPeriod
  • 14. Model Comparison • Sometimes, we may have more than 1 model that we could consider applying to the data • 2 or more competing theoretical models • Exploratory analysis where we don’t yet know which model would be appropriate
  • 15. Dataset ! Social support & health (e.g., Cohen & Wills, 1985) ! lifeexpectancy.csv: ! Longitudinal study of 1000 subjects – some siblings from same family, so 517 total families ! Perceived social support (z-scored) ! Lifespan ! And several control variables
  • 16. Nested Models ! Three possible models of life expectancy: ! Amount of weekly exercise ! Amount of weekly exercise & perceived social support ! Amount of weekly exercise, perceived social support, years of education, conscientiousness, yearly income, and number of vowels in your last name ! These are nested models—each one can be formed by subtracting variables from the one below it (“nested inside it”)
  • 18. Nested Models ! Three possible models of life expectancy: ! Amount of weekly exercise ! Amount of weekly exercise & perceived social support ! Amount of weekly exercise, perceived social support, years of education, conscientiousness, yearly income, and number of vowels in your last name ! Which set of information would give us the most accurate fitted() values?
  • 19. Nested Models ! Three possible models of life expectancy: ! Amount of weekly exercise ! Amount of weekly exercise & perceived social support ! Amount of weekly exercise, perceived social support, years of education, conscientiousness, yearly income, and number of vowels in your last name • The “biggest” nested model will always provide predictions that are at least as good • Adding info can only explain more of the variance
  • 20. Nested Models • The “biggest” nested model will always provide predictions that are at least as good • Adding info can only explain more of the variance • Might not be much better (“number of vowels” effect zero or close to zero) but can’t be worse Slope of regression line relating last name vowels to life expectancy is near 0 But that merely fails to improve predictions; doesn’t hurt them
  • 21. Week 4.1: Model Comparison ! Lab: Interactions Practice ! Model Comparison ! Nested Models ! Hypothesis Testing ! REML vs ML ! Non-Nested Models ! Shrinkage ! The Problem ! Solutions
  • 22. Hypothesis Testing ! Let’s think about our first two models: ! Comparing these two statistical models closely relates to our research question: Which theoretical model best explains data? ! The theoretical model where social support does affect life expectancy ! The model where social support doesn’t affect life expectancy E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport model1 E(Yi(j)) = γ00 + γ10HrsExercise model2
  • 23. Hypothesis Testing ! Let’s think about our first two models: ! What are some possible values of γ20 (the SocSupport effect) in model1? ! 3.83 ! -1.04 ! 0 – there is no social support effect E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport model1 E(Yi(j)) = γ00 + γ10HrsExercise model2
  • 24. ! Let’s think about our first two models: ! What happens when γ20 is equal to 0? ! Anything multiplied by 0 is 0, so SocSupport just drops out of the equation ! Becomes the same thing as model2 E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport Hypothesis Testing 0 E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport model1 E(Yi(j)) = γ00 + γ10HrsExercise model2
  • 25. Hypothesis Testing ! Let’s think about our first two models: ! model2 is just a special case of model1 ! The version of model1 where γ20 happens to be 0 ! One of many possible versions of model1 ! Why we say model2 is “nested” in model1 E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport model1 E(Yi(j)) = γ00 + γ10HrsExercise model2 0
  • 26. Hypothesis Testing ! Let’s think about our first two models: ! This also helps show why model1 always fits as well as model2 or better ! model1 can account for the case where γ20 = 0 ! But it can also account for many other cases, too E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport E(Yi(j)) = γ00 + γ10HrsExercise + γ20SocSupport model1 E(Yi(j)) = γ00 + γ10HrsExercise model2 0
  • 27. Likelihood Ratio Test ! We can compare nested models (only) using the likelihood-ratio test ! Remember that likelihood is what we search for in fitting an individual model (find the values with the highest likelihood) ! First, fit each of the models to be compared ! model1 <- lmer(Lifespan ~ 1 + HrsExercise + SocSupport + (1|Family), data=lifeexpectancy) ! model2 <- lmer(Lifespan ~ 1 + HrsExercise + (1|Family), data=lifeexpectancy)
  • 28. Likelihood Ratio Test • Then, compare them with anova(): • anova(model1, model2) • Order doesn’t matter • Differences in (log) likelihoods are distributed as a chi-square • d.f. = # of variables added or removed • Here, χ2 (1) = 8.67, p = .003 Log likelihood will also be somewhat higher (better) for the complex model … but is it SIGNIFICANTLY better? We’ll discuss what this means in a moment (don’t worry; it’s what we want)
  • 29. Likelihood Ratio Test • t-test and LR test are very similar! • t-test: Tests whether an effect differs from 0, based on this model • Likelihood ratio: Compare to a model where the effect actually IS constrained to be 0 • With an infinitely large sample, these two tests would produce identical conclusions • With small sample, t-test is less likely to detect spurious differences (Luke, 2017) • But, large differences uncommon
  • 30. Likelihood Ratio Test • t-test and LR test are very similar! • t-test: Tests whether an effect differs from 0, based on this model • Likelihood ratio: Compare to a model where the effect actually IS constrained to be 0 p-value from likelihood ratio test: .0032 p-value from lmerTest t- test: .0033
  • 31. Likelihood Ratio Test • t-test and LR test are very similar! • t-test: Tests whether an effect differs from 0, based on this model • Likelihood ratio: Compare to a model where the effect actually IS constrained to be 0 • Guidance: • LR test is useful for testing groups of variable • model1 <- lmer(Lifespan ~ 1 + HrsExercise …) • model3 <- lmer(Lifespan ~ 1 + HrsExercise + SocSupport + YrsEducation + Conscientiousness …) • If testing just one variable at a time, use t-test— slightly less likely to produce Type I error
  • 32. Week 4.1: Model Comparison ! Lab: Interactions Practice ! Model Comparison ! Nested Models ! Hypothesis Testing ! REML vs ML ! Non-Nested Models ! Shrinkage ! The Problem ! Solutions
  • 33. REML vs ML • Technically, two different algorithms that R can use “behind the scenes” to get the estimates # REML: Restricted Maximum Likelihood • Assumes the fixed effects structure is correct • Bad for comparing models that differ in fixed effects # ML: Maximum Likelihood • OK for comparing models • But, may underestimate variance of random effects • Ideal: ML for model comparison, REML for final results • lme4 does this automatically for you! • Defaults to REML. But automatically refits models with ML when you do likelihood ratio test.
  • 34. REML vs ML • The one time you might want to mess with this: • If you are going to be doing a lot of model comparisons, can fit the model with ML to begin with • model1 <- lmer(DV ~ 1 + Predictors, data=lifeexpectancy, REML=FALSE) • Saves refitting for each comparison • Remember to refit the model with REML=TRUE for your final results
  • 35. Week 4.1: Model Comparison ! Lab: Interactions Practice ! Model Comparison ! Nested Models ! Hypothesis Testing ! REML vs ML ! Non-Nested Models ! Shrinkage ! The Problem ! Solutions
  • 36. Non-Nested Models • Which of these pairs is not a case of nested models? • A • Accuracy ~ SentenceType + Aphasia + SentenceType:Aphasia • Accuracy ~ SentenceType + Aphasia • B • MathAchievement ~ SocioeconomicStatus • MathAchievement ~ TeacherRating + ClassSize • C • Recall ~ StudyTime • Recall ~ StudyTime + StudyStrategy
  • 37. Non-Nested Models • Which of these pairs is not a case of nested models? • A • Accuracy ~ SentenceType + Aphasia + SentenceType:Aphasia • Accuracy ~ SentenceType + Aphasia • B • MathAchievement ~ SocioeconomicStatus • MathAchievement ~ TeacherRating + ClassSize • Each of these models has something that the other doesn’t have.
  • 38. Non-Nested Models • Models that aren’t nested can’t be tested the same way • A non-nested comparison: • What would support 1st model over 2nd? • γ20 is significantly greater than 0, but also γ10 is 0 • But remember we can’t test that something is 0 with frequentist statistics … can’t prove the H0 is true • Parametric statistics don’t apply here $ E(Yi(j)) = γ00 + γ10YrsEducation + γ20IncomeThousands E(Yi(j)) = γ00 + γ10YrsEducation + γ20IncomeThousands 0 0
  • 39. Non-Nested Models: Comparison • Can be compared with information criteria • Remember our fitted values from last week? • fitted(model2) • What if we replaced all of our observations with just the fitted (predicted) values? • We’d be losing some information • However, if the model predicted the data well, we would not be losing that much • Information criteria measure how much information is lost with the fitted values (so, lower is better)
  • 40. Non-Nested Models: Comparison • AIC: An Information Criterion or Akaike’s Information Criterion • -2(log likelihood) + 2k • k = # of fixed and random effects in a particular model • A model with a lower AIC is better Akaike, 1974
  • 41. Non-Nested Models: Comparison • AIC: An Information Criterion or Akaike’s Information Criterion • -2(log likelihood) + 2k • k = # of fixed and random effects in a particular model • A model with a lower AIC is better • Doesn’t assume any of the models is correct • Appropriate for correlational / non-experimental data • BIC: Bayesian Information Criterion • -2(log likelihood) + log(n)k • k = # of fixed & random effects, n = num. observations • A model with a lower BIC is better • Typically prefers simpler models than AIC • Assumes that there’s a “true” underlying model in the set of variables being considered • Appropriate for experimental data Yang, 2005; Oehlert, 2012
  • 42. Non-Nested Models: Comparison • Can also get these from anova(model1, model2) • Just ignore the chi-square if non-nested models • AIC and BIC do not have a significance test associated with them • The model with the lower AIC/BIC is preferred, but we don’t know how reliable this preference is
  • 43. Week 4.1: Model Comparison ! Lab: Interactions Practice ! Model Comparison ! Nested Models ! Hypothesis Testing ! REML vs ML ! Non-Nested Models ! Shrinkage ! The Problem ! Solutions
  • 44. Shrinkage • The “Madden curse”… • Each year, a top NFL football player is picked to appear on the cover of the Madden NFL video game • That player often doesn’t play as well in the following season • Is the cover “cursed”?
  • 45. Shrinkage • The “Madden curse”… • Each year, a top NFL football player is picked to appear on the cover of the Madden NFL video game • That player often doesn’t play as well in the following season • Is the cover “cursed”?
  • 46. Shrinkage • What’s needed to be one of the top NFL players in a season? • You have to be a good player • Genuine predictor (signal) • And, luck on your side • Random chance or error • Top-performing player probably very good and very lucky • The next season… • Your skill may persist • Random chance probably won’t • Regression to the mean • Madden video game cover imperfect predicts next season’s performance because it was partly based on random error
  • 47. Shrinkage • Our estimates (& any choice of variables based on them) always partially reflect random chance in the dataset we used to obtain them • Won’t fit any later data set quite as well … shrinkage • Problem when we’re using the data to decide the model
  • 48. Shrinkage • Our estimates (& any choice of variables based on them) always partially reflect random chance in the dataset we used to obtain them • Won’t fit any later data set quite as well … shrinkage • “If you use a sample to construct a model, or to choose a hypothesis to test, you cannot make a rigorous scientific test of the model or the hypothesis using that same sample data.” (Babyak, 2004, p. 414)
  • 49. Shrinkage—Examples • Relations that we observe between a predictor variable and a dependent variable might simply be capitalizing on random chance • U.S. government puts out 45,000 economic statistics each year (Silver, 2012) • Can we use these to predict whether US economy will go into recession? • With 45,000 predictors, we are very likely to find a spurious relation by chance • Especially w/ only 15 recessions since the end of WW II
  • 50. Shrinkage—Examples • Relations that we observe between a predictor variable and a dependent variable might simply be capitalizing on random chance • U.S. government puts out 45,000 economic statistics each year (Silver, 2012) • Can we use these to predict whether US economy will go into recession? • With 45,000 predictors, we are very likely to find a spurious relation by chance • Significance tests try to address this … but with 45,000 predictors, we are likely to find significant effects by chance (5% Type I error rate at ɑ=.05)
  • 51. Shrinkage—Examples • Adak Island, Alaska • Daily temperature here predicts stock market activity! • r = -.87 correlation with the price of a specific group of stocks! • Completely true—I’m not making this up! • Problem with this: • With thousands of weather stations & stocks, easy to find a strong correlation somewhere, even if it’s just sampling error • Problem is that this factoid doesn’t reveal all of the other (non- significant) weather stations & stocks we searched through • Would only be impressive if this hypothesis continued to be true on a new set of weather data & stock prices Vul et al., 2009
  • 52. Shrinkage—Examples • “Puzzlingly high correlations” in some fMRI work • Correlate each voxel in a brain scan with a behavioral measure (e.g., personality survey) • Restrict the analysis to voxels where the correlation is above some threshold • Compute final correlation in this region with behavioral measure—very high! • Problem: Voxels were already chosen based on those high correlations • Includes sampling error favoring the correlation but excludes error that doesn’t Vul et al., 2009
  • 53. Week 4.1: Model Comparison ! Lab: Interactions Practice ! Model Comparison ! Nested Models ! Hypothesis Testing ! REML vs ML ! Non-Nested Models ! Shrinkage ! The Problem ! Solutions
  • 54. Shrinkage—Solutions • One solution: Select model(s) in advance (perhaps even pre-registered) • A theory is valuable for this • Adak Island example is implausible in part because there’s no causal reason why an island in Alaska would relate to stock prices “Just as you do not need to know exactly how a car engine works in order to drive safely, you do not need to understand all the intricacies of the economy to accurately read those gauges.” – Economic forecasting firm ECRI (quoted in Silver, 2012)
  • 55. Shrinkage—Solutions • One solution: Select model(s) in advance (perhaps even pre-registered) • A theory is valuable for this • Not driven purely by the data or by chance if we have an a priori reason to favor this variable “There is really nothing so practical as a good theory.” -- Social psychologist Kurt Lewin (Lewin’s Maxim)
  • 56. Shrinkage—Solutions • One solution: Select model(s) in advance (perhaps even pre-registered) • A theory is valuable for this • Not driven purely by the data or by chance if we have an a priori reason to favor this variable • Based on some other measure (e.g., another brain scan)
  • 57. Shrinkage—Solutions • One solution: Select model(s) in advance (perhaps even pre-registered) • A theory is valuable for this • Not driven purely by the data or by chance if we have an a priori reason to favor this variable • Based on some other measure (e.g., another brain scan) • Based on research design • For factorial experiments, typical to include all experimental variables and interactions • Research design implies you were interested in all of these
  • 58. Shrinkage—Solutions • For more exploratory analyses: Show that the finding replicates • On a second dataset • Test a model obtained from one subset of the data applies to another subset (cross-validation) • e.g., training and test sets • A better version: Do this with many randomly chosen subsets • Monte Carlo methods • Reading on Canvas for some general ways to do this in R