Recommended
More Related Content
What's hot
Similar to Correcting for unreliability & partial invariance: A two-stage path analysis approach
Similar to Correcting for unreliability & partial invariance: A two-stage path analysis approach (20)
Recently uploaded
Recently uploaded (19)
Correcting for unreliability & partial invariance: A two-stage path analysis approach
- 1. 1 CORRECTING FOR UNRELIABILITY & PARTIAL INVARIANCE: A TWO-STAGE PATH ANALYSIS APPROACH Hok Chio (Mark) Lai, Winnie Wing-Yee Tse, Gengrui Zhang, Yixiao Li, and Yu-Yu Hsiao
- 3. 3 Effect of Noninvariance Biased/Spurious group differences Biased/Spurious interactions (Hsiao* & Lai, 2018)
- 4. 4 Unreliability Bias regression slopes (e.g., Cole & Preacher, 2014) Can happen differentially across groups
- 5. 5 Solution 1: Joint Measurement and Structural Modeling
- 6. 6 Joint Modeling Not Always Practical Need a large model E.g., 3 structural coefficients, but ~ 100 parameters with JM Sample size implication
- 7. 7 Joint Modeling Not Always Practical
- 8. 8 Joint Modeling Not Always Practical Computational challenges with discrete indicators Maximum likelihood numerical integration with high dimensions Weighted least squares may need N >= 200 missing data handling
- 9. 9 Solution 2: Two-Stage Path Analysis (2S-PA) Lai & Hsiao* (2021, Psychological Methods); Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 10. 10 2S-PA First stage: Obtain one indicator ( ) per latent construct ( ) E.g., regression scores; EAP scores Adjust for noninvariance Second stage: path modeling with and standard error of measurement/reliability Available in standard psychometric software Lai & Hsiao* (2021, Psychological Methods); Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 11. 11 2S-PA Stage 1: η σ2 ε Lai & Hsiao* (2021, Psychological Methods); Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 12. 12 2S-PA Stage 1: Stage 2: η* 1 σ2 ε Lai & Hsiao* (2021, Psychological Methods); Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 13. 13 What Data Can Use 2S-PA? Normal continuous indicators Discrete items Individual-specific error variance Handled with definition variables η* 1 σ2 ε Lai & Hsiao* (2021, Psychological Methods); Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 14. 14 Stage 2 Estimation Maximum likelihood (Mplus, OpenMx) Bayesian (e.g., Stan) Lai & Hsiao* (2021, Psychological Methods); Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 15. 15 Simulation Treatment (X, observed) Mediator (ηM, 6 continuous indicators) Outcome (ηY, 16 binary indicators) Two groups (n = 50, 100, 300 per group) 3 noninvariant items for ηM, 5 for ηY Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 16. 16 Results Convergence rate (%) n SEM (JM) 2S-PA 50 10.73 92.44 100 33.47 100 300 82.64 100 Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 17. 17 Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 18. 18
- 19. 19 Summary Separate estimation seems a good alternative strategy for adjust both noninvariance and unreliability Plus better small sample performance Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
- 20. 20 References Cole, D. A., & Preacher, K. J. (2014). Manifest variable path analysis: Potentially serious and misleading consequences due to uncorrected measurement error. Psychological Methods, 19(2), 300–315. https://doi.org/10.1037/a0033805 Hsiao, Y.-Y., & Lai, M. H. C. (2018). The impact of partial measurement invariance on testing moderation for single and multi- level data. Frontiers in Psychology, 9, Article 740. https://doi.org/10.3389/fpsyg.2018.00740 Lai, M. H. C., & Hsiao, Y.-Y. (2021). Two-stage path analysis with definition variables: An alternative framework to account for measurement error. Psychological Methods. https://doi.org/10.1037/met0000410 Lai, Tse*, Zhang*, Li*, & Hsiao* (under review)
Editor's Notes
- Good morning, thank you for having me. It’s been a great learning experience for me hearing from the previous presentations. Today I'm excited to share with you a recent work on two-stage path analysis. Specifically, I’ll talk about how to use this approach to correct for both unreliability and partial invariance. This is joint work with my grad students, Winnie Tse & Gengrui Zhang, Yixiao Li who’s an undergrad in my lab, & my colleague Yu-Yu Hsiao
- As we know, measures in the social and behavioral sciences are usually not 100% accurate, and there could be systematic differences due to different individual and contextual factors, which we call violations of measurement invariance, or simply noninvariance For example, the graph here shows that for the same true depression level, females, the red line, tend to get a higher observed score than males. Therefore, any gender differences we found on the test scores could be just due to noninvariance.
- As a result, noninvariance can lead to biased or spurious group differences, and can also lead to biased or spurious interactions. Consider the graph here with two groups, where the regression line is the same for the two groups if we have an accurate measurement of Y. When noninvariance exists, it can push the scores for group 1 upward, and the association between the test and the latent variable may change.
- The effect of measurement noninvariance is usually accompanied by the presence of unreliability, because our measures are rarely, if ever, perfectly reliable. It is well known in the literature that unreliability can bias regression slopes. But when the reliability is different across groups, it creates differential impact, as shown on the right, where the reliability is lower for G2, in blue than for G1, in red. Therefore, we can get very misleading results if we don’t pay attention to measurement bias and measurement unreliability.
- A usual solution is to do what we call joint modeling, where we model both the relation between the indicators and the latent variables, which is the measurement model, and also the relations among the latent variables, or the structural model, simultaneously
- However, as many of you who have experience with SEM modeling may know, joint modeling is not always practical. For one thing, joint modeling requires using all indicators, and the number of indicators is usually much larger than the number of constructs. So for the graph in the last page, even though there is only three constructs, we need to estimate like 100 parameters. As such, we may need a large sample size for stable estimation
- And in small samples, the more complex the model, the more likely we run into convergence issues
- In addition, there are computational challenges when we have discrete items or indicators. In the frequentist framework, one usually either uses maximum likelihood or weighted least squares estimations. The challenge with ML is that it requires numerical integration, and so is not feasible in high dimensional problems. With WLS, it generally requires even a larger sample size, and have a stricter requirement on missing data.
- As an alternative, we can consider a two-stage estimation strategy, using what we call two-stage path analysis, abbreviated as 2S-PA. We first proposed this in a recent paper, currently in press in psychological methods As opposed to joint modeling, the 2S-PA approach only requires one indicator per construct, so the structural model is much easier to specify. And by reducing the model size, it also has a smaller sample size requirement.
- In the first stage of 2S-PA, we use psychometric analyses to obtain the best estimates of the latent variables. We call these estimates eta tilde. For example, we can obtain different kinds of factor scores with factor analysis or item response theory. It is also important in this stage that we account for measurement noninvariance, like using a partial invariance model. In the second stage, we treat eta tilde as single indicators of the latent variables in the path model. In addition, we also need the standard error of measurement, or the reliability, of eta tilde. These are usually available in standard psychometric software. So with these, we can account for partial invariance in the first stage, and the unreliability in the second stage.
- Here’s a bit more detail. In the first stage, say we have a construct, eta, of interest. For each construct, we perform a psychometric analysis to obtain an estimate or prediction, which we call eta titlde, of each person’s latent variable score. For many models, eta tilde can be expressed as a times eta plus error, where a is a scaling constant, and the error follows classical measurement error with mean 0, and is uncorrelated with eta. As an example, the first column here shows the factor scores for a latent variable fm, the second column is the standard error of measurement, and the third is the reliability of the factor scores. When the indicators are continuous, the standard error of measurement is constant, but for discrete indicators, the standard error is usually person-specific, like the numbers in the last two columns.
- After we obtain an indicator for each construct in the first stage, the second stage is basically a structural equation model with single indicators. However, notice the subscript i here, which allows the standard error of measurement to be different across observations. This is needed when the indicators were obtained using models like IRT with discrete indicators. With single indicators, the model is identified by constraining the measurement error variance based on the standard error of measurement values in the first stage. The indicator loading is set to 1 for convenience, so the latent variables are generally scaled differently than in the first stage. However, this is not a problem if we’re interested in the standardized coefficients, meaning we rescale the parameters so that the latent variables have variances of 1.
- While the idea of two stage estimation is not new, previous methods usually do not consider both noninvariance and unreliability. Also, an advantage of 2S-PA is it can handle not just continuous indicators, but also discrete items. With discrete items, the error variance is usually different for different individuals, but this can be handled using definition variables. The idea is, instead of setting a common constraint for every observation, we can allow the constraint to depend on a variable, like the one represented in diamond in the diagram, for individual-specific error variance.
- The estimation can be done in standard software for structural equation modeling, such as Mplus and OpenMx, with maximum likelihood. It can also be done with Bayesian estimation such as in Stan.
- To compare 2S-PA and joint modeling, we conducted a simulation study. The paper is currently under review. Actually we just got the invitation to revise a few days ago. We simulated data from a mediation model with three variables, the one shown in an earlier slide. The treatment variable, X, is observed, while the mediator, eta M, is measured by six continuous indicators, and the outcome, eta Y, is measured by 16 binary indicators. The simulated data have two groups, with sample size conditions of 50, 100, and 300 per group. There were three noninvariant items for eta M, and five noninvariant items for eta Y, across the two groups.
- Here’s a summary of the results. First we look at the convergence rate. As you can see in the table, joint modeling has a lot of convergence issues, like when n is 50 per group, the convergence rate was only 10%. On the other hand, with 2S-PA it was 92%. While convergence rates improved when the sample size increased, even with 300 observations, joint modeling was still only at about 80% convergence rate, whereas 2S-PA achieved close to 100% convergence rate with just 100 observations per group.
- The graph here shows the parameter bias. The methods are shown in the x-axis. FS-PA means factor score path analysis without adjusting for measurement error, which, as expected, gave biased estimates when the true coefficients were not zero. This is the classic attenuation due to measurement error. For JM and 2S-PA, we can see they were close to unbiased when the sample sizes are large, but in smaller samples, 2S-PA outperformed JM.
- We also look at inferences. Here I only compare 2S-PA and JM, as FS-PA does not give consistent estimates. The x-axis is the sample size, and the y-axis is the coverage of the 95% CI. We want the empirical coverage rates to be close to or above 95%. As you can see, with JM, which is represented by the blue lines, the coverage of the 95% confidence intervals was quite low in small samples. On the other hand, with 2S-PA, it performed quite well even in small samples. This includes all the path coefficients, as well as the product term beta_1 beta_3, or the indirect effect.
- To summarize the study, we found that the separate estimation strategy in 2S-PA seems a good alternative for adjusting both noninvariance and unreliability, especially in small samples.
- I want to end by thanking Oi-man for organizing this great opportunity, and allowing me to be part of it. Thank you, you audience, for staying with me. I’m looking forward to your questions and suggestions.