Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multidirectional survey measurement errors: the latent class MTMM model

459 views

Published on

Perhaps one day we will figure out how to ask perfect survey questions. In the meantime, survey analyses are biased by random and correlated measurement errors, and evaluating the extent of such errors is therefore essential, both to remove the bias and to improve our question design.

When there is no gold standard, these errors are often estimated using multitrait-multimethod (MTMM) experiments or longitudinal data by applying linear or ordinal factor models, which assume that (latent) measurement is linear and that the only type of method bias is one that pushes the answers monotonely in a particular direction—that of acquiescence, for example. However, not all measurement is linear and not all method bias is monotone. Extreme response tendencies, for example, are nonmonotone, as are primacy and recency effects, which act on just one category. Just as the monotone kind, such method effects will also lead to spurious dependencies among different survey questions, distorting their true relationships. Diagnosing, preventing, or correcting for such distortions therefore calls for a model that can account for them.

For this purpose I will discuss the latent class MTMM model (Oberski 2011). In it, a latent loglinear modeling approach is combined with the MTMM design to yield a model that provides detailed information about the measurement quality of survey questions while also dealing with nonmonotone method biases. I will discuss the method's assumptions and demonstrate it on a few often-used survey questions. Standard software for latent class analysis can be used to estimate this model, so that evaluating the extent of nonlinear random and correlated measurement errors is now a reasonably user-friendly experience for survey researchers.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Multidirectional survey measurement errors: the latent class MTMM model

  1. 1. Multidirectional errors Latent class MTMM Conclusions Multidirectional survey measurement errors: the latent class MTMM model Daniel Oberski / doberski@uvt.nl Department of methodology & statistics AAPOR 2015 Latent class MTMM Daniel Oberski / doberski@uvt.nl
  2. 2. Multidirectional errors Latent class MTMM Conclusions ..1 Multidirectional survey measurement errors ..2 Latent class multitrait-multimethod model ..3 Conclusions Latent class MTMM Daniel Oberski / doberski@uvt.nl
  3. 3. Multidirectional errors Latent class MTMM Conclusions Stochastic errors Two respondents who: • Went to the doctor the same number of times • Have the same opinion about the role of women in society give different answers to these questions. This will bias estimates of relationships between the variables. Latent class MTMM Daniel Oberski / doberski@uvt.nl
  4. 4. Multidirectional errors Latent class MTMM Conclusions Stochastic errors • Latent variables can never be recovered Latent class MTMM Daniel Oberski / doberski@uvt.nl
  5. 5. Multidirectional errors Latent class MTMM Conclusions Stochastic errors • Latent variables can never be recovered • But latent variable models can recover the amount of influence they exert Latent class MTMM Daniel Oberski / doberski@uvt.nl
  6. 6. Multidirectional errors Latent class MTMM Conclusions Stochastic errors • Latent variables can never be recovered • But latent variable models can recover the amount of influence they exert • This is useful to remove bias Latent class MTMM Daniel Oberski / doberski@uvt.nl
  7. 7. Multidirectional errors Latent class MTMM Conclusions Stochastic errors • Latent variables can never be recovered • But latent variable models can recover the amount of influence they exert • This is useful to remove bias Latent class MTMM Daniel Oberski / doberski@uvt.nl
  8. 8. Multidirectional errors Latent class MTMM Conclusions Stochastic errors • Latent variables can never be recovered • But latent variable models can recover the amount of influence they exert • This is useful to remove bias • Multitrait-multimethod (MTMM) approach to estimating this influence (Andrews 1984; Saris & Andrews 1991; Saris & Gallhofer 2007); Latent class MTMM Daniel Oberski / doberski@uvt.nl
  9. 9. Multidirectional errors Latent class MTMM Conclusions Stochastic errors • Latent variables can never be recovered • But latent variable models can recover the amount of influence they exert • This is useful to remove bias • Multitrait-multimethod (MTMM) approach to estimating this influence (Andrews 1984; Saris & Andrews 1991; Saris & Gallhofer 2007); • Quasi-simplex approach (Wiley & Wiley 1970; Alwin 2007). Latent class MTMM Daniel Oberski / doberski@uvt.nl
  10. 10. Multidirectional errors Latent class MTMM Conclusions Stochastic errors • Latent variables can never be recovered • But latent variable models can recover the amount of influence they exert • This is useful to remove bias • Multitrait-multimethod (MTMM) approach to estimating this influence (Andrews 1984; Saris & Andrews 1991; Saris & Gallhofer 2007); • Quasi-simplex approach (Wiley & Wiley 1970; Alwin 2007). Latent class MTMM Daniel Oberski / doberski@uvt.nl
  11. 11. Multidirectional errors Latent class MTMM Conclusions Stochastic errors • Latent variables can never be recovered • But latent variable models can recover the amount of influence they exert • This is useful to remove bias • Multitrait-multimethod (MTMM) approach to estimating this influence (Andrews 1984; Saris & Andrews 1991; Saris & Gallhofer 2007); • Quasi-simplex approach (Wiley & Wiley 1970; Alwin 2007). ↑ Linear models Latent class MTMM Daniel Oberski / doberski@uvt.nl
  12. 12. Multidirectional errors Latent class MTMM Conclusions Stochastic errors in the literature • Random mistakes • Acquiescence • Answering in the socially desirable direction • Tending to choose the first/last of several categories • Extreme response (outer categories) • Avoiding some particular category for whatever reason • Preferring the midpoint • Heaping (rounding) • ... Problem: Not all of these are linear Stochastic errors are multidirectional Latent class MTMM Daniel Oberski / doberski@uvt.nl
  13. 13. Multidirectional errors Latent class MTMM Conclusions Example of the effect of a nonlinear error on estimate of relationships True relationship 1 2 3 4 1 136 112 91 75 2 112 75 50 34 3 91 50 28 15 4 75 34 15 7 Polychoric correlation = -0.25 Relationship with ERS 1 2 3 4 1 374 41 34 205 2 41 4 3 12 3 34 3 1 6 4 205 12 6 19 Polychoric correlation = -0.43 Latent class MTMM Daniel Oberski / doberski@uvt.nl
  14. 14. Multidirectional errors Latent class MTMM Conclusions Problem: Stochastic errors can strongly bias relationship estimates; Problem: Linear latent variable models assume errors are all one-way, so bias is not appropriately removed. Solution: Latent class models to allow for multidirectional errors. Here MTMM but could also be quasi-simplex Latent class MTMM Daniel Oberski / doberski@uvt.nl
  15. 15. Multidirectional errors Latent class MTMM Conclusions The latent class MTMM model Oberski, Hagenaars & Saris, to appear in Psychological Methods. Latent class MTMM Daniel Oberski / doberski@uvt.nl
  16. 16. Multidirectional errors Latent class MTMM Conclusions M1 M2 M3 T1 T2 T3 y11 y21 y31 y12 y22 y32 y13 y23 y33 • Latent variables are discrete (categorical) • Observed may be continuous or discrete • Relationships (can be) nonparametric Latent class MTMM Daniel Oberski / doberski@uvt.nl
  17. 17. Multidirectional errors Latent class MTMM Conclusions Experimental design: split-ballot MTMM Method 1 Method 2 Method 3 Random group 1 . . Random group 2 . . Latent class MTMM Daniel Oberski / doberski@uvt.nl
  18. 18. Multidirectional errors Latent class MTMM Conclusions Opinion about the role of women experiment: Main questionnaire (first method) Latent class MTMM Daniel Oberski / doberski@uvt.nl
  19. 19. Multidirectional errors Latent class MTMM Conclusions Opinion about the role of women experiment: Supplementary group 1 (second method) Latent class MTMM Daniel Oberski / doberski@uvt.nl
  20. 20. Multidirectional errors Latent class MTMM Conclusions Opinion about the role of women experiment: Supplementary group 2 (third method) Latent class MTMM Daniel Oberski / doberski@uvt.nl
  21. 21. Multidirectional errors Latent class MTMM Conclusions Effect of trait on item distribution, Greece Men more right, positive agree-disgree 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Latent class MTMM Daniel Oberski / doberski@uvt.nl 1 2 3 4 5 Factor score: 0.00 Category Proportion 0.00.20.40.60.81.0
  22. 22. Multidirectional errors Latent class MTMM Conclusions Effect of trait on item distribution, Slovenia Men more right, negative agree-disgree 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Latent class MTMM Daniel Oberski / doberski@uvt.nl 1 2 3 4 5 Factor score: 0.00 Category Proportion 0.00.20.40.60.81.0
  23. 23. Multidirectional errors Latent class MTMM Conclusions Method factors have a nonlinear influence on the items 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Cut down, method 1 Trait Method 1 2 34 5 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Cut down, method 2 Trait Method 1 2 3 4 5 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Cut down, method 3 Trait Method 1 2 3 4 5 Latent class MTMM Daniel Oberski / doberski@uvt.nl
  24. 24. Multidirectional errors Latent class MTMM Conclusions Prevalence of method behavior Latent class MTMM Daniel Oberski / doberski@uvt.nl
  25. 25. Multidirectional errors Latent class MTMM Conclusions Conclusion Latent class MTMM Daniel Oberski / doberski@uvt.nl
  26. 26. Multidirectional errors Latent class MTMM Conclusions Conclusion • Stochastic errors are important when you are interested in relationships; • Need latent variable models to estimate their extent so their effects can be removed statistically; • In traditional MTMM and quasi-simplex models, these effects are all one-way; • Latent class models allow for multidirectional errors. • Example: the latent class MTMM model (Oberski et al., to appear - see http://daob.nl/publications) Latent class MTMM Daniel Oberski / doberski@uvt.nl
  27. 27. Multidirectional errors Latent class MTMM Conclusions Thank you for your attention! doberski@uvt.nl @DanielOberski See http://daob.nl/publications for preprints Supported by Veni grant number 451-14-017 from the Netherlands Organization for Scientific Research (NWO). Latent class MTMM Daniel Oberski / doberski@uvt.nl

×