0
PRINCIPLES OF QUESTIONNAIRE DEVELOPMENT: MULTIVARIATE APPROACH Dr. D. Dutta Roy, Ph.D. Psychology Research Unit INDIAN STA...
Questionnaire development  is an Art. Develop Multivariate Temperament
Items are flying, feel,catch and paint them with your inner voice.
Observation  <ul><li>Observe everywhere, and ask question: </li></ul><ul><ul><li>Why does it happen? </li></ul></ul><ul><u...
Exploration <ul><li>Do extensive literature survey; Organize them systematically  </li></ul><ul><li>and explore. like a ch...
Dreaming <ul><li>Spend time alone and dream.  </li></ul><ul><li>Dream life style of your population for research. </li></u...
Experience your dream
Respect Classics  <ul><li>Review Pioneering articles; </li></ul><ul><li>Study their perspectives; </li></ul><ul><li>Do tre...
Perspective  <ul><li>Study same object of research from multiple perspective.  </li></ul>
Painting & Writing your feelings Your writing will help you to develop items/statements of questionnaire
Questionnaire Development is a process of structuring and restructuring items
Statistical Modeling (Hypothetical item linkage)  D1 D2 D3 D4 I2 I1 I4 I5 I6 I10 I11 I7 I8 I9 I3 D: Dimensions; I: Item <u...
Statistical Modeling (Hypothetical Regression) D1= i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10+i11w11 D2=...
Find out variables for Host’s response to items Agent Host Environment
Dynamics of variables  Initial E X P R E S S I O n Middle Terminal A E H H H
Make sampling in such a fashion so that role of intervening variables on item response can be controlled Can we assess the...
Observe and interview sample to find out underlying reasons behind item-response variation
MULTIVARIATE ANALYSIS OF QUESTIONNAIRE RESPONSE
Myths about Multivariate Statistical Models
MYTHS <ul><li>Statistical treatment of more than 2 variables is multivariate statistics; </li></ul><ul><ul><li>No, when mo...
Myth 2 <ul><li>Purpose of multivariate statistics is to establish correlation among sets of variables. </li></ul><ul><ul><...
Myth 3 <ul><li>Loss of original score  </li></ul><ul><ul><li>Accepted, if analysis extracts more latent properties within ...
What is Multivariate Statistics ? <ul><li>MVS refers to the set of statistical tools that simultaneously analyze multiple ...
Assumptions for Multi-variate Regression Models <ul><li>linearity of relationships,  </li></ul><ul><li>homoscedasticity (s...
Multivariate model for item analysis  <ul><li>Multivariate model helps researcher to develop insight about possible impact...
Reliability Analysis <ul><li>Reliability refers to the consistency of scores. </li></ul><ul><li>Types : Time and Internal ...
Test-Retest Multi-item response <ul><li>All items do not behave in same fashion always. </li></ul><ul><li>Identify inconsi...
Test-Retest Multi-item response Consistency (8 months interval)
Test-Retest Multi-Trait Consistency  (8 months interval) After 8 moths Tool: Reading motivation questionnaire (Dutta Roy, ...
Split-half  <ul><li>Upper and lower part of the questionnaire sometimes differ in item content. </li></ul><ul><li>All item...
Split-half Canonical correlation <ul><li>Split-half Canonical correlation provides knowledge about the percent of variance...
Study: Split-half Canonical correlation <ul><li>12-item Likert type 5 point scale assessing attitude towards workers educa...
Internal Consistency <ul><li>It measures whether several items that propose to measure the same general construct produce ...
Item-Item correspondence: Internal consistency among 42 items of Reading Motivation questionnaire Correspondence Map shows...
Correspondence map of traits
Validity <ul><li>Can  questionnaire predict change in criterion ? </li></ul><ul><li>Can questionnaire make difference ?  <...
Types of Validity  Expert Judgment Content Item-Total correlation Construct Factorial  Criterion Concurrent Predictive Con...
Validity  <ul><li>Validity denotes the extent to which an instrument is measuring what it is supposed to measure It indica...
Content validity  Colors of all items are not same Some item behaves differently  <ul><li>Content Validity (It is concerne...
Predictive validity  <ul><li>Questionnaires measuring same variable provide different results.  </li></ul><ul><li>Which qu...
Concurrent validity  <ul><li>Collect data simultaneously with two instruments measuring similar construct.  </li></ul><ul>...
Construct validity  <ul><li>Questionnaire should measure the underlying theories of research construct.  </li></ul>
Factorial validity <ul><li>Items can be classified by respondent’s latent phenomena.  </li></ul>
Factorial validity <ul><li>Two questionnaires can be mixed with respondent’s frame of reference. </li></ul><ul><li>Princip...
Studies on Validity of questionnaire
Studies CONTENT  Item-Rating correspondence of questionnaire measuring computer programming task taxonomy. CRITERION RELAT...
Content validity: Item-Rating Correspondence Input data for CA Dutta Roy,D. (2002). Correspondence between item and rating...
Criterion related validity:Stepwise Multiple regression  Inter correlation matrix of reading motivation variables  (n=200)...
Correspondence analysis to explore latent structure of RMQ Chi-square (36) = 3041.4, p< 0.00001
Discriminative validity of Attitude towards School Infrastructure questionnaire (ASIQ) <ul><li>ASIQ would be able to diffe...
Multivariate model is the Journey to Harmony  THANK YOU
Upcoming SlideShare
Loading in...5
×

Multivariate Models in Questionnaire Development

6,213

Published on

Statistics, Psychology, Questionnaire, Multivariate,
Canonical, Dutta Roy

Published in: Education
2 Comments
3 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,213
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
225
Comments
2
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Multivariate Models in Questionnaire Development"

  1. 1. PRINCIPLES OF QUESTIONNAIRE DEVELOPMENT: MULTIVARIATE APPROACH Dr. D. Dutta Roy, Ph.D. Psychology Research Unit INDIAN STATISTICAL INSTITUTE 203, B.T. Road, Kolkata – 700 108 E-mail: ddroy @ isical .ac.in http://www.isical.ac.in/~ddroy Venue: P.G. Department of Psychology, Dr. Babasaheb Ambedkar Marathwada University, Aurangabad (MS) 431004, India
  2. 2. Questionnaire development is an Art. Develop Multivariate Temperament
  3. 3. Items are flying, feel,catch and paint them with your inner voice.
  4. 4. Observation <ul><li>Observe everywhere, and ask question: </li></ul><ul><ul><li>Why does it happen? </li></ul></ul><ul><ul><li>How can I control it ? </li></ul></ul><ul><li>Observe carefully, you may find light in the shadow behind light. </li></ul>
  5. 5. Exploration <ul><li>Do extensive literature survey; Organize them systematically </li></ul><ul><li>and explore. like a child. </li></ul>
  6. 6. Dreaming <ul><li>Spend time alone and dream. </li></ul><ul><li>Dream life style of your population for research. </li></ul><ul><li>Relate each property or dimension of your research construct to their life style. </li></ul><ul><li>You may find multi-dimensions. </li></ul>
  7. 7. Experience your dream
  8. 8. Respect Classics <ul><li>Review Pioneering articles; </li></ul><ul><li>Study their perspectives; </li></ul><ul><li>Do trend analysis of construct development; </li></ul>
  9. 9. Perspective <ul><li>Study same object of research from multiple perspective. </li></ul>
  10. 10. Painting & Writing your feelings Your writing will help you to develop items/statements of questionnaire
  11. 11. Questionnaire Development is a process of structuring and restructuring items
  12. 12. Statistical Modeling (Hypothetical item linkage) D1 D2 D3 D4 I2 I1 I4 I5 I6 I10 I11 I7 I8 I9 I3 D: Dimensions; I: Item <ul><li>Classify items by content analysis; </li></ul><ul><li>Write Dimension and hypothesize item linkages across different quadrants. </li></ul><ul><li>Each dimension represents specific trait of research construct. </li></ul>
  13. 13. Statistical Modeling (Hypothetical Regression) D1= i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10+i11w11 D2=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10 +i11w11 D3=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10+ +i11w11 D4=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10 +i11w11 D:Dimension; I:Item; w=Beta weight
  14. 14. Find out variables for Host’s response to items Agent Host Environment
  15. 15. Dynamics of variables Initial E X P R E S S I O n Middle Terminal A E H H H
  16. 16. Make sampling in such a fashion so that role of intervening variables on item response can be controlled Can we assess their perception, beliefs and attitudes ?
  17. 17. Observe and interview sample to find out underlying reasons behind item-response variation
  18. 18. MULTIVARIATE ANALYSIS OF QUESTIONNAIRE RESPONSE
  19. 19. Myths about Multivariate Statistical Models
  20. 20. MYTHS <ul><li>Statistical treatment of more than 2 variables is multivariate statistics; </li></ul><ul><ul><li>No, when more than 2 variables are interrelated with each other, we can use multivariate statistics.In questionnaire, multiple questions measuring same issue are used. So MVS assumptions are more effective in questionnaire development. </li></ul></ul>
  21. 21. Myth 2 <ul><li>Purpose of multivariate statistics is to establish correlation among sets of variables. </li></ul><ul><ul><li>True. But it’s purpose is not limited in determining relation among sets of variables. It tends to partial out the effect of some intervening variables on relationship among sets of variables. Response to items varies with other than the construct measured by questionnaire, therefore, control of intervening variable on item response is necessary. </li></ul></ul>IV INTR_V DV It measures intervening vars and gives insight to rectify
  22. 22. Myth 3 <ul><li>Loss of original score </li></ul><ul><ul><li>Accepted, if analysis extracts more latent properties within the variable. </li></ul></ul>
  23. 23. What is Multivariate Statistics ? <ul><li>MVS refers to the set of statistical tools that simultaneously analyze multiple measurements on each individual or object under investigation. </li></ul><ul><li>It is the linear combination of variables with empirically determined weights. </li></ul><ul><ul><li>Variate value= w1X1+ w2X2+ w3X3 – wnXn </li></ul></ul><ul><ul><ul><li>W=weight determined by the multivariate technique; X=observed variable </li></ul></ul></ul><ul><li>MVS is the extension of univariate (central tendency, SD, variance) and bivariate analysis (cross tabulation, correlation, ANOVA, simple regression). </li></ul><ul><li>In MVS, all the variables must be random and interrelated in such ways that their different effects can not meaningfully be interpreted separately. </li></ul>
  24. 24. Assumptions for Multi-variate Regression Models <ul><li>linearity of relationships, </li></ul><ul><li>homoscedasticity (same level of relationship for the full range of the data), </li></ul><ul><li>interval or near-interval data, </li></ul><ul><li>untruncated variables, proper specification of the model, </li></ul><ul><li>lack of high multicollinearity, and </li></ul><ul><li>multivariate normality for purposes of hypothesis testing. </li></ul>
  25. 25. Multivariate model for item analysis <ul><li>Multivariate model helps researcher to develop insight about possible impact of rejecting individual item on the set of items in which specific item belongs to. This is specially true when researcher assumes interdependence among sets of items. </li></ul><ul><li>When questionnaire assesses multi-traits, multivariate analysis helps to understand the latent structure or inherent relations among the different traits. So, it represents psychological map of the respondents. </li></ul>
  26. 26. Reliability Analysis <ul><li>Reliability refers to the consistency of scores. </li></ul><ul><li>Types : Time and Internal consistency; </li></ul>
  27. 27. Test-Retest Multi-item response <ul><li>All items do not behave in same fashion always. </li></ul><ul><li>Identify inconsistent items in the set across periods.. </li></ul>Last supper: Leonardo Da Vinci
  28. 28. Test-Retest Multi-item response Consistency (8 months interval)
  29. 29. Test-Retest Multi-Trait Consistency (8 months interval) After 8 moths Tool: Reading motivation questionnaire (Dutta Roy, 2002); N=72 students of same school
  30. 30. Split-half <ul><li>Upper and lower part of the questionnaire sometimes differ in item content. </li></ul><ul><li>All items do not reflect same content always. </li></ul>
  31. 31. Split-half Canonical correlation <ul><li>Split-half Canonical correlation provides knowledge about the percent of variance in the one set explained by the other set of variables along a given dimension . </li></ul>
  32. 32. Study: Split-half Canonical correlation <ul><li>12-item Likert type 5 point scale assessing attitude towards workers education was administered to 1600 rural workers of WB. </li></ul><ul><li>Split-half rtt=0.85; Cronbach’s alpha = 0.87 </li></ul><ul><li>Canonical correlation coefficient between the sets (first 6 and last 6 items) = 0.78, Chisq(36)=1558.3, p<0.0000. </li></ul>
  33. 33. Internal Consistency <ul><li>It measures whether several items that propose to measure the same general construct produce similar scores. For example, if a respondent expressed agreement with the statements &quot;I like to ride bicycles&quot; and &quot;I've enjoyed riding bicycles in the past&quot;, and disagreement with the statement &quot;I hate bicycles&quot;, this would be indicative of good internal consistency of the test. </li></ul>
  34. 34. Item-Item correspondence: Internal consistency among 42 items of Reading Motivation questionnaire Correspondence Map shows cluster of intrinsic reading motivation items and extrinsic motivation items are scattered widely.
  35. 35. Correspondence map of traits
  36. 36. Validity <ul><li>Can questionnaire predict change in criterion ? </li></ul><ul><li>Can questionnaire make difference ? </li></ul>
  37. 37. Types of Validity Expert Judgment Content Item-Total correlation Construct Factorial Criterion Concurrent Predictive Convergent Discriminative
  38. 38. Validity <ul><li>Validity denotes the extent to which an instrument is measuring what it is supposed to measure It indicates extent of relationship between a scale and the measure of independent criterion variable. It is assumed that criterion variable is reliable and valid. </li></ul><ul><li>Bi-variate techniques: </li></ul><ul><li>Content Validity (It is concerned with the relevance of contents of items, individually and as a whole): </li></ul><ul><ul><li>Correlating experts’ judgement; </li></ul></ul><ul><ul><li>Item-item or item-total correlation </li></ul></ul><ul><li>Criterion - related Validity (Correlating questionnaire scores with criterion variable): </li></ul><ul><ul><li>Correlating the test with criteria during data collection (Concurrent validity); </li></ul></ul><ul><ul><li>before or after (keeping time gap) (Predictive validity). </li></ul></ul><ul><li>Construct Validity : It is concerned with the extent to which questionnaire measures a theoretical construct or trait. Construct is a sort of concept, which is formally proposed with definition and is related to empirical data. </li></ul><ul><ul><li>Techniques:Factor analysis (Factorial validity); </li></ul></ul><ul><ul><li>Correlating with other theoretical measure with which the developing instrument should correlate (convergent validity) </li></ul></ul><ul><ul><li>Correlating with other theoretical measure with which the developing instrument should not correlate (Discriminant validity) </li></ul></ul>
  39. 39. Content validity Colors of all items are not same Some item behaves differently <ul><li>Content Validity (It is concerned with the relevance of contents of items, individually and as a whole): . </li></ul><ul><li>Correlating experts’ judgement; </li></ul><ul><li>Item-item or item-total correlation </li></ul>
  40. 40. Predictive validity <ul><li>Questionnaires measuring same variable provide different results. </li></ul><ul><li>Which questionnaire predicts success ? </li></ul>
  41. 41. Concurrent validity <ul><li>Collect data simultaneously with two instruments measuring similar construct. </li></ul><ul><li>Use canonical correlation if product moment correlation is significant. </li></ul>
  42. 42. Construct validity <ul><li>Questionnaire should measure the underlying theories of research construct. </li></ul>
  43. 43. Factorial validity <ul><li>Items can be classified by respondent’s latent phenomena. </li></ul>
  44. 44. Factorial validity <ul><li>Two questionnaires can be mixed with respondent’s frame of reference. </li></ul><ul><li>Principal component analysis can be used to find common properties. </li></ul>
  45. 45. Studies on Validity of questionnaire
  46. 46. Studies CONTENT Item-Rating correspondence of questionnaire measuring computer programming task taxonomy. CRITERION RELATED Step-wise Multiple regression in predicting academic achievement through reading motivation. Predictive CONSTRUCT Correspondence analysis to explore latent structure of RMQ Factorial Discriminative Discriminative validity of Attitude towards School Infrastructure questionnaire (ASIQ)
  47. 47. Content validity: Item-Rating Correspondence Input data for CA Dutta Roy,D. (2002). Correspondence between item and rating on the checklist of relative importance of computer programming tasks., Journal of Psychometry , 16,2,67-76.
  48. 48. Criterion related validity:Stepwise Multiple regression Inter correlation matrix of reading motivation variables (n=200) Data collected from Bengali medium schools
  49. 49. Correspondence analysis to explore latent structure of RMQ Chi-square (36) = 3041.4, p< 0.00001
  50. 50. Discriminative validity of Attitude towards School Infrastructure questionnaire (ASIQ) <ul><li>ASIQ would be able to differentiate between good and poor infrastructure schools. </li></ul><ul><li>Methods: ASIQ was administered to the students of primary schools with good and poor infrastructures. Good and poor criteria was determined by the 8 indices available in District Information System of Education (DISE). Based on Median rank, schools were classified into high and low infrastructure categories. </li></ul><ul><li>Results: stepwise discriminant function analysis ( [Wilks’ Lambda=0.77, Rao’s R (9,153)=5.13, p<0.00] ) extracted five most important attitudes namely, reliability, equal opportunity, comfort, safety, cleanliness in predicting differences between good and poor infrastructure schools. The overall classification accuracy is 82.8% . This suggests high predictive capacity of the Discriminant function obtained. </li></ul>Fisher’s Linear Discriminant Functions for differentiating Schools with Good and Poor Infrastructure
  51. 51. Multivariate model is the Journey to Harmony THANK YOU
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×