PRINCIPLES OF QUESTIONNAIRE DEVELOPMENT: MULTIVARIATE APPROACH Dr. D. Dutta Roy, Ph.D. Psychology Research Unit INDIAN STATISTICAL INSTITUTE 203, B.T. Road, Kolkata – 700 108 E-mail: ddroy @ isical .ac.in http://www.isical.ac.in/~ddroy Venue: P.G. Department of Psychology, Dr. Babasaheb Ambedkar Marathwada University, Aurangabad (MS) 431004, India

Questionnaire development is an Art. Develop Multivariate Temperament

Items are flying, feel,catch and paint them with your inner voice.

Observation Observe everywhere, and ask question: Why does it happen? How can I control it ? Observe carefully, you may find light in the shadow behind light.

Observe everywhere, and ask question:

Why does it happen?

How can I control it ?

Observe carefully, you may find light in the shadow behind light.

Exploration Do extensive literature survey; Organize them systematically and explore. like a child.

Do extensive literature survey; Organize them systematically

and explore. like a child.

Dreaming Spend time alone and dream. Dream life style of your population for research. Relate each property or dimension of your research construct to their life style. You may find multi-dimensions.

Spend time alone and dream.

Dream life style of your population for research.

Relate each property or dimension of your research construct to their life style.

You may find multi-dimensions.

Experience your dream

Respect Classics Review Pioneering articles; Study their perspectives; Do trend analysis of construct development;

Review Pioneering articles;

Study their perspectives;

Do trend analysis of construct development;

Perspective Study same object of research from multiple perspective.

Study same object of research from multiple perspective.

Painting & Writing your feelings Your writing will help you to develop items/statements of questionnaire

Questionnaire Development is a process of structuring and restructuring items

Statistical Modeling (Hypothetical item linkage) D1 D2 D3 D4 I2 I1 I4 I5 I6 I10 I11 I7 I8 I9 I3 D: Dimensions; I: Item Classify items by content analysis; Write Dimension and hypothesize item linkages across different quadrants. Each dimension represents specific trait of research construct.

Classify items by content analysis;

Write Dimension and hypothesize item linkages across different quadrants.

Each dimension represents specific trait of research construct.

Statistical Modeling (Hypothetical Regression) D1= i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10+i11w11 D2=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10 +i11w11 D3=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10+ +i11w11 D4=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10 +i11w11 D:Dimension; I:Item; w=Beta weight

Find out variables for Host’s response to items Agent Host Environment

Dynamics of variables Initial E X P R E S S I O n Middle Terminal A E H H H

Make sampling in such a fashion so that role of intervening variables on item response can be controlled Can we assess their perception, beliefs and attitudes ?

Observe and interview sample to find out underlying reasons behind item-response variation

MULTIVARIATE ANALYSIS OF QUESTIONNAIRE RESPONSE

Myths about Multivariate Statistical Models

MYTHS Statistical treatment of more than 2 variables is multivariate statistics; No, when more than 2 variables are interrelated with each other, we can use multivariate statistics.In questionnaire, multiple questions measuring same issue are used. So MVS assumptions are more effective in questionnaire development.

Statistical treatment of more than 2 variables is multivariate statistics;

No, when more than 2 variables are interrelated with each other, we can use multivariate statistics.In questionnaire, multiple questions measuring same issue are used. So MVS assumptions are more effective in questionnaire development.

Myth 2 Purpose of multivariate statistics is to establish correlation among sets of variables. True. But it’s purpose is not limited in determining relation among sets of variables. It tends to partial out the effect of some intervening variables on relationship among sets of variables. Response to items varies with other than the construct measured by questionnaire, therefore, control of intervening variable on item response is necessary. IV INTR_V DV It measures intervening vars and gives insight to rectify

Purpose of multivariate statistics is to establish correlation among sets of variables.

True. But it’s purpose is not limited in determining relation among sets of variables. It tends to partial out the effect of some intervening variables on relationship among sets of variables. Response to items varies with other than the construct measured by questionnaire, therefore, control of intervening variable on item response is necessary.

Myth 3 Loss of original score Accepted, if analysis extracts more latent properties within the variable.

Loss of original score

Accepted, if analysis extracts more latent properties within the variable.

What is Multivariate Statistics ? MVS refers to the set of statistical tools that simultaneously analyze multiple measurements on each individual or object under investigation. It is the linear combination of variables with empirically determined weights. Variate value= w1X1+ w2X2+ w3X3 – wnXn W=weight determined by the multivariate technique; X=observed variable MVS is the extension of univariate (central tendency, SD, variance) and bivariate analysis (cross tabulation, correlation, ANOVA, simple regression). In MVS, all the variables must be random and interrelated in such ways that their different effects can not meaningfully be interpreted separately.

MVS refers to the set of statistical tools that simultaneously analyze multiple measurements on each individual or object under investigation.

It is the linear combination of variables with empirically determined weights.

Variate value= w1X1+ w2X2+ w3X3 – wnXn

W=weight determined by the multivariate technique; X=observed variable

MVS is the extension of univariate (central tendency, SD, variance) and bivariate analysis (cross tabulation, correlation, ANOVA, simple regression).

In MVS, all the variables must be random and interrelated in such ways that their different effects can not meaningfully be interpreted separately.

Assumptions for Multi-variate Regression Models linearity of relationships, homoscedasticity (same level of relationship for the full range of the data), interval or near-interval data, untruncated variables, proper specification of the model, lack of high multicollinearity, and multivariate normality for purposes of hypothesis testing.

linearity of relationships,

homoscedasticity (same level of relationship for the full range of the data),

interval or near-interval data,

untruncated variables, proper specification of the model,

lack of high multicollinearity, and

multivariate normality for purposes of hypothesis testing.

Multivariate model for item analysis Multivariate model helps researcher to develop insight about possible impact of rejecting individual item on the set of items in which specific item belongs to. This is specially true when researcher assumes interdependence among sets of items. When questionnaire assesses multi-traits, multivariate analysis helps to understand the latent structure or inherent relations among the different traits. So, it represents psychological map of the respondents.

Multivariate model helps researcher to develop insight about possible impact of rejecting individual item on the set of items in which specific item belongs to. This is specially true when researcher assumes interdependence among sets of items.

When questionnaire assesses multi-traits, multivariate analysis helps to understand the latent structure or inherent relations among the different traits. So, it represents psychological map of the respondents.

Reliability Analysis Reliability refers to the consistency of scores. Types : Time and Internal consistency;

Reliability refers to the consistency of scores.

Types : Time and Internal consistency;

Test-Retest Multi-item response All items do not behave in same fashion always. Identify inconsistent items in the set across periods.. Last supper: Leonardo Da Vinci

All items do not behave in same fashion always.

Identify inconsistent items in the set across periods..

Test-Retest Multi-item response Consistency (8 months interval)

Test-Retest Multi-Trait Consistency (8 months interval) After 8 moths Tool: Reading motivation questionnaire (Dutta Roy, 2002); N=72 students of same school

Split-half Upper and lower part of the questionnaire sometimes differ in item content. All items do not reflect same content always.

Upper and lower part of the questionnaire sometimes differ in item content.

All items do not reflect same content always.

Split-half Canonical correlation Split-half Canonical correlation provides knowledge about the percent of variance in the one set explained by the other set of variables along a given dimension .

Split-half Canonical correlation provides knowledge about the percent of variance in the one set explained by the other set of variables along a given dimension .

Study: Split-half Canonical correlation 12-item Likert type 5 point scale assessing attitude towards workers education was administered to 1600 rural workers of WB. Split-half rtt=0.85; Cronbach’s alpha = 0.87 Canonical correlation coefficient between the sets (first 6 and last 6 items) = 0.78, Chisq(36)=1558.3, p<0.0000.

12-item Likert type 5 point scale assessing attitude towards workers education was administered to 1600 rural workers of WB.

Split-half rtt=0.85; Cronbach’s alpha = 0.87

Canonical correlation coefficient between the sets (first 6 and last 6 items) = 0.78, Chisq(36)=1558.3, p<0.0000.

Internal Consistency It measures whether several items that propose to measure the same general construct produce similar scores. For example, if a respondent expressed agreement with the statements &quot;I like to ride bicycles&quot; and &quot;I've enjoyed riding bicycles in the past&quot;, and disagreement with the statement &quot;I hate bicycles&quot;, this would be indicative of good internal consistency of the test.

It measures whether several items that propose to measure the same general construct produce similar scores. For example, if a respondent expressed agreement with the statements &quot;I like to ride bicycles&quot; and &quot;I've enjoyed riding bicycles in the past&quot;, and disagreement with the statement &quot;I hate bicycles&quot;, this would be indicative of good internal consistency of the test.

Item-Item correspondence: Internal consistency among 42 items of Reading Motivation questionnaire Correspondence Map shows cluster of intrinsic reading motivation items and extrinsic motivation items are scattered widely.

Correspondence map of traits

Validity Can questionnaire predict change in criterion ? Can questionnaire make difference ?

Can questionnaire predict change in criterion ?

Can questionnaire make difference ?

Types of Validity Expert Judgment Content Item-Total correlation Construct Factorial Criterion Concurrent Predictive Convergent Discriminative

Validity Validity denotes the extent to which an instrument is measuring what it is supposed to measure It indicates extent of relationship between a scale and the measure of independent criterion variable. It is assumed that criterion variable is reliable and valid. Bi-variate techniques: Content Validity (It is concerned with the relevance of contents of items, individually and as a whole): Correlating experts’ judgement; Item-item or item-total correlation Criterion - related Validity (Correlating questionnaire scores with criterion variable): Correlating the test with criteria during data collection (Concurrent validity); before or after (keeping time gap) (Predictive validity). Construct Validity : It is concerned with the extent to which questionnaire measures a theoretical construct or trait. Construct is a sort of concept, which is formally proposed with definition and is related to empirical data. Techniques:Factor analysis (Factorial validity); Correlating with other theoretical measure with which the developing instrument should correlate (convergent validity) Correlating with other theoretical measure with which the developing instrument should not correlate (Discriminant validity)

Validity denotes the extent to which an instrument is measuring what it is supposed to measure It indicates extent of relationship between a scale and the measure of independent criterion variable. It is assumed that criterion variable is reliable and valid.

Bi-variate techniques:

Content Validity (It is concerned with the relevance of contents of items, individually and as a whole):

Correlating experts’ judgement;

Item-item or item-total correlation

Criterion - related Validity (Correlating questionnaire scores with criterion variable):

Correlating the test with criteria during data collection (Concurrent validity);

before or after (keeping time gap) (Predictive validity).

Construct Validity : It is concerned with the extent to which questionnaire measures a theoretical construct or trait. Construct is a sort of concept, which is formally proposed with definition and is related to empirical data.

Techniques:Factor analysis (Factorial validity);

Correlating with other theoretical measure with which the developing instrument should correlate (convergent validity)

Correlating with other theoretical measure with which the developing instrument should not correlate (Discriminant validity)

Content validity Colors of all items are not same Some item behaves differently Content Validity (It is concerned with the relevance of contents of items, individually and as a whole): . Correlating experts’ judgement; Item-item or item-total correlation

Content Validity (It is concerned with the relevance of contents of items, individually and as a whole): .

Correlating experts’ judgement;

Item-item or item-total correlation

Predictive validity Questionnaires measuring same variable provide different results. Which questionnaire predicts success ?

Questionnaires measuring same variable provide different results.

Which questionnaire predicts success ?

Concurrent validity Collect data simultaneously with two instruments measuring similar construct. Use canonical correlation if product moment correlation is significant.

Collect data simultaneously with two instruments measuring similar construct.

Use canonical correlation if product moment correlation is significant.

Construct validity Questionnaire should measure the underlying theories of research construct.

Questionnaire should measure the underlying theories of research construct.

Factorial validity Items can be classified by respondent’s latent phenomena.

Items can be classified by respondent’s latent phenomena.

Factorial validity Two questionnaires can be mixed with respondent’s frame of reference. Principal component analysis can be used to find common properties.

Two questionnaires can be mixed with respondent’s frame of reference.

Principal component analysis can be used to find common properties.

Studies on Validity of questionnaire

Studies CONTENT Item-Rating correspondence of questionnaire measuring computer programming task taxonomy. CRITERION RELATED Step-wise Multiple regression in predicting academic achievement through reading motivation. Predictive CONSTRUCT Correspondence analysis to explore latent structure of RMQ Factorial Discriminative Discriminative validity of Attitude towards School Infrastructure questionnaire (ASIQ)

Content validity: Item-Rating Correspondence Input data for CA Dutta Roy,D. (2002). Correspondence between item and rating on the checklist of relative importance of computer programming tasks., Journal of Psychometry , 16,2,67-76.

Criterion related validity:Stepwise Multiple regression Inter correlation matrix of reading motivation variables (n=200) Data collected from Bengali medium schools

Correspondence analysis to explore latent structure of RMQ Chi-square (36) = 3041.4, p< 0.00001

Discriminative validity of Attitude towards School Infrastructure questionnaire (ASIQ) ASIQ would be able to differentiate between good and poor infrastructure schools. Methods: ASIQ was administered to the students of primary schools with good and poor infrastructures. Good and poor criteria was determined by the 8 indices available in District Information System of Education (DISE). Based on Median rank, schools were classified into high and low infrastructure categories. Results: stepwise discriminant function analysis ( [Wilks’ Lambda=0.77, Rao’s R (9,153)=5.13, p<0.00] ) extracted five most important attitudes namely, reliability, equal opportunity, comfort, safety, cleanliness in predicting differences between good and poor infrastructure schools. The overall classification accuracy is 82.8% . This suggests high predictive capacity of the Discriminant function obtained. Fisher’s Linear Discriminant Functions for differentiating Schools with Good and Poor Infrastructure

ASIQ would be able to differentiate between good and poor infrastructure schools.

Methods: ASIQ was administered to the students of primary schools with good and poor infrastructures. Good and poor criteria was determined by the 8 indices available in District Information System of Education (DISE). Based on Median rank, schools were classified into high and low infrastructure categories.

Results: stepwise discriminant function analysis ( [Wilks’ Lambda=0.77, Rao’s R (9,153)=5.13, p<0.00] ) extracted five most important attitudes namely, reliability, equal opportunity, comfort, safety, cleanliness in predicting differences between good and poor infrastructure schools. The overall classification accuracy is 82.8% . This suggests high predictive capacity of the Discriminant function obtained.

Multivariate model is the Journey to Harmony THANK YOU