Designing Questionnaire  and Survey Research (updated Jan 2011)

Quantitative Research Methodologies (4/6): Designing Questionnaire and Survey Research Prof. Dr. Hora Tjitra & Dr. He Quan www.SinauOnline.com

ICS Project Best • Research Objectives Practices – The primary objective of this research is to examine cross-cultural variations in the perceived importance of ICC of tour guides. Another objective is to build a model of tour guides’ ICC from Asian perspective. • Research Methods – The study on tour guide intercultural communication competence consists of two parts. First part is group discussion (Flick, 2002) to develop a cultural sensitive questionnaire on TCC. Second part is quantitative survey using the questionnaire that is developed in the first part. • Research Results – There are four results from this research. First of all, what constructs tour guides’ ICC from non American perspective (Especially from Chinese) will be get; Secondly, a cultural sensitive questionnaire will be developed to identify Tour Guide ICC; Then the questionnaire will be translated into three languages which are Chinese, English and German; Finally, the cultural differences of tour guide’s ICC will be compared. 2

Behavioral Expectation Model (Pavitt & Haight,1985) • Individuals hold cognitive notions of what comprises competence in particular contexts. • They then form impressions and evaluations of others’ behaviors based on these notions. • The degree to which the ideal meets the expected is the degree to which behavior is to judged competent 3

Tour Guides’ Intercultural Communication Competence Theoretical Background Empirical Research Qualitative Method 1. Theories of ICS focus on 1. TCC (Leclerc & Martin, 2004) 1. Group Discussion: new inputs communication 2. Method: Questionnaire from non US perspective 2. Research from western 3. Result: cultural differences of TCC 2. Content analysis among DE,FR,US perspective (Martin a. Inductive analysis &Hammer,1989) 4. Question: appropriateness of b. Deductive analysis applying US TCC framework to other cultures Cultural Sensitive Questionnaire on TCC The survey will be conducted from June to July in 2005. Respondents are international tourists. At least 200 respondents are expected to participate in this study. TCC Research Objectives 1. Compare cultural differences on Tour Guides’ Intercultural Communication Competence 2. Building Intercultural Communication Competence Model 4

Group Discussion (GD) GD Design Data Procedure Number of participants: 62 Videos Introduction Number of groups: 11 Flipcharts Short movies of tour guide Pictures of Pure Chinese Groups (5) Participants Preliminary discussion Chinese - Other Asian Question 1 Groups Chinese with Korean Divided discussion Chinese with Malaysian Question 2 & 3 Chinese with Pakistan Chinese - Western Groups Chinese with German (2) Plenary discussion Chinese with Australian Question 4 Closing 5

Data Collection and Analysis • Basing on group discussion and Leclerc’s results, develop New Questionnaire • Sampling and distributing the questionnaires • Analyze the data: reliability, MANOVA, etc. • Explain the research results 6

Survey Research • Survey research: a specific type of field study that involves the collection of data from a sample of elements drawn from a well-defined population through the use of a questionnaire The reasons to conduct survey research… • Study a representative sample through field research is relatively easy and practical • Research can confidently generalize the finding from selected sample to entire population • It provides ideal conditions for exploration of process × individual difference interaction 7

Types of Surveys …the broad area of survey research encompasses any measurement procedures that involve asking questions of respondents. Questionnaires: Questionnaires are usually paper-and-pencil instruments that the respondent completes. Interviews (constructed): Interviews are completed by the interviewer based on the respondent says. 8

The Issues in Survey Research • Population Issues • Sampling Issues ♪ Can the population be enumerated? ♪ What data is available? ♪ Is the population literate? ♪ Can respondents be found? ♪ Who is the respondent? ♪ Are there language issues? ♪ Can all members of population be sampled? ♪ Will the population cooperate? ♪ Are response rates likely to be a problem? ♪ What are the geographic restrictions? • Question Issues • Content Issues ♪ What types of questions can be asked? ♪ Can the respondents be expected to know about ♪ How complex will the questions be? the issue? ♪ Will screening questions be needed? ♪ Will respondent need to consult records? ♪ Can question sequence be controlled? ♪ Will lengthy questions be asked? ♪ Will long response scales be used? • Bias Issues • Administrative Issues ♪ Can social desirability be avoided? ♪ Costs ♪ Can interviewer distortion and subversion be ♪ Facilities controlled? ♪ Time ♪ Can false respondents be avoided? ♪ Personnel 9

The Types of Measurement Scaling is the assignment of objects to numbers according to a rule there are a variety possible response Likert or "Summative" scaling scales (1-to-7, 1-to-9, 0-to-4). Thurstone or Equal-appearing rate each statement on a 1-to-11 scale interval scaling in terms, then choose “yes” or “No” the answering item is “Yes” or “No” Guttman or “Cumulative” scaling 10

The Basic Steps in Developing a Likert or "Summative" Scale. Defining the Focus: define what it is you are trying to measure Generating the Items: create the set of potential scale items. These should be items that can be rated on a 1-to-5 or 1-to-7 Disagree-Agree response scale. Rating the Items: have a group of judges rate the items. Usually you would use a 1-to-5 rating scale where:1= strongly unfavorable to the concept ; 5= strongly favorable to the concept Selecting the Items. compute the intercorrelations •Throw out any items that have a low correlation with the between all pairs of items, based on the ratings of the judges. total (summed) score across all items • For each item, get the average rating for the top quarter of judges and the bottom quarter. Then, do a t-test of the differences between the mean value for the item for the top and bottom quarter judges. Administering the Scale: reversal items 1= strongly disagree 2= disagree 3= undecided 4= agree 5= strongly agree 11

Example: The Employment Self Esteem Scale INSTRUCTIONS: Please rate how strongly you agree or disagree with each of the following statements by placing a check mark in the appropriate box. 1. I feel good about my work on the job. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 2. On the whole, I get along well with others at work. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 3. I am proud of my ability to cope with difficulties at work. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 4. When I feel uncomfortable at work, I know how to handle it. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 5. I can tell that other people at work are glad to have me Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree there. 6. I know I'll be able to cope with work for as long as I want. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 7. I am proud of my relationship with my supervisor at work. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 8. I am confident that I can handle my job without constant Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree assistance. 9. I feel like I make a useful contribution at work. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 10. I can tell that my coworkers respect me. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 12

Questionnaire Design :The six steps for preparing the questionnaire 6. Edit the questionnaire and specify the procedure for its use 5. Pretest the questionnaire 4. Reexamine and revise the questionnaire 3. Write a first draft of the questionnaire 2. Decide what type of questionnaire should be used 1. Decide what information should be sought 13

Question Content • Is the Question Necessary/ Useful? • Does the Question Need to be ♪ Do you need the age of each child or just the More Specific? number of children under 16? ♪ How well did you like the book? ♪ Do you need to ask income or can you ♪ Did you recommend the book to others? estimate? • Are Several Questions • Is Question Biased or Loaded? Needed? ♪e.g. What do you see as the benefits of a tax cut? ♪ double-barreled question (eg. Teacher & parents) ♪e.g. What do you see as the disadvantages of eliminating welfare? ♪ does not cover all possibilities ♪ does not give you enough context ♪ does not determine the intensity • Do Respondents Have the • Is Question Sufficiently General? Needed Information? ♪ You should ask a filter question first (e.g., Have you ever watched the show ER?) before asking them their opinions about it. • Will Respondent Answer Truthfully? 14

Question Words One of the major difficulty in writing good survey questions is getting the wording right ♪ Can the Question be Misunderstood? e.g. What kind of headache remedy do you use? ♪ What Assumptions Does the Question Make? ♪ Is the time frame specified? Whenever using the words "will", "could", "might", or "may" in a question, you might suspect that the question asks a time-related question. ♪ Is the wording too direct? E.g. How did you feel about being in the war? How well did the equipment hold up in the field? How well were new recruits trained? ♪ Does the question contain difficult or unclear terminology?….. 15

Question Placement Considering following questions: ♪ Is the answer influenced by prior questions? ♪ Does question come too early or too late to arouse interest? ♪ Does the question receive sufficient attention? A Checklist of Considerations ✓ start with easy, nonthreatening questions ✓ put more difficult, threatening questions near end ✓ never start a mail survey with an open-ended question ✓ for historical demographics, follow chronological order ✓ ask about one topic at a time ✓ when switching topics, use a transition ✓ reduce response set (the tendency of respondent to just keep checking the same response) ✓ for filter or contingency questions, make a flowchart The Golden Rule Do unto your respondents as you would have them do unto you! 16

Other Important Issues • Question Order ♪ Grouping Questions by topic may be useful ♪ It is easier for the respondents to think in the same topic • Rating Scale Formats ♪ For bipolar scales ( e.g., running from positive to negative with neutral in the middle), reliability and validity are highest for about 7 point.( Matell & Jacoby, 1971) ♪ The reliability and validity of unipolar scales (e.g., running from no importance to very high importance) seem to be optimized for a bit shorter scales, approximately 5 points long ( Wikman & Waarneryd, 1990) ♪ A good number of studies suggest that data quality is better when all scales points are labeled with words than only some are. ( Krosnick & Berent, 1993) 17

Sampling in Survey Research 18

Statistical Sampling Terms Variable: is a specific measurement value that a sampling unit supplies Statistic: the responses that we get for our entire sample, Parameter: the measurement the entire population 19

Approaches to Sampling Probability Sampling Nonprobability Sampling  Any method of sampling that  Methods do not involve random utilizes some form of random selection selection  Simple Random Sampling  Convenience Sampling  Stratified Random Sampling  Purposive Sampling  Systematic Random Sampling  Cluster (Area) Random Sampling 20

Probability Sampling • any method of sampling that utilizes some form of random selection Some Definitions ♪ N = the number of cases in the sampling frame ♪ n = the number of cases in the sample ♪ NCn = the number of combinations (subsets) of n from N ♪ f = n/N = the sampling fraction 21

Simple Random Sampling • Objective: To select n units out of N such that each NCn has an equal chance of being selected. • Procedure: Use a table of random numbers, a computer random number generator, or a mechanical device to select the sample. 22

Stratified Random Sampling • Objective: Divide the population into non-overlapping groups (i.e., strata) N1, N2, N3, ... Ni, such that N1 + N2 + N3 + ... + Ni = N. Then do a simple random sample of f = n/N in each strata. • Procedure :divide your population into homogeneous subgroups and then taking a simple random sample in each subgroup. 23

Systematic Random Sampling • number the units in the population from 1 to N • decide on the n (sample size) that you want or need • k = N/n = the interval size • randomly select an integer between 1 to k • then take every kth unit 24

Cluster (Area) Random Sampling • Divide population into clusters (usually along geographic boundaries) Beijing ShijiaZhuang Tianjin  (Hebei)   Qingdao • Randomly sample clusters Zhengzhou Jinan (Henan) Nanjing   Suzhou Xi’an Hefei Shanghai (Shaanxi) (Anhui) Hangzhou  Chongqing Wuhan Chengdu • Measure all units within  Changsha   Xiamen Kunming  Shantou sampled clusters Guiyang Shenzhen  Zhanjiang Eg. Measure Chinese personality 25

Nonprobability Sampling • does not involve random selection Purposive Sampling: Convenience Sampling: sample with a purpose in mind. traditional "man on the street" •Modal Instance Sampling or called Accidental, •Expert Sampling •Quota Sampling Haphazard sampling •Snowball Sampling •they are not representative of the populations 26

Advantage and Disadvantage of Sampling Approach probability sampling Nonprobability sampling  Advantage  Advantage  Research can be confident  Convenient that a selected is representative  Flexible  It permits researchers to precisely estimate the amount of variance present in a given data set that is due to sampling error Disadvantage Disadvantage  we may or may not represent  Calculation the population well,  it will often be hard for us to know how well we've done so 27

Survey-Research Design Cross-Sectional Design: the collection of data at a single point in time from a sample drawn from a specified population Successive Independent Samples Design: a series of cross- sectional survey are conducted over time Longitudinal Design: the same sample of respondents is surveyed more than once 28

Total survey error The ultimate goal of survey research is to accurately measure particular constructs within a sample of people who represent the population of interest • Coverage Error: the bias that can result when the pool of potential survey participants from which a sample is selected does not include some portions of the interest • Sampling Error: the random differences that invariably exist between any sample and the population from which it was selected • Nonresponse Error: the bias that can result when data are not collected from all of the members of a sample • Measurement Error: all distortions in the assessment of the construct of interest, including systematic biases and random variance that can be brought by respondents’ own behavior, interviewer behavior, and the questionnaire 29

Thanks You An y co m m e n ts & q u e s ti o n s are welcome Contact me at hora_t@sianuonline.com www.SinauOnline.com @ Tjitra, 2010 30

http://kmaharjan.com.np	10
http://sinau.me	7
http://horatjitra.com	1
http://www.brijj.com	1
http://tjitra.com	1
https://aamu.blackboard.com	1

Designing Questionnaire  and Survey Research (updated Jan 2011)

by Hora Tjitra

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 21

Statistics