Carma internet research module   n-bias
Upcoming SlideShare
Loading in...5

Carma internet research module n-bias






Total Views
Views on SlideShare
Embed Views



1 Embed 6 6



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Carma internet research module   n-bias Carma internet research module n-bias Presentation Transcript

    • N-BIAS Nonresponse BiasAssessment Techniques
      CARMA Internet Research Module
      Jeffrey Stanton
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-2)
      N-BIAS Methods
      Rogelberg, S. G, & Stanton, J. M. (2007). Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195-209.
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-3)
      N-BIAS Methods
      Wave Analysis
      Archival Analysis
      Follow-up survey of non-respondents
      Passive non-responseanalysis (time-to-respond vs. substantive)
      Active non-respondentpre-study analysis
      Interest level analysis
      Worst case resistance
      Detect, estimate, and optionally compensate for the presence, direction, and magnitude of non-response bias. Gives clearer picture of extent of bias but few good options for eliminating it.
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-4)
      N-BIAS Methods
      N-BIAS comprises eight techniques
      Archival Analysis
      Follow-up Approach
      Wave Analysis
      Passive Nonresponse Analysis
      Interest Level Analysis
      Active Nonresponse Analysis
      Worst Case Resistance
      Demonstrate Generalizability
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-5)
      Technique 1: Archival Analysis
      Most common technique
      The researcher identifies an archival database that contains the members of the whole survey sample (e.g. personnel records).
      That data set, usually containing demographic data, can be described:
      50% Female; 40% Supervisors, etc
      After data collection, code numbers on the returned surveys (or access passwords) can be used to identify respondents, and by extension nonrespondents. Using this information, the archival database can be partitioned into two segments: 1) data concerning respondents; and 2) data concerning nonrespondents.
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-6)
      So, if you found the above do you have nonresponse bias?
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-7)
      Technique 2: Follow-up Approach
      Using identifiers attached to returned surveys (or access passwords), respondents can be identified and by extension nonrespondents.
      The follow-up approach involves randomly selecting and resurveying a small segment of nonrespondents often by phone. The full or abridged survey is then administered.
      In the absence of identifiers, telephone a small random sample and ask whether they responded or not to the initial survey. Follow-up with survey relevant questions
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-8)
      Technique 3: Wave Analysis
      By noting in the data set whether each survey was returned before the deadline, after an initial reminder note, after the deadline, and so on, responses from pre-deadline surveys can be compared with the late responders on actual survey variables (e.g. compare job satisfaction levels).
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-9)
      Technique 4: Passive Nonresponse Analysis
      Rogelberg et al. (2003) found that the vast majority of nonresponse can be classified as being passive in nature (approx. 85%).
      Passive nonresponse does not appear to be planned.
      When asked (upon receipt of the survey), these individuals indicate a general willingness to complete the survey – if they have the time. Given this, it is not surprising that they generally do not differ from respondents with regard to job satisfaction or related variables.
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-10)
      Technique 5: Interest Level Analysis
      Researchers have repeatedly identified that interest level in the survey topic is one of the best predictors of a respondent’s likelihood of completing the survey.
      As a result, if interest level is related to attitudinal standing on the topics making up the survey, the survey results are susceptible to bias.
      E.g., if low interest individuals tend to be more dissatisfied on the survey constructs in question, results will be biased “high”
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-11)
      Technique 6: Active Nonresponse Analysis
      Active nonrespondents, in contrast to passive nonrespondents, are those that overtly choose not to respond to a survey effort. The nonresponse is volitional and a priori (i.e. it occurs when initially confronted with a survey solicitation).
      Active nonrespondents tend to differ from respondents on a number of dimensions typically relevant to the organizational survey researcher (e.g. job satisfaction)
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-12)
      Technique 7: Worst Case Resistance
      Given the data collected from study respondents in an actual study, one can empirically answer the question of what proportion of nonrespondents would have to exhibit the opposite pattern of responding to adversely influence sample results.
      Similar philosophy as what occurs in meta-analyses when considering the “file-drawer problem”
      By adding simulated data to an existing data set, one can explore how resistant the dataset is to worst case responses from non-respondents.
    • Technique 8
      Using measures with norms for the population under examination, compare means and standard deviations of the collected sample to the norms
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-14)
      Technique 9: Demonstrate Generalizability
      By definition, nonresponse bias is a phenomenon that is peculiar to a given sample under particular study conditions.
      Triangulating with a sample collected using a different method, or varying the conditions under which the study is conducted should also have effects on the composition of the nonrespondents group.
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-15)
      Exercise 2: Item Writing
      Write three Likert scaled items (each) to tap into the following response-related constructs:
      Busyness (how busy the respondent feels)
      Topical interest (how interested the respondent is in the topic of the survey)
      Satisfaction with the sponsoring entity (the group who is running the survey)
      Response modality preference (e.g., got paper survey but preferred web)
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-16)
      Demographic Profile:
      1) Archival analysis
      Attitudinal Profile:
      1) Follow-up approach
      2) Wave analysis
      Warning Signs:
      1) Interest level analysis
      2) Passive NR analysis
      3) Active NR analysis
      4) Worst-case analysis
    • May 15-17, 2008
      Internet Data Collection Methods (Day 2-17)
      Important Footnote
      In many of the N-BIAS techniques described above, one is amassing evidence for the absence of non-response bias by making comparisons and tests that preferably lead to non-significant statistical results.
      What is the problem with this?