Preparing for Manuscript Submission<br />CARMA Internet Research Module<br />Jeff Stanton<br />
May 18-20, 2006<br />Internet Data Collection Methods (Day 2-2)<br />Standard Preparation for MS Submission<br />Report an...
May 18-20, 2006<br />Internet Data Collection Methods (Day 2-3)<br />Common Objections that Reviewers or Editors may Mount...
May 18-20, 2006<br />Internet Data Collection Methods (Day 2-4)<br />Common Objections that Reviewers or Editors may Mount...
May 18-20, 2006<br />Internet Data Collection Methods (Day 2-5)<br />Common Objections that Reviewers or Editors may Mount...
May 18-20, 2006<br />Internet Data Collection Methods (Day 2-6)<br />Common Objections that Reviewers or Editors may Mount...
May 18-20, 2006<br />Internet Data Collection Methods (Day 2-7)<br />Strengthening Research Plans for Web Studies<br />A c...
May 18-20, 2006<br />Internet Data Collection Methods (Day 2-8)<br />Useful References I<br />Birnbaum, M. H. (1999). Test...
May 18-20, 2006<br />Internet Data Collection Methods (Day 2-9)<br />Useful References II<br />Smith, M. A., & Leigh, B. (...
Upcoming SlideShare
Loading in …5
×

Carma internet research module preparing for manuscript submission

730 views
631 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
730
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Carma internet research module preparing for manuscript submission

  1. 1. Preparing for Manuscript Submission<br />CARMA Internet Research Module<br />Jeff Stanton<br />
  2. 2. May 18-20, 2006<br />Internet Data Collection Methods (Day 2-2)<br />Standard Preparation for MS Submission<br />Report analysis of a cross validation sample<br />Assess non-response bias; analyze and report<br />Do duplicate detection; report<br />Do malicious data detection; report<br />Read and cite Internet research method reviews<br />
  3. 3. May 18-20, 2006<br />Internet Data Collection Methods (Day 2-3)<br />Common Objections that Reviewers or Editors may Mount in R&R and Your Rebuttals<br />Lack of access control leads to junk data<br />You used a password protected consent form<br />You filtered responses using timestamp and IP address<br />You detected similarities in (or identical) response patterns:<br />Flip data, run correlations, look for high values<br />StudyResponse studies ranged from 0% to 6.9% duplicates using this screening method<br />Simulations showed that repeats would be needed on more than 20% of cases to substantially disturb means/correlations<br />
  4. 4. May 18-20, 2006<br />Internet Data Collection Methods (Day 2-4)<br />Common Objections that Reviewers or Editors may Mount in R&R and Your Rebuttals II<br />Your sample is bogus because of coverage errors<br />You argue that sample representativeness is a challenge in all research and that purposive sampling is a better goal anyway<br />Make the sample fit the question: An Internet survey of migrant workers? Coal miners?<br />You show the consistency of results between web and cross-validation samples<br />You argue that a typical group of Internet respondents has to be an improvement over pure undergrad samples<br />You cite demographic studies of Internet: increasing normalization to the general population over time <br />
  5. 5. May 18-20, 2006<br />Internet Data Collection Methods (Day 2-5)<br />Common Objections that Reviewers or Editors may Mount in R&R and Your Rebuttals III<br />No one has ever demonstrated the equivalence of the measures you used when administered over the web<br />You cite research that factor structures replicate, substantive conclusions replicate, correlations generally replicate within the limits of sampling error, be wary of mean comparisons<br />You argue this is a higher standard than many other published studies in which:<br />Researchers routinely make up their own items<br />Modify items or response options of existing scales<br />Trim scale lengths and field abridged versions<br />
  6. 6. May 18-20, 2006<br />Internet Data Collection Methods (Day 2-6)<br />Common Objections that Reviewers or Editors may Mount in R&R and Your Rebuttals IV<br />All Internet research participants are volunteers by definition and therefore volunteer bias makes your sample unusable<br />Belmont report and federal legislation require all research to be conducted on volunteers, so volunteer bias is endemic to the whole social research enterprise<br />Volunteer bias can substantially limit projectability of means, but my study doesn’t care about means<br />Studies and simulations of the effect of volunteer bias generally show that correlations are reduced in magnitude because of restriction of range effects that volunteer bias causes<br />
  7. 7. May 18-20, 2006<br />Internet Data Collection Methods (Day 2-7)<br />Strengthening Research Plans for Web Studies<br />A cross-validation sample using traditional RMs is never a bad thing<br />Use your web sample only to make tests of correlative structures and self-referential comparisons of means (e.g., within subjects)<br />Don’t compare means from web study to means from prior paper and pencil study without formal equating<br />Speeded and objective tests need careful testing and cross-validation<br />Assess correlations between substantive variables to demographics: If they don’t correlate, then the non-response bias may carry less weight<br />
  8. 8. May 18-20, 2006<br />Internet Data Collection Methods (Day 2-8)<br />Useful References I<br />Birnbaum, M. H. (1999). Testing critical properties of decision making on the Internet. Psychological Science, 10, 399-407.<br />Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.<br />Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and Word-Wide Web samples on determinants of female attractiveness. Behavior Research Methods, Instruments, and Computers, 29, 264-269.<br />Pasveer, K. A., & Ellard, J. H. (1998). The making of a personality inventory: Help from the WWW. Behavior Research Methods, Instruments, and Computers, 30, 309-313.<br />
  9. 9. May 18-20, 2006<br />Internet Data Collection Methods (Day 2-9)<br />Useful References II<br />Smith, M. A., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments, & Computers, 29, 496-505.<br />Stanton, J. M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.<br />Yost, P.R. & Homer, L.E. (1998, April). Electronic versus Paper Surveys: Does the Medium Affect the Response? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology. Dallas, TX.<br />

×