Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reducing Response Burden

732 views

Published on

Brief deck describing approaches and constraints for reducing response burden.

Published in: Education, Technology, Business
  • Be the first to comment

  • Be the first to like this

Reducing Response Burden

  1. 1. Fred Oswald<br />Rice University – Department of Psychology<br />Jeff Stanton<br />Syracuse University – School of Information Studies<br />Kauffman Roundtable on Establishment Surveys<br />August 9, 2011<br />Reducing Response Burden<br />
  2. 2. Reducing Response Burden<br />Goals<br />Reduce time of administration and associated costs<br />Increase response rates, compliance, fewer missing data, <br />Decrease survey fatigue and increase willingness to re-engage over time<br />Effects<br />Increase information yield per unit of response time/effort<br />Balance multiple stakeholder survey goals based on “lean” and less reliable information sources<br />
  3. 3. Approaches and Constraints <br />Goal: No practical loss in reliability or validity at the desired aggregate level (unit, sub-unit/cluster)<br />Approaches<br />Reduce instructions/explanatory material<br />Reduce item redundancy<br />Distribute subsets of items strategically across units, using available data or imputation to complete analyses <br />Automate field completion with NLP and scrapers<br />Constraints<br />Each approach has intrinsic weaknesses<br />Stakeholders wedded to particular items<br />Longitudinal comparability limits changes<br />
  4. 4. Research on Response Burden<br />Example: How to determine efficient item assignments to sub-samples, given available item information, precision goals, and stakeholder concerns?<br />Given different approaches (e.g., multiple imputation) what is the maximum reduction in item content before loss of precision becomes intolerable?<br />How redundant are components of a profile (so certain items ‘drive’ a profile)?<br />What are stakeholder reactions to various imputation/analysis strategies?<br />Our Expertise<br />Methods generalists who examine technology and statistical tools to inform practical goals of survey development and test characteristics<br />Statistical tools: Item response theory, multi-level and cross-classified models, meta-analysis, machine learning<br />Technology tools: Web-based surveys, data scrapers, alternative data collection methods<br />
  5. 5. Selected Papers<br />Non-response issues<br />Converse, P. D., Wolfe, E. W., Huang, X., & Oswald, F. L. (2008). Response rates for mixed-mode surveys using mail and email/web. American Journal of Evaluation, 29, 99-107.<br />Rogelberg, S. G, & Stanton, J. M. (2007). Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195-209.<br />Wolfe, E. W., Converse, P. D., & Oswald, F. L. (2008). Item-level non-response rates in an attitudinal survey of teachers delivered via mail and web. Journal of Computer-Mediated Communication, 14, 35-66.<br />Shortening measures<br />Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The Mini-IPIP scales: Tiny-yet-effective measures of the Big Five factors of personality. Psychological Assessment, 18, 192-203. <br />Russell, S.R., Spitzmüller, C., Lin, L.F., Stanton, J.M., Smith, P.C., & Ironson, G. H. (2004). Shorter can also be better: The abridged Job in General Measure. Educational and Psychological Measurement, 64 (5), 878-893.<br />Stanton, J. M., Sinar, E. F., Balzer, W. K., & Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55 (1), 167-193.<br />Stanton, J. M. (2000). Empirical distributions of correlations as a tool for scale reduction. Behavior Research Methods, Instruments, and Computers, 32, 403-406.<br />Technology issues<br />Rogelberg, S. G., Church, A. H., Waclawski, J., & Stanton, J. M. (2002). Organizational Survey Research: Overview, the Internet/intranet and present practices of concern. In S. G. Rogelberg (Ed.), Handbook of Research Methods in Industrial and Organizational Psychology. Oxford: Blackwell.<br />Stanton, J. M., & Rogelberg, S. G. (2001). Using Internet/Intranet Web Pages to Collect Organizational Research Data. Organizational Research Methods, 4, 199-216.<br />Stanton, J. M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.<br />Parallel forms (item banking, test security)<br />Oswald, F. L., Friede, A. J., Schmitt, N., Kim, B. K., & Ramsay, L. J. (2005). Extending a practical method for developing alternate test forms using independent sets of items. Organizational Research Methods, 8, 149-164. <br />

×