• Like
  • Save
Moving Beyond the Monkey: An Overview of the Latest Online Survey Software Products and their Implications for Evaluators
Upcoming SlideShare
Loading in...5
×
 

Moving Beyond the Monkey: An Overview of the Latest Online Survey Software Products and their Implications for Evaluators

on

  • 181 views

Washington Evaluators Brown Bag

Washington Evaluators Brown Bag
by Stephen Axelrad

Statistics

Views

Total Views
181
Views on SlideShare
175
Embed Views
6

Actions

Likes
0
Downloads
0
Comments
0

3 Embeds 6

http://washingtonevaluators.org 4
http://www.washingtonevaluators.roundtablelive.org 1
http://www.washingtonevaluators.org 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Motivation for doing the session – searching for the right online survey software solution for a previous employer <br /> No real guide for researchers and evaluators <br /> Myriad of choices to make sense of <br />
  • If flip chart is available, record comments on the flip chart. <br /> If flip chart is not available, record comments on a blank slide or document <br />
  • The Ritter and Sue chapter illustrate concepts, theories, and methodological techniques <br /> Most examples are given using low-cost, simple solutions such as SurveyMonkey or Zoomerang <br /> Most evaluation projects have complex features that go beyond the capabilities of those tools <br />
  • If you get one valuable thing from this presentation, I hope this slide will do that <br /> Software vendors offer a multitude of descriptions for essentially the same features – the requirements listed on this slide help distill the core features that most survey researchers would need <br /> Research needs should determine the technology benefits <br />
  • The Witte chapter provides a series of informative methodological studies on different online survey techniques that can provide conceptual and theoretical rationale for designing certain online survey features <br />

Moving Beyond the Monkey: An Overview of the Latest Online Survey Software Products and their Implications for Evaluators Moving Beyond the Monkey: An Overview of the Latest Online Survey Software Products and their Implications for Evaluators Presentation Transcript

  • Washington Evaluators Evening Brown Bag Session Moving Beyond the Monkey An overview of the latest online survey software products and their implications for evaluators Stephen Axelrad Senior Consultant Organization & Strategy Survey & Assessment Services Washington, DC May 27, 2009
  • Getting started  Your name  Your organization  Your typical evaluands  Online survey software – What products do you use? – What evaluation projects do you use online surveys? – What have been the benefits? – What have been the challenges? Booz Allen Hamilton Proprietary 2
  • Agenda  Online survey concepts and issues  Choosing an online survey software solution  Examples  Concluding points and guidance Booz Allen Hamilton Proprietary 3
  • Moving Beyond the Monkey  Online survey concepts and issues  Choosing an online survey software solution  Examples  Concluding points and guidance Booz Allen Hamilton Proprietary 4
  • In the Online Surveys issue of New Directions in Evaluation, Ritter and Sue (2007) identify the respondent and survey characteristics that are necessary for online surveys to work  Evaluators can use online surveys when the following conditions are true for potential respondents – Respondents have regular access to the Internet (e.g., home or office) – Respondents have provided their e-mail addresses to an organization who is authorized to forward such list for research purposes – Respondents in the sampling frame are geographical dispersed – Respondents cannot be accessed easily through mail or interviewing modes of administration (e.g., highly mobile or no land lines)  Evaluators can use online surveys when the following characteristics are present in the survey – Survey is self-administered and contains skip logic – Survey includes open-ended questions which may be easier to type than handwrite – Survey questions solicit responses to an image or audio/video clip Booz Allen Hamilton Proprietary 5
  • Ritter and Sue (2007) illustrate how online surveys solve some old challenges associated with other survey modes but also introduce some new ones  Online surveys overcome several obstacles associated with other survey methods – Online surveys are less expensive than mail, telephone, or in-person methods – Online surveys collect data that can be more easily recovered and stored than paper-based surveys – Online surveys allow for confidential and sensitive information to be collected without the need for an interviewer and stored in electronically secure data warehouses – Online surveys can include complex screening and branching options without confusing the respondent and avoid interviewer error  Online surveys may introduce new challenges not associated with other methods – Online surveys may not work with populations that have less access or familiarity with the Internet – Online surveys need to have designs that are compatible with diverse browser setups – Online surveys posted directly to the Web make it very difficult to obtain a sampling frame Booz Allen Hamilton Proprietary 6
  • Discussion Questions  What types of evaluations are more suited to online surveys?  Just because evaluators can use online surveys in an evaluation, should evaluators use online surveys?  Are evaluators over-using online surveys?  Is it possible to have a credible and rigorous evaluation with just online surveys? Should evaluators aim to integrate multiple modes (e.g., mail, in-person, telephone)?  How can online surveys enhance the value of evaluations to organizations and the constituents they reach? Booz Allen Hamilton Proprietary 7
  • Moving Beyond the Monkey  Online survey concepts and issues  Choosing an online survey software solution  Examples  Concluding points and guidance Booz Allen Hamilton Proprietary 8
  • Evaluators have limited available resources for choosing an online survey software solution  A Google search for “online*survey*software” yielded 180,000 results  Many software product review sites are not helpful to evaluators – Some of these “product review” sites are created by software vendors and reviews are written deceptively to favor their products – Other legitimate sites evaluate the technical qualities but are written from an informationtechnology perspective and not an applied research or evaluation perspective  The American Evaluation Association site lists more than 60 different online survey software solutions – The AEA site does not offer guidance on the quality or suitability of the different products or vendors – The AEA site includes products that range dramatically in cost and features  Evaluators cannot rely upon their organization’s IT departments because these individuals have little if any expertise in survey research or evaluation Booz Allen Hamilton Proprietary 9
  • Evaluators must choose a software solution that includes features, which parallel the steps in the survey research process  Steps in the survey research process Identify potential Create Specify response Invite respondents Administer survey Analyze resultsReport results respondents survey questions options  Important features in an online survey software solution (i.e., functional requirements)  Address books  Order flexibility  Restricting ability  E-mail invitations  Changing format  Reminder scheduling  Prepopulation  Completion status  Translation  Visibility & Rotation  Quota & conditional sampling  Question library  Response library  Unique IDs  Skip logic  Import lists  Conditional branching  Duplication  Audio/video detection  Number of responses stored Booz Allen Hamilton Proprietary  Panel setup  Log in  Browser restrictions  Survey navigation  Saving  Chart & table formatting  Missing data  Look and feel  Partially completed responses  Reporting over time  Data validation  Built-in statistics  Exporting to SPSS, SAS  Setting templates  Exporting to Word, PDF or PowerPoint 10
  • Evaluators must consider the range of costs and user requirements in selecting a software solution  Evaluators need to develop a budget that can cover the range of costs associated with different software solutions – Product costs: survey publishing, web hosting, subscription fees, upgrades – User costs: number of licenses, training, technical support – Capacity costs: number of surveys, number of responses  Evaluators need to develop user requirements that establish a hierarchy of users with varying degrees of privileges that match their responsibilities in the survey research process – Group administrators – Survey editors/approvers – Survey authors – Survey analysts – Survey translators – Report viewers Booz Allen Hamilton Proprietary 11
  • Evaluators must weigh the benefits and drawbacks of choosing internal or off-the-shelf solutions Internal Solutions Commercial Off-The-Shelf (COTS) Solutions  Benefits  Benefits – You can design a product that matches your unique research and evaluation needs – You are not as dependent on your organization’s IT department – You can develop surveys that use data from your organization’s database – You can consult with colleagues from other organizations about the quality of the product  Drawbacks – Your organization must have an IT department that has qualified and available application developers – Your organization must have networks and servers capable of hosting the survey and handling large amounts of data – Your organization must be willing to wait several months before a solution is available to implement – Your organization must be willing to invest several thousand dollars up front Booz Allen Hamilton Proprietary – You can purchase an advanced software solution that can satisfy most if not all of your research and evaluation needs  Drawbacks – You may have to develop workarounds for some research and evaluation projects – You may encounter a software product that was not designed with researchers and evaluators in mind – You may have to pay extensive fees in product upgrades, technical support packages and training 12
  • Discussion Questions  Should AEA and other professional associations with researchers and evaluators as members have online survey product guides?  Given some of the advanced features in online survey software packages and the need to interact with IT professionals, what technological capabilities do evaluators need to have in order to decide on and implement online survey software solutions?  How can evaluators determine if COTS products were designed with input from researchers and evaluators?  What factors do evaluators need to consider when determining the return on investment they could get from internally developed solutions versus COTS packages? Booz Allen Hamilton Proprietary 13
  • Moving Beyond the Monkey  Online survey concepts and issues  Choosing an online survey software solution  Examples  Concluding points and guidance Booz Allen Hamilton Proprietary 14
  • Examples  Internally developed solutions – Needs assessment survey – Implementation evaluation survey  COTS packages – Gravic Remark Web Survey – Prezza Tech Checkbox Survey – Snap Professional Edition Booz Allen Hamilton Proprietary 15
  • COTS Example – Gravic Remark Web Survey Booz Allen Hamilton Proprietary 16
  • COTS Example – Prezza Tech Checkbox Survey Booz Allen Hamilton Proprietary 17
  • COTS Example – Prezza Tech Checkbox Survey Booz Allen Hamilton Proprietary 18
  • COTS Example – Snap Survey Professional Edition Booz Allen Hamilton Proprietary 19
  • COTS Example – Snap Survey Professional Edition Booz Allen Hamilton Proprietary 20
  • Moving Beyond the Monkey  Online survey concepts and issues  Choosing an online survey software solution  Examples  Concluding points and guidance Booz Allen Hamilton Proprietary 21
  • Evaluators should develop an online survey software strategy and examine different solutions based on their research and evaluation needs  Evaluators should develop an online survey software strategy – Identify the aspects of your evaluation practice that involve online surveys – Understand the implications of using online surveys – Develop a budget – Develop user and functional requirements based on the survey research process  Evaluators should examine the range of online survey software solutions – Consider the benefits and drawbacks for using internally developed versus COTS solutions – Compare different software features to your user and functional requirements – Be open to technological innovations only if they help answer your evaluation question Booz Allen Hamilton Proprietary 22
  • References and additional reading to learn more about research and practice related to online surveys  E-mail me at axelrad_stephen@bah.com for an electronic copy of this presentation or give me your business card at the end of the session  American Evaluation Association (n.d.). Surveys. Retrieved April 11, 2009, from http://www.eval.org/Resources/surveylinks.asp.  Ritter, L.A., & Sue, V.M. (2007). Using online surveys in evaluation. New Directions in Evaluation, 115, 1-64.  Witte, J.C. (Ed.; 2009). Special issue: Web surveys. Sociological Methods & Research, 283455. Booz Allen Hamilton Proprietary 23
  • Feedback  Questions  Comments  Implications for Your Current Work Booz Allen Hamilton Proprietary 24