Your SlideShare is downloading. ×
0
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Edmedia2011 online.surveys
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Edmedia2011 online.surveys

444

Published on

Presentation by Dr. Valerie Irvine at EdMedida 11 in Lisbon, Portugal for the Graduate Training Stream

Presentation by Dr. Valerie Irvine at EdMedida 11 in Lisbon, Portugal for the Graduate Training Stream

Published in: Education, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
444
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. Online Surveys: Best Practices and Ethics Dr. Valerie Irvine Co-Director, TIE Research Lab University of Victoria, Canada http://tie.uvic.ca [email_address] Twitter: _valeriei Acknowledgement to Dr. John Williams, University of Northern Iowa
  2. Some Advantages… <ul><li>Large Sample Sizes </li></ul><ul><li>High Statistical Power </li></ul><ul><li>Diverse and Distant Geographical Areas </li></ul><ul><li>External Validity and Generalizability </li></ul><ul><li>Relatively Easy (relatively!!!) </li></ul>
  3. Use of Internet for Research <ul><li>From 1998 to 2003, 425% increase in listed studies on one website (Birnbaum, 2004) </li></ul><ul><li>Google “Online Experiment” </li></ul><ul><ul><li>42,100 pages in 2007 </li></ul></ul><ul><ul><li>107,000,000 in 2011 </li></ul></ul><ul><ul><li>Explore some, but note not to be a participant if you are clicking through it </li></ul></ul>
  4. Online vs. Traditional <ul><li>Are online studies fundamentally different than traditional studies? </li></ul><ul><ul><li>Direct contact between researcher and participant? </li></ul></ul><ul><ul><li>Use of computers? </li></ul></ul><ul><ul><li>Age verification? </li></ul></ul><ul><ul><li>Electronic data transmission potentially less confidential? </li></ul></ul>
  5. Ethical Considerations <ul><li>Ethical considerations are generally the same for online studies as for traditional studies. </li></ul><ul><li>Various ethical guidelines still apply </li></ul><ul><ul><li>For example, US Code of Federal Regulations: Protection of Human Subjects or Canada SSHRC Tri-Council Guidelines for Research Ethics </li></ul></ul><ul><ul><li>Guild Guidelines </li></ul></ul><ul><ul><ul><li>American Psychological Association’s Ethical Principles of Psychologist’s and Code of Conduct </li></ul></ul></ul><ul><ul><li>Institutional/Ethical Review Boards (IRB or ERB) </li></ul></ul>
  6. Lack of Direct Contact Beneficial? <ul><li>May increase response rate because of increased perception of anonymity </li></ul><ul><li>May increase ability to discontinue a study because of less social pressure </li></ul>
  7. Lack of Direct Contact Detrimental? <ul><li>Harder to judge understanding of consent form or debriefing </li></ul><ul><li>Harder to provide immediate clarification of participant questions </li></ul><ul><li>Harder to verify age </li></ul><ul><li>These problems however are not unique to online studies and also occur in mailed surveys, telephone interviews, and group testing </li></ul>
  8. Seven Ethical Issues <ul><li>Informed Consent </li></ul><ul><li>Deception </li></ul><ul><li>Debriefing </li></ul><ul><li>Right to Withdraw </li></ul><ul><li>Security of test materials </li></ul><ul><li>Confidentiality and Anonymity </li></ul><ul><li>Avoiding Harm </li></ul>
  9. Informed Consent <ul><li>There is a difference between asking for consent and documenting consent </li></ul><ul><ul><li>Asking for consent is easy, an informed consent statement on a webpage will do. </li></ul></ul><ul><ul><li>Documenting consent is more involved. </li></ul></ul>
  10. Options to Documentation of Consent <ul><li>ERBs typically require some indication of consent. </li></ul><ul><ul><li>A webpage with “I consent” and a “I do not consent” options (buttons) may be sufficient. </li></ul></ul><ul><ul><li>Emailed consent may be adequate </li></ul></ul>
  11. Deception <ul><li>The issue with deception in online studies is related to debriefing </li></ul><ul><ul><li>Participants may not read or understand debriefing information </li></ul></ul><ul><ul><li>Researchers may have difficulty addressing emotional turmoil related to deception. </li></ul></ul><ul><li>Studies with minimal deception </li></ul><ul><ul><li>Weigh risks vs. possible harm </li></ul></ul><ul><ul><li>If justified for other methods with limited contact than probably justified for online studies </li></ul></ul>
  12. Right to Withdraw <ul><li>In-person studies may exert social pressure for participants to stay. </li></ul><ul><li>Online studies may exert little pressure to remain and easy ways to exit a study should be provided </li></ul><ul><ul><li>“ Quit the Study” links on each page </li></ul></ul>
  13. Withdrawing Data <ul><li>Withdrawing data is more problematic </li></ul><ul><ul><li>Some studies may collect data throughout the study. Withdrawing data from these studies will require manual deletion </li></ul></ul>
  14. Withdrawing Data <ul><ul><li>Studies which collect data upon completion of the study may have links to exit and not submit data if the participant so choses </li></ul></ul><ul><ul><ul><li>E.g., surveymonkey </li></ul></ul></ul><ul><ul><ul><li>Watch out for inadvertently losing participants should they miss clicking the submit button*** </li></ul></ul></ul>
  15. Withdrawing Data <ul><li>The consent form could tell participants that if they withdraw by clicking a “Quit the study” button, they will be asked whether they would like to withdraw their data, but if they withdraw by closing the browser window, the data they have already submitted will be analyzed. </li></ul>
  16. Confidentiality and Anonymity <ul><li>Keeping Data Secure </li></ul><ul><ul><li>The steps for data security in online studies is the same as with traditional studies </li></ul></ul><ul><ul><ul><li>Restrict access to data </li></ul></ul></ul><ul><ul><ul><li>Separate identifying information </li></ul></ul></ul><ul><ul><li>Issue with U.S. Patriot Act </li></ul></ul><ul><ul><ul><li>See surveymonkey’s response </li></ul></ul></ul><ul><ul><ul><li>See Mt. Alison’s University’s response </li></ul></ul></ul>
  17. Confidentiality and Anonymity <ul><li>Keeping Data Secure </li></ul><ul><ul><li>Restricting Access </li></ul></ul><ul><ul><ul><li>Requires physical security of the server </li></ul></ul></ul><ul><ul><ul><li>Requires electronic security of the server collecting data and the computer storing the data </li></ul></ul></ul><ul><ul><li>Identifying Information </li></ul></ul><ul><ul><ul><li>Names, email address, addresses, or even usernames or locations </li></ul></ul></ul><ul><ul><ul><li>Identifying information should be stored in separate files, databases, or computers </li></ul></ul></ul>
  18. Confidentiality and Anonymity <ul><li>The level of security needed with directly depend upon the sensitive of the information collected and the amount of identifying information. </li></ul><ul><ul><li>Data can be encrypted on the server and during transmission. </li></ul></ul><ul><ul><li>Data values and labels can be assigned in a way that is meaningless to others without the key </li></ul></ul><ul><ul><ul><li><input type=”radio” name=”Item3” value=”b”> </li></ul></ul></ul><ul><ul><ul><li>rather than </li></ul></ul></ul><ul><ul><ul><li><input type= “radio” name= “How often have you stolen a car?” </li></ul></ul></ul><ul><ul><ul><li>value= “More than once”> </li></ul></ul></ul>
  19. Confidentiality and Anonymity <ul><li>Public use computers may reduce confidentiality and anonymity. </li></ul><ul><ul><li>Pages may be cached on the computer. </li></ul></ul><ul><ul><li>Others using the computer may be able to view study information in URLs. </li></ul></ul><ul><ul><li>Corporations and businesses have legal rights to search and view information on the computers owned by them. Participants using these computers may be revealing information about themselves. </li></ul></ul><ul><ul><li>IP addresses are sometimes considered identifying information. </li></ul></ul>
  20. Confidentiality and Anonymity <ul><li>Establishing Identity </li></ul><ul><ul><li>Asking for information that only the participant knows (i.e. student number) </li></ul></ul><ul><ul><li>Sending individualized participation invitations </li></ul></ul><ul><ul><ul><li>Recipient of invitation may NOT be the individual expected </li></ul></ul></ul>
  21. Avoiding Harm <ul><li>Because much online research is widely advertised, consider additional risks to participants from other countries and cultures. </li></ul><ul><ul><li>Everyday topics, words, and pictures may be distressing or illegal in the participant’s country. </li></ul></ul><ul><ul><li>Forewarn potential participants of such content through the use of appropriate study titles and descriptions. </li></ul></ul>
  22. Avoiding Harm <ul><li>Researchers should provide detailed information regarding threats to confidentiality, if interception of the material could result in severe social or legal consequences for participants in some countries. </li></ul>
  23. Measurement <ul><ul><li>T = O + E (true score equals observed score plus error) </li></ul></ul><ul><ul><li>Double-barrel items: Agree or Disagree to “I like cats and dogs” </li></ul></ul><ul><ul><li>Numerical ranges: Your Age: 19-25, 25-35, 35-50, 50-65 </li></ul></ul>
  24. Measurement <ul><ul><li>Presentation of Likert Scales </li></ul></ul><ul><ul><ul><li>Horizontal and keep ‘don’t know’ out of it for spatial response formation </li></ul></ul></ul><ul><ul><li>Provide definitions to avoid confusion </li></ul></ul><ul><ul><ul><li>Put the definition BEFORE the question </li></ul></ul></ul>
  25. Getting Read <ul><ul><li>Avoid high-risk words that may send it to SPAM </li></ul></ul><ul><ul><li>Send HTML and plain text email invitations </li></ul></ul><ul><ul><li>Insert a large START HERE link </li></ul></ul><ul><ul><li>Have a familiar person or person of status send the email </li></ul></ul><ul><ul><li>Do NOT harvest email addresses from the web </li></ul></ul><ul><ul><li>Be sure to send reminder prompt </li></ul></ul>
  26. Common Mistakes in Conducting Surveys (online) <ul><li>Increasing risk of losing participants with submit button </li></ul><ul><ul><li>(e.g., saying “thank you for your participation” ABOVE the submit button) </li></ul></ul><ul><li>In some delivery modes, naming two fields the same </li></ul><ul><li>Ability to check more than one option (radio vs. checkboxes) </li></ul>
  27. Common Mistakes in Conducting Surveys (online) <ul><li>Survey webpages being long and overwhelming </li></ul><ul><ul><li>Participants will skip questions or enter false data </li></ul></ul><ul><ul><li>Use “skip logic” to shorten # of questions </li></ul></ul><ul><li>Avoid stupid questions in online versions </li></ul><ul><ul><li>“ I completed this survey: </li></ul></ul><ul><ul><ul><li>By paper and pencil </li></ul></ul></ul><ul><ul><ul><li>Online” </li></ul></ul></ul><ul><li>Use of open-ended when structured answers could prevent long data coding time </li></ul>
  28. Pilot it first <ul><li>Complete a few online surveys first to see pitfalls and good practices </li></ul><ul><li>Get friends and colleagues to pilot the survey BEFORE you deploy it </li></ul>
  29. Examples of Online Research <ul><li>http://psych.hanover.edu/research/exponnet.html </li></ul><ul><ul><li>See attitudes toward online counselling (surveymonkey) </li></ul></ul><ul><li>http://psychexps.olemiss.edu (needs plug-ins….) </li></ul>
  30. References Birnbaum, M. H., & Reips, U. -D. (2005). Behavioral research and data collection via the Internet. In R. W. Proctor and K.-P. L. Vu (Eds.), The handbook of human factors in Web design (pp. 471-492). Mahwah, New Jersey: Erlbaum. Buchanan, T., Johnson, J. A., & Goldberg, L. E. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21 , 115-127. Ess, C. (2007). Internet Research Ethics. In A. Joinson, K. McKenna, T. Postmes, and U.-D. Reips (Eds.), The Oxford Handbook of Internet Psychology , pp 487-502. New York: Oxford University Press. Krantz, J. H. & Dalal, R. (2000). Validity of web-based psychological research. In M.H. Birnbaum (Ed.), Psychological Experiments on the Internet (pp. 35 – 60). San Diego, CA: Academic Press.

×