Your SlideShare is downloading. ×
How to design an effective survey: The good the bad and the ugly
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

How to design an effective survey: The good the bad and the ugly

280
views

Published on

Published in: Education, Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
280
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • This questionnaire was given to participants at the beginning of the workshop when they completed the ‘Before presentation’ section. The ‘After presentation’ section was completed immediately after the presentation. While this is a good way to gather process data you should not give participants the questions on the same sheet pre and post presentation. Usually for process it is enough to collect data at post presentation and ask participants to reflect on how the presentation has made them think about changes to their knowledge, attitudes etc. If you do want to collect at pre and post presentation see the Wheatbelt Resiliency Program document for an example as to how to do this. The questions are very broad. For example ‘I can identify factors associated with suicide.’ It is better to relate back to perceived knowledge. When conducting workshops it is important to collect information about the workshop – e.g. was the workshop useful, well facilitated etc. These types of questions help with the planning of subsequent workshops. Questions can be asked using a Likert scale. Avoid too many open ended questions, however it is always useful to ask what participants liked and what could be improved.
  • Transcript

    • 1. How to design an effective survey Workshop Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J 26.02.2013
    • 2. Workshop overview 10.00-10.15 Introduction to survey design 10.15-11.15 How to design an effective survey Writing good questions 11.15-11.30 Morning tea 11.30-12.30 Tips and tricks How to maximise response rate Getting the most out of your data 12.30-1.00 Q&A and Close Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 3. Introduction So you want to create a survey...  What is the purpose of the survey? • Needs assessment • Impact assessment • Knowledge assessment • Attitudes assessment • Behaviour assessment • Client satisfaction / program feedback • Program evaluation  What survey mode will you use? • Telephone, online, paper, interview Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 4. When to use a survey  To collect information from a large number of people or records  To collect this information relatively inexpensively  To say something about the population from which the sample was drawn  When you do not need very in-depth information  When you want the data collection procedure to be as standardised as possible Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 5. Some advantages of surveys  Anonymity  Asking sensitive questions  Standardised data to monitor trends  Quick to administer  Simple data analysis (%, frequencies)  Data can be presented as graphs/charts  Multiple distribution channels (online, paper, social networking sites) Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 6. Some limitations of surveys  Limited explanations for the data  Comments open to a range of interpretations  Don‟t know why questions were skipped or why survey was partially completed  No opportunity to explain questions that people don‟t understand  Can‟t guarantee the survey was even received or completed by the right person  Basic literacy levels needed Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 7. What makes a ‘good’ survey? Have you ever created a survey and felt disappointed with the data you collected? Have you ever been asked to answer a survey and felt irritated with the questions asked?  A good survey considers both your needs for data and the capabilities and experience of the respondent.  Collects complete, accurate, and reliable data. Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 8. What is a ‘bad’ survey?  Too wordy  Language is hard to understand  Bad timing - e.g. respondents tired at the end of a session  No „Other‟ response category  Questions that seem irrelevant to the topic  Repetitive  Boring – e.g. same structure throughout, no images  Does not consider the capabilities of the respondents Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 9. Is a long survey a bad survey? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 10. HOW TO DESIGN AN EFFECTIVE SURVEY IN 3 EASY STEPS! Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J
    • 11. Step 1. Establish your survey goals! Ask yourself:  Why am I creating this survey?  What do I hope to learn or accomplish?  How do I plan on using the data that I collect?  How will the data influence my decisions? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 12. Step 2. Create a ‘dummy table’ A tool you may not have used before. The dummy table helps ensure all survey objectives are addressed and all survey questions are relevant. Consider data analysis before, during and after survey development. Areas of inquiry Questions Demographics Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 13. A dummy table … • Ensures you can interpret data • Ensures each question will collect the information you need • Ensures you leave nothing of importance out • Ensures you don‟t collect more information than you need Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 14. Example: Health Care Experience of the LGBTI* Community in WA Activity – Find questions related to: Area of inquiry Questions LGBTI health issues Demographics and lifestyle Current experience of healthcare services How and where to reorient current health services *LGBTI – lesbian, gay, bisexual, transgender, intersex Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 15. Answers Area of inquiry Questions LGBTI health issues 10,11,24 Demographics and lifestyle 1,2,3,4,5,6,7,8,9,15,16 Current experience of healthcare services 12,13,14,20,21,22,23 How and where to reorient current health services 17,18,19,25 Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 16. Step 3. Write good questions Try to use a range of question types:  Open-ended questions • To explore issues • Open to a variety of responses (comment boxes)  Closed-ended questions • Dichotomous • Likert scale • Guttman scale • Multiple choice (one response, multiple responses) • Rank order Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 17. Example - Dichotomous Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 18. Example – Likert Scale Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 19. Example - Guttman Scale  Agreement with any one item implies agreement with the lower-order items.  E.g. are you willing to permit _______ to... - live in your country? - live in your community? - live next door to you? - marry your child? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 20. Example – Multiple Choice (one response) Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 21. Example – Multiple Choice (multi-response) Evaluation of Health Promotion to design an effective survey How 382 Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J
    • 22. Example – Rank Order  Rank the following methods of receiving a positive STI diagnosis in order of preference from highest to lowest (1=your most preferred method; 5=your least preferred method) • SMS______ • Phone call ______ • In person ______ • Email ______ • Letter ______ Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 23. Developing a Client Satisfaction Survey Example: The Wheatbelt Resilience Program  Included a series of community workshops  A survey was developed to find out how useful the workshops were in terms of increasing awareness of the Wheatbelt Resilience Program; understanding of suicide and suicidal behaviours; and opportunities to help with this issue in the community Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 24. Can you identify any ways in which this feedback sheet could be improved? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 25. The questionnaire was adapted to capture more specific information and to also gauge satisfaction with the workshop. The pre-questionnaire was on a separate document to the post-workshop evaluation. Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 26. Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 27. WRITING GOOD QUESTIONS Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J
    • 28. What do you want to know? What do you need to know?  Don‟t waste the opportunity to collect good quality data by asking vague questions  Be economical with the number of questions but make sure you collect all the information required Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 29. Activity: Work in small groups Re-write these questions and include closed ended responses:  (a) “How often do you participate in physical activities on average?”  (b) “When were you last physically active?” Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 30. What to avoid  Double-barreled questions  Biased/leading questions  Questions with a double negative  Loaded questions Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 31. Example: double-barreled question Please indicate how much you agree or disagree with each of the following statements about our service: I feel welcomed by staff and other youth at the centre Example: leading question Negotiating condom use with a sexual partner is difficult. Did this session help you feel more confident to negotiate condom use successfully? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 32. Example: double negative How do you feel about the following statement? We should not reduce investment in sexual health services. Strongly Agree Agree Disagree Strongly Disagree Example: Loaded questions Which of the following services most urgently requires improvement? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 33. What to avoid  Questions which could create a response set i.e. the tendency for a respondent to answer later questions the same way they answered earlier questions  Technical language/jargon Use simple, common words  Emotive language Do you approve of the government‟s oppressive immigration policy? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 34. What to avoid  Using “and”, “or”, “but” in the same question  Long, complex, multi-part questions  Questions which contain a negative and a positive If you were in a foreign country and did not speak the language but knew someone who did would you… Huh?! a. Let the person that spoke the language speak for you b. Try to learn the language on your own c. Not try to learn the language or let the person that you knew that spoke the language speak for you. Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 35. What to avoid Response categories that are not exhaustive What is your current age? 10 to 20 21 to 30 31 to 40 I‟m 41! Response categories that are not mutually exclusive What is your current age? 10 or less 10 to 20 20 to 30 30 or greater Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J I‟m 30! How to design an effective survey
    • 36. Avoid fuzzy words in the question  Those words which are open to different interpretation by the respondents. • Have you ever had any major injuries during the past year?  Avoid words like: important, major, better, regular, often, occasional (subjective interpretation).  Best to give specific time & details Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 37. Irrelevant questions  Questions which are irrelevant to the survey objectives • E.g. do you really need to know someone‟s income?  Questions which are irrelevant to the respondent • e.g. Don‟t ask nurses to compare the effectiveness of different patient management techniques if they have no experience of the alternatives. Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 38. Presuming knowledge  Don‟t assume people either know or have appropriate experience to answer the question. What is • Have you been bullied this year? bullying? • How many standard drinks have you drunk in the last week? • How many cigarettes have you smoked in the last week?  Use filter questions or provide more information to avoid these problems Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey What is a standard drink? I don‟t smoke!
    • 39. Presuming knowledge  Hypothetical or predictive questions • Would you like to try marijuana? • Avoid - many people might like to try anything once.  Generally avoid: • Would you like to... • What would you do if..? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 40. Personal questions  Be sensitive!  May need to preface personal questions to give respondent „permission‟ to respond truthfully  Consider the structure of response choices • people don‟t like to consider themselves at extreme ends of a scale Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 41. Activity: Design a brief survey to evaluate this workshop Workshop objectives • Increase participants’ knowledge and skills in effective survey design • Assess participants’ satisfaction with the SiREN training workshop Areas of inquiry Questions Demographics Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 42. TIPS AND TRICKS Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J
    • 43. Critically assess your survey questions  Is the respondent‟s understanding of response categories likely to be influenced by more than words?  The vaguer the question and answer categories, the greater potential for error / poor data  Is the survey information being collected in more than one mode? E.g. phone, interview, email Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 44. Critically assess your survey questions  Is changing a question acceptable to the survey sponsor?  Is the question necessary?  Will it improve the validity of your study if you do ask more than one question? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 45. Critically assess your survey questions  How easy is it to get an accurate answer?  Can people accurately recall and report past behaviours?  Will the respondent be willing to reveal the requested information?  Will the respondent feel threatened or defensive by the question?  Will the respondent feel motivated to answer each question? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 46. Tips and tricks  Insert dummy questions • These may be irrelevant to the research to avoid respondents guessing desirable answers  Reverse the wording of some survey items • This technique can avoid creating response sets I always enjoy going to work I dread going to work Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 47. Tips and tricks  The order of questions may influence responses • Put interesting questions first • Sensitive questions towards end • Demographic information – location of these questions depends on importance • Similar themes together  Put instructions for questions before the question not at the beginning of the survey Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 48. Tips and tricks  Be specific Example: How can we improve our service? This is too vague and may elicit single responses about many different aspects of the service without much guidance to help you decide which ones are important. Instead, consider asking respondents to rank from 1-5 the quality of different aspects of the service  What claims will you use the data to support? Be honest and consider if this is realistic or credible. This technique helps you to clarify the wording of questions Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 49. HOW TO MAXIMISE RESPONSE RATE Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J
    • 50. Aim to get a 65% response rate  That is: 65% of your sample complete and return a questionnaire  Remember non-respondents may bias your results  Choose a sample size that takes into account response rates  Sample size for pre- and post- surveys should be large enough in case of attrition  If possible, have participants complete survey at the time of distribution Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 51. How to maximise response rate  Involve your target group in design if possible  Pre-test!  Ask questions that people can and will answer  Uplift, motivate and encourage the respondents to give information  Avoid bad survey questions  Allow sufficient time for responding to the survey and for participants to complete the survey Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 52. How to increase responses  Plan multiple contacts • Pre-notification (tell people you are planning to survey) • Reminders for non-responders  Consider incentives • Individual incentives more effective than prize draws e.g. incentive given for getting others to complete the survey • Dependent on resources  Obtain ethics approval for your survey Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 53. A note about survey ethics… Considerations may include:  Information and consent forms  Confidentiality and anonymity  Samples which include minors  How to ask questions about potentially illegal activities  Length of survey / burden on respondents  Format of survey - e.g. acceptability, culturally appropriate Please contact siren@curtin.edu.au for more advice about survey ethics Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 54. GETTING THE MOST OUT OF YOUR SURVEY DATA Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J
    • 55. Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 56. Planning and evaluation cycle Needs assessment Re-planning Survey data Program planning Survey data Survey data Evaluation Implementation Survey data Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey Use different types of survey data as part of a continuous program planning and evaluation cycle
    • 57. How will you use the data?  Use survey data to inform the program planning cycle  Adopt a mindset of continuous program improvement  After implementing the survey, critically evaluate the quality and completeness of the data collected • Were any questions ambiguous? • Were all the questions necessary? • Did any questions appear to cause confusion? • Did you get some unexpected responses? • Were the data useful? • What decisions could the data inform? • Are there still gaps in the data? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey Ask yourself what will you change?
    • 58. Close the feedback loop  Finally, it is important to „close the loop‟ when collecting data by following up respondents to thank them for their input and to share the results of the survey  If you intend to survey the same group several times, this feedback loop is particularly important and may influence the response rate Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 59. Getting the most out of your data  Don‟t just file the survey data away until the next activity report for your funding provider is due! Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 60. Health promotion short course – example survey data How confident do you feel to undertake a needs assessment to plan a health promotion program? How confident do you feel to contribute to the design and implementation of an evaluation of a health promotion initiative? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J Pre course Post course 11 11 5 3 2 0 Not confident Confident Very confident Pre course Not Confident Very confident Confident Post course 9 10 3 7 3 0 Not confident Confident Very Confident How to design an effective survey Not Confident Very confident confident
    • 61. Health promotion short course – example survey data Pre course How confident do you feel to work collaboratively with your community and with other organisations? Post course 13 12 4 3 0 Not confident 0 Confident Very Confident Not confident Confident Very confident  What could you conclude from these data?  What decisions about the course would you make as a result of these data?  Are these decisions valid? What other data might you want to collect? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 62. Any questions? Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 63. FOLLOW UP SESSION Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J
    • 64. Follow up session SURVEY CLINIC Tuesday 12th March 2013, 10am – 12 noon Curtin University (Room TBA)  In two weeks‟ time!  Aim to put what you have learned into practice  Either: 1. Critically review a survey you have used previously OR 2. Design a new survey using what you have learned.  Bring your surveys and any questions to the survey clinic Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 65. Workshop evaluation Please draw around your hand (or both hands) and provide some feedback on today‟s workshop. Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey
    • 66. THANKS FOR COMING! Please don’t forget to complete the workshop evaluation! Thank you  Curtin University is a trademark of Curtin University of Technology CRICOS Provider Code 00301J How to design an effective survey

    ×