The good the bad and the ugly: how to design an effective survey

1,979 views

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,979
On SlideShare
0
From Embeds
0
Number of Embeds
64
Actions
Shares
0
Downloads
32
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • This questionnaire was given to participants at the beginning of the workshop when they completed the ‘Before presentation’ section. The ‘After presentation’ section was completed immediately after the presentation. While this is a good way to gather process data you should not give participants the questions on the same sheet pre and post presentation. Usually for process it is enough to collect data at post presentation and ask participants to reflect on how the presentation has made them think about changes to their knowledge, attitudes etc. If you do want to collect at pre and post presentation see the Wheatbelt Resiliency Program document for an example as to how to do this. The questions are very broad. For example ‘I can identify factors associated with suicide.’ It is better to relate back to perceived knowledge. When conducting workshops it is important to collect information about the workshop – e.g. was the workshop useful, well facilitated etc. These types of questions help with the planning of subsequent workshops. Questions can be asked using a Likert scale. Avoid too many open ended questions, however it is always useful to ask what participants liked and what could be improved.
  • The good the bad and the ugly: how to design an effective survey

    1. 1. The Good, the Bad, and the Ugly:How to design an effective surveyWorkshop 26.02.2013Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
    2. 2. The Good, the Bad, and the Ugly: How to design an effective surveyCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    3. 3. Workshop overview10.00-10.15 Introduction to survey design10.15-11.15 How to design an effective survey Writing good questions11.15-11.30 Morning tea11.30-12.30 Tips and tricks How to maximise response rate Getting the most out of your data12.30-1.00 Q&A and CloseCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    4. 4. Introduction So you want to create a survey...  What is the purpose of the survey? • Needs assessment • Impact assessment • Knowledge assessment • Client satisfaction / program feedback • Program evaluation  What survey mode will you use? • Telephone, online, paper, interviewCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    5. 5. When to use a survey To collect information from a large number of people or records To collect this information relatively inexpensively To say something about the population from which the sample was drawn When you do not need very in-depth information When you want the data collection procedure to be as standardised as possibleCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    6. 6. Some advantages of surveys  Anonymity  Asking sensitive questions  Standardised data to monitor trends  Quick to administer  Simple data analysis (%, frequencies)  Data can be presented as graphs/charts  Multiple distribution channels (online, paper, social networking sites)Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    7. 7. Some limitations of surveys  Limited explanations for the data  Comments open to a range of interpretations  Don‟t know why questions were skipped or why survey was partially completed  No opportunity to explain questions that people don‟t understand  Can‟t guarantee the survey was even received or completed by the right person  Basic literacy levels neededCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    8. 8. What makes a ‘good’ survey? Have you ever created a survey and felt disappointed with the data you collected? Have you ever been asked to answer a survey and felt irritated with the questions asked?  A good survey considers both your needs for data and the capabilities and experience of the respondent.  Collects complete, accurate, and reliable data.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    9. 9. What is a ‘bad’ survey? Too wordy Language is hard to understand Bad timing - e.g. respondents tired at the end of a session No „Other‟ response category Questions that seem irrelevant to the topic Repetitive Boring – e.g. same structure throughout, no images Does not consider the capabilities of the respondentsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    10. 10. Is a long survey a bad survey?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    11. 11. HOW TO DESIGN AN EFFECTIVE SURVEY IN 3 EASY STEPS!Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
    12. 12. Step 1.Establish your survey goals! Ask yourself:  Why am I creating this survey?  What do I hope to learn or accomplish?  How do I plan on using the data that I collect?  How will the data influence my decisions? Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J
    13. 13. Step 2.Create a ‘dummy table’ To ensure all objectives are being addressed and all questions address objectives. Consider data analysis before, during and after survey development. Areas of inquiry Questions DemographicsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    14. 14. A dummy table … • Ensures you can interpret data • Ensures each question will collect the information you need • Ensures you leave nothing of importance out • Ensures you don‟t collect more information than you needCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    15. 15. Example: Health Care Experience of theLGBTI* Community in WAActivity – Find questions related to: Area of inquiry Questions LGBTI health issues Demographics and lifestyle Current experience of healthcare services How and where to reorient current health services *LGBTI – lesbian, gay, bisexual, transgender, intersexCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    16. 16. Answers Area of inquiry Questions LGBTI health issues 10,11,24 Demographics and lifestyle 1,2,3,4,5,6,7,8,9,15,16 Current experience of healthcare 12,13,14,20,21,22,23 services How and where to reorient current 17,18,19,25 health servicesCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    17. 17. Step 3.Write good questionsTry to use a range of question types: Open-ended questions • To explore issues • Open to a variety of responses (comment boxes) Closed-ended questions • Dichotomous • Likert scale • Guttman scale • Multiple choice (one response, multiple responses) • Rank orderCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    18. 18. Example - DichotomousCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    19. 19. Example – Likert ScaleCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    20. 20. Example - Guttman Scale Agreement with any one item implies agreement with the lower-order items. E.g. are you willing to permit _______ to... - live in your country? - live in your community? - live next door to you? - marry your child?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    21. 21. Example – Multiple Choice (one response)Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    22. 22. Example – Multiple Choice(multi-response) Evaluation of Health Promotion to design an effective survey HowCurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J 382
    23. 23. Example – Rank Order Rank the following methods of receiving a positive STI diagnosis in order of preference from highest to lowest (1=your most preferred method; 5=your least preferred method) • SMS______ • Phone call ______ • In person ______ • Email ______ • Letter ______Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    24. 24. Developing a Client SatisfactionSurveyExample: The Wheatbelt Resilience Program Included a series of community workshops A survey was developed to find out how useful the workshops were in terms of increasing awareness of the Wheatbelt Resilience Program; understanding of suicide and suicidal behaviours; and opportunities to help with this issue in the community Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J
    25. 25. Can you identify any ways in which this feedback sheet could be improved?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    26. 26. The questionnaire was adapted to capture more specific information and to also gauge satisfaction with the workshop. The pre-questionnaire was on a separate document to the post-workshop evaluation.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    27. 27. Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    28. 28. WRITING GOOD QUESTIONSCurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
    29. 29. What do you want to know?What do you need to know? Don‟t waste the opportunity to collect good quality data by asking vague questions Be economical with the number of questions but make sure you collect all the information required Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J
    30. 30. Activity: Work in small groups Re-write these questions and include closed ended responses: (a) “How often do you participate in physical activities on average?” (b) “When were you last physically active?”Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    31. 31. What to avoid Double-barreled questions Biased/leading questions Questions with a double negative Loaded questionsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    32. 32. Example: double-barreled question Please indicate how much you agree or disagree with each of the following statements about our service: I feel welcomed by staff and other youth at the centre Example: leading question Negotiating condom use with a sexual partner is difficult. Did this session help you feel more confident to negotiate condom use successfully?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    33. 33. Example: double negative How do you feel about the following statement? We should not reduce investment in sexual health services. Strongly Agree Agree Disagree Strongly Disagree Example: Loaded questions Which of the following services most urgently requires improvement?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    34. 34. What to avoid Questions which could create a response set i.e. the tendency for a respondent to answer later questions the same way they answered earlier questions Technical language/jargon Use simple, common words Emotive language Do you approve of the government‟s oppressive immigration policy?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    35. 35. What to avoid Using “and”, “or”, “but” in the same question Long, complex, multi-part questions Questions which contain a negative and a positive If you were in a foreign country and did not speak the language but knew someone who did would you… a. Let the person that spoke the language speak for you Huh?! b. Try to learn the language on your own c. Not try to learn the language or let the person that you knew that spoke the language speak for you.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    36. 36. What to avoidResponse categories that are not exhaustive What is your current age? 10 to 20 21 to 30 I‟m 41! 31 to 40Response categories that are not mutually exclusive What is your current age? 10 or less 10 to 20 I‟m 30! 20 to 30 30 or greaterCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    37. 37. Avoid fuzzy words in the question Those words which are open to different interpretation by the respondents. • Have you ever had any major injuries during the past year? Avoid words like: important, major, better, regular, often, occasional (subjective interpretation). Best to give specific time & detailsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    38. 38. Irrelevant questions Questions which are irrelevant to the survey objectives • E.g. do you really need to know someone‟s income? Questions which are irrelevant to the respondent • e.g. Don‟t ask nurses to compare the effectiveness of different patient management techniques if they have no experience of the alternatives.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    39. 39. Presuming knowledge Don‟t assume people either know or have appropriate experience to answer the question. What is bullying? • Have you been bullied this year? What is a • How many standard drinks have you drunk in the standard drink? last week? • How many cigarettes have you smoked in the I don‟t last week? smoke! Use filter questions or provide more information to avoid these problemsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    40. 40. Presuming knowledge Hypothetical or predictive questions • Would you like to try marijuana? • Avoid - many people might like to try anything once. Generally avoid: • Would you like to... • What would you do if..?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    41. 41. Personal questions Be sensitive! May need to preface personal questions to give respondent „permission‟ to respond truthfully Consider the structure of response choices • people don‟t like to consider themselves at extreme ends of a scaleCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    42. 42. Activity: Design a brief survey toevaluate this workshop Workshop objectives • Increase participants’ knowledge and skills in effective survey design • Assess participants’ satisfaction with the SiREN training workshop Areas of inquiry Questions DemographicsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    43. 43. MORNINGTEACurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    44. 44. TIPS AND TRICKSCurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
    45. 45. Critically assess your surveyquestions Is the respondent‟s understanding of response categories likely to be influenced by more than words? The vaguer the question and answer categories, the greater potential for error / poor data Is the survey information being collected in more than one mode? E.g. phone, interview, email Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J
    46. 46. Critically assess your surveyquestions Is changing a question acceptable to the survey sponsor? Is the question necessary? Will it improve the validity of your study if you do ask more than one question?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    47. 47. Critically assess your surveyquestions How easy is it to get an accurate answer? Can people accurately recall and report past behaviours? Will the respondent be willing to reveal the requested information? Will the respondent feel threatened or defensive by the question? Will the respondent feel motivated to answer each question?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    48. 48. Tips and tricks Insert dummy questions • These may be irrelevant to the research to avoid respondents guessing desirable answers Reverse the wording of some survey items • This technique can avoid creating response sets I always enjoy going to work I dread going to workCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    49. 49. Tips and tricks The order of questions may influence responses • Put interesting questions first • Sensitive questions towards end • Demographic information – location of these questions depends on importance • Similar themes together Put instructions for questions before the question not at the beginning of the surveyCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    50. 50. Tips and tricks  Be specific Example: How can we improve our service? This is too vague and may elicit single responses about many different aspects of the service without much guidance to help you decide which ones are important. Instead, consider asking respondents to rank from 1-5 the quality of different aspects of the service  What claims will you use the data to support? Be honest and consider if this is realistic or credible. This technique helps you to clarify the wording of questionsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    51. 51. HOW TO MAXIMISE RESPONSE RATECurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
    52. 52. Aim to get a 65% response rate That is: 65% of your sample complete and return a questionnaire Non-respondents may bias your responses Need to consider response rates when selecting sample size Need to consider attritionCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    53. 53. How to maximise response rate Involve your target group in design if possible Pre-test! Ask questions that people can and will answer Uplift, motivate and encourage the respondents to give information Avoid bad survey questions Allow sufficient time to complete surveyCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    54. 54. How to increase responses Plan multiple contacts • Pre-notification (tell people you are planning to survey) • Reminders for non-responders Consider incentives • Individual incentives more effective than prize draws e.g. incentive given for getting others to complete the survey • Dependent on resources Obtain ethics approval for your surveyCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    55. 55. A note about survey ethics…Considerations may include: Information and consent forms Confidentiality and anonymity Samples which include minors How to ask questions about potentially illegal activities Length of survey / burden on respondents Format of survey - e.g. acceptability, culturally appropriatePlease contact siren@curtin.edu.au for more advice about survey ethicsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    56. 56. GETTING THE MOST OUT OF YOUR SURVEY DATACurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
    57. 57. Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    58. 58. Planning and evaluation cycle Needs assessment Survey data Use different Re-planning Program types of survey planning data as part of a continuous Survey data program Survey data planning and evaluation cycle Evaluation Implementation Survey dataCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    59. 59. How will you use the data? Use survey data to inform the program planning cycle Adopt a mindset of continuous program improvement After implementing the survey, critically evaluate the quality and completeness of the data collected • Were any questions ambiguous? • Were all the questions necessary? • Did any questions appear to cause confusion? Ask yourself what • Did you get some unexpected responses? will you change? • Were the data useful? • What decisions could the data inform? • Are there still gaps in the data?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    60. 60. Getting the most out of your data Don‟t just file the survey data away until the next activity report for your funding provider is due!Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    61. 61. Health promotion short course –example survey data How confident do you Pre course Post course 11 11 feel to undertake a needs assessment to plan a 3 5 2 health promotion 0 program? Not Confident Very Not Confident Very confident confident confident Confident How confident do you feel to Pre course Post course 10 9 contribute to the design and 7 implementation of an 3 3 evaluation of a health 0 promotion initiative? Not Confident Very Not Confident Very confident Confident confident confident Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J
    62. 62. Health promotion short course –example survey data Pre course Post course How confident do you feel to 13 12 work collaboratively with your community and with 4 3 other organisations? 0 0 Not Confident Very Not Confident Very confident Confident confident confident  What could you conclude from these data?  What decisions about the course would you make as a result of these data?  Are these decisions valid? What other data might you want to collect? Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J
    63. 63. Any questions?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    64. 64. FOLLOW UP SESSIONCurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
    65. 65. Follow up session SURVEY CLINIC Tuesday 12th March 2013, 10am – 12 noon Curtin University (Room TBA) In two weeks‟ time! Aim to put what you have learned into practice Either: 1. Critically review a survey you have used previously OR 2. Design a new survey using what you have learned. Bring your surveys and any questions to the survey clinicCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    66. 66. Workshop evaluation Please draw around your hand (or both hands) and provide some feedback on today‟s workshop.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J
    67. 67. THANKS FOR COMING! Please don’t forget to complete the workshop evaluation! Thank you Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J

    ×