More Related Content
What's hot
Similar to How to create a well constructed survey final copy
Similar to How to create a well constructed survey final copy (20)
Recently uploaded
Recently uploaded (20)
How to create a well constructed survey final copy
- 1. USING SURVEYS TO GATHER QUANTITATIVE DATA Sierra Patterson Sr. UX Researcher
- 2. AGENDA o Why surveys? o Steps to designing a survey o Step 1 – Defining the objective o Step 2 – Creating questions o Step 3 – Structuring the survey o Step 4 – Testing and analysis o Questions?
- 3. WHY SURVEYS o Surveys are used frequently… by everybody o They are an effective tool if designed properly o Poor design = bad data The following slides will walk you through how to design a survey in a way that minimizes bias, and maximizes efficiency and data validity.
- 4. TYPES OF SURVEYS AND COLLECTION METHODS o Descriptive surveys o A=A, B=B, C=C, ETC. o Explorative surveys o A+B=C Mail/email Face to face Online Telephone
- 5. STEP 1: DEFINING THE OBJECTIVE Defining the objective
- 7. DEFINE YOUR OBJECTIVE Clearly define your objective “What should dog owners take into consideration while at the park with their dog?”
- 8. STEP 2: CREATE QUESTIONS Rules and Question types
- 9. WRITE YOUR QUESTIONS o What will the answer to this question tell me? o Am I asking this question because I am curious about it, or because it is a valid question supported by existing data? o Is there a clear purpose to my question?
- 10. RULES o Be unambiguous: o Ambiguous: “How often do you take your dog out?” o Unambiguous: “How often do you take your dog to the park?” o Avoid jargon and acronyms o Avoid double barreled questions: o “Dogs should be kept on a leash at all times, so laws should be passed to restrain dogs in parks” o “Dogs should be kept on a leash at all times.” o “Laws should be passed to restrain dogs in parks.”
- 11. o Don’t ask leading or loaded questions: o “Don’t you agree that dogs should not be allowed off of the leash at the park?” o Do you agree or disagree that dogs should not be allowed off of the leash at the park?” o Avoid questions with too many options o Avoid questions without an ‘out’ Yes/No Scales without a ‘Neutral’ or ‘I don’t know’ or both
- 12. TYPES OF QUESTIONS o Open ended questions o Yes/No o Radio button options o Checklists o Scales Strongly Agree Agree Neutral Disagree Strongly Disagree I don’t know For the following questions please circle your answer with 1=Once a month, 2=Once a week, 3=Multiple times a week, 4=Daily, 5=Other, please explain in the space below. 1 2 3 4 5Question:
- 13. STEP 3: STRUCTURE THE SURVEY Survey flow and organization
- 14. STRUCTURING AND ORGANIZING YOUR QUESTIONSo Flow from low to high recall/sensitivity rate o Keep questions types together o Keep topics grouped together o It is better to group topics together and find new ways to ask a question. Questions to ‘warm up’ Demographic questions Questions answered easily Yes/No/Out Moderate questions Radio, Checklist, Scales High recall, sensitive, or new information Open Ended
- 15. STEP 4: TESTING AND ANALYSIS Dry run and test your survey. Analyze the results
- 16. Look for Inconsistencies Organizational issues Areas or points of confusion Broken rules! You cannot change a survey once it has been delivered!
- 17. Based on the above analysis, it is clear that roughly 60% of dog owners believe that all owners should keep their dog on a leash at all times. While 30% believe that dogs should be kept on a leash outside of dog parks. With the remaining 10% claiming that they do not think that pet owners should not be required to keep a dog on a leash in any circumstance. When should Pet Owners be Required to use a Leash Leash at all times Leash only in public Should not require a leash
- 19. Questions?
- 20. Bernard H., Russell Social Research Methods. SAGE Publications: 2013 Goodman, Elizabeth; Kuniavsky, Mike; Moed Andrea Observing the User Experience. Elsevier: 2012
Editor's Notes
- Surveys are a frequently used tool, and can lead to powerful data if they are designed properly. They can be used to inform on a population Gather further insight Or validate existing knowledge However, surveys are oftentimes misused and poorly designed, leading to invalid or misleading data. Overuse is another issues when it comes to survey deployment, and this is typically a result of poorly design surveys. If you have a poorly designed survey, it will not address you objective and you will have to go back to your sample population with additional questions – hence the overuse. Therefore, understanding how to design and create effective surveys is critical to using them as a tool in your research The following slides will walk you through how to design a survey in a way that minimizes bias, and maximizes efficiency and data validity.
- Surveys can provide either descriptive or explorative data. Descriptive data focuses on one variable at a time to characterize a population. Explorative data, will describe a population by identifying and exploring the relationships between variables in your survey. For the purposes of this presentation, we will focus on descriptive surveys. This is likely the survey type you will be working with in UX. There are a number of different ways to distribute surveys. You can send them through the mail, conduct telephone surveys, have users’ complete online surveys, or deliver surveys face-to-face. The latter is my preferred method as it ensures that you will receive a complete survey, and allows participants to ask questions if something is unclear to them. Regardless of the method of distribution, there are some standard methods and if you will ‘rules’ to follow to create a survey that will increase the reliability of the data collected. So let’s say you know you want to conduct a survey to better understand how respondents feel about dogs in public. We are assuming you already have a representative sample population identified – we’ll say it’s dog owners - and you are moving into survey creation.
- Utilize qualitative data to give you an idea of what your survey should be asking. Qualitative data gives you an idea about the existing terminology, mental models, behaviors, desires or pain points that exist in your population Begin with existing work. If you have the opportunity to use existing research to inform your new survey – do it (as long as it isn’t out dated and no longer valid)! If not, then conducting qualitative research before you begin designing a survey should be your first step. I would argue that it should be done regardless of previous work, however sometimes you just don’t have to time to conduct original qualitative research, so using existing work is your next best option. Either way, if you get the opportunity to conduct qualitative research prior to creating your survey, you can begin with some qualitative methods such as: Focus groups Interviews – semi-structured or unstructured Participatory observation or observational research This can be done in a short amount of time, or can be a full-fledged research effort. So now that we have some data to better understand what we need, we can begin to think about our objective
- Before you even begin thinking about what questions you want to ask, you need to have a clearly defined objective of which the questions will be structured around. So building off of our initial goal of better understanding how respondents feel about dogs in public. Let’s say your qualitative data is showing that pet owners frequently take dogs to parks where other activities are occurring. It also tells you that dog owners have differing of opinions when it comes to their responsibility for their dog in public. We can then focus our objective to address the following: What should pet owners take into consideration while at the park with their dog?
- Begin by writing all of the questions you can think of, don’t worry about structure, just get them all written down. After you have written down all of the questions you can think of, start to edit them. Meaning begin to assess whether each question will lead you to answering your objective. For example, if you are looking at a question ask yourself, “What will the answer to the question give me?” “Am I asking it because I am curious about it, or because it is a valid question supported by data?” AND “is there a clear purpose to my question.” Referring back to your qualitative data can also help you edit out your questions. Does the data support the question you are asking?
- Once your questions are written, but before you can start to place them into question types or styles, make sure that your questions follow these rules: Be unambiguous. “How often do you take your dog out” can be ambiguous – take your dog to the park, the vet, your friend’s house? Where are you taking the dog? You need to make sure you are clear and direct, if respondents can interpret a question differently from what you intend to ask, they likely will. Avoid jargon and acronyms Unless you are 100% confident that your population will be familiar with these questions, avoid them, and spell acronyms out at the first time. Do not use double barreled questions: Double barreled questions really asks two questions in one. A respondent may agree or disagree with the first half of the question , but then agree or disagree with the second half – how do you know which half of the question they are agreeing or disagreeing with?
- Do not use loaded or leading questions: Leading questions will bias responses as they lead or bait informants to provide you with the answer you want Questions with too many options – this can overwhelm respondents and lead to incorrect data Provide your respondents with an out – meaning give them the opportunity to provide their own information, or select a neutral or ‘I don’t know’ answer. This will cause less bias in your answers, as a question that only gives respondents the option between yes and no, does not give them an out, and can lead to misleading data as respondents would rather give you an answer than not. If your questions adhere to the rules we just laid out, you can now begin to structure them into actual survey questions
- After you have written down every question you can think of, it is now time to structure them into question types. There are a number of types of questions you can ask in your survey. If appropriate, you can have a survey with the same types of questions, or a survey with a combination of these questions, it depends on what you want to know and how that information is best gathered. Question types: Open ended Yes/No Radio button questions Checklist Scales – such as a likert scales When using scales, always explain how you want the respondent to answer. For example, if your scale has the answers written out in the scale itself, then little to no direction is necessary. However, for numbered scales, or scales without the answers directly in the scale, directions must be given as to how the respondent should answer each question and what each step of the scale stands for. As mentioned in the previous slide; no matter which question type you use, you always have to make sure you give your respondent an ‘out’. For example, a question that only gives respondents the option between yes and no, does not give them an out, and can lead to misleading data as respondents would rather give you an answer than not. Rather than yes/no, ask your respondents yes, no, I don’t know, and other with a field for open ended entry. This gives you information regarding whether respondents know about the information you’re asking them, and gives you an opportunity to learn about other variables you have not taken into account. As far as the radio button, checklist, and scale options, you still want to give an out in ways appropriate for the question type. Radio buttons or scales can both have a neutral option, and an I don’t know option. I like to add other options wherever possible, so that could be an option for radio button and check list options. Open ended questions have an automatic out
- Now that you have all of your questions, it is time to structure them into a survey. This typically happens simultaneously with your question creation. As you create your questions types, you will start to put questions next to one another and loosely structure your entire survey. However, there is still room for editing and reorganizing. It is surprising how important the flow of a survey is, and even with what I think are well structured and organized surveys, I still have to edit and change question types when I start working on the flow of the survey. That being said, it is important to make sure your survey flows from low to high recall/sensitivity rate, that topics are grouped together, and that like question types are kept together. Make certain that you are not asking your respondent to jump between topics throughout your survey For example: we wouldn’t want to ask respondents three radio button questions about their activities in the park, followed by a radio button question asking about what type of dog food they use just so we can keep all of the radio button questions together. It is better to group topics together and find new ways to ask a question, then it is to jump topics. I am mentioning this a lot, because it is very important and frequently overlooked. As Jumping topics reduces recall rate and can lead to misleading data – not to mention it irritates your respondents. To begin, you want to give your respondents a chance to ‘warm up’ before you get to any sensitive or difficult questions, or questions that may require higher recall rates. This is because if you don’t give your respondent’s time to think about the topic you are asking them about, they may not recall deeper insights that they would if you were to slowly get them into the harder questions. It is similar to a movie in that you can’t start at the end even though that is what you are watching the movie for – you want to know how it ends. But if you just see the ending you don’t know what is happening and aren’t able to make much sense about what is put in front of you. Additionally, respondents may be offended or put off if you begin asking more sensitive or higher recall questions right off the bat. This is particularly true for sensitive topic issues. Starting the survey off with the deeper questions you are looking to answer can also scare your respondent off – they see the first question and how much cognitive stress it puts on them and they can only attribute the rest of the survey will follow this same pattern. To avoid this it is best to start your survey off with the ‘lighter’ questions such as demographic questions. After demographic questions, your respondent is now ‘in’ the survey and you can begin to ask questions that will lead them through the survey. Example of a good flow would be as follows: Demographics Questions that are answered easily are typically heavily supported by your qualitative research – you expect that respondents will find these questions easy to answer, but it also introduces them to the topic you will be exploring – Typically yes/no questions Moderate questions – questions that came up in qualitative research, but you would like to explore in more detail Typically radio buttons, check lists, and scales Finally, the high recall, sensitive, or new information questions. Typically open ended questions because we may not have as much information about these topics going into the survey.
- Once your questions are written, and the design of your survey is complete, it is time to test run your survey. One of the best ways to find out if you have any inconsistencies, organizational issues, areas or points of confusion or, broken any of the rules for asking questions, is to ask a coworker – or multiple coworkers – to take the survey for you. It would be best to provide them with a bit of background first, but if they are able to understand the survey and do not run into any obstacles without complete domain knowledge, then your survey should be good to go! Before you send your survey to your sample population, however, make sure you are 100% happy with it – or as close as you can be – because you cannot change a survey once it has been delivered. This will invalidate your data and prevent comparison between your variables.
- Descriptive analysis uses data visualization – such as graphs, data tables, and summaries to understand and describe the data. Descriptive analysis is not intended to make inferences about anything that is outside of your data set – meaning you can’t infer something about the rest of the world simply based on the data from your sample population and survey results. Your analysis should include univariate analysis – analysis that focuses on examining individual independent variables in detail. This is the easiest analysis to do by hand as you are simply looking for patterns in data for data visualizations. Univariate analysis deals with one set of data. For example if you ask when pet owners should be required to keep their dog on a leash, and provide respondents with options that are exhaustive to your question, it is possible that you will get the following breakdown in which 60% of dog owners believe owners should keep their dogs on a leash at all times 30% believe that they should only have to keep their dog on a leash outside of dog parks and 10% of dog owners believe that they should never be required to keep their dog on a leash. You are looking at this question independently of the other questions you have asked, and you are not comparing that variable with another. When analyzing any open ended questions, you will want to look for common patterns, themes, or words that your respondents have used to answer your question. This will tell you whether or not there is commonality among your respondents, and can provide you with rich insight into the actual cultural constructs and domains of your respondents. Other types of analysis include: Bivariate (looking at associations between pairs of variables) and Multivariate (understanding the effects of more than one independent variable at a time on a dependent variable) are best carried out with a statistical analysis tool such as SPSS which can tell you – among other things - if there is actually statistical significance between variables and associations or not. Again, this type of heavier analysis may not be necessary for your work in UX as it is.