Chapter 6 addresses frequency claims and focuses on the two validities that are important for interrogating a frequency claim. First, we ask about the construct validity of the questions in a survey, or the construct validity of the observations researchers make about other people or animals. Second, we ask about the external validity of these claims by asking about the sampling techniques used. I cover this material in about two days. On the second day, students complete an easy homework assignment, which is to find, print, and bring in a copy of a poll or survey they found on the internet. (See slide 25)
The detailed learning objectives appear in the Instructor’s Manual, but not in the text. I do not spend much time reading these aloud to students (that’s why the font is small here); they are mainly here to guide students’ studying. (I give students access to slides via our course management system.) The objectives show students what they are expected to learn in each chapter. I usually highlight in bold text the learning objectives that I will prioritize on exams. I may also indicate whether we will cover a learning objective in class vs. whether they should study it only in the book. Questions in the test bank are keyed to which of these learning objectives they are meant to address. That way, you can more easily select questions that address the learning objectives you chose to focus on.
The detailed learning objectives appear in the Instructor’s Manual but not in the text. I do not spend much time reading these aloud to students (that’s why the font is small here); they are mainly here to guide students’ studying. (I give students access to slides via our course management system.) The objectives show students what they are expected to learn in each chapter. I usually highlight in bold text the learning objectives that I will prioritize on exams. I may also indicate whether we will cover a learning objective in class vs. whether they should study it only in the book. Questions in the test bank are keyed to which of these learning objectives they are meant to address. That way, you can more easily select questions that address the learning objectives you chose to focus on.
I expect students to read and study the textbook before coming to class so we can use our class time working through examples and doing active learning exercises rather than lecturing. To encourage students to do the reading in advance, I present them with these reading quizzes on the first day we cover a chapter. Students can write their answers on 3 x 5 note cards, or you could use clickers (personal response systems). The questions are meant to be simple enough for students to answer if they have read and understood the book. After students turn in their answers, we discuss the correct answers right away for immediate feedback. In my course, these quizzes and homework combined count for 10% of the total grade—they are low-stakes. When I post slides on the course management system, I delete the quiz slide so that the questions do not circulate. Although these are intended to be “check your reading” quizzes, they might also be used to support “just in time teaching.” If most students miss a question, I’ll know I need to explain it well during class time. If most students get a difficult question right, then I may decide to go more quickly through that material. Key for this quiz: 1.) c 2.) a 3.) b 4.) a and b
Good question wording for surveys and polls is the first major topic in this chapter. This topic addresses the construct validity of surveys and polls.
Headings for this section.
Screenshot from p. 147
I often cover question wording on the same day I go over the first exam. That leaves less time for the topic of question wording. Most students find this material fairly easy to understand so I don’t lecture extensively on it. Instead, I use the activity, “How Amazing My Class Is,” discussed in the Instructor’s Manual, p. 60. I present the survey first and ask students to nominate the problems with it. (I take out the bold headings that are shown in the sample in the IM). As students suggest different problems, I point to the heading on this slide or the next. We might suggest alternative ways of wording the questions to improve them. The activity takes 10 to 15 minutes.
I often cover question wording on the same day I go over the first exam. That leaves less time for the topic of question wording. Most students find this material fairly easy to understand so I don’t lecture extensively on it. Instead, I use the activity, “How Amazing My Class Is”, discussed in the Instructor’s Manual, p. 60. I present the survey first, and ask students to nominate the problems with it. (I take out the bold headings that are shown in the sample in the IM). As students suggest different problems, I point to the heading on this slide or the next. We might suggest alternative ways of wording the questions to improve them. The activity takes 10 to 15 minutes.
I use this figure from the text ( Figure 6.2 ) as a way to review key elements of a good survey. I might discuss here how the questions are simple, the rating system is straightforward. And I might mention how there is no reverse-worded item in this scale, and what that might mean for construct validity. (I usually help students to see that in the context of the other evidence supporting the construct validity of this scale—reviewed in Chapter 5 — the lack of a reverse-worded item is not that much of a problem).
Most of my students find the material on behavioral observations very straightforward. To save time, I usually do not lecture on this topic in class. Students in my colleagues’ classes often enjoy an activity in which they observe behavior on campus, such as how people carry their books or how much food people take at a cafeteria, and so on. They can form hypotheses and also test the inter-rater reliability of their observations in small groups.
Headings for this section.
Clear coding manuals can help improve the inter-rater reliability of coders in a behavioral observation. Figure 6.7 Excerpt from Bowker and colleagues’ (2009) description of coding hockey fan comments. This information was included in the empirical journal article’s Method section.
Sampling is usually covered on the second day devoted to this chapter. This topic addresses the external validity of surveys and polls. The focus is on generalizing to other people (the population). Other chapters discuss generalizing to other settings or contexts.
Headings for this section.
I almost always start a new topic with a concrete example. I started the topic of sampling with an example from the blog (www.everydayresearchmethods.com). Most recently I used this one, “Survey on Disaster Preparedness.” The survey had 375,000 respondents, but it was not a random sample . This example provided an opportunity to confront the common perception that even if a sample is very large, it is not representative unless it is selected using random sampling techniques. http://www.everydayresearchmethods.com/2011/07/online-poll-on-disaster-preparedness.html
Here I have copied the key questions from the blog entry. Students can discuss the answers in pairs or small groups before we discuss them as a class. This activity takes about 10 minutes to conduct and discuss. We focus especially on question b. Students have the die-hard habit of believing that a survey with so many people will be generalizable, but it’s not. We focus on how it’s the “how” not “how many” that makes a sample generalizable.
Using this figure ( Figure 6.12 ) as a guide, I do a short overview of the different sampling techniques in the book. I explain how all the different types of sampling covered in the book are organized. The colors in this figure are meant to suggest that unbiased samples look like the population but biased samples do not.
It’s worth reminding students that we always talk about “a population” (a specific group such as registered voters, students in this classroom, Target shoppers, chemistry majors, Republican men, and so on) rather than “the population” (i.e., the population of the entire world).
The homework assignment due on this day asked students to locate, print, and bring in an example of a poll. In their small groups, they discussed their examples using the questions above. Later, groups nominated the best and worst polls in their group (as determined by their construct and external validity). This activity can take 15 to 30 minutes to conduct and discuss.
Sometimes I include “more practice” at the end of the slide sets. Even if I don’t go over them in class, these provide students with additional problems to work on as they prepare for exams. This activity is Learning Actively #3 in Chapter 6, page 178 . By working with this activity (either as homework or as an in-class group exercise) students start to understand the differences between the different kinds of sampling. This activity takes about 20 to 30 minutes to conduct and discuss. My students struggled especially with the distinction between stratified random samples and cluster samples. In fact, they didn’t know that they struggled with this distinction until they did this exercise!
Detailed Learning Objectives: Question Wording and Observations1. Give examples of how question wording can change the results of a survey orpoll.2. Describe the different ways questions can be worded: open-ended, forced-choice, and using rating scales.3. Explain how to increase the construct validity of questions by wording themcarefully and by avoiding leading questions, double-barreled questions, anddouble negatives.4. Explain how question order can change the meaning (and validity) of a question.5. Explain ways to increase the construct validity of questions by preventingrespondent shortcuts (such as nay-saying), biases (such as trying to look good), orsimple inability to report.6. Describe how observational techniques for measurement are different fromsurvey techniques.7. Explain ways to improve the construct validity of observations by reducingobserver bias and target reactivity.
Detailed Learning Objectives: Sampling8. Indicate the population that a given sample might represent.9. Define three sampling problems that lead to biased samples.10. Understand why a random sample is more likely to be arepresentative sample and why representative samples have externalvalidity to a particular population.11. Explain five techniques for random sampling: simple random,multistage, cluster, stratified random sampling, and oversampling.12. Describe three techniques of nonrandom sampling: purposive,convenience, and snowball sampling.13. Explain why representative samples may be especially importantfor many frequency claims.14. Consider times when an unrepresentative sample may beappropriate for a frequency claim.15. Explain why it is more important to ask how a sample wascollected rather than how large the sample is.
1. A response set can be in the form of a) Open ended questions b) Leading questions c) Yay-saying answers d) A Likert scale2. A survey question that has you rate a response from agree to disagree is an example of a) A Likert scale b) A double barreled question c) The semantic differential format3. Which of the following is NOT a way to prevent or avoid observer bias a) Masked study design b) Inclusion of filler questions c) Measuring traces that the behavior leaves behind d) Make unobtrusive observations4. Choose the following that are correctly matched (can be more than one) a) Random sampling—external validity b) Random assignment—internal validity c) Random sampling—internal validity d) Random assignment—external validity
Writing Well-Worded QuestionsHow can you • Leading questions write items • Double-barreled questions that avoid • Double negatives (vs. negatively these worded items) problems? • Question order
Encouraging Accurate ResponsesWhat are • Response sets (three types) some – Yea-saying/ nay-saying solutions to – Fence sitting these • Faking good or bad problems? • Saying more than we can know • Measuring subjectivity vs. objectivity
Construct Validity of Behavioral Observations• Examples of frequency claims based on observational data• Are observations better than self-reports?• Good observations are both reliable and valid
Some Examples of Frequency Claims Based on Observational Data
Are Observations Better than What are some solutions to these Self-Reports? problems?Observers may see what they expect to seeObservers can affect what they see Masked research designsObserver effects Unobtrusive observations Wait it out Measure results of a behavior Figure 6.6 Unobtrusive observations This one-wayObserve ethically mirror allows researchers to unobtrusively record the behaviors of children in a preschool classroom.
Good Observations Are Both Reliable and ValidFigure 6.8 Table 1 from Campos and colleagues’ (2009) study of familyinteractions. The table depicts the degree of interrater reliability for each of thebehaviors coded.
Generalizing to Others: Sampling Participants• Populations and samples• How to get a representative sample• When does external validity matter?• When a representative sample is not the top priority• Are bigger samples better samples?
Key Questionsa. Read the first three paragraphs carefully.What do you think of the sampling techniquesused in this poll? To what population can wegeneralize the results from this poll?b. Your friend comments, “wow! 375,000 peopleresponded to this survey! We can definitelygeneralize this to the American population.”What do you say?
Populations and SamplesPopulation is ?Sample is ?
HomeworkWorking with • What do you think of the your own construct validity of the polls your examples group brought in? • What do you think about the external validity of the polls your group brought in?
More PracticeImagine that you are planning to estimate the price of the averagebook at the UD college bookstore. The bookstore carries 13,000 titles,but you plan to sample only 200 books. You will select a sample of 200books, record the price of each book, and use the average of the 200books to estimate the average price of the 13,000 books in thebookstore.Based on this information, answer the following questions:a. What is the sample in this study, and what is the population?b. How might you collect a simple random sample of books?c. How might you collect a stratified random sample?d. How might you collect a convenience sample?e. How might you collect a systematic random sample?f. How might you collect a cluster sample of 200 books?