Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Experiments and Test Markets
Recently uploaded
Recently uploaded (20)
Experiments and Test Markets
- 1. 11-1 Chapter 11 Experiments and Test Markets
- 2. 11-2 Learning Objectives Understand . . . • uses for experimentation • advantages and disadvantages of the experimental method • seven steps of a well-planned experiment • internal and external validity with experimental research designs
- 3. 11-3 Learning Objectives • three types of experimental designs and the variations of each • functions and types of test marketing used in experimenting with new marketing products and services
- 4. 11-4 Causal Evidence Agreement between IVs and DVs Time order of occurrence Extraneous variables did not influence DVs
- 5. 11-5 Evaluation of Experiments Advantages Disadvantages • Ability to manipulate • Artificiality of labs IV • Non-representative • Use of control group sample • Control of extraneous • Expense variables • Focus on present and • Replication possible immediate future • Field experiments • Ethical limitations possible
- 6. 11-6 Exhibit 11-1 Experimentation in the Research Process
- 7. 11-7 Conducting an Experiment Specify treatment levels Specify treatment levels Control environment Choose experimental design Select and assign participants Pilot-test, revise, and test Collect data Analyze data
- 8. 11-8 Exhibit 11-2 Experiment of Placement of Benefits Module
- 9. 11-9 Selecting and Assigning Participants Random Matching assignment
- 11. 11-11 Exhibit 11-3 Quota Matrix Example
- 12. 11-12 Measurement Options Observation Physiological Physiological Paper-and- Paper-and- measures measures pencil tests pencil tests Options Options Scaling Self- Scaling techniques administered techniques instruments
- 13. 11-13 Validity in Experimentation Internal External
- 14. 11-14 Threats to Internal Validity Maturation History Experimental Testing mortality Threats Statistical Instrumentation regression Selection
- 15. 11-15 Additional Threats to Internal Validity Diffusion of treatment Diffusion of treatment Compensatory equalization Compensatory equalization Compensatory rivalry Compensatory rivalry Resentful disadvantaged Resentful disadvantaged Local history Local history
- 16. 11-16 Threats to External Validity Reactivity of testing on X Interaction of selection and X Other reactive factors
- 17. 11-17 Experimental Research Designs Pre-experiments True experiments Field experiments
- 18. 11-18 After-Only Case Study X O Pre-experiment
- 19. 11-19 One Group Pretest- Posttest Design O1 X O2 Pre-experiment
- 20. 11-20 Static Group Comparison X O1 O2 Pre-experiment
- 21. 11-21 Pretest-Posttest Control Group Design R O1 X O2 R O3 O4 True experiment
- 22. 11-22 Posttest-Only Control Group Design R X O1 R O2 True experiment
- 23. 11-23 Nonequivalent Control Group Design O1 X O2 O3 O4 Field experiment
- 24. 11-24 Separate Sample Pretest- Posttest Design R O1 (X) R X O2 Field experiment
- 25. 11-25 Group Time Series Design R O1 O2 O3 X O4 O5 O6 R O7 O8 O9 O10 O11 O12 Field experiment
- 26. 11-26 Test Market Selection Over-testing Control of Control of distribution distribution Representative Representative Criteria Criteria Isolation Multiple Multiple Isolation locations locations Media coverage
- 27. 11-27 Types of Test Markets Standard Standard Controlled Controlled Electronic Electronic Simulated Simulated Virtual Virtual Web-enabled Web-enabled
- 28. 11-28 Exhibit 11-5 Test Market Cities
- 29. 11-29 Key Terms • Blind • Experiment • Control group • Experimental treatment • Controlled test market • External validity • Dependent variable • Field experiment • Double-blind • Hypothesis • Environmental control • Independent variable • Internal validity
- 30. 11-30 Key Terms • Matching • Test market • Operationalized – Electronic test market • – Simulated test market Quota matrix – Standard test market • Random assignment – Virtual test market • Replication • Treatment levels • Web-enabled test market
Editor's Notes
- This chapter presents the uses of experimentation and the various types of experimental designs. It also discusses the advantages and disadvantages of the experimental method.
- An experiment is a study involving intervention by the researcher beyond that required for measurement. The usual intervention is to manipulate some variable in a setting and observe how it affects the participants or subjects being studied. There is at least one independent variable (IV) and one dependent variable (DV) in a causal relationship. There are three types of evidence necessary to support causality. First, there must be an agreement between independent and dependent variables. The presence or absence of one is associated with the presence or absence of the other. Second, beyond the correlation of independent and dependent variables, we consider the time order of the occurrence of the variables. The effect on the dependent variable should not precede the manipulation of the independent variable. The effect and manipulation may occur simultaneously or the manipulation may occur before the effect. The third source of support comes when researchers are confident that other extraneous variables did not influence the dependent variable. To ensure that these other variables are not the source of influence, researchers control their ability to confound the planned comparison.
- This slide presents the advantages and disadvantages of experiments. Replication is the process of repeating an experiment with different participant groups and conditions to determine the average effect of the IV across people, situations, and times. A field experiment is a study of the dependent variable in actual environmental conditions. The artificiality of a lab is possibly the greatest disadvantage of experiments. Also, experiments typically use small convenience samples which cannot be generalized to a larger population. Compared to surveys, they are expensive. They also cannot deal with past events or predict events in the far-off future. Finally, marketing research is often concerned with the study of people and there are limits to the types of manipulation and controls that are ethical.
- Exhibit 11-1 depicts the use of experiments in the research process.
- These are the seven activities the researcher must accomplish to make an experiment a success. In the first step, the researcher is challenged to 1) select variables that are the best operational definitions of the original concepts, 2) determine how many variables to test, and 3) select or design appropriate measures for the chosen variables. The selection of measures for testing requires a thorough review of the available literature and instruments. In an experiment, participants experience a manipulation of the independent variable, called the experimental treatment. The treatment levels are the arbitrary or natural groups the researcher makes within the independent variable. A control group can provide a base level for comparison. A control group is a group of participants that is measured but not exposed the independent variable being studied. Environmental control means holding the physical environment of the experiment constant. When participants do not know if they are receiving the experimental treatment, they are said to be blind. When neither the participant nor the researcher knows, the experiment is said to be double-blind. The design is then selected. Several designs are discussed on the next several slides. The participants selected for the experiment should be representative of the population to which the researcher wishes to generalize the study’s results. Random assignment is required to make the groups as comparable as possible.
- Exhibit 11-2 illustrates an experimental design.
- Random assignment uses a randomized sample frame for assigning participants to experimental and control groups. Matching is an equalizing process for assigning participants to experimental and control groups. Matching employs a nonprobability quota sampling approach. The object of matching is to have each experimental and control participant matched on every characteristic used in the research. Quota matrix is a means of visualizing the matching process. If matching does not alleviate assignment problems, a combination of matching, randomization, and increasing the sample size may be useful. Examples of each are provided on the following slides.
- The sampling frame is often small for experiments and the participants may be self-selected. However, if randomization is used, those assigned to the experimental group are likely to be similar to those assigned to the control group. Random assignment allows one to make the groups as comparable as possible. The random assignment means that participants have an equal and known chance of being assigned to any of the groups in the experiment.
- Exhibit 11-3 presents an example of a quota matrix. One-third of the participants from each cell of the matrix would be assigned to each of the tree groups.
- Researchers have several measurement and instrument options with experiments.
- Internal validity exists when the conclusions drawn about a demonstrated experimental relationship truly implies cause. External validity exists when an observed causal relationship can be generalized across persons, settings, and times.
- History: During the time that an experiment is taking place, some events may occur that confuse the relationship being studied. Maturation: Changes may also occur within the participant that are a function of the passage of time and are not specific to any particular event. A participant may become hungry, bored, or tired and these conditions can affect response results. Testing: The process of taking a test can affect the scores of a second test. Instrumentation: This threat to internal validity results from changes between observation in either the measuring instrument or the observer. Selection: An important threat to internal validity is the selection of participants for experimental and control groups. The groups should be equivalent in every respect. Statistical regression: This factor operates especially when groups have been selected by their extreme scores. Experimental mortality: This occurs when the composition of the study groups changes during the test. These seven threats to internal validity can largely be addressed through random assignment.
- Five additional threats to internal validity are independent of whether or not one randomizes. Diffusion or imitation of treatment: If people in the experimental and control groups talk, then those in the control group may learn of the treatment, eliminating the difference between the groups. Compensatory equalization: Where the experimental treatment is much more desirable, there may be an administrative reluctance to deprive the control group members. Actions to compensate the control group may confound the experiment. Compensatory rivalry: This may occur when members of the control group know they are in the control group. This may generate competitive pressures, causing the control group members to try harder. Resentful demoralization of the disadvantaged: When the treatment is desirable and the experiment is conspicuous, control group members may become resentful that they are deprived and lower their cooperation and output. Local history: The regular history effect already mentioned impacts both experimental and control groups alike. When one assigns all experimental persons to one group session and all control group people to another, there is a chance for some peculiar event to confound results.
- External validity is concerned with the interaction of the experimental treatment with other factors and the resulting impact on the ability to generalize to (and across) times, settings, or persons. Three major threats to external validity are presented in the slide. The reactive effect refers to sensitizing participants via a pretest so that they respond to the experimental stimulus (X) in a different way. For instance, people who participate in a web survey like the one from Kohl’s shown in the slide may then be sensitized to store displays and organization. The process by which test participants are selected for an experiment may be a threat to external validity. The population from which one selects participants may not be the same as the population to which one wishes to generalize the results. The experimental settings themselves may have a biasing effect on a participant’s response to X. An artificial setting can produce results that are not representative of larger populations. If participants know they are participating in an experiment, there may be a tendency to role-play in a way that distorts the effects of X. Another reactive effect is the possible interaction between X and participant characteristics.
- Experimental designs vary widely in their power to control contamination of the relationship between the independent and dependent variables. Experiments can be categorized as pre-experiments, true experiments, and field experiments based on the characteristic of control. Various designs in each category are presented on the following slides.
- X refers to the treatment or manipulation of the independent variable (more than one X refers to a different level of treatment ). O refers to the observation or measurement of the dependent variable. An example is a media campaign about a product’s features without a prior measurement of consumer knowledge. Results would reveal only how much target consumers know after the media campaign, but there is no way to judge the effectiveness of the campaign. The lack of a pretest and control group makes this design inadequate for establishing causality. Dryel was test-marketed prior to full commercialization. Its placement in a market followed by observations of sales is another example of this design.
- This design meets the threats to internal validity better than the one-shot case study, but it is still a weak design. For example, a researcher examining the effect of a commercial on brand liking would begin by taking a pre-test to determine current levels of brand liking among the participants. The commercial would be shown. Then a post-test would measure brand liking after the commercial. A comparison between the post-test and the pre-test shows the change in liking. However, any changes in liking are not necessarily due to the commercial. The act of giving a pre-test could have influenced liking (testing effect).
- This design provides for two groups, one of which receives the experimental stimulus while the other serves as a control. For example, imagine that a new type of cheeseburger is being introduced, and an ad is run in one-fourth of the 210 designated market areas (DMAs) in the United States. After the ad airs, those who remember seeing it would be in the experimental group (X). Those who have no recall of the ad would be in the control group. The intent of each group to purchase the cheeseburger would be measured. The main weakness of this design is that there is no way to be certain that the two groups are equivalent or that the individuals are representative.
- True experimental designs use randomly assigned groups to ensure equivalence. In this design, groups are randomly assigned and there is a control group for comparison. The effect of the experimental variable is E = (O 2 -O 1 ) – (O 4 -O 3 ). This design deals with many of the threats to internal validity, but local history, maturation, and communication among groups can still lead to problems. External validity is threatened because there is a chance for a reactive effect from testing. For example, this meat counter may wish to test the effectiveness of adding a display of recipes for use with different cuts of meat. Participants are randomly assigned to either the control group or the experimental group. Those in the control group are not exposed to the display of recipes. Sales are measured for the control group at the start of the study and again the following week. Sales associated with the experimental group are also measured at the start of the study and again after exposure to the recipe display.
- In this design, the pretest measurements are omitted. Pretests are well established in classical research design but are not really necessary when it is possible to randomize. The experimental effect is measured by the difference between O1 and O2. Internal validity threats from history, maturation, selection, and statistical regression are controlled adequately by the random assignment. Different mortality rates could cause a problem. Buick dealerships wish to determine the effectiveness of a special “test-drive” incentive. Buick dealerships nationwide are randomly assigned to either the control group or the experimental group. Those in the experimental group use a promotion to encourage test drives. The control group does not use any such promotions. The number of test drives throughout are measured and compared to determine if the promotion resulted in significantly more test drives.
- This is a strong and widely used quasi-experimental design. It differs from the pretest and posttest control group design because the test and control groups are not randomly assigned. There are two varieties: intact equivalent design and self-selected experimental group design. In the intact equivalent design, the membership of the experimental and control groups is naturally assembled. The self-selected experimental group design is weaker because volunteers are recruited to form the experimental group, while non-volunteer participants are used for control. A comparison of the pretest results for each group is one indicator of the degree of equivalence between test and control groups. For example, children from two different classes in school may be asked to test a toy. Participants are pre-tested on their interest in the toy. The experimental group spends time playing with the toy while the control group is not exposed to the toy. A post-test then measures interest in the toy.
- This design is most applicable when we cannot know when and to whom to introduce the treatment but we can decide when and whom to measure. The parenthesized treatment (X) is irrelevant to the purpose of the study but is shown to suggest that the experimenter cannot control exposure to the treatment. This is not a strong design because several threats to internal validity are not handled adequately. History can confound the results. There is no way to restrict the application of the treatment. For example, an new advertising campaign for a prescription drug is introduced on television. Awareness of the brand name is measured prior to the campaign introduction. After the campaign ends, awareness is measured again.
- A time series design introduces repeated observations before and after treatment and allows participants to act as their own controls. The single treatment group design has before-after measurements as the only controls. There is also a multiple design with two or more comparison groups as well as the repeated measurements in each treatment group. This format is especially useful where regularly kept records are a natural part of the environment and are unlikely to be reactive. The time series approach is also good way to study unplanned events in an ex post facto manner. The internal validity problem for this design is history. To reduce this risk, we keep a record of possible extraneous factors and attempt to adjust the result to reflect their influence., For example, if the federal government were to begin price controls, we could still study the effects of this action on gasoline prices later if we had regularly collected records for the period before and after the advent of price control.
- A test market is a controlled experiment conducted in a carefully chosen marketplace to measure marketplace response and predict sales or profitability of a product. There are several criteria to consider when selecting test market locations. These are listed in the slide. The location and characteristics of participants should be representative of the market in which the product will compete. Multiple locations are often required for optimal demographic balance. The market should not have been over-tested because savvy participants’ prior experience may cause atypical responses. The location should not have spill-over media coverage from other markets. It should be isolated to minimize uncontrollable factors. The control of distribution affects test locations and types of test markets.
- There are six major types of test markets. A standard test market is a real-time test of a product through existing distribution channels. It offers the advantage of using actual distribution channels but costs a lot and takes a long time to get results. A controlled test market is a real-time test of a product through arbitrarily selected distribution partners. It is conducted by a specialty research supplier that guarantees distribution of the test product through outlets in selected cities. Controlled test markets cost less than traditional ones and reduce the likelihood of competitor monitoring. An electronic test market is a test system that combines store distribution services, consumer scanner panels, and household-level media delivery in specifically designated markets. Simulated test markets test a product in a laboratory setting designed to simulate a traditional shopping environment. A virtual test market uses a computer simulation and hardware to replicate the immersion of an interactive shopping experience in a three-dimensional environment. Web-enabled test markets are product tests using online distribution.