Downloaded 13 times
Uploaded byRachel Chung
MBA724 s6 w1 experimental design
This document defines key concepts in experimental design including experiments, causation versus correlation, control groups, random assignment, internal and external validity, and the pros and cons of experimental methods. It explains that experiments manipulate independent variables to determine their effect on dependent variables and discusses strategies for establishing evidence of causality, such as time order of events and ruling out other possible causes. The document also covers how much control researchers have in different types of studies from observations to full experiments and notes important considerations for conducting experiments like specifying treatment variables, levels, and designs and ensuring validity.
More Related Content
![Are you ready for the private cloud? [WHITEPAPER]](https://cdn.slidesharecdn.com/ss_thumbnails/privatecloudwhitepaperen-120425094801-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to MBA724 s6 w1 experimental design
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
MBA724 s6 w1 experimental design
- 1. Experimental Design Learning Objectives Define experimentation Explain pros and cons of the experimental method Become familiar with key design principles of experiments Define internal validity Define external validity
- 2. Experiments Manipulation of onevariable to determine its effect on other variables e.g. the impact of insurance plan (has insurance vs. no insurance) on auto repair quote e.g. the design of suspect identification (line-up vs. one by one) on eyewitness accuracy
- 3.
- 4. Causation and ExperimentalDesign Control/ Random Matching Assignment 6-4
- 5. Evidence of Causality Covariation between A and B Time order of events No other possible causes of B
- 6. Classic Experiment Strategy •Pre-test • Control group
- 7. Non-Experiments After-Only Case Study “Let’s do something and see what happens” X O
- 8. How much theresearcher manipulates and controls the independent variables None – observations, document analysis, case studies*, surveys* Little – ex post facto experiments (or quasi- experiments) Full control – experiments *Surveys and case studies can be experimental depending on whether researcher manipulation is introduced
- 9. Conducting an Experiment Specify treatment variables Specify treatment levels Control environment Choose experimental design
- 10. Validity in Experimentation Internal External
- 11. Experiments – Prosand Cons Advantages Disadvantages • Ability to manipulate IV •Artificiality of labs • Use of control group •Non-representative • Control of extraneous sample variables •Expense • Replication possible •Focus on present and • Field experiments immediate future possible •Ethical limitations
- 12. Discussion If yourgroup were to address your research questions using the experimental methodology, how would you design the experiment? Discuss the following: Independent variable(s)? (also called manipulation) Dependent variable(s)? (also called outcome variables) Confounding variable(s)? (potential alternative explanations) Extent of control: full experiment or ex post facto? Would you prefer a lab or field experiment? How would you achieve random assignment? How do you ensure blindness (or double-blindness?) How do you make sure it’s not just a non-experiment? (e.g., an after-only case study)
- 13.
Editor's Notes
- #3 The next set of slides is review from Session 3 – Research DesignLet’s first see if you can reconstruct from what we discussed a couple of weeks ago about experimentsWhat are the basic elements of an experiment?What does an experiment involve?An experiment is an attempt to establish cause and effectIt’s the best design available for a causal studyMany experiments have been conducted in different areas of social sciences (e.g., psychology, economics, etc). When done well, they often offer the strongest evidence supporting any given claim or argument.
- #6 Why do businesses and social scientists obsess so much about causality?Understanding causal relationships is extremely critical to problem solving. We can't solve a problem unless we understand what the root cause is! For example, if we want to know why employees have low levels of productivity, we need to find the potential causes – health issues? Motivation? Compensation? Organizational culture? We can attempt to improve productivity only after we have identified the most important causes.(However having said that, single linear causal models have its limitations - Refer to Peter Senge's systems thinking)To make a causality statement, you need strong support for your argumentWe need at least 3 pieces of evidence:1. Your independent variable correlates with your dependent variable in some way. For example, the temperature goes up, the likelihood of getting sick goes down, and vice versa. This is “negative” correlation. It’s a necessary but not sufficient condition for causality2. The independent variable must precedes the dependent variable. If you turn 30 before eating a birthday cake, it’s hard to explain why eating a cake causes you to get older!3. Control for confounding variables - You should be able to demonstrate that no other variables could possibly cause the observed changes in the dependent variable. This is hard to do with an observational study like the temperature and likelihood of getting sick. You see a pattern, but because you didn’t control other factors, you can’t really say for sure that there’s no other causes hidden somewhere.This is why “fully controlled” experiments are so highly valued in scientific studies because they provide the best way to determine causality with certainty. Unfortunately they are not always feasible, especially in the social sciences where manipulation of people is not always ethical.
- #7 Most of us are familiar with experimental strategy simply because we read the medical news all the time. A group of patients with a specific illness are given current treatment X while another group of patients are given treatment Y. Typically, a third group called control group, are given treatment Z (which is the same treatment X but patients are told it is a new treatment). If the group with treatment Y shows a statistically difference (hopefully improvement) then we say that treatment X is effective in treating the specific illness (note – the control group is needed because some people improve just because they are told they are being given a new state of the art medicine!)Note there are two key design elements here:The control group – we discussed this aboveThe pre-test – measurement of the baseline (how patients did before the treatment) is important in establishing whether significant improvement is made
- #9 Causal studies are differentiated by their ability to control and manipulate variables. Causal studies may be experiments or ex post facto studies. Experiments are studies involving the manipulation of one or more variables to determine the effect on another variable. For example, direct marketers can use split tests on mailings to test which mailing resulted in the highest response rate. Ex post facto designs are evaluations made after-the-fact based on measured variables. For example, you can create different stories about your insurance coverage and see how repair shops react differently when giving you quotes. You as a researcher play an active role in influencing and changing how the participant (i.e. repair shop owner) responds. That would make it an experiment. Because you have full control of who gets to what version of your cover-up story, you are running a real experiment.With quasi-experiments, you expect different groups of people to react/behave differently, but you have no control who is in which group. For example, if you’d like to see the impact of financial problems on marriage, ideally you’d like to run an experiment. You can cause half of your group to have financial problems (perhaps by stealing their 401K fund), and the other half not, and see which group has more marriages falling apart. Unfortunately this is not really considered an ethical thing to do and so you can’t really run this experiment, no matter how clever the design is.Your best compromise is to have a quasi-experiment. You let people decide which treatment group they end up being in – the victim groupor the control group. Then you compare their marriage quality across groups. Same thing with gender differences. You can’t assign people to the female group or the male group. Your best compromise is to go with whatever gender they are instead of mandating a sex change operation!With surveys, observations, document analysis, case studies, etc, the researchers observe and analyze the observations. They do not play an active role in influencing the study participants behaviors, reactions, or feelings. Caution: Surveys and case studies can be experimental depending on whether researcher manipulation is introduced. A clever way of conducting experiment is giving people two different versions of a questionnaire by embedding the experimental manipulation in the survey.