Experimental Method - is basically a collection of research designs, guidelines for using them, principles and procedures for determining statistical significance, and criteria for determining the quality of a study For some researchers , the experimental method is the premier method, all others being ‘ground clearing’ operations, that is, preliminary data collection and interpretation exercises to prepare for a formal experiment.
Experiments are generally conducted in order to test the strength of relationships between variables. We also saw that when the researcher is testing the influence of one variable on another , the variable doing the influencing is called the independent variable, while the other being influenced is called the dependent variable.Example: “In a study of the effect of two different methods forteaching grammar, the teaching method would me theindependent variable, and the students’ performance on a test ofgrammar knowledge would be the dependent variable”.
QUALITY CONTROL ISSUES: THREATS TO INTERNAL VALIDITY Quality Control – in the experimental method, is largely a matter of understanding the many things that can go wrong and taking steps to prevent or minimize those threats.Threats to internal validity can be divided to threecategories: Experience Bias factors Participant Bias Instrumentation Bias
Threats to Internal Validity Based on Experience Experience Bias factors are those “based on what occurs within a research study as it progresses”.Three Experience BiasFactors - refers to events –things that happen during an experiment which may influence the results. - can also have an unintended influence on the outcomes of the treatment. Example : “If classes in the study are disrupted due to natural disasters or political unrest, the research will be affected”
- refers to the fact that taking a pre-test mayinfluence the subjects’ performance on post-test.That is in addition to learning from thetreatment, learners may do better on the post-testbecause the pre-test alerted them to what was beinginvestigated in the study. - A treatment may appear to increase learningeffectiveness as compared to that of a control orcomparison group, not because it really boostseffectiveness but because either the experimenter orthe subjects believe that it does and behave accordingto this expectation ( Tuckman, 1999)
Threats to Internal Validity Based on ParticipantsParticipant Bias is a result of the “characteristics ofthe people on whom the study is conducted”Five participant bias factors: -as a threat to Internal validity is the idea that somehow the groups to be compared turn out to be different before the treatment Safeguard: Random selection and random assignment are used to combat this problem.
-refers to the normal development people undergowhether or not they are receiving a treatment of some kind. Safeguard: This problem is addressed through the use of control group, which is at the same development level as the treatment group and goes through the same experiences for the period of time. -is the name of a tendency for people’s test scores to change whether or not the knowledge, skill or ability to being measured truly changes. Safeguard: Using subjects who represent an entire range of ability levels in your design is one way to avoid this problem.
- Is the problem of losing subjects from the study. It canbe especially worrisome if the groups end up being quitedifferent sizes because people dropped out. Safeguard: To deal with this threat, researchers often try to recruit more people for a study than they may actually need. -happens when more than one threat is present in a study. Safeguard: Careful planning is needed to avoid these sorts of problems.
Threats to Internal Validity Based on Instrumentation Instrumentation Bias has to do with “the way thedata are collected” The threat of instrumentation is also sometimes called instability of measures, because it occurs if the measurement or recording processes change during the experiment. Safeguard: Observer training, rater training, and the careful piloting of all questionnaires and data collection devices.
QUALITY CONTROL ISSUES:THREATS TO EXTERNAL VALITY
There are four issues concern about externalvalidity: - It is related to the testing threat ( or practice effect to internal validity. In this situation, the presence of a pre- test may give the effects of the treatment boost. Then, when the outcomes of the experiment are transferred to the real world where no pre-test is involved, that extra boost will be lacking. -This threat is very interesting problem. This idea refers to the fact that sometimes just knowing that one is in an experiment is enough to cause a difference that may be captured by the dependent variable , whether or not one is in the treatment group.
-occurs when the sample in an experiment is notreally representative of the population from which it wasdrawn. -This term means that before we select peoplefrom the population to be in the sample, we determinewhat the relevant characteristics of the population are andmake sure that the levels in the sample represents thepopulation appropriately. -This threat is hard to manage in the classroomresearch , especially in second language settings wherestudents have access to the target language outside ofclass.
In order to deal with these threats, theexperimental method includes manydifferent research designs to counteract thepossible confounding variables that couldinfluence the internal and external validityof a study. The various designs havedifferent strength and weaknesses. Anyonewho chooses to do an experiment mustbalance the focus of the research questionagainst the time resources available forconducting the study in order to choose thebest design.
Scenario 1:The One-Shot Case Study DesignScenario 2:The One-Group Pre-Test Post-Test DesignScenario3:The Intact Groups DesignScenario 4 :The nonequivalent Control(comparison)Groups design
Scenario 5:The Time Series DesignScenario 6:The Equivalent Time Samples DesignScenario 7:Post- Test Only Control GroupDesignScenario 8:Pre-test Post Test ControlDesign