Innovations in assessment: an investigation into the role of blended learning...
Kopcha Sullivan 2008 Reaction
1. Learner Preferences and Prior Knowledge in Learner-Controlled Computer-Based Instruction Reaction of Kopcha & Sullivan (2008) Educational Technology Research and Development Nicola Ritter Texas A&M University This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
4. Questions 1. Do learners achieve better in a full version of a computer-based instructional program in which they have no control over the amount of instruction (program control) or in a full version in which they have control over the number of examples and amount of practice they receive (learner control)? 2. Do learners achieve better in a computer-based program when their preference for amount of learner control is matched with the amount they receive? 3. Are there any statistical interactions between learners’ prior knowledge on the instructional task, learner preference for control, and type of control (learner or program) that affect learner achievement? 4. Do amount of preference for control and prior knowledge influence learners’ choices of amount of instruction in a computer-based instructional program?
8. Results Question #1: Achievement higher in learner-controlled or program-controlled environments Findings: No statistically significant (p = .39) (η²= .01)difference between students using a learner controlled environment and program controlled environment
9. Results Question #2: Achievement higher when preference for control is matched Findings: No statistically significant (p = .48) (η² = .01) difference between matched preference for control and unmatched preference for control
10. Results Question #3: Are there any interactions that affect learner achievement? Findings: A statistically significant (p < .05) (η² = .07) interaction: prior knowledge x control preference x control type High prior knowledge learners scored higher when their control preference matched Low prior knowledge learners scored lower when their control preference matched
11. Results Question #4: Do amount of preference for control and prior knowledge influence learners’ choices of amount of instruction in a computer-based instructional program? Findings: There were statistically significant relationships between control preference and both the number of options used and time in the program. While prior knowledge influenced learner attitudes about the instructional program.
12. Strengths Utilized mixed methods Minimized novelty effects All assumptions of the analyses conducted were met and discussed. Scores were reliable and reported appropriately.
13. Weaknesses Generalizability is poor. No validity information on the scores. Limitations were not expressed Internal validity Pretest sensitization due to content External validity Selection-treatment interaction Ecological validity Non-random sample
14. References Kopcha, T. J., & Sullivan, H. (2008). Learner preferences and prior knowledge in learner-controlled computer-based instruction. Educational Technology Research & Development, 56(3), 265-286. doi:10.1007/s11423-007-9058-1
Editor's Notes
Kopcha and Sullivan (2008) investigated the effects of learner control and program control in computer-based instruction (CBI) on mathematics achievement of students with different amounts of prior knowledge and different preferences for control.
This study builds on prior research in the areas of learner controlled environments, learner preferences, and prior knowledge on learner achievement. The authors investigated the effects of learner controlled environments on learner achievement and attitudes. The authors presented numerous studies (Corbalan et. al 2006; Van Gog, et. al 2005) that suggest learner control environments improve learner achievement and attitudes. The authors also counteracted with findings (Farrel and Moore 2000-2001; Morrison et. al. 1992) that learner control does not necessarily lead to improved achievement. With this discrepancy, the authors investigate whether or not learners achieve more in learner controlled environments. The authors suggested that learner control preferences may also effect learner achievement. The authors present several studies that support (Van Merrienboer et. al 2002) and oppose (Schnackenberg et. al 1998) this theory. The authors present previous studies that demonstrate the effect prior knowledge has on learner achievement in learner-controlled environments.
This article relates to the course because the authors consider prior knowledge and investigate the impact of cognitive load via learner control on learner achievement. Cognitive theorists acknowledge the influence prior knowledge has on long-term memory storage (Ormrod, 2008). The authors consider prior knowledge as one factor that may determine learner achievement in CBI. Cognitive load theory is also relevant to the study. Cognitive theorists believe giving learners control over the instruction allows the learner to manage the cognitive demand of the activity (Kopcha & Sullivan, 2008). The authors reference other studies (Bannert 2002; Salden et al. 2006; van Gog et al. 2005) that attribute learner control to a reduction in cognitive load. The findings from this study suggest learners with higher prior knowledge have a better understanding of their preferences that helps them achieve more. This study also provides evidence that assigning instruction that matches high prior knowledge learner’s control preferences is an effective strategy to reduce the cognitive load of the learners in computer-based instruction. The authors did not make any predictions because they felt that the findings from previous research in prior knowledge and control preferences were not strong enough to support predictions about the effects of these factors (Kopcha & Sullivan, 2008). The motivation for this study is more applied than basic. The authors recommend designing learner-controlled environments that offer more control as the learners’ level of knowledge increases. This study demonstrates how such environments have the potential to increase learner achievement and attitudes about the received instruction.
The authors proposed four research questions: 1. Do learners achieve better in a full version of a CBI program in which they have no control over the amount of instruction (program control) or in a full version in which they have control over the number of examples and amount of practice they receive (learner control)?2. Do learners achieve better in a computer-based program when their preference for amount of learner control is matched with the amount they receive?3. Are there any statistical interactions between learners’ prior knowledge on the instructional task, learner preference for control, and type of control (learner or program) that affect learner achievement?4. Do amount of preference for control and prior knowledge influence learners’ choices of amount of instruction in a computer-based instructional program? (Kopcha & Sullivan, 2008, p. 268).
There were 99 sixth and seventh grade students from a middle school in the northeastern United States that participated. Of the 99 students, 44 were male and 53 were female. The authors provide no demographics of the sample but did provide demographics of the school population of the sample. The school population (n = 700) is approximately 70% Caucasian, 18% Hispanic, 9% African American, and 3% Asian and a socio-economic status of middle to upper-middle class. The authors did not specify the techniques used in selecting samples. However, it is likely the sample used in the quantitative analysis was selected by convenience or purposive sampling and in the qualitative analysis was selected by criterion sampling.
The independent variables in this study were level of prior knowledge categorized as high or low, preference for controls, and type of control received both classified as learner-controlled or program-controlled. The dependent variables in this study were math achievement, operationally defined as the observed scores on the researcher-developed posttest, time in program, and options used. The authors explain the coding used for each instrument. The Learner Preference Scale and the attitude questionnaire, developed by the authors, used a five-point Likert-type scale coding strongly agree = 4 to strongly disagree = 0. The pretest and posttest which were also developed by the authors included 12 items and 24 items, respectively, covering the content that was going to be taught using the CBI program. The items were coded as correct or incorrect. The CBI program included an electronic data-tracking system that tracked the amount of time each participant spent in the program and the options used under the learner-controlled version. The interviews with participants that used the learner-controlled version consisted of 6 questions asking about the choices they made during the CBI program.
The Learner Preference Scale and the pretest were administered to the participants three weeks prior to interaction with the CBI program. Four groups were randomly formed based on identified prior knowledge, learner’s control preferences, and assigned type of CBI program. “The study took place over three 45-min class periods, one per day on consecutive days” (Kopcha and Sullivan, 2008). Participants completed two modules during the first two class periods and were administered a post-test and attitude questionnaire during the third class period. The researchers interviewed 25 volunteers who used the learner-controlled program. Volunteers were selected by alternating between students with a high and low preference for control.
1. Do learners achieve better in a full version of a computer-based instructional program in which they have no control over the amount of instruction (program control) or in a full version in which they have control over the number of examples and amount of practice they receive (learner control)?Findings:Results showed there was no statistically significant (p = .39) (η² = .01) differences between students using a learner controlled environment and program controlled environment in computer-based instruction.
2. Do learners achieve better in a computer-based program when their preference for amount of learner control is matched with the amount they receive?In addition, there was not a statistically significant (p = .48) (η² = .01) difference between matched preferences of control and unmatched preferences of control.
3. Are there any statistical interactions between learners’ prior knowledge on the instructional task, learner preference for control, and type of control (learner or program) that affect learner achievement?The general linear model (GLM) showed a statistically significant (p < .05) (η² = .07) three way interaction between prior knowledge, control preference, and type of control. Students with high prior knowledge scored higher when their control preference matched the type of control they received. While students with low prior knowledge scored higher when their control preference did not match the type of control received.
4. Do amount of preference for control and prior knowledge influence learners’ choices of amount of instruction in a computer-based instructional program? There were statistically significant relationships between control preference and both the number of options used and time in the program. The researchers also looked at the relationship between time in the program and control preferences while using prior knowledge as a covariate. The GLM analysis indicated a statistically significant (p < .01) (η² = .13) difference between time in the program and the student’s preference for control. The high control preference students spent statistically significant (p < .05) greater amount of time per example and practice screen than low control preference students. However, there was no statistically significant difference between the two groups on the amount of time spent on the information screens. Lastly, the authors found a statistically significant control preference x prior knowledge interaction effect on options viewed. These results suggest that the Learner Preference Scale developed by the researchers did differentiate between low and high preferences for control in the amount of option use and time in the program. Participants with high control preferences selected more optional elements under the learner-controlled program and spent more time in the CBI program. While prior knowledge influenced the interaction: attitudes x control preference x control type. At a low level of prior knowledge, high preference-for-control students had higher agreement across items on an attitude scale when their preference for control was mismatched than when matched. High-prior-knowledge students with a high preference for control had higher agreement when they control preference was matched than when mismatched. (Authors suggested the reasoning behind this may be due the difference in metacognitive skills between the two groups. High prior knowledge learners tend to have great metacognitive skills that allow them to know what works best for them.
The school was selected because the students were familiar with using computers in a school setting. The instructional materials were aligned to the regular classroom mathematics curriculum. Both conditions would alleviate novelty or disruptioneffects that can influence ecological validity and is often a concern in educational technology studies. All assumptions of the analyses conducted were met and discussed. (e.g. Assumptions met to conduct a GLM, while assumptions were not met to run an ANCOVA) Reliability of the scores used in this study had acceptable internal consistency. The Learner Preference Scale developed for the study had a Cronbach’s alpha equal to .82. The pretest and posttest had an acceptable alpha of .59 and .76, respectively. Additionally, the attitude questionnaire reported an acceptable alpha coefficient of .88.
Generalizability is poor. (e.g. small sample size, method of sample selection, and replication not tested)No validity informationLimitations were not expressed: threats to internal and external validityReliability and validity are valuable components when considering the results of a study. The authors report reliability coefficients for all scores. The reliability coefficients reported are appropriate for the types of scores produced by the instruments. Although the authors report reliability coefficients there is no mention of the validity of the scores. In order to have validity the scores must be reliable, but just because the scores are reliable does not imply that the scores are valid. Validity consists of multiple components: content validity, criterion-related validity, and construct validity. (Gay, Mills, & Airasian, 2006). Content validity determines the degree in which an instrument measures an intended content domain. This is often determined by experts in the content domain. The authors refer to the expertise of the CBI program’s instructional designer and two other expert reviewers to confirm the content validity of the instructional program. While the authors reference the primary investigator as a source to confirm content validity on the instruments used. The authors should have another expert review the instruments similar to what the authors did in the review of the instructional material. This would build a stronger case for content validity on these instruments. Criterion-related validity relates scores on one instruments to scores on another instrument. The authors did not provide any evidence to have criterion-related validity support on the instruments the authors developed. Construct validity estimates the degree to which scores from an instrument measure the intend construct. There is also no evidence that the instruments demonstrate construct validity. The authors mentioned that one of the reasons the school was selected was due to the content in the instructional program which was part of the regular mathematics curriculum. The researchers could have compared the instruments in this study with assignments or tests from the school’s regular curriculum that addressed the same intended constructs as the instructional program. In addition to score validity, the authors did not address plausible threats to experimental validity. An experiment is internally and externally valid if and only if the results are due solely to the manipulated independent variable and the results are generalizable. Random selection of participants, a researcher’s assignment of participants, and control of other variables are powerful strategies to guard against such threats. Although the authors did not randomly select participants, the authors did utilized the later two strategies. The authors checked for differences between participants before the study began. This is good practice to rule out possible differential selection of participants which can threaten internal validity. One plausible threat to internal validity in this study that was not addressed was the possibility of pretest sensitization. Pretest sensitization occurs when improved posttest outcomes are due to exposure to a pretest. Pretest sensitization most likely occurs when the time between the pretest and posttest are short and where facts are recalled (Gay, Mills, & Airasian, 2006). In this study the time between the pretest and posttest was three weeks, three days. This timeframe is long enough given the length of the experiment. Although the time between testings is appropriate, the content in the pretest may lead to pretest sensitization. The pretest content assesses the recall of adding and subtracting integers. The heavy reliance on the recall of addition and subtraction of integers in the pretest may impact the performance observed from the posttest and thus can threaten the internal validity of this study. Another concern is the presence of threats to external validity of this study. Selection-treatment interaction is a plausible threat. The method used to select the sample could hinder external validity. The sample was not randomly selected, which makes this study susceptible to extraneous variable effects. The authors state that the sample was selected based on the sample’s familiarity with CBI and the instructional materials covered in the program corresponded to the regular curriculum of the school. Although the sample was not randomly selected, the authors guarded against threats to validity by randomly selecting participants into groups. Replication, Generalizability, and InterpretationsThe information provided about the study provides sufficient information to replicate the study. Unfortunately, the authors did not conduct any replicablility analyses. The generalizability of this study is questionable due to the non-randomized sample and small sample size. When non-probability samples are used, it is difficult to describe the populations from which the sample was drawn thus making it difficult to generalize results. Since the authors did not specify the type of sampling used it is difficult to form generalizations. If the sample was of convenience, no generalizations can be formed. However, if the sample was purposive, generalizations may be formed for 6th and 7th graders that are familiar with CBI. Since there is no demographic information about this sample, there is still a problem making generalizations to particular ethnic groups. Even if the authors included demographic information about the sample, the sample size is still a barrier in forming generalizations. Although the authors determined there were no differences between the groups formed using an ANOVA, the sample size may be too small to detect a difference between groups.
