A COMPARISON OF STUDENT ACHIEVEMENT AND RETENTION IN AN INTRODUCTORY MATH COURSE DELIVERED IN ONLINE, FACE-TO-FACE AND BLENDED MODALITIES by Russell R. Paden, Ph.D. & Mary I. Dereshiwsky, Ph.D.

Presented to the Teaching, Colleges and Community (TCC) 2007 Worldwide Conference: April 17 -19, 2007

Background of the Study Explosive growth of online learning (2 million students in 2003, increasing to 2.6 million in 2004, according to Allen & Seaman, 2004) Same source reports annual growth rates of 20 - 25% Blended instructional models (combination of online and face-to-face) are also growing in popularity Little data exist to support conclusions regarding impact on retention of online instruction, in whole or in part (blended) Certain subject matter such as mathematics is challenging to teach entirely online More studies are needed regarding impact of instructional modality on student achievement in math instruction

Explosive growth of online learning (2 million students in 2003, increasing to 2.6 million in 2004, according to Allen & Seaman, 2004)

Same source reports annual growth rates of 20 - 25%

Blended instructional models (combination of online and face-to-face) are also growing in popularity

Little data exist to support conclusions regarding impact on retention of online instruction, in whole or in part (blended)

Certain subject matter such as mathematics is challenging to teach entirely online

More studies are needed regarding impact of instructional modality on student achievement in math instruction

Research Questions Research Question #1 : Is there a difference in student retention for an introductory math course based on whether it is taught in an online, blended, or traditional format?’ Research Question #2 : Does gender make a difference in student retention for an introductory math course based on whether it is taught in an online, blended, or traditional format?

Research Question #1 : Is there a difference in student retention for an introductory math course based on whether it is taught in an online, blended, or traditional format?’

Research Question #2 : Does gender make a difference in student retention for an introductory math course based on whether it is taught in an online, blended, or traditional format?

Research Questions, con’t. Research Question #3 : Is there a difference in student achievement, as measured by course final grades, in an introductory math course based on whether it is taught in an online, blended or traditional format? Research Question #4: Does gender make a difference in student achievement, as measured by course final grades, in an introductory math course based on whether it is taught in an online, blended or traditional format?

Research Question #3 : Is there a difference in student achievement, as measured by course final grades, in an introductory math course based on whether it is taught in an online, blended or traditional format?

Research Question #4: Does gender make a difference in student achievement, as measured by course final grades, in an introductory math course based on whether it is taught in an online, blended or traditional format?

Setting and Context A large, regionally accredited private university (“Private U”) Undergraduate mathematics course titled Math 200 Offered in three modalities: online, blended and traditional face-to-face format Standardized curriculum across all 3 modalities Three-year time frame of study: 2004 through 2004 (no significant changes to curriculum, instructional models or instructor training during this time)

A large, regionally accredited private university (“Private U”)

Undergraduate mathematics course titled Math 200

Offered in three modalities: online, blended and traditional face-to-face format

Standardized curriculum across all 3 modalities

Three-year time frame of study: 2004 through 2004 (no significant changes to curriculum, instructional models or instructor training during this time)

Population and Sample A total of 79,545 students attempted to take Math 200 at Private U from January 2002 through December 2004. Of these students: -- 38,056 attempted the course in an online format, --1,310 attempted the course in a blended format, --40,179 attempted the course in a traditional or face-to-face format

A total of 79,545 students attempted to take Math 200 at Private U from January 2002 through December 2004. Of these students:

-- 38,056 attempted the course in an online format,

--1,310 attempted the course in a blended format,

--40,179 attempted the course in a traditional or face-to-face format

Measures of Variables for the Study All necessary data were provided electronically by the registrar’s office in Excel spreadsheet documents The first document included assigned student identification numbers with an associated grade for the course in the form of a numeric quality score (e.g., a grade of “A” equals 4.00, “A-” equals 3.66, “B+” equals 3.33, etc.). The second document included student identification numbers with an associated field indicating whether the student completed or dropped the course . Based on this information, the student was identified as retained or not retained . Both documents included each student’s associated with the student’s identification number. gender

All necessary data were provided electronically by the registrar’s office in Excel spreadsheet documents

The first document included assigned student identification numbers with an associated grade for the course in the form of a numeric quality score (e.g., a grade of “A” equals 4.00, “A-” equals 3.66, “B+” equals 3.33, etc.).

The second document included student identification numbers with an associated field indicating whether the student completed or dropped the course . Based on this information, the student was identified as retained or not retained .

Both documents included each student’s associated with the student’s identification number. gender

Findings for Research Question #1: Are there differences in retention by instructional modality of delivery? Students in traditional courses had the highest retention rate at 91.9% Students in online courses had the lowest retention rate at 85.6% Students in the blended courses fell in the middle , with a retention rate of 88.2% These differences in retention rate are statistically significant ( χ 2 (2, N = 79, 545) = 796.63, p < .05)

Students in traditional courses had the highest retention rate at 91.9%

Students in online courses had the lowest retention rate at 85.6%

Students in the blended courses fell in the middle , with a retention rate of 88.2%

These differences in retention rate are statistically significant ( χ 2 (2, N = 79, 545) = 796.63, p < .05)

Findings for Research Question #2: Does gender make a significant difference in student retention in an introductory math course depending on whether it is taught in an online, blended, or traditional format? There was a slight difference in the percentages of retained students by gender across all modalities pooled together (89.1% of male students were retained, and 88.7% female students were retained) This difference was not statistically significant χ 2 (2, N = 79, 545) = 2.47, < critical value of chi square of 3.841 For only the online instruction there was a small difference in the percentage of retained students by gender (86.0% of males and 85.3% of females were retained)

There was a slight difference in the percentages of retained students by gender across all modalities pooled together (89.1% of male students were retained, and 88.7% female students were retained)

This difference was not statistically significant χ 2 (2, N = 79, 545) = 2.47, < critical value of chi square of 3.841

For only the online instruction there was a small difference in the percentage of retained students by gender (86.0% of males and 85.3% of females were retained)

Findings for Research Question #2, con’t.: Does gender make a significant difference in student retention in an introductory math course depending on whether it is taught in an online, blended, or traditional format? The difference in retention by gender in the online format was marginally significant χ 2 (1, N = 38,056) = 3.846, p < .05 For only the blended instruction , 86.0% of males and 89.8% of females were retained The difference in retention by gender in the blended format was statistically significant χ 2 (1, N = 1310) = 4.46, p < .05 For only the traditional format, 92.2% of males and 91.8% of females were retained This difference was not statistically significant χ 2 (2, N = 40,179) = 2.47, < critical value of chi square of 3.841

The difference in retention by gender in the online format was marginally significant χ 2 (1, N = 38,056) = 3.846, p < .05

For only the blended instruction , 86.0% of males and 89.8% of females were retained

The difference in retention by gender in the blended format was statistically significant χ 2 (1, N = 1310) = 4.46, p < .05

For only the traditional format, 92.2% of males and 91.8% of females were retained

This difference was not statistically significant χ 2 (2, N = 40,179) = 2.47, < critical value of chi square of 3.841

Findings for Research Question #3.: Are there differences in student achievement (measured by course final grade) by instructional modality of delivery? Traditional instruction had the highest average grade (3.028) Online instruction had the lowest average grade (2.967) Once again, blended instruction fell in the middle (3.017) Analysis of variance (ANOVA) revealed that this difference in average grade by instructional modality was statistically significant (F=30.321; p < .0001) Follow-up testing revealed that: --there were no statistically significant differences between online and blended courses or blended and traditional courses ; but: --there was a statistically significant difference in academic achievement when online and traditional instruction were compared

Traditional instruction had the highest average grade (3.028)

Online instruction had the lowest average grade (2.967)

Once again, blended instruction fell in the middle (3.017)

Analysis of variance (ANOVA) revealed that this difference in average grade by instructional modality was statistically significant (F=30.321; p < .0001)

Follow-up testing revealed that:

--there were no statistically significant differences between online and blended courses or blended and traditional courses ; but:

--there was a statistically significant difference in academic achievement when online and traditional instruction were compared

Findings for Research Question #4: Does gender make a significant difference in student achievement (measured by course final grade) depending on whether it is taught in an online, blended, or traditional format? There was a slight difference in the mean grade by gender across all modalities pooled together (mean grade of 3.022 for men and 2.951 for women) This difference was statistically significant ( F [1, 79,543] = 17.70, p < .0001) For only the online instruction there was also a slight difference in the mean grade by gender (mean grade of 3.048 for men and 2.915 for women)

There was a slight difference in the mean grade by gender across all modalities pooled together (mean grade of 3.022 for men and 2.951 for women)

This difference was statistically significant ( F [1, 79,543] = 17.70, p < .0001)

For only the online instruction there was also a slight difference in the mean grade by gender (mean grade of 3.048 for men and 2.915 for women)

Findings for Research Question #4, con’t.: Does gender make a significant difference in student achievement (measured by course final grade) depending on whether it is taught in an online, blended, or traditional format? For only the blended instruction , there was also only a slight difference in average grade by gender (males: 3.022; females: 3.01) For only the traditional instruction , there was also only a slight difference in average grade by gender (males: 3.108; females; 2.980) Analysis of variance revealed that the interaction of gender and modality of instruction (only online, blended, and traditional one at a time instead of pooled) was not statistically significant (F=1.972; p = 0.1392)

For only the blended instruction , there was also only a slight difference in average grade by gender (males: 3.022; females: 3.01)

For only the traditional instruction , there was also only a slight difference in average grade by gender (males: 3.108; females; 2.980)

Analysis of variance revealed that the interaction of gender and modality of instruction (only online, blended, and traditional one at a time instead of pooled) was not statistically significant (F=1.972; p = 0.1392)

Discussion There appears to be a strong relationship between instructional format and student retention Traditional instruction produces higher retention rates than either online or blended formats A greater percentage of men retained in the online and traditional models as compared to women in the same models Somewhat surprisingly, women retained at a significantly higher percentage rate than men only in the blended model No clear explanation (other than much smaller sample size) for why a blending of those models produced a superior retention rate among women The current study is more in line with York’s 2003 finding that women retain better in face-to-face courses than in online courses

There appears to be a strong relationship between instructional format and student retention

Traditional instruction produces higher retention rates than either online or blended formats

A greater percentage of men retained in the online and traditional models as compared to women in the same models

Somewhat surprisingly, women retained at a significantly higher percentage rate than men only in the blended model

No clear explanation (other than much smaller sample size) for why a blending of those models produced a superior retention rate among women

The current study is more in line with York’s 2003 finding that women retain better in face-to-face courses than in online courses

Discussion, con’t. With respect to final grades, the academic achievement of students in the traditional classroom was superior to that of the students in the online format This finding contradicts results of other studies that online instruction is at least equal and possibly superior to traditional instruction Also contradicts studies about courses in fields outside of mathematics that found no significant difference between scores in online and traditional courses There may be some subject areas , such as mathematics, in which students will not perform as well in an online environment Adding a face-to-face component to an online mathematics course may mitigate any potential shortcomings in academic achievement

With respect to final grades, the academic achievement of students in the traditional classroom was superior to that of the students in the online format

This finding contradicts results of other studies that online instruction is at least equal and possibly superior to traditional instruction

Also contradicts studies about courses in fields outside of mathematics that found no significant difference between scores in online and traditional courses

There may be some subject areas , such as mathematics, in which students will not perform as well in an online environment

Adding a face-to-face component to an online mathematics course may mitigate any potential shortcomings in academic achievement

Discussion, con’t. Gender did make a difference in student achievement in Math 200 but not in conjunction with instructional format ( only across all modalities pooled) Interestingly, although the blended modality produced the highest academic achievement for women , it produced the lowest academic achievement for men Blended instruction was the only modality in which the retention of women exceeded that of men

Gender did make a difference in student achievement in Math 200 but not in conjunction with instructional format ( only across all modalities pooled)

Interestingly, although the blended modality produced the highest academic achievement for women , it produced the lowest academic achievement for men

Blended instruction was the only modality in which the retention of women exceeded that of men

Critical Reflections Instructional modality played a significant role in both retention and achievement (most pronounced for the blended format which is a combination of online and traditional) Gender had no significant role in the interactive effect between achievement and instructional modality This study stands in contrast to the advocates of the “no significant difference” phenomenon The subject matter of a course may play a more significant role in the success of students in alternative instructional formats than some are willing to acknowledge Some content areas may be more challenging for some students, and require better academic support for both students and faculty , to achieve equivalent academic outcomes in an online format

Instructional modality played a significant role in both retention and achievement (most pronounced for the blended format which is a combination of online and traditional)

Gender had no significant role in the interactive effect between achievement and instructional modality

This study stands in contrast to the advocates of the “no significant difference” phenomenon

The subject matter of a course may play a more significant role in the success of students in alternative instructional formats than some are willing to acknowledge

Some content areas may be more challenging for some students, and require better academic support for both students and faculty , to achieve equivalent academic outcomes in an online format