OCWC Global Conference 2013: Assessing the Educational Value of OER
Assessing the Educational Value of OERThe Open Education Grouphttp://openedgroup.org/• T.J. Bliss• Lane Fischer• John Hilton III• T. Jared Robinson• David Wiley
Recent Studies• Scottsdale Community College• Houston Community College• Virginia State University• Kaleidoscope Open Course Initiative• Mercy College• Nebo School District
Houston Community College(paper available)• Houston Community College(HCC) is a large communitycollege with more than70,000 students. 33% of itsstudents are Hispanic, 33%are African American, 17%are white, 14% Asian, and 3%are classified as ‘other.’ Fifty-nine per cent of the studentsare female, and 41% aremale.
Textbook Adoption• In 2011 HCC’sPsychologydepartment chose toadopt Flat WorldKnowledge’s (FWK)textbook Introductionto Psychology.• During a pilot study 690students used the FWKtext.• So what happened?
HCC – Student Feedback• 84% of students surveyed agreed with thestatement that “Having a free online book helpsme go to college.”• Of the 108 students who responded to aquestion regarding the difficulty of using anonline text, 45 (42%) said that it was easy, 28(26%) said that it was moderately easy, 26 (24%)said that it was neither hard nor easy, and nine(8%) said that it was moderately difficult.
HCC – Results Comparing Students from twoinstructors who taught in Spring 2011 and Fall2011
Virginia State University(paper available)• Virginia State University(VSU) has 5,300 students.They offer 55 baccalaureateand masters degreeprograms, two doctoraldegree programs and threecertificates.• VSU’s Business Departmentadopted nine FWK texts andpaid $30.00 per student foran unlimited seat license forall FWK content.
VSU – Results• 95% of students accessed at least some of FWKresources (in contrast with 47% of studentspurchasing textbooks one year previously).• Students downloaded a variety of resources
VSU – ResultsIn total there were seven courses that used FWK textbooks (COBU101, COBU200, COBU201, COBU210,COBU300, COBU301, COBU302, COBU310, MISY350), and ten courses that did not use FWK textbooks(COBU110, COBU111, COBU155, COBU170, COBU202, COBU260, COBU304, COBU342, COBU343, COBU400).While these were clearly different courses, they were roughly equivalent in terms of course difficulty.ABC_ inCoreCoursesusingFWKDFW_incorecoursesusingFWKTotal DFW %withFWKABC_incorecoursesw/oFWKDFW_incorecoursesw/oFWKTotal DFW %notusingFWKFall2010611 151 762 24.7%830 270 1100 32.5%Spring2011531 100 631 18.8%834 242 1076 29.0%
Kaleidoscope Open Course Initiative(paper available)Kaleidoscope Open CourseInitiative(http://www.project-kaleidoscope.org/) is an openeducation initiative initiallycomprised of eightcommunity colleges and openaccess four-year institutions.These colleges work togetherto create courses that replacetraditional, textbooks withOER.
Kaleidoscope Open Course InitiativeDuring the 2011-2012academicyear, Kaleidoscope OpenCourse Initiative impactedabout 4,000 studentsacross 80 teachers in eightinstitutions.
Kaleidoscope Open Course Initiative• 58 teachers from 8PK institutionscompleted at leastsome items on aquestionnaire.• 490 studentscompleted at leastsome items on aquestionnaire.
Total number of students: 14,606Kscope 3,876Non-kscope 10,739Quantifying Textbook Costs
• Average book cost per class section:$92.90• Total potentially spent by non-kaleidoscope students: $997,653.10• Total potentially saved bykaleidoscope students: $360,080.40• Total potential savings:$1,357,733.50
Kaleidoscope Open Course Initiative – TeacherPerceptions
Teachers: How do the OER textbookscompare with other textbooks youhave used?0102030405060Worse The Same BetterPercentofteachersTeacher perceptions of quality
Students: How do the OER textbookscompare with other textbooks youhave used?0102030405060Worse The Same BetterStudent perceptions of text qualityPercentofstudents
Kaleidoscope Open Course Initiative– Student Perceptions• 160 students provided a description of whatthey thought made the OER texts better.Their responses clustered in six majorcategories: technical advantages(8%), learning aides (9%), customization(10%), cost (20%), access (26%), andquality/presentation (27%).
Kaleidoscope Open Course Initiative– Student Perceptions• Conversely, 20 students provided adescription of what they thought made theOER texts worse than other texts they hadused in the past. Their responses clusteredin 2 major categories; six students describedtechnology issues and 15 students tookissue with text quality.
05101520253035404550Subjects students commented onNumberofstudentcomments
Mercy College MathAbout Mercy: Mercy College isdedicated to making a high-qualityeducation attainable offering morethan 90 undergraduate andgraduate programs within its fiveschools: Business, Education, Healthand Natural Sciences, LiberalArts, and Social and BehavioralSciences. The New Yorkmetropolitan area college of nearly10,000 students has four campuslocations including: DobbsFerry, Bronx, Manhattan, andYorktown Heights and offers morethan 25 degree programs online.
No OER All OERMath116 Fall 2011 Fall 2012TOTAL STUDENTS 697 695Pass (A-C) 443 479Not Pass (D/F/FW) 254 216% Pass 63.60% 68.90%% Not Pass 36.40% 31.10%Mercy College Math
Nebo School DistrictThe Nebo School District isa public school districtlocated about one hoursouth of Salt Lake City. Itserves 40 schools and30,500 students.
Nebo School DistrictDoes the adoption of openscience textbooks significantlyaffect science learning outcomesfor secondary students?
• 4,137 students took thestate Criterion ReferencedTests (CRTs) in earthsystems, biology, orchemistry. (Excludes APsections, specialeducation sections, andbio-agriculture sections)• 2,350 students usedtraditional textbooks,while 1,787 used OERtextbooks.
Method• Quasi-experimental design with– Treatment and control groups– Groups matched by propensity score• Dependent variable: 2012 science CRT score• Independent variable: Textbook condition• Covariates: age, gender, special education,English language proficiency, 2011 test data,2011 GPA, race, and free and reduced lunch
Matching• Propensity Score Matching• Based on the conditional probability ofassignment to treatment condition, given thecovariates• Increased group balance by 98%• Approximates random assignment
Summary• Houston Community College – Exploring avariety of other OER.• Virginia State University – Continues to useFWK texts.• Kaleidoscope Open Course Initiative – 28schools• Mercy Math – Expanding work with OER.• Nebo School District – Utah has approved thetextbooks for statewide use; preliminarysurveys suggest up to 75,000 students will usethe open textbooks.