Restricting access to the “best” students can increase quality.Increasing cost can increase quality.How can we have high quality while allowing open access and containing costs? Answer: scalability and decoupling
Assessment quality is not just about decoupled assessments. These aspects of quality apply in bricks-and-mortar and online courses too: anything that is used for academic credit needs to have good assessments. Note, though, that over the course of a semester, there are usually multiple assessments, so the individual assessments need not all be extremely rigorous: the aggregation of multiple assessments can lend reliability in cases in which a single assessment may not be highly reliable. However, in decoupled assessments, there is usually an “all your eggs in one basket” approach, so the basket needs to be very, very good.
The big advantage courses have over decoupled assessments is that the first issue is usually not a problem!Formative assessments (ones given throughout a course) need not worry about depth—aggregated, though, they should cover everything.
These two characteristics are the biggest pitfalls for the lower-stakes assessments usually seen as part of courses. Instructors often have no training in assessment.
Appropriate here is on two levels. It’s important for the assessment to match the content of what examinees are expecting to learn. But it’s also important for the objectives to be valuable to credit-granting bodies. If I build a beautiful assessment for an OER course on water management that relates nicely to the course objectives, that assessment will still not be appropriate for someone seeking credit specifically for a degree in literature. Validity in this area often balances with validity in scalability: it’s often hard to do a great job on appropriateness (good matching) while at the same time providing a truly decoupled assessment
Our typical student brings 5 transcripts to the table and may start a BA/BS program with about 60 credits completed. The Academic Advisor is the generic job at EC, and they specialize in helping students identify what they need.
We began developing our current models to serve 1) an international agency and 2) the challenge of providing a $10K degree. But we don’t develop “exclusives.”
An assessment based_degree
AN ASSESSMENT-BASED DEGREE November 1, 2012 WCET Annual Conference
Scalability of Content3 Open Educational Resources: accessible to millions Independent Study Requires that learners be self-motivated
Types of OER Lecture notes Physics lecture Videos of classroom lectures Lessons designed for OER CoursesOpenStudy Discussion groups Khan Academy Open University
What is OER’s value? Learning CertificatesBut… OER by itself does not typically award formal educational credit Why not?
Academic credit6 What is credit? Assurance that someone knows something The something must be appropriate for the particular academic program To provide that assurance, both the someone and the something must be verified
Decoupling assessment from the7 classroom “What you know is more important than where or how you learned it.” Credit should be based on knowledge, not attendance Knowledge gained through independent study can be independently assessed
Aspects of validity for credit by assessment8 Identity verification Assessment quality Appropriateness of knowledge tested for a particular degree program Scalability
Identity verification9 Ryan Ruppe Jeffery Turner Steve Winton
Threats to validity--courses10 Did the person actually go through the course? Did the person do his/her own work? Is the person who took the course the same person who is presenting the credential?
Threats to validity--11 assessments Is the person taking the assessment the same person who is claiming the knowledge? Is the person claiming the knowledge the same person who is presenting the credential?
Assessment quality12 A good assessment Measures knowledge of the subject Does not measure irrelevant characteristics Gives a person the same score regardless of which form is taken Gives people of the same ability the same score
Threats to assessment quality13 Assessments may not measure quite the same content as the material studied Not all things in the world labeled “Sociology” cover the same topics! Assessments may not cover material in enough depth A 10-question quiz is unlikely to cover the equivalent of a college semester’s worth of content
Threats to assessment quality14 The assessment might not be scored consistently enough Different instructors have different standards Assessments might measure irrelevant characteristics Unnecessarily complicated questions Questions with “giveaway” answers Trick questions
Appropriateness of content15 Match of assessment to OER How close are the assessment specifications to the learning objectives of the OER material? Match of assessment to credit-granting body How close are the assessment specifications/learning objectives to what is taught at the institution where credit is sought?
Efficient assessment Machine-scored competency/proficiency exams can handle large numbers of examinees and be used for multiple OER sources Machine-scored exams need not measure simply mindless regurgitation of facts For some types of OER, more specific assessments may be needed Portfolios for highly specialized content Cost of a 3-credit multiple-choice assessment:
Scalability - Generalizability Lessgeneralizable Course final exams and homework Third-party assessments designed for a specific course Competency/proficiency assessments Moregeneralizable
Scalability – Large groupsLess scalable Individual assessments Portfolios Research papers Oral examinations Human-scored group assessments Short answer questions Essay questions Machine-scored assessments Multiple-choice exams Machine-scored constructed responseMore scalable exams
What is Charter Oak State College? Founded in 1973 in Connecticut Enrollment about 2,000 Over 11,000 degrees awarded, with alumni from all 50 states and a dozen countries Partnerships with military branches Associate and Baccalaureate degrees New England Association of Schools and Colleges accreditation
What is Charter Oak State College? Aggregation model - with Cornerstone and Capstone Outcome assessment courses Writing, research, program planning Transfer credit CBE, portfolio assessment, ACE and National CCRS, CCAP, institutionally reviewed training and certifications
Student Profile CONCENTRATION PLAN OF STUDY (CPS) FORM Business Administration MINIMUM Number and Level of MINIMUM SUBJECT REQUIRED SOURCE: TEST OR INSTITUTION COURSE/TEST NO. & TITLE GRADE Credits Propose REQUIREMENTS CREDITS Lower UpperAccounting: 3 credits CLEP Financial Accounting Prop (3) Financial * Managerial * 3 credits StraighterLine Managerial Acct. B 3Finance: 3 credits DSST Prin. Finance Prop (3) Principles * or Managerial * or CorporateManagement: 3 credits Excelsior Coll. Organizational Behav Prop 3 Organizational BehaviorManagement: 3 credits Northwood U. MGT 230 Prin. Mgt. A 2.7 Intro/Principles Management*Marketing: 3 credits Northwood U. MKT 208 Prin. Mark. C+ 2.7 Principles of Marketing*Strategic Processes 3 credits COSC MGT 499 Strategic Prop 3Capstone Course Management
Program PlanningHi Karen,As per your request, Ive included a list below of tests/ independentstudies courses I am planning on taking for my degree.• Pending exams/courses: • Cornerstone course (IDS 101)- in progress • College Algebra - Straighterline, in progress • English Composition II - Straighterline, in progress • Mgmt Info Systems - DSST (area of concentration) • Environment & Humanity – DSST • *Business Ethics in Society - DSST (UL area of concentration) • Organizational Behavior - Excelsior Exam(UL area of concentration) • Principles of Public Speaking - DSST
Class of 2011-12 & CBE2011-12 Charter Oak StateCollege Graduates: Average of Test Credit Graduates Average of Test Count HoursNo Testing 310 64%Took 1 or more tests 176 5.23 17.64 36%Grand Total 486 5.23 17.64
What Is Excelsior College? Independent institution serving adult students and other non-traditional learners since 1971 Enrollment 34,000; alumni 141,000 Up to one-third of EC students are Members of historically under-represented groups Military or veterans Average age of graduates: late 30s Associate, Baccalaureate, and Masters degrees Middle States accreditation
What Is Excelsior College? Aggregation model EC online courses (available since 2004) Two national exam series developed in house: ECE and UExcel Transfer credit Consistently ranked #1 or #2 in acceptance of credit from other accredited institutions Other CBE, portfolio assessment, ACE and National CCRS, institutionally reviewed training and certifications Largest pre-licensure nursing program in the country, entirely assessment based
Excelsior College Exam-Based Degree Paths New degree pathways emphasize Excelsior’s own CBE products, along with a single required Capstone course Degree planning templates show suggested sequence of exams; students with transfer credit have wider choices Supported Independent Study model is based on Open Courseware options reviewed and recommended for all exams Initial degree templates for BS and BA in Liberal Studies and AS and BS in General Business
Infrastructure Needs29 Common definitions of assessment-based learning among the Regional Accreditors Access to Federal Financial Aid Proposal:a demonstration program that lets a set of low cost assessment approaches prove their merits. National Database of existing credential and training assessments
A New Narrative30 We need to add a chapter to the higher education biography. The story begins with Socratic learning, moves to libraries, adds faculty, morphs universities, carves land grants from the wilderness, upscales adult education into Community Colleges, and invests in for-profits. Now the story must add competency-based credentialing