This document summarizes a research agenda to develop new measures of educational quality across higher education institutions. It involves a three-phase pilot study: 1) A dual-institution pilot using surveys, syllabus analysis, classroom observations, and student work to measure academic rigor, teaching quality, and learning outcomes. 2) A multi-institution benchmarking pilot. 3) A national study with publicly reported data. The goal is to provide more comprehensive public data on the educational core of institutions - teaching, rigor, and student experiences - to inform stakeholders and promote institutional self-reflection. The pilots will test frameworks for rigor, teaching, and essential learning outcomes developed from Bloom's Taxonomy and AAC&U guidelines.

1. It Can’t Be Done: Assessing
Educational Quality Across Institutions
Corbin M. Campbell, Assistant Professor
Theresa Cruz Paul, Research Team Member
Higher and Postsecondary Education Program
Teachers College, Columbia University

2. Advisory Board Members
• Jennifer Glaser
• Director of Student Services, Fairfax High School
• Dr. Wendell Hall
• Deputy Director
• Institute for Higher Education Policy
• Dr. Karen Inkelas
• Director, Center for the Advanced Study of Teaching and Learning
• University of Virginia
• Dr. Christine Keller
• Executive Director, Voluntary System of Accountability; Associate
Vice President for Academic Affairs
• Association of Public and Land-grant Universities
• Sharon La Voy
• Director of Assessment
• University of Maryland
• Dr. Jennifer Lindholm
• Special Asst. to the Dean & Accreditation Coordinator
• University of California Los Angeles
• Dr. George Mehaffy
• Vice President for Academic Leadership and Change
• American Assoc. of State Colleges and Universities
• Dr. Jessica Mislevy
• Research Social Scientist
• SRI International
• Dr. Daniel Newhart
• Senior Researcher & Associate Director
• Center for the Study of Student Life
• The Ohio State University
• Dr. Anna Neumann
• Professor of Higher Education
• Teachers College, Columbia University
• Dr. KerryAnn O'Meara
• Associate Professor of Higher Education
• University of Maryland, College Park
• Dr. Aaron Pallas
• Professor of Sociology and Education
• Teachers College, Columbia University
• Dr. Stephen Porter
• Professor of Higher Education
• North Carolina State University
• Dr. Priscilla Wohlstetter
• Distinguished Research Professor
• Teachers College, Columbia University

3. Claim
There is currently no comprehensive way for
the public and prospective students and
families to know about the quality of the
education that is happening inside the walls
of a college or university—and how that
quality compares to the quality at other colleges
and universities.

4. The Black Box of Higher Education
Extensive accountability data is largely unseen by
the public:
• Accreditation
• Collegiate Learning Assessment (CLA)
• Course-based learning outcomes
• Course evaluations

5. Impacts of the Black Box [& Rising Costs]
• Spellings Commission
• Academically Adrift (Arum & Roksa, 2010)
• Numerous articles in top newspapers regarding
questions to academic rigor in higher education.
• 12/10/12: “Who will hold colleges
accountable?” (Carey, NYT, p. A27)
• “Affront on Higher Education”

6. How do prospective students, parents, and the
public decide which institution has the highest
quality education?
• US News
• World Rankings
• Other ranking venues: Princeton Review
• Word of Mouth—Family, Friends, High School
Guidance Counselors
• Reputation
[Newer, but under used forms: NSSE?, VSA/College
Portrait?]

7. US News & World Report’s Formula
1. Graduation & retention
rates-20%
1. Average graduation rates-80%
2. Average freshman retention rate-
20%
2. Financial resources-10%
1. Average educational expenditure
per student-100%
3. Alumni giving-5%
4. Graduation rate
performance–5%
5. Peer assessment-25%
6. Student selectivity-15%
1. Acceptance rate-10%
2. High school ranking-40%
3. SAT/ACT scores-50%
7. Faculty resources-20%
1. Faculty compensation-35%
2. % faculty with top terminal
degrees-15%
3. Percent full time faculty-5%
4. Student/Faculty ratio-5%
5. Class size 1-19 students-30%
6. Class size 50 or more-10%

8. Possible unintended consequences of
past and current measures….
Policy: No Child Left Behind (and Collegiate Learning
Assessment—CLA)
• Teaching to the test
• Altering curriculum
Public: US News & World Report
• Students and parents use rank instead of fit to select which
college to attend.
Institutions: US News & World Report
• Mission creep/Striving
• Over-reliance on SAT (GWU, for example)
• Numbers Manipulation: Cornell removed non-graduates from
the alumni list

9. Questions Absent in These Measures…
• What is the level of academic rigor?
• What is the quality of teaching?
• What are the educational practices that an
institution employs that affect student learning?
[Maybe a few examples of rankings that use surveys to measure
these items, for example, Princeton Review]

10. Why are these questions absent?
• These data are difficult to obtain!!
• These data are expensive to obtain!!
• Colleges and Universities are protective of
academic freedom and are insular with data about
the educational core.

11. Enter NSSE, CLA, VSA/College Portrait
The National Survey of Student Engagement (NSSE): Surveys
institutions about their effective educational practices
CLA: Measures student’s critical thinking skills pre and post college via
a standardized test.
Voluntary System of Accountability (VSA)/College Portrait: Created
to ward off imposed and mandating higher education testing. Aimed at
the public transparency of higher education—compiles several data
sources: NSSE, CLA, Grad/retention
PROBLEMS:
1) Concerns with validity
2) Relies on a single data collection method
3) Not primarily intended for the public
4) Assumes learning is due to college environment
5) Missing data / limited data

12. Purpose
This research agenda aims to create
alternative, innovative, and comprehensive
measures of educational quality across institutions
that could contribute to public understanding of
college and university quality.

13. What are the intended consequences of
this new educational quality measure?
1) A stronger focus on the educational core of institutions:
teaching and academic rigor, and educational
experiences
2) Public access to comprehensive data about teaching,
academic rigor, and educational experiences in higher
education at the institutional level
3) Administrators having an in-depth understanding of how
their institution compares to others in terms of
teaching quality, academic rigor, and educational
experiences

14. Three Phases:
• Dual-Institution Pilot (Spring 2013)
• One large, public, research extensive institution
• One medium private research extensive institution
• Multi-Institution Peer Benchmarking Pilot
(Spring/Fall 2014)
• National Study with Publicly posted Data
(Spring 2016)

15. Dual-Institution Pilot
1) Student and faculty survey: NSSE FSSE
2) Syllabus analysis
3) Experience sampling method
4) Class observations
5) Analyzing Student Work
6) Course Evaluations*
Academic Rigor, Teaching Quality, Learning Outcomes

16. Academic Rigor
• Based on the cognitive complexity required by students in
the coursework as defined by the revised Bloom’s
Taxonomy (Anderson & Krathwohl, 2001)

17. Teaching Quality
• Based on Anna Neumann’s claims on teaching and
learning (2012).
• According to this framework, quality teaching entails:
(Part I) Orchestrating an encounter of subject matter ideas
(Part II) Connecting student’s learning to prior knowledge
(Part III) Supporting students in working through the
cognitive and emotional features of encounters between
their own long-held understandings and new ones gained
during the course.

18. Essential Learning Outcomes
• Based on the American Association of Colleges and
Universities’ (AAC&U) Essential Learning Outcomes (ELO).
• This framework was developed by AAC&U through
engagement with hundreds of institutions, accreditors, and
higher education stakeholders (AAC&U, 2004).
• Four Parts:
• ELO Part I: Knowledge of Human Cultures and the Physical and
Natural World
• ELO Part II: Intellectual and Practical Skills
• ELO Part III: Personal and Social Responsibility
• ELO Part IV: Integrative and Applied Learning

19. Where we are now
• At this point we have observed 100 classrooms
• In the process of collecting 300+ syllabi
• Collecting NSSE FSSE data
• Discovering that Experience Sampling may not be a
viable option
• Collecting student work
Next steps
• Preparing to collect data from a second pilot institution
• Searching for a multi-site pilot institution that would like
to compare data to defined peer institutions

20. Questions? Comments?
Corbin M. Campbell, Assistant Professor
campbell2@tc.columbia.edu
Theresa Cruz Paul, Research Team Member
Higher and Postsecondary Education Program
Teachers College, Columbia University