Understanding the U.S. News & World Report “Best Colleges” 2007Presentation Transcript
Understanding the U.S. News & World Report “Best Colleges” 2007: How it Works, Problems, and Implications for Institutional Research Matthew J. Hendrickson Ball State University
Introduction: Ranking Systems
Many companies have made attempts at trying to provide the college “consumer” with guides for selecting a institution. These systems were created to aid in making “one of the most important decisions of your life” (USNWR-primer, 2007)
U.S. News, Insider’s Guide, Fiske, Peterson’s, Princeton Review, College Blue Book, etc.
The most controversial is U.S. News & World Report (USNWR) because it offers a ranking system that the others do not.
How the Rankings Work
Uses a factor system to assign ranks to institutions in different categories.
Surveys 8 types of programs associated with student learning.
Relies on quantitative measures of academic quality.
Divides institutions into categories based on mission and region (if necessary).
The Carnegie Foundation for Advancement of Teaching system is used to determine the categories.
Gather info for up to 15 indicators of academic excellence.
Lastly, they are ranked against each other to create a numerical standing.
U.S. News & World Report: Factor Weights
These weights were selected based on years of reporting about education, on reviews of research about education, and after consultation with experts in higher education (USNWR-cofaq_brief, 2007).
More weight has been placed on the outcome measures versus the input measures.
Definitions Morse & Flanigan, 2007
Peer Assessment (25%): assessed by peer surveys completed by the administration.
Retention (20%): measures average six-year graduation rates and freshman retention.
Faculty Resources (20%): considers compensation, terminal degree, student/faculty ratio, class size, and percent full-time faculty.
Student Selectivity (15%): based on acceptance rates, high school class standing, and SAT/ACT scores.
Financial Resources (10%): uses average expenditures per-student.
Graduation Rate Performance (5%): considers actual versus expected six-year graduation rates.
Alumni Giving (5%): percentage of alumni donating to the institution.
Problems with the Rankings
Does not include all institutions.
Methodology changes frequently, making it difficult to compare previous versions to the most current version.
Only ranks the top 50% of institutions. The lower 50% is placed into two different tiers, neither of which are numerically ranked.
If an institution does not provide the data, USNWR will fill in that data, or use the previous data supplied by the institution.
Justifications for Rankings From Morse & Flanigan (2007) unless otherwise noted.
Peer Assessment: assumes that those in the field are the most knowledgeable about their competition and the status of higher education.
Retention: the larger the number of returning freshman and those who graduate the university shows the institution to offer classes and services the students need to succeed.
Faculty Resources: the more satisfied students are about their contact with professors, the more likely they will learn, making it more likely they will graduate.
Student Selectivity: a institution’s academic atmosphere and abilities are limited to the students who attend.
Financial Resources: it is assumed that a generous per-student spending indicates that a college can offer a wide variety of programs and services.
Graduation Rate Performance: if a institution has a low graduation rate, there must be other issues impacting the institution’s overall ability.
Alumni Giving Rate: as an indirect measure, it assumes the more alumni who donate to the institution, the more satisfied they were with their experience.
Cheating the Rankings Ehrenberg (2005) and Woodbury (2003)
Peer Assessment: Talk up your institution to others. Host a 1-2 year spending “boost” of rankings to increase opinion.
Retention: Relax standards. Spend retention money on other factors to boost overall opinion.
Faculty Resources: Pay tenure faculty high salaries and have instructors teach many low level courses (faculty/student ratio). When selecting course size, do not allow more than 19 or less than 100 students.
Student Selectivity: Increase application numbers. Pre-acceptance statements and the “safety school”. Reporting of SAT/ACT optional, but only accept high scores. Accept as few students as possible. Go for high “yield.”
Financial Resources: Have faculty get outside grants, spend institution funds elsewhere.
Graduation Rate Performance: This is taken care of through the “fixes” proposed in the retention and student selectivity sections.
Alumni Giving Rate: Have as many alumni donate as possible; get them to donate ANYTHING to increase your percentage.
General: Accept only non-traditional students.
Implications for IR
The creation of a standardized Common Data Set.
Allows for competition for increasing educational methods.
Provides information to potential college applicants about specific institutions.
Upward pressure on tuition due to “spending races.”
Competition for rankings can create negative emphasis.
Penalizes schools for not participating.
There have been criticisms about the use and validity of the USNWR rankings.
Carnegie Foundation for Advancement of Teaching, Classification System
Ehrenberg, R.G. (2005). Method or madness? Inside the U.S. News & World Report college rankings. Journal of College Admission, Fall, 29-35.
Hamrick, F. A., Schuh, J. H., & Shelley, M. C. (2004, May 4). Predicting higher education graduation rates from institutional characteristics and resource allocation. Education Policy Analysis Archives, 12, 19. Retrieved [12/8/2006] from http://epaa.asu.edu/epaa/v12n19/.
Morse, R.J. & Flanigan, S. (2007). How we do the rankings. http://www.usnews.com/usnews/edu/college/rankings/about/07rank_brief/php
Stuart, D.L. (1995). Reputational rankings: Background and Development. In D. Walleri, & M.K. Moss (Eds.), New Directions for Institutional Research: Evaluating and Responding to College Guidebooks and Rankings (13-20). Jossey-Bass, San Francisco, Winter 1995, No. 88.
U.S. News & World Report, Best Colleges 2007. http://www.usnews.com/usnews/edu/college/rankings/about/cofaq_brief.php