Understanding the U.S. News & World Report “Best Colleges” 2007 - HandoutDocument Transcript
U.S. News and World Report 1 Understanding the U.S. News & World Report “Best Colleges” 2007: How it Works, Problems, and Implications for Institutional Research Matthew J. Hendrickson Ball State University December 11, 2006Introduction:Many companies have made attempts at trying to provide the college “consumer” with guides for selecting an institution. These systems were created to aid in making “one of the most important decisions of your life” (USNWR-primer, 2007). -Some examples of these guides include: U.S. News, Insider’s Guide, Fiske, Peterson’s, Princeton Review, College Blue Book.The most controversial is U.S. News & World Report (USNWR) because it offers a ranking system that the others do not. Since its inception in 1983, this powerhouse has taken over the college rankings field (Stuart, 1995).How the rankings work:Survey: Identified eight types of programs associated with student success: 1) 1st year experiences, 2) learning communities, 3) writing in the disciplines, 4) senior capstone, 5) study abroad, 6) internships/cooperative education, 7) opportunities for undergraduate research, and 8) service learning.Relies on quantitative measures of academic quality. These measures are explicit and measurable, serving as true indicators that can be compared across institutions.Divides institutions into categories based on mission and region (if necessary). These categorizations serve to put like institutions together by the type, and in some cases location, of the institution.The Carnegie Foundation for Advancement of Teaching system is used to determine the categories. This classification was developed in order to create rough classifications to make comparisons between different types of institutions.USNWR gathers information for up to 15 indicators of academic excellence. These indicators are outlined below, including the seven main categories and their respective subcategories. I refer to these as factors.Lastly, they are ranked against each other for a numerical standing, in order to determine “the best” colleges. The first two tiers are numerically ranked, whereas the last two tiers are simply referred to as the third and fourth tiers. These two tiers do not have a numerical ranking.
U.S. News and World Report 2Factor Weights: 5% 5% 25% 10% 15% 20% 20% Peer Assessment Retention Faculty Resources Student Selectivity Financial Resources Graduation Rate Performance Alumni GivingThese weights were selected based on years of reporting about education, on reviews of research about education, and after consultation with experts in higher education (USNWR-cofaq_brief, 2007).More weight has been placed on the outcome measures (i.e., graduation rate, alumni giving, etc.) versus the input measures (i.e., entering test scores, financial resources, etc.).Factor Definitions: (Morse & Flanigan, 2007)Peer Assessment (25%): survey top academic leaders for opinions of other institutions. Data collected by Synovate (Chicago). For 2007, there was a 58% response rate (total sent out = 4,089). The instructions ask administrators (president, provost, and dean of admissions) to rate items from 1 (marginal) to 5 (distinguished), an “I don’t know” option is present if the administrator does not know enough about the institution.Retention (20%): check collected data from the institutions and completed blank data from national reporting sources. Includes two components; six-year graduation rate (80% of final retention score) and freshman retention rate (20% of final retention score).Faculty Resources (20%): includes six subcategories to compile the total score. These categories are: faculty compensation/salary/benefits (35%), percent with terminal degree (15%), percent full-time faculty (5%), student/faculty ratio (5%), class size of 1-19 students (30%), an class size of 50+ students (10%).Student Selectivity (15%): includes three subcategories; SAT/ACT score (50%), high school class standing-top 10 percent (40%), and acceptance rate (10%).Financial Resources (10%): assesses per-student spending. This evaluates average spending per-student on instruction, research, student services, and related educational expenditures.
U.S. News and World Report 3Graduation Rate Performance (5%): looks at six-year graduation rates versus a predicted rate. This predicted rate is based on the characteristics of the entering class and characteristics of the institution. This process uses the said characteristics and performs regression analyses to compile predictions. This is a very complex process and varies depending on the individual institution.Alumni Giving (5%): counts the average percent of alumni who gave to the university during a specified year. This was assessed by creating a weighted sum of scores. Then the scores were rescaled, giving the top institution a value of 100, the next highest a value of 99, and so on. Final scores were rounded to the nearest whole number and ranked in a descending order. “Tied” institutions are then listed alphabetically.Problems with the Rankings:Does not include all institutions. They must be regionally accredited and have a total enrollment of at least 200 students. Those others that are not counted include: institutions with almost all non-traditional students; the ones that mostly grant fine arts, performing arts, business, or engineering degrees; and the uniformed service academies (USNWR-cofaq_brief, 2007)The methodology changes on a yearly basis. This creates difficulties in comparing different years’ rankings. On a positive note, this is in the attempt to improve the ranking system.The system only ranks approximately the top 50% of institutions. This composes the first two tiers, which are assigned numerical ranks. Tiers three and four are each lumped alphabetically into two broad groups, one for each tier.When institutions do not report specific pieces of data, USNWR puts “an educated guess” for the proper number in that space. This data is collected from the Council for Aid to Education, the National Collegiate Athletic Association, the National Center for Education Statistics, data collected from previous USNWR issues, and data pulled from the institution’s website.Justifications for the Rankings:Uses reported data from the institution as well as extra data from national institution data commissions. These sources included the National Survey of Student Engagement (NSSE), American Association of University professors, the National Collegiate Athletic Association, the Council for Aid to Education, and the U.S. Department of Education’s National Center for Education Statistics.Peer Assessment: assumes that those in the field are the most knowledgeable about their competition and the status of higher education.Retention: the larger the number of returning freshman and those who graduate the university shows the institution to offer classes and services the students need to succeed.Faculty Resources: the more satisfied students are about their contact with professors, the more likely they will learn, making it more likely they will graduate.Student Selectivity: a institution’s academic atmosphere and abilities are limited to the students who attend.
U.S. News and World Report 4Financial Resources: it is assumed that a generous per-student spending indicates that a college can offer a wide variety of programs and services.Graduation Rate Performance: if a institution has a low graduation rate, there must be other issues impacting the institution’s overall ability.Alumni Giving Rate: as an indirect measure, it assumes the more alumni who donate to the institution, the more satisfied they were with their experience. --The justifications for each factor from Morse & Flanigan (2007).Cheating the Rankings:These points were adapted and created from Ehrenberg (2005) and Woodbury (2003).Peer Assessment: Talk up your institution to others. The better light you can shine, the more (positive) attention you can get. Also, by hosting a one to two year “boost” of your ratings by pouring out vast amounts of money into your institution, this will also make you look good and provide for the long haul.Retention: Relax standards and make it easier to graduate, thus increasing both retention and graduation rates. One could also spend money on retention programs, but this could be better spent on a short term rankings boost, systematically alleviating this problem by attracting “better” and more determined students.Faculty Resources: Pay tenure faculty high salaries and hire instructors to teach many low level courses. This decreases the student/faculty ratio and increases faculty compensation. When enrolling for classes, do not allow more than 19, or less than 100, in any course to make your class size factors look better.Student Selectivity: Talk up your school to as many students as possible, increasing your application numbers. Also, focus your moderate institution as the “ideal backup institution,” increasing your application numbers. Have these students sign pre- acceptance statements, ensuring the enrollment of many high-quality students. Make reporting of SAT/ACT scores optional; the ones with high scores will report them, while the ones with lower scores will not drag down your average. In this line, accept only those with high SAT/ACT scores, as well as exemplary high school rankings and elevated GPAs. Finally, accept as few students as possible. This will increase your selectivity and also increase your per-student spending ratio. Make attempts to find the high “yield;” which is the selection of those with the greatest likelihood of actually enrolling.Financial Resources: Have faculty gain numerous outside grants, allowing more of the institution’s budget to be spent on other rank raising factors.Graduation Rate Performance: This is taken care of through the “fixes” proposed in the retention and student selectivity sections. By creating recruitment and selection methods to find the high “yield” and increasing general retention.Alumni Giving Rate: Have as many alumni donate as possible. It does not matter how much they donate, you just need to increase the percentage of those who donate anything.General: Accept only non-traditional students. By avoiding these students, many factors that hurt your rankings can be avoided (i.e. decreased six-year graduation rates, alumni giving, selectivity, peer assessment, etc.). Lobby to those at U.S. News to change the formula in ways that will help your institution.
U.S. News and World Report 5Implications for IR:Pros:A standardized Common Data Set was created for the standardization of reporting figures to USNWR. This allows the institutions to use the “same” numbers. The problem here is that the definitions vary at the institutional level, creating added error and variance.Allows for competition for increasing educational methods. In an attempt to improve rankings, the institution may implement programs that not only make the school look better, but also provide for a better college experience and, hopefully, better learning.Provides information to potential college applicants about specific institutions. This information is provided by statistics, a biography of the institution, demographics of the institution, and even the rankings.Cons:Upward pressure on tuition due to “spending” races by institutions to increase rank.Pressure on applicants to accept enrollment prematurely, as well as avoid challenging “pre-college” courses to maintain class rank.Penalizes institutions for not participating in the program. This occurs through either using previous submissions by that institution, or will fill in the data from the Common Data Set, if available.There have been many criticisms of the validity and usefulness of the USNWR rankings. These issues have made this system unpopular in the academic field, although its popularity among parents and students continues.References:Carnegie Foundation for Advancement of Teaching, Classification System http://www.carnegiefoundation.org/about/sub.asp?key=18&subkey=405#4Ehrenberg, R.G. (2005). Method or madness? Inside the U.S. News & World Report college rankings. Journal of College Admission, Fall, 29-35.Hamrick, F. A., Schuh, J. H., & Shelley, M. C. (2004, May 4). Predicting higher education graduation rates from institutional characteristics and resource allocation. Education Policy Analysis Archives, 12, 19. Retrieved [12/8/2006] from http://epaa.asu.edu/epaa/v12n19/.Morse, R.J. & Flanigan, S. (2007). How we do the rankings. http://www.usnews.com/usnews/edu/college/rankings/about/07rank_brief/phpStuart, D.L. (1995). Reputational rankings: Background and Development. In D. Walleri, & M.K. Moss (Eds.), New Directions for Institutional Research: Evaluating and Responding to College Guidebooks and Rankings (13-20). Jossey-Bass, San Francisco, Winter 1995, No. 88.U.S. News & World Report, Best Colleges 2007. http://www.usnews.com/usnews/edu/college/rankings/about/cofaq_brief.php http://www.usnews.com/usnews/edu/college/rankings/about/primer_brief.php http://www.usnews.com/usnews/edu/college/rankings/rankindex_brief.phpWoodbury, R.L. (2003). How to make your college no. 1 in U.S. News & World Report…and lose your integrity in the process. New England Board of Higher Education, Spring, 18-20.