Dont let yourself get snookered by education statistics


Published on

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Dont let yourself get snookered by education statistics

  1. 1. Don't Let Yourself Get Snookered by Education Statistics! By Gerald W. Bracey From Principal I N seventeen-century France and England, some people took to collecting numbers they thought would reflect the health of the state—births, deaths, mar- known as statists. They were usu- ally collected—or created—for a purpose, sometimes benign, some- times not. Shortly after the creation of sta- riages, and so on—that came to be tistics, British Prime Minister Ben- Gerald W. Bracey (, an independent education re- searcher based in Alexandria, Virginia, is author of Reading Educational Research: How to Avoid Getting Statistically Snookered (Heinemann, 2006), from which this article is adapted. Condensed from Principal, 85 (May/June 2006), 24-27. Published by the National Association of Elementary Schooi Principals, 1615 Duke St., Alexandria, VA 22314. 44
  2. 2. Grasping Education Statistics jamin Disraeli is alleged to have ric, it lists 13 indicators of risk. All declared, "There are three kinds of relate to test scores and many of lies: lies, damned lies, and statis- them are highly selective. tics." For example: "There was a steady I Statistics have always had to de- decline in science achievement fend their honor, which makes it scores of U.S. 17-year-olds as mea- curious that so many people are sured by national assessments...." accepting of statistics despite This was true, but why did the re- Disraeli's aphorism. That certainly port highlight only science and only seems to be true about education 17-year-olds? statistics, and especially true when Well, it seems that there was no eiducation statistics imply some- "steady decline" in science for 9- thing negative about American pub- and 13-year-olds, the other ages lic schools. tested at the time. There also was no hint of decline for any of the Lacking the Time three age groups for reading or Sometimes we accept statistics math. The commissioners who because we are not in a position to wrote the report had in their pos- challenge them, although this is far session nine trend lines (three sub- less true today, thanks to a variety jects by three ages), only one of of online research tools. Other which could be used to support times—and I think this is especially crisis rhetoric. That was the only true for harried principals—we ac- one they used. cept them because we lack the time to ferret out the truth. Beware of mutant statistics. I To interpret statistics in the Perhaps the most dangerous statis- hope of giving non-researchers a tic is the "mutant statistic." This is time-saving tool with which to ap- a figure that begins life as an honest proach statistics with both skepti- number but somehow morphs into cism and wisdom, I have identified something inaccurate. what I call my Principles of Data An example is the widely circu- Interpretation. lated statistic that 150,000 Ameri- can women die each year of anor- The first principle is to question exia. But a visit to ihe Statistical statistics that, whiie accurate Abstract of the United States, an an- enough, do not teii the whole story. nual compendium from the Census For example, the 1983 federal gov- Bureau, showed that 8,500 women ernment report, A Nation at Risk, is aged 15 to 24 die each year of all a veritable treasury of slanted, spun, causes, as do 47,000 women aged and distorted statistics. After an 25 to 44. opening blast of cold-warrior rheto- Given that anorexia is a disorder September 2006 45
  3. 3. THE EDUCATION DIGEST usually found in young women, and dren because they believe they are given that 55,500 women aged 15 to better." 44 die of all causes, how likely is it Elxcept that they don't choose that 150,000 women annually die of private schools. If you look at anorexia alone? In fact, the initial Doyle's own numbers, you see that and possibly legitimate statistic was 10.6% of public school teachers and that 150,000 American women suf- 12.1 % of the general public had chil- fer from anorexia. dren in private schools in 2000, Mutant statistics afflict educa- compared with 12.1% of teachers tion. Here's an example: Columnist and 13.1% of the general public in George Will, writing in The Wash- 1990. ington Post, reported that nearly What is remarkable about these half of the public school teachers in numbers is that, despite all the Chicago sent their kids to private bashing of public schools and soar- schools. This was true; the figure at ing rhetoric about vouchers and the time was 43%—the highest pro- privatization in the 1990s, teachers portion in the nation. used private schools less than the But six months later. Will mu- general public, and that a smaller tated the statistic when he wrote, proportion of both teachers and "nationally, almost half of all urban the general public were using public school teachers with private schools in 2000 than in school-age children send their chil- 1990. dren to private schools." This was nof true. But, as with drug-resistant Are group comparisons valid? bacteria, this mutant has been hard The example above leads to yet to kill. another Principle of Data Interpre- tation: When comparing groups, Do the numhers match the make sure the groups are compa- rhetoric? This leads to another Prin- rable. In this case, public school ciple of Data Interpretation: Make teachers and the general public are certain the rhetoric and the num- not truly comparable. While slightly bers match. Those with agendas more than a quarter of the general often use rhetorical flourishes to public has at least a bachelor's de- distract the reader away from the gree, virtually all public school actual numbers. teachers do. And while almost 20% Denis Doyle, from whom Will got of the general public lives in pov- his original statistic, wrote, "With erty, teachers on average make teachers choosing private schools, about 2.5 times the poverty thresh- the truth is self-evident: while they old and are often not the sole wage work in public schools, they choose earner in a family. private schools for their own chil- As a variant on the principle of 46 www.ed(
  4. 4. Grasping Education Statistics group comparisons, make sure ever the aggregate data show one groups that are comparable Initially pattern, and subgroup data, the stay comparable over time. Con- opposite. sider this statistic: In 2004, Massa- How could this happen? The chusetts claimed that 96% of high resolution of the paradox lies in the school seniors passed its gradua- changing makeup of the two groups. tion exit test. The figure was true, In 1981, whites made up 85% of SAT but only for those who were se- test-takers. If we compare the 1981 niors at the start of the 2003-04 scores with those in 2005, when school year. only 63% of the test-takers were , To get a more meaningful pic- white, we see Simpson's Paradox in ture of the condition of education action. The same result will also as it exists here, we should ask be seen in any examination of scores what percentage of the class of 2004 from the National Assessment of as ninth-graders graduated in 2004? Educational Progress (NAEP): Eth- In Massachusetts, the class of 2004 nic group increases are consistently numbered about 60,000 seniors, greater than national increases. while it once contained 78,000 ninth- graders. Where did those other Do the arithmetic. There are 18,000 students go? many more Principles of Data Inter- Looking at the number of ninth- pretation than I have space to graders in the class of 2004 com- present here, but here's one you pared with the numbers who were can work out for yourself. I call it graduating, the graduation rates "Do the arithmetic." range from 54% for Latino students In 1994, an author wrote in a to 80% for white students, with an respected educational periodical, overall rate of 74%, considerably "Every year since 1950, the number lower than the 96% which was of American children gunned claimed by the state and praised by down has doubled." This statement test advocates. suffers a failing of many statistical statements—a lack of specificity. I Watch out for Simpson's Para- What ages define "children" and dox. It is particularly important to what does "gunned down" mean? watch for changes in groups over Now sit down with a piece of time in an ethnically diversifying paper and a calculator. On the pa- country like the United States. Look- per, list the years from 1950 to 1994 ing only at changes at the national (the year of the article). Now, as- level can obscure changes that are sume that one child was gunned tciking place within ethnic groups. down in 1950 and double that The phenomenon known as number each successive year. Have Sitnpson's Paradox occurs when- fun! effl September 2006 47