Abundant scientific, scholarly research on the uses and effects of educational testing dates to the 1890s; randomized experiments comprise much of the volume. Comprehensive reviews of the accumulating research literature date as far back as the 1930s. By the 1970s, however, the character of some of the more widely cited reviews had changed. This presentation chronicles a half-century of relatively successful efforts to suppress much of the research on the uses and beneficial effects of educational testing. This presentation: chronicles with several examples from literature reviews from the past 80 years the parallel trends of a decline in the volume of research cited and a rise in the volume of research dismissal claims; describes the methods used to dismiss and suppress research; and includes excerpts from invited testimony before US Congressional Committees. Only a skewed subset of the relevant research literature was consulted in crafting highly consequential US educational testing policies.

1. The Successful Degradation of Evidence
on Educational Testing in the United States
Richard P. Phelps
International Test Commission
Virtual 12th Conference, July 2021

2. International Test Commission, 12th (virtual) Conference, 2
July 2021
1890s+ “Scientific” study of educational tests
Two Strands:
Aptitude: Binet & Simon, France
Achievement: Rice, Thorndike, USA (& Prussia)

3. International Test Commission, 12th (virtual) Conference, 3
July 2021
1910s+ Mass commercialization of school tests
Examples:
Courtis Series of Tests in Arithmetic (1909)
Hillegas-Thorndike Composition Scale (1912)
Ayres Spelling List and Scale (1915)

4. International Test Commission, 12th (virtual) Conference, 4
July 2021

5. International Test Commission, 12th (virtual) Conference, 5
July 2021

6. International Test Commission, 12th (virtual) Conference, 6
July 2021

7. International Test Commission, 12th (virtual) Conference, 7
July 2021
Dismissing testing effects literature - 1971
“ … a review of the literature revealed only a few small-scale
and somewhat peripheral empirical studies.”
-- “The Effects of Tests on Students and Schools,”
Review of Educational Research

8. International Test Commission, 12th (virtual) Conference, 8
July 2021
0
100
200
300
400
500
600
700
1942 Ross 1971 Review of Ed Research
The disappearing research on testing in US schools, 1971

9. International Test Commission, 12th (virtual) Conference, 9
July 2021
Dismissing testing effects literature - 1980
“… Today there remains … a dearth of empirical support on
actual test use practices.”
-- “Review of the Research Literature on Test Use,”
University research center

10. International Test Commission, 12th (virtual) Conference, 10
July 2021
The disappearing research on testing in US schools, 1980
0
100
200
300
400
500
600
700
1942 Ross 1971 Review of Ed
Research
1980 University center

11. International Test Commission, 12th (virtual) Conference, 11
July 2021
The disappearing research on testing in US schools, 1991
0
100
200
300
400
500
600
700
1942 Ross 1971 Review of
Ed Research
1980 University
center
1991 National
center

12. International Test Commission, 12th (virtual) Conference, 12
July 2021
” … We are just getting started in terms of solid research
on standards, testing and accountability.”
-- Brookings Institution scholar
Dismissing the testing effects literature - 2001

13. International Test Commission, 12th (virtual) Conference, 13
July 2021
The disappearing research on testing in US schools, 2001
0
100
200
300
400
500
600
700
1942 Ross 1971 Review of
Ed Research
1980 University
center
1991 National
center
2001 Political
advisors

14. International Test Commission, 12th (virtual) Conference, 14
July 2021
“Several types of effects” of educational tests:
“A. Curricular and instructional reform
“B. Motivation of teachers
“C. Motivation of students
“D. Improvement in student learning
“E. Public confidence”
Professor Smith literature review: 1997–2003

15. International Test Commission, 12th (virtual) Conference, 15
July 2021
Study: The Effect of Testing on Student
Achievement, 1910--2010
• analyzed:
• over 800 separate studies, and
• more than 1,700 separate effects
• majority from RCTs

16. International Test Commission, 12th (virtual) Conference, 16
July 2021
0 20 40 60 80 100 120
Program evaluation journal
Testing in the professions journal
Language testing
Non-education association
Higher education journal/publication
Education magazine
Governmental agency
Book
Dissertation or thesis
Education association
Research/advocacy org
Psychology journal
Education journal
Number of testing effect studies, 2002 and earlier, by category
Phelps meta-analysis Smith article

17. International Test Commission, 12th (virtual) Conference, 17
July 2021

18. International Test Commission, 12th (virtual) Conference, 18
July 2021
“[The Center] is a stellar center, in my view, and is doing very
important work on testing and assessment.
-- 1st Director, Institute for Education Sciences, 2003

19. International Test Commission, 12th (virtual) Conference, 19
July 2021
“Prior to IES’s leadership, RCTs were rarely conducted in
education and not valued among many researchers.”
-- Chair, Institute for Education Sciences (IES) &
Harvard U economist, 2013

20. International Test Commission, 12th (virtual) Conference, 20
July 2021
“… education has not, until recently, benefitted from
rigorous, scientific research.”
“IES has made remarkable progress. The Institute is only able
to convene top experts … because researchers believe [IES]
now promotes scientifically-grounded research. ”
-- Stanford U economist, 2011

21. International Test Commission, 12th (virtual) Conference, 21
July 2021
[Spacing effect]
[Testing effect]
“I could provide many more pages of example of things we know now
about education that we did not know 15 years ago.”
-- 1st Director, Institute of Education Sciences, 2011
“… examples of things we’ve learned from recent education research …

22. International Test Commission, 12th (virtual) Conference, 22
July 2021

23. International Test Commission, 12th (virtual) Conference, 23
July 2021
“The act of testing alone, irrespective of other factors, tends
to improve achievement.”
-- C. C. Ross, Measurement in Today’s Schools, 1941

24. International Test Commission, 12th (virtual) Conference, 24
July 2021
Capsule Summary:
History of Federal Educational Testing Policy in the USA
(1890–1950) Psychologists develop and study educational testing
(1920–1980) Education schools diverge from origins in Psychology
(1970+) Some ed school professors suppress positive research on testing
(2000–2003) Ed testing becomes high profile issue; Economists interested
(2001+) Economists believe Ed school professors; ignore Psych research

25. International Test Commission, 12th (virtual) Conference, 25
July 2021
Contact:
richard (at) nonpartisaneducation (dot) org
On Frank Dempster, see Yana Weinstein, Learning Scientists
Blog
https://www.learningscientists.org/blog/2018/8/31-1