The document discusses the design of an effective assessment system, emphasizing the importance of frequent testing and retrieval practice for enhancing long-term retention of knowledge. It presents cognitive psychology's findings on learning, along with strategies for effective assessment, such as spaced practice and providing feedback. Additionally, the document highlights the role of large-scale testing and quality control measures in educational assessment while cautioning against potential negative impacts on formative assessments.

1. Designing an Assessment System
Richard P. Phelps
International Research-to-Practice Conference
Nazarbayev Intellectual Schools AEO
Astana, Kazakhstan
October, 2016
© 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 1

2. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 2
“If a thing exists, it
exists in some
amount. If it exists in
some amount, then it
is capable of being
measured.”
−−René Descartes,
Principles of
Philosophy, 1664

3. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 3
Image of Protein Molecules Forming Memories
Albert Einstein College of Medicine, New York, January 2014

4. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 4
Image of Protein Molecules Forming Memories
Albert Einstein College of Medicine, New York, January 2014

5. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 5
Learning Curve

6. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 6
Forgetting Curve (1870s)

7. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 7

8. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 8
Ebbinghaus:
“Learning usually
requires rehearsal
or repetition”

9. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 9
Cognitive Load Theory
John Sweller, 1980s
Working Memory Capacity
George Miller, 1950s

10. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 10
Working Memory:
Ability to temorarily hold and
manipulate information for
cognitive tasks
Working Memory is challenged by:
new, unfamiliar information and
quantity of discrete bits of information

11. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 11
I am thinking of a type of object, what is it?
They are shapes, geometric plane figures,
polygons, quadrilaterals, and parallelograms
with opposite equal acute angles, opposite
equal obtuse angles, and four equal sides
Description 1:

12. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 12
I am thinking of a type of object, what is it?
Description 2:

13. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 13

14. Two centuries of research on learning concludes…
© 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 14
“…repeated retrieval during learning is the key to
long-term retention.”
— Henry L. “Roddy” Roediger

15. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 15
Cognitive Scientists’ 6 Strategies for Effective Learning
Retrieval Practice
Spaced Practice
Dual Coding
Interleaving
Concrete Examples
Elaboration

16. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 16
Retrieval Practice

17. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 17

18. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 18

19. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 19
Implications for Teachers 1
Most teachers should test more
frequently, …with smaller,
shorter, low-stakes tests
Understand that useful
assessment can be short and
simple.

20. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 20
Implications for Teachers 2
Does the test format
matter?
• multiple-choice?
• essay?
• short answer?
• oral?
• demonstration?
• …etc.?
Not so much.

21. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 21
Tests provide
feedback to teachers
about what works
and what does not
Implications for Teachers 3
Just like students can learn by testing each other;
teachers can help each other by reviewing each
others’ tests.

22. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 22
Cognitive Psychology
experiments were
conducted with
“formative” tests in
schools and classrooms

23. What about systemwide, large-scale tests?
First priority:
do no harm to the
formative testing
programs in schools
and classrooms

24. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 24
The effect of testing on student learning
• 12-year study, read >3,000 documents
• analyzed close to 700 separate studies, and
more than 1,600 separate effects
• 2,000 other studies were reviewed and
found incomplete or inappropriate
• hundreds of other studies remain to be
reviewed

25. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 25
245 Qualitative studies
813 Surveys or Polls
640 Quantitative Studies:
Experiments:
School- and classroom-level
Multivariate studies:
Large-scale testing programs
The effect of testing on student learning

26. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 26
Meta-analysis
A method for
summarizing a large
research literature, with
a single, comparable
measure.
( 0.5 effect size ≈ 1 grade level of learning )

27. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 27
Findings from Phelps (2012):
• Survey study effect sizes average >1.0
• Over 90% of qualitative studies positive
• For quantitative studies, univariate effect sizes positive and
stronger when:
– Testing more frequently
– Testing with feedback
– Testing with stakes

28. 28
Findings from Phelps & Silva (2015)
For quantitative studies, effect sizes vary
between 0.55 and 0.88:
+++ testing more frequently
++ testing with stakes
+ testing with feedback
International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016© 2016, Richard P PHELPS

29. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 29
• size of study population
• small +0.34 over large
• scale of test administration
• small-scale +0.14 over large-scale
• responsible level of government
• local tests +0.29 over state tests
Effect of scale on testing benefits

30. Large-scale test, tight security
© 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 30

31. Large-scale test, lax security
© 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 31

32. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 32
Besides, systemwide tests are needed for
other purposes, such as…
…selection to programs with limited number of places
…monitoring and system diagnosis
…workforce planning
…accountability
…credentialing
That’s enough!

33. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 33
Some large-scale test advantages
On per-student basis, inexpensive
Cognitive laboratory pre-testing possible
Standardization offers comparisons across schools and regions.
May produce high-quality items that schools and teachers can use.
MOST IMPORTANT:
provides reliable, comparative information to all those not involved in
a particular school

34. The more systemwide decision points, the better ?
Figure 1: Average TIMSS Score and Number of Quality Control
Measures Used, by Country
0
10
20
30
40
50
60
70
80
0 5 10 15 20
Number of Quality Control Measures Used
AveragePercentCorrect(grades7&8)
Top-Performing Countries Bottom-Performing Countries
SOURCE: Phelps, Benchmarking to the best in mathematics, Evaluation Review, 2001
© 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 34

35. Quality control has proportionally greater effect in poorer countries
Figure 2: Average TIMSS Score and Number of Quality Control
Measures Used (each adjusted for GDP/capita), by Country
Number of Quality Control Measures Used (per GDP/capita)
AveragePercentCorrect(grades7&8)
(perGDP/capita)
SOURCE: Phelps, Benchmarking to the best in mathematics, Evaluation Review, 2001
© 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 35

36. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 36
TIMSS, PIRLS, CIVED, SITES, ICILS,
PPP, ECES, TEDS
IEA:
OECD PISA:
World Bank:
PISA, PISA for schools
PISA for development
READ, SABER
…provides funding for PISA

37. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 37
The effect of international testing programs
Freedomtodesignyourtesting
school
tests
international
tests
state and national tests

38. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 38
OECD and World Bank are run by economists
How well do economists understand PSYCH-ometrics?
Some interesting examples:
Chile’s national testing
program, funded by the
World Bank
OECD’s “Synergies for
Better Learning” project

39. © 2016, Richard P PHELPS International Research-to-Practice Conference, Astana, Kazakhstan, October, 2016 39
Some interesting oddities:
World Bank educational
assessment chiefs are always
Irish nationals affiliated with
Boston College in the USA.
PISA is universally interpreted as
an achievement test, even by
the OECD. In reality, it has been
an unvalidated aptitude test.

40. Designing an Assessment System
richard {at} nonpartisaneducation {dot} org