Slides for the AMET-NAMA session

1. MATHEMATICAL TASKS HAVE
BECOME TOO WORDY:
EVIDENCE FROM
INTERNATIONAL COMPARISONS
AMET/NAMA
6 March 2021
9:30-10:30
Dr Christian Bokhove
Southampton Education School
University of Southampton

2. Who am I?
• Dr Christian Bokhove
• From 1998-2012 teacher mathematics & computer science, head of
ICT secondary school Netherlands
• PhD Utrecht University 2011
‘Use of ICT for acquiring, practicing and assessing algebraic expertise’
• Associate Professor at University of
Southampton
• Mathematics education
• Technology use
• Large-scale assessment (PISA/TIMSS)
• Research methods

3. This workshop
• The role of language in mathematics
• We look at some example tasks
• National assessments
• International assessments
• The influence of context and language on achievement
(PISA and TIMSS)
• What does this mean for teaching mathematics?
• Conclusion

4. Context
• Whether we like it or not, mathematics has become more
wordy….
• Word problems play a big role
(Fuchs et al., 2015: text comprehension plus specific word
problem vocabulary)
• Mathematics in newspapers…
• Emphasis on mathematical literacy (e.g. OECD’s PISA)
But….
• This might have unintended (?) consequences.
• Let’s explore…

5. SATs KS2 reasoning paper 2019
GCSE maths – foundation- non-calc

6. Other examples
Arcavi (1994): “The findings in Clement [1982] show that more
than 30% of the 150 freshman engineering students who answered
the test failed to solved this problem correctly. The typical wrong
answer reported was 6S = P.”
He noted there was a
“language distractor, i e. the order of the key words (six times as
many students) which may bias our sense towards a "word-by-
word translation" as 6S = P”
“Six wolfs catch six lambs in six minutes. How many wolves will be
needed to catch sixty lambs in sixty minutes?”

7. Examples from international assessments
• PISA – Mathematical literacy (Definition)
• By the OECD
• More aimed at ‘mathematical literacy’. “Mathematical literacy is an
individual’s capacity to reason mathematically and to formulate,
employ and interpret mathematics to solve problems in a variety of
real-world contexts.” (from the PISA 2021 mathematics framework).
• 15-year olds
• Trends in Mathematics and Science Study (TIMSS)
• By the IEA
• More curriculum-oriented
• Grades 4 and 8
• Cognitive domains: Knowing, Applying, Reasoning.
• Content domains

8. TASK
• Breakout rooms
(depends on size
of the group)
• Look at these two
tasks from
respectively PISA
and TIMSS. What
literacy aspects
can you distil
from these two
tasks?
• Discuss
TIMSS
PISA

9. SHARING
Can be in audio/video or chat

10. PISA - context
“A logical consequence of wanting to make a fair
international test is that an item cannot be used if it
behaves in an “unfair” fashion. While this is a sensible
argument from a statistical point of view, it also implies that
items that are too close to real life contexts of some
countries, but not in others, have to be removed.”
(Sjøberg, 2019)

11. Different contexts
• Kreiner and Christensen (2013)
• Focussed on reading
• Differential item functioning (DIF) is a statistical
characteristic of an item that shows the extent to which
the item might be measuring different abilities for
members of separate subgroups.
• “Our analyses provide strong evidence of misfit of the
PISA scaling model and very strong evidence of DIF.”

12. TIMSS-PIRLS relationship report
• In 2011 same grade 4
sample sat TIMSS and
PIRLS (reading equivalent
of TIMSS).
• Could study the relationship
between reading and
mathematics.
• Profiles of countries
• The impact of reading ability
• Effective schools
• Home background
https://timssandpirls.bc.edu/timsspirls2011/international-database.html

13. The impact of reading ability
Reading demand – indicators
• Number of words.
• Symbolic language. These can include numerals (e.g., 3, 5,
40) as well as other symbols and abbreviations (e.g., +, =,
cm).”
• Vocabulary. The use of particular vocabulary terms can
contribute to reading demand
• Visual displays. Visual displays included the following: 1)
pictorial representations of real world things, 2) geometric
shapes and figures, 3) models and diagrams, 4) tables, and
5) graphs.

14. (Martin et al., 2013)

15. (Martin et al., 2013)

16. (Martin et al., 2013)

17. (Martin et al., 2013)

18. (Martin et al., 2013)

19. Findings
• On average across countries, the mathematics
achievement difference between poor and good readers
was larger on the high reading demand items than on the
low reading demand items.
• While the poorest readers consistently achieved at a
lower level in mathematics than the best readers, they
were additionally disadvantaged on the mathematics
items that required more reading.
• For most countries, better readers have a significantly
greater advantage over poorer readers on mathematics
items with high reading demands.

20. Secondary school
Mathematics
Items (N=90)
Number Mean Maximum
Total words 27 91
Symbols 11 36
Vocab 1 9
Visual displays 7 49
• Worked on an equivalent
‘reading demand’ classification
for grade 8.
• However, note this is not the
same sample.
• Simply focussed on achievement
for tasks with a higher reading
demand.
• Achievement better for lower
reading demand.
• Correlation Home Educational
Resources (proxy for SES) with
reading demand, stronger for low
reading demand item
achievement.
(Bokhove & Watson, still in prep after all those years)

21. TASK
• Breakout rooms (depends on size of the group)
• What do you think the implications are for teachers and
teacher educators?
• Has mathematics become too wordy?
• Discuss.

22. SHARING
Can be in audio/video or chat (pollev?)

23. Conclusions
• Language plays a big role in mathematics now.
• This has differential effects on students.
• This is important to know when interpreting
results of international largescale assessments.
• Is it desirable that mathematical tasks so wordy?
• Maybe too much influence?
• Balance of symbolic maths and applied maths.
• Awareness is a first step.

24. Thank you - Questions
• C.Bokhove@soton.ac.uk
• Southampton Education School
• Twitter: @cbokhove
• Website: www.bokhove.net
(I don’t blog so often)
• Check out my research column in
TES magazine as well.

25. References
• Arcavi, A. (1994). Symbol sense: Informal sense-making in formal mathematics. For
the learning of Mathematics, 14(3), 24-35.
• Clement, J [1982] Algebra word problem solutions: thought processes underlying a
conunon misconception. Journal for Research in Mathematics Education, Vol 13(1), pp.
16-30.
• Fuchs, Lynn S., Douglas Fuchs, Donald L. Compton, Carol L. Hamlett, and Amber Y.
Wang. 2015. “Is Word-Problem Solving a Form of Text Comprehension?” Scientific
Studies of Reading 19: 204–223. doi: 10.1080/10888438.2015.1005745
• Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness. A new
look at the PISA scaling model underlying ranking of countries according to reading
literacy. Psychometrika, 79(2), 210–231. https://doi.org/10.1007/s11336-013-9347-z
• Martin, M. O., & Mullis, I. V. (2013). TIMSS and PIRLS 2011: Relationships among
Reading, Mathematics, and Science Achievement at the Fourth Grade--Implications for
Early Learning. International Association for the Evaluation of Educational
Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands.
• Pimm, D. (2019). Routledge Revivals: Speaking Mathematically (1987):
Communication in Mathematics Clasrooms (Vol. 4). Routledge.
• Sjøberg, S. (2019). The PISA-syndrome–How the OECD has hijacked the way we
perceive pupils, schools and education. Confero: Essays on Education, Philosophy and
Politics, 7(1), 12-65.