Research colloquium presentation at the Language Testing Research Colloquium, Auckland, New Zealand, July 2018

1. A NATIONAL ASSESSMENT OF
ACADEMIC LITERACY AND ITS
TRANSFORMATORY ROLE IN THE
DEVELOPMENT OF DISCIPLINARY
TEACHING
Alan Cliff, University of Cape Town, July 2018

2. The South African context
 Massification of higher education off a ‘low’ base
 Participation rates under 20% of school-leaving population
 Hugely unequal rates of participation by demographic
background
 High levels of diversity by language, school background,
readiness
 Difficult to interpret school-leaving examination results

3. Different purposes of AL testing
 As a contribution towards applicant selection
 Towards placement in a particular kind of curriculum:
foundational; augmented; skills-based; reduced-load
 For diagnostic purposes
 For teaching purposes
 ‘Snapshot’ vs longitudinal purposes
 Generic vs disciplinary purposes

4. The National Benchmark Tests Project (NBTP)
HESA, 2006
 To assess entry-level academic and quantitative literacy and mathematics
proficiency of students
 To assess the relationship between Higher education (HE) entry level
requirements and school-level exit outcomes
 To provide a service to HE institutions requiring additional information to
assist in admission (selection and placement) of students in appropriate
curricular routes (regular, extended, augmented, or any other routes)
 To assist with curriculum development, particularly in relation to
foundation and augmented courses (or similar)

5. What is Academic Literacy?
Academic literacy1 here refers to the ability of students to
negotiate the grammatical and textual structure of the language
of instruction and to understand its functional and sociolinguistic
bases.
In a higher education context, this means:
1 Cliff, Weideman, Sebolai, Van Dyk, Scholtz, Yeld

6. What is Academic Literacy?
 negotiate meaning at word, sentence, paragraph and whole-
text level;
 understand discourse and argument structure and the text
“signals” that underlie this structure;
 extrapolate and draw inferences beyond what has been stated
in text;
 separate essential from non-essential and super-ordinate from
sub-ordinate information;

7. What is Academic Literacy?
 understand and interpret visually encoded information, such as
graphs, diagrams and flow-charts;
 understand and manipulate numerical information;
 understand the importance and authority of “own voice”;
 understand and encode the metaphorical, non-literal and
idiomatic bases of language;

8. What is Academic Literacy?
 negotiate and analyse text genre.

9. Language Knowledge (Bachman and
Palmer 1996)
Language Knowledge Specifications (Yeld 2001)
Grammatical
Vocabulary
Morphology
Syntax
Vocabulary: ‘unknown’ vocabulary (deriving
meanings from context); ‘known’ vocabulary (i.e. no
context provided); spelling as it affects meaning
Syntax: understanding the syntactic basis of the
language
Organisational
Knowledge Textual
Cohesion
Rhetorical
organisation
Understanding relations between parts of text (e.g.
through devices of cohesion such as pronoun
reference, particularly demonstratives, referring to
statements/propositions or ‘entities’, and/or by
recognising indicators in discourse, especially for
introducing, developing, transition and conclusion of
ideas, and signalling relations between phenomena).
Skimming and scanning (e.g. using macro features of
text such as headings, illustrations) to get gist of
passage, locating particular pieces of information
Extrapolation and application (e.g. drawing
conclusions/applying insights derived from texts,
seeing trends)
Inferencing: (understanding ideas/information in a
text, implied but not explicitly stated).

11. NBT AL specification
Vocabulary Students’ abilities to derive/work out word meanings from
their context
Metaphorical Expression Students’ abilities to understand and work with metaphor
in language. This includes their capacity to perceive
language connotation, word play, ambiguity, idiomatic
expressions, and so on
Extrapolation, application and inferencing Students’ capacities to draw conclusions and apply
insights, either on the basis of what is stated in texts or is
implied by these texts.
Understanding the communicative function of sentences Students’ abilities to ‘see’ how parts of sentences /
discourse define other parts; or are examples of ideas; or
are supports for arguments; or attempts to persuade

12. NBTP AL specification
Understanding relations between parts of text Students’ capacities to ‘see’ the structure and organisation
of discourse and argument, by paying attention – within
and between paragraphs in text – to transitions in
argument; superordinate and subordinate ideas;
introductions and conclusions; logical development
Understanding the grammatical / syntactical basis of
language
Students’ abilities to ‘see’ / analyse the way in which
sentence structure / word, phrase order affects meaning
and emphasis in language
Understanding text genre Students’ abilities to perceive ‘audience’ in text and
purpose in writing, including an ability to understand text
register (formality / informality) and tone (didactic /
informative / persuasive / etc.)

13. NBTP AL specification
Separating the essential from the non-essential Students’ capacities to ‘see’ main ideas and supporting
detail; statements and examples; facts and opinions;
propositions and their arguments; being able to classify,
categorise and ‘label’
Understanding information presented visually Students’ abilities to understand graphs, tables, diagrams,
pictures, maps, flow-charts

14. Reproducing orientation Transformative orientation
Level 1:
Knowing
Level 2:
Routine application
Level 3:
Multi-step
application
Level 4:
Reasoning and
reflecting
1+ 1- 2+ 2- 3+ 3- 4+ 4-
Vocabulary
(10%)
Metaphorical
(15%)
Inferencing
(15%)
Communicati
ve function
(15%)
Relations
cohesion
(5%)
Relations
discourse
(10%)
Grammar/synt
ax
(5%)
Text genre
(5%)
Essential /
non-essential
(20%)
Number of
items

15. 100% Performance in domain areas suggests that academic
performance will not be adversely affected.
If admitted, students should be placed on regular
programmes of study.
Proficient
Intermediate
Challenges in domain areas are identified such that it is
predicted that academic progress will be affected.
If admitted, students’ educational needs should be met in a
way deemed appropriate by the institution (e.g. extended or
augmented programmes, special skills provision).
Basic
Serious learning challenges are identified: it is predicted that
students will not cope with degree level study without
extensive & long-term support, perhaps best provided
through bridging programmes or FET colleges.
Institutions registering students performing at this level
would need to provide such support.
0%
NBT Performance Levels
15

23. Essential / non-essential
cluster
1
47
52
0
61
39
0
53
47
1
42
57
1
35
64
0 25 50 75 100
Percentage
All writers
Proficient
Intermediate upper
Intermediate lower
Basic
Essential/non-esential cluster by benchmark category
Wrong Correct NoResponse

24. 0.0
20.0
40.0
60.0
80.0
100.0
3 10 16 27 31 91 93 97 103 114
Test item number
Basic Int lwr Int uppr Prof All

25. Inferencing
1
50
49
0
67
33
0
59
41
1
45
55
1
30
69
0 25 50 75 100
Percentage
All writers
Proficient
Intermediate upper
Intermediate lower
Basic
Inferencing cluster by benchmark category
Wrong Correct NoResponse

26. 0.0
20.0
40.0
60.0
80.0
100.0
1 6 8 11 18 19 21 24 28 96 101 105 110 116
Test item number
Basic Int lwr Int uppr Prof All

27. Academic discourse
1
43
56
0
64
36
1
50
49
1
37
62
2
28
71
0 25 50 75 100
Percentage
All writers
Proficient
Intermediate upper
Intermediate lower
Basic
Discourse cluster by benchmark category
Wrong Correct NoResponse

28. 0.0
20.0
40.0
60.0
80.0
100.0
4 14 30 33 118
Test item number
Basic Int lwr Int uppr Prof All

29. Vocabulary
0
59
41
0
70
29
0
63
37
0
56
44
1
45
54
0 25 50 75 100
Percentage
All writers
Proficient
Intermediate upper
Intermediate lower
Basic
Vocabulary cluster by benchmark category
Wrong Correct NoResponse

30. 0.0
20.0
40.0
60.0
80.0
100.0
9 12 13 25 86 98
Test item number
Basic Int lwr Int uppr Prof All

31. Metaphorical language
4
48
48
1
73
26
3
55
42
5
41
54
6
30
64
0 25 50 75 100
Percentage
All writers
Proficient
Intermediate upper
Intermediate lower
Basic
Metaphorical expression cluster by benchmark category
Wrong Correct NoResponse

32. 0.0
20.0
40.0
60.0
80.0
100.0
2 5 22 89 102 106 112 122 123 124 125
Test item number
Basic Int lwr Int uppr Prof All

33. Textual cohesion
0
55
45
0
72
28
0
63
37
0
51
49
1
34
65
0 25 50 75 100
Percentage
All writers
Proficient
Intermediate upper
Intermediate lower
Basic
Cohesion cluster by benchmark category
Wrong Correct NoResponse

34. 0.0
20.0
40.0
60.0
80.0
100.0
20 23 26 32 117
Test item number
Basic Int lwr Int uppr Prof All