Changing issues, changing questions, changing approaches An overview of research in the Alternative Admissions Research Project

An overview of research in the Alternative Admissions Research Project

The late 1980s to about 2001 The main questions were about: Access Finding ways to identify talented students from poor schools (focus on the DET)

The main questions were about:

Access

Finding ways to identify talented students from poor schools (focus on the DET)

(extract: Placement Test in English for Education Purposes construct) aim to predict the performance of candidates in future situations in which language will be an important, but not sole, variable (language-as-vehicle rather than language-as-target); acknowledge the effects on cognitive functioning of the quantity and quality of prior mediated learning opportunities experienced by an individual (i.e. take seriously the impact of educational disadvantage by building in mediated learning opportunities). be based on a notion of knowing and learning which views learners as actively involved as individuals and in collaboration with others, in creating and negotiating meaning in a wide variety of settings. This process of conceptual development is seen as highly dependent on specific areas of expertise involving knowledge and information, and on the connections between these. be based on a componential model of language ability, which comprises topical knowledge and language knowledge, mediated by strategic competence (metacognitive strategy use) and affective schemata.

aim to predict the performance of candidates in future situations in which language will be an important, but not sole, variable (language-as-vehicle rather than language-as-target);

acknowledge the effects on cognitive functioning of the quantity and quality of prior mediated learning opportunities experienced by an individual (i.e. take seriously the impact of educational disadvantage by building in mediated learning opportunities).

be based on a notion of knowing and learning which views learners as actively involved as individuals and in collaboration with others, in creating and negotiating meaning in a wide variety of settings. This process of conceptual development is seen as highly dependent on specific areas of expertise involving knowledge and information, and on the connections between these.

be based on a componential model of language ability, which comprises topical knowledge and language knowledge, mediated by strategic competence (metacognitive strategy use) and affective schemata.

The main assessment related challenges were: producing a greater range of scores (spread), so that capable students could be more clearly differentiated from weaker students improving (raising) the level of stronger students’ scores increasing the predictive validity of the test (that is, did the test correctly distinguish between weaker and stronger students)

producing a greater range of scores (spread), so that capable students could be more clearly differentiated from weaker students

improving (raising) the level of stronger students’ scores

increasing the predictive validity of the test (that is, did the test correctly distinguish between weaker and stronger students)

Towards ‘dynamic’ testing: the old ELPT and the new PTEEP The same three major writing tasks (summary, description and contrast/comparison) The same additional text types (graph and diagram) The PTEEP included structured and sequenced tasks designed to act as mediation for the writing assignments. The ELPT did not include these ‘scaffolding’ tasks. The same prose texts Differences Similarities

The same three major writing tasks (summary, description and contrast/comparison)

The same additional text types (graph and diagram)

The same prose texts

ASSESSING THE IMPACT OF THE SCAFFOLDING APPROACH Questionnaire evidence from pilot groups suggests that candidates found the new test to be more ‘user-friendly’. Correlations between first-year academic (UCT) performance and the selection tests strengthened. Stronger candidates use preparatory tasks whereas weaker candidates do not see the connections between tasks (such tasks therefore help to widen the gap). Based on scores from the same tasks (summary, short description and one page essay), the range of scores increased. Candidate perceptions Predictive validity Task preparation study Range of scores

ELPT and PTEEP Scores

Quantitative approaches used at this stage and in early 2000s Problems with correlational analysis A move to ‘survival analysis’ Classical item analysis Psychometrically ‘naïve’ tests relying on ranking – limitations for score use Reliance on group performance for ‘standards’

Problems with correlational analysis

A move to ‘survival analysis’

Classical item analysis

Psychometrically ‘naïve’ tests relying on ranking – limitations for score use

Reliance on group performance for ‘standards’

The early 2000s Expansion of “AARP” testing Introduction of Quantitative Literacy test Major innovations in the area of test delivery (use of innovative technology etc) Pressure on tests reduced ‘potential’ aspect Development of Health Science Consortium group of tests

Expansion of “AARP” testing

Introduction of Quantitative Literacy test

Major innovations in the area of test delivery (use of innovative technology etc)

Pressure on tests reduced ‘potential’ aspect

Development of Health Science Consortium group of tests

2005 -> the main questions … Concerns largely shifted from access to throughput Placement function emphasised – implications for curriculum (focus mainly on extended progs) Concerns about the new school curriculum and qualification – target now all applicants (mass test) Increasing tensions with Department of Education (with stronger FET curriculum links)

Concerns largely shifted from access to throughput

Placement function emphasised – implications for curriculum (focus mainly on extended progs)

Concerns about the new school curriculum and qualification – target now all applicants (mass test)

Increasing tensions with Department of Education (with stronger FET curriculum links)

Issues and challenges Academic literacy/ies construct widened to include greater focus on quantitative literacy (and reading not writing the focus) For the first time direct testing in the school curriculum (Mathematics & Maths Literacy) Radically different development trajectories, research traditions for the new NBTs (achievement tests) and TAPs (re-designing the old PTEEPs)

Academic literacy/ies construct widened to include greater focus on quantitative literacy (and reading not writing the focus)

For the first time direct testing in the school curriculum (Mathematics & Maths Literacy)

Radically different development trajectories, research traditions for the new NBTs (achievement tests) and TAPs (re-designing the old PTEEPs)

Research approaches: quantitative From Classical Test Theory to Item Response Theory Psychometric test development approaches

From Classical Test Theory to Item Response Theory

Psychometric test development approaches

CTT vs IRT (Hambleton and Jones, Comparison of CTT and IRT and their applications to test development)

Item and Test Characteristic Curves; Test Information Functions and Equating

THE NATIONAL BENCHMARK TESTS PROJECT

THE NATIONAL BENCHMARK

TESTS PROJECT

Why did HESA commission the NBTP? Demonstrable inefficiencies in Higher Education itself (low throughput etc.) Difficulties in identifying students’ educational needs Lack of appropriate curriculum flexibility at entry to meet these needs Concerns about how to interpret the new NSC

Demonstrable inefficiencies in Higher Education itself (low throughput etc.)

Difficulties in identifying students’ educational needs

Lack of appropriate curriculum flexibility at entry to meet these needs

Concerns about how to interpret the new NSC

In summary …. The NBTP is about higher education getting its own act in order – it is not about pointing fingers at the school system It sets out to do this by providing information about the competence of entering students in terms of 3 core domains of knowledge / skills it is important to note that higher education’s ‘take’ on what these core sets are, and at what level they should be mastered, will in all probability differ somewhat from those deemed most salient by the school-leaving system.

The NBTP is about higher education getting its own act in order – it is not about pointing fingers at the school system

It sets out to do this by providing information about the competence of entering students in terms of 3 core domains of knowledge / skills

it is important to note that higher education’s ‘take’ on what these core sets are, and at what level they should be mastered, will in all probability differ somewhat from those deemed most salient by the school-leaving system.

What do the NBTs aim to do? Provide additional information about performance in core, underlying areas (additional to NSC information) The core (domain) areas are: Academic literacy one 3-hr test Quantitative literacy Mathematics one 3-hr test

Provide additional information about performance in core, underlying areas (additional to NSC information)

The core (domain) areas are:

Academic literacy one 3-hr test

Quantitative literacy

Mathematics one 3-hr test

Example - Competency Specification Proportions NBT Quantitative Literacy Competence area Knowing Applying routine procedures in familiar contexts Applying multistep procedures in a variety of contexts Reasoning and reflecting 1+ 1- 2+ 2- 3+ 3- 4+ 4- n % Comprehending: identifying or locating Vocabulary 5 0 7 7 2 4 1 5 31 19.0 15-20 Representations of numbers and operations 1 0 1 2 1 1 1 0 7 4.3 5-10 Conventions for visual representations 4 0 10 6 1 5 1 5 32 19.6 20-25 Acting, interpreting, communicating Using representations of data 2 0 6 3 1 4 1 4 21 12.9 20-25 Computing (and estimating?) 0 0 4 10 2 4 0 2 22 13.5 15-20 Conjecturing 0 0 0 0 0 0 0 1 1 0.6 0-5 Interpreting 5 0 10 5 3 5 2 5 35 21.5 10-15 Reasoning 0 0 1 0 0 0 1 4 6 3.7 5-10 Representing quantitative information 0 0 3 1 0 0 2 0 6 3.7 5-10 Describing quantitative relationships 0 0 1 0 1 0 0 0 2 1.2 0-5 n 17 0 43 34 11 23 9 26 163 % 10.4 0.0 26.4 20.9 6.7 14.1 5.5 16.0 % 10.4 47.2 20.9 21.5

Mathematical and statistical ideas Quantity, number and operations (LO1) 1 0 6 9 3 4 1 2 26 31.0 25-30 Shape, dimension and space (LO3) 3 0 4 3 0 2 1 1 14 16.7 10-15 Relationships, pattern, permutation (LO2a) 0 0 5 2 1 0 2 1 11 13.1 10-15 Change and rates (LO2b) 0 0 4 2 1 0 1 1 9 10.7 10-15 Data representation and analysis (LO4a) 2 0 5 3 1 4 0 6 21 25.0 20-25 Chance and uncertainty (LO4b) 0 0 1 0 0 2 0 0 3 3.6 5-10 n 6 0 25 19 6 12 5 11 84 % 7.1 0.0 29.8 22.6 7.1 14.3 6.0 13.1 % 7.1 52.4 21.4 19.0 15-20 25-30 25-30 25-30

How are the benchmark (cut-off points) derived? The process is fundamentally different to the examination paper design procedures, and the norm referenced standardising and resulting processes of the NSC All items need to have been through rigorous review (fairness, content etc) and be statistically robust All items need to have been piloted The benchmark setting process is NOT about whether students can pass an item or not - the process is based on a set of probability assessments made by first year lecturers, with the core questions being: “ if a student can’t pass this item / do this, will s/he experience academic difficulties – and if so, how severe?”

The process is fundamentally different to the examination paper design procedures, and the norm referenced standardising and resulting processes of the NSC

All items need to have been through rigorous review (fairness, content etc) and be statistically robust

All items need to have been piloted

The benchmark setting process is NOT about whether students can pass an item or not - the process is based on a set of probability assessments made by first year lecturers, with the core questions being:

“ if a student can’t pass this item / do this, will s/he experience academic difficulties – and if so, how severe?”

NBT information INDIVIDUAL LEVEL Benchmark level (Basic, Intermediate, Proficient) Description of what this means for each domain (ie what does being in the ‘Basic’ category mean a student knows and can do in Mathematics) Clear recommendations about the type and extent of support needed GROUP LEVEL At the level of a faculty, or qualification, or institution …. Give clear indication of the needs and strengths of entering cohorts, either before entry, or at registration: useful for placement into existing courses, and/or with course design or modification.

INDIVIDUAL LEVEL

Benchmark level (Basic, Intermediate, Proficient)

Description of what this means for each domain (ie what does being in the ‘Basic’ category mean a student knows and can do in Mathematics)

Clear recommendations about the type and extent of support needed

GROUP LEVEL

At the level of a faculty, or qualification, or institution ….

Give clear indication of the needs and strengths of entering cohorts, either before entry, or at registration: useful for placement into existing courses, and/or with course design or modification.

DATA BASED ON FEB 2009 PILOTS

DATA

BASED ON

FEB 2009 PILOTS

ACADEMIC LITERACY (overall) N = 12,202 Participating institutions: Mangosuthu, Rhodes, Stellenbosch, UCT, UKZN, UWC, Wits. Serious learning challenges – long term, pre-tertiary intervention needed. Challenges identified such that it is predicted that academic progress will be adversely affected. If admitted, students’ educational needs should be met in a way deemed appropriate by the institution (eg extended or augmented programmes, special skills provision) Performance such that academic performance will not be affected. If admitted, students should be placed on regular programmes of study. 851 5571 5780 0 1000 2000 3000 4000 5000 6000 7000 Total ACADEMIC LITERACY NBT Benchmark Levels, February 2009 Basic Intermediate Proficient

ACADEMIC LITERACY by Faculty

Quantitative Literacy N = 12,202

Quantitative Literacy by Faculty

Mathematics (overall)

Mathematics overall [Intermediate ‘Top’ and ‘Bottom’]

Mathematics by Faculty