The Cambridge companion to language assessment by Thom Hudson chapter 34 prepared by Shaho Hoorijani

1. In Lord We Trust
Language Assessment
Topic: Criterion-Referenced Approach to Language Assessment
Thom Hudson
Prepared by Shaho Hoorijani

2. Presentation Outline
• Introduction
• Background
• CRM Interpretation Assessments
• Characteristics Which Differentiate Criterion-Referenced
from Norm-Referenced Tests
• CRM Projects that are applied to Different Scale Contexts
• Challenges in the Development of CRM approach
2

3. Introduction
Criterion-Referenced Approach to Language Assessment
Definition of keyterms
• Criterion:1660s, from Latinized form of Greek kriterion "means
for judging.
• Criterion: represents the underlying skill or knowledge base to
which the test is being generalized, not necessarily the level of
performance.
• Referenced: Something for which a name or designation stands.
• Domain: A set of skills or dispositions that examinees display when
called to do so in a testing situation
3

4. Introduction
Characteristics of Criterion-Referenced Approach to Language Assessment
• Measures student performance against a fixed set of predetermined
criteria or learning standards.
• Determine which examinee have mastered a particular domain of
knowledge or skill.
• Measures degree of competence attained by a particular student which
is independent of reference to the performance of others.
• The CRM test assesses the content and skill standards for the domain
of interest.
4

5. Background
Why Criterion-referenced Language assessment developed?
• It is a reaction against the misuse of psychometric methods (especially
reliability theory) that were developed for assessing individual differences
in aptitudes and abilities....
• Items that are answered correctly have lower discrimination indexes, they
were often eliminated solely on that empirical criterion....
• Many achievement test developers lost sight of content validity － that the
items must first and foremost be representative of the domain (content) and
possess objectives to be assessed, and focused on high reliability as an end
in itself. (pp.184-185 emphasis in original)
Hopkins, Stanley and Hopkins (1990)
5

6. Background
Why Criterion-referenced Language assessment developed?
The growing influence of outcomes-based and competency-based assessment
has raised an awareness of the need for language assessment to reflect the
specific learning objectives of a program or the particular abilities needed for
language proficiency certification. The need to assess targeted features of a
domain is the focus of CRM.
6

7. Background
Test Administration Reasons
• Norm-referenced interpretation
• Criterion-referenced interpretation
Which one is better?
Depends on the desired assessment
7

8. Background
Classroom Setting
• CRM assessment is the primary form of assessment carried out by teachers in
classroom settings.
• Most language teachers are less interested in how their students rank in relation to
other students than in what their students can do with the language feature that
has been taught.
• The lack of descriptive clarity resulting from NRM tests does not provide teachers
with clear objectives at which to aim specific instruction.
8

9. Background
Criterion-referenced in 1960s and 1970s
Behavioral psychology, audiolingualism, structural linguistics, and learning systems:
– Its identification with the atomization of skills into discrete units and narrow
instances of learned information.
– It was not useful for higher order skills or complex learning
Change from focus on narrow skills to broader concepts
– Cognitive language processing and sociolinguistic ability
– Importance of performance, task-based or authentic assessment
– Assessment tasks that reflect the real-world criterion
– Attention to qualifications regarding modality, social status, and language
complexity
9

10. Characteristics Which Differentiate Criterion-Referenced
from Norm-Referenced Tests
1.Test Purpose
NRT
• The score is interpreted in terms of other examinees score
• These tests are predominantly used for admission and placement purposes
CRT
• CRM is absolute rather than relative
• Tests are given to determine how much course material a student has learned and/or
tasks are assigned and evaluated to see if they have been completed to established
standards
10

11. Characteristics Which Differentiate Criterion-Referenced
from Norm-Referenced Tests
2.Test Content and Structure
NRT
• Very general, and students should have no idea of what will be on the test.
• A few relatively long subsets with heterogeneous item content in each test
CRT
• Measure specific well-defined domain
• The content should be drawn from the course, and students should have a good
idea of what they will encounter on the test.
• The content of the tests are constrained by the results of the needs analysis
• Short, well-defined subsets with homogeneous item content in each subset.
11

12. Characteristics Which Differentiate Criterion-Referenced
from Norm-Referenced Tests
3.Test Development
NRT
• Test development is primarily psychometric driven
• Importance is placed on statistical considerations ( item discrimination and score
distribution)
• Designed to providing instruments that allow comparisons among examinees
12

13. Characteristics Which Differentiate Criterion-Referenced
from Norm-Referenced Tests
3.Test Development
1. Scores on tests that are interpreted in terms of percentiles, such as the Test of English
as a Foreign Language (TOEFL), would not be considered CRM. This is perhaps the
least satisfactory meaning of the term in that it does not relate the measurement to a
specified domain.
2. A focus of this sense of CRM is on the decision to be made and the cut score
that is established. Such a test should be based on a clear real-world activity,
perhaps identified through a needs analysis.
3. Status on the criterion is assessed by sampling the well-defined domain and
setting a cut score.
4. This level refers to proficiency levels that are the basis for the criterion
referencing. The levels of proficiency are viewed as cumulatively relating to
development. Each stage of development would thus have a real-world
proficiency dimension and a relationship to other stages of development. 13

14. Examples of CRM Projects
This projects show examples of assessment attempts to provide tests designed to
specify what examinees can do in relation to the criteria developed for each
project.
1. ACCESS for ELLs test
((Assessing Comprehension and Communication in English State-to-State for
English Language Learners ))
• Assessing proficiency levels in foreign language learning
• Designed ELPS for ELLs in Kindergarten through grade 12 (K-12) developed
by WIDA Consortium.
What is WIDA acronym stands for?
It stands for World-class Instructional Design and Assessment
14

15. Examples of CRM Projects
WIDA and the ELP Standards
– The ACCESS for ELLs is the test operationalization of the WIDA ELPS
frame- work
– The ELPS are the basis for test and item specifications.
– Items written from the PIs reflect the form of the academic language
requirements in the standards.
What is the responsibility of WIDA consortium?
• Promote the development of assessment and educational materials for ELLs
• Create English language proficiency standards for ELLs that were tied to
academic content standards
• Assessments of ELLs in K-12 annually on their language proficiency
15

16. Examples of CRM Projects
WIDA and the ELP Standards
Three-dimensional ELP Constructs in WIDA:
1. Language complexity
2. Cognitive engagement
3. Context
ELPS reflects the:
language needed for the classroom and school functions, and
Language for learning in school content areas
Learning the language and learning school contents concurrently in 4 skills domain
16

17. Examples of CRM Projects
WIDA and the ELP Standards
ELPS Frameworks :
a) Large-scale state assessment
b) Classroom assessment
Common features of these two framework:
1. English language Proficiency standards
2. Language domains
3. Grade level clusters
4. Language proficiency levels
17

18. Five WIDA ELP Standards
Standard 1- SIL: English language learners communicate for SOCIALAND
INSTRUCTIONAL purposes within the school setting.
Standard 2 – LoLA: English language learners communicate information, ideas,
and concepts necessary for academic success in the content area of LANGUAGE
ARTS.
Standard 3 – LoMA: English language learners communicate information, ideas,
and concepts necessary for academic success in the content area of
MATHEMATICS.
Standard 4 – LoSC: English language learners communicate information, ideas,
and concepts necessary for academic success in the content area of SCIENCE.
Standard 5 – LoSS: English language learners communicate information, ideas,
and concepts necessary for academic success in the content area of SOCIAL
STUDIES.
18

19. Examples of CRM Programs
2. The Common European Framework of Reference (CEFR)
– It describes in a comprehensive way what language learners have to learn to do
in order to use a language for communication and what knowledge and skills
they have to develop so as to be able to act effectively.
– The description also covers the cultural context in which language is set.
The Framework also defines levels of proficiency which allow learners’
progress to be measured at each stage of learning and on a lifelong basis.
– The text of the CEFR emphasizes learner’s ‘communicative needs’, including
dealing with the business of everyday life, exchanging information and ideas,
and achieving wider and deeper intercultural understanding. This is to be
achieved by ‘basing language teaching and learning on the needs, motivations,
characteristics and resources of learners.’
19

21. Examples of CRM Programs
3. Assessment of Language Performance (ALP)
– A small-scale project designed to use CRM approach in task-based
language assessment
– Provide examples of an array of possible task contexts in general topics
to demonstrate how task-based assessment can be developed
– The focus is on the functions of the real-world tasks on examinee’s
language ability in use
– Task-based test are goal-oriented use outside of language test situation
– Predetermined real-world criterion elements
21

22. Examples of CRM Programs
3. Assessment of Language Performance (ALP)
– Task difficulty is controlled through:
a) Language code complexity
b) Cognitive complexity: adding additional material not relevant to the completion
of the task
c) Communicative stress
– Task factors varied due to their difficulty
Needs Analysis plays an important role in task criteria
22

23. Challenges in the Development of CRM approach
– Domain Specification, and Standards identification which causes:
– Domain overspecifications
– Too many item specifications
– content standards may not be written in a way that facilitates the generation of
items.
– Hindering effects of item specification curricular change. That is, once the
framework for assessment is refined, there can be a reluctance to make
changes in the educational approach that will necessitate a reworking and
rethinking of the assessment system.
23

24. Challenges in the Development of CRM approach
Solution
Test and item writers must collaborate with teachers,
administrators, and other stakeholders in clarifying
what specific types of examinee performances will lead
to inferences about the examinee’s language and ability
to engage in scientific inquiry.
24

25. Thank You
25