This article summarizes the evaluation of the CALL product Longman English Online 3 (LEO 3) using a set of principles and criteria. The evaluation examined how LEO 3 operationalized criteria related to language learning potential, interaction, production, and assessment. It was found that LEO 3 received good ratings for enhanced input, interaction between people and computers, planning production, and the construct validity of its assessment. Some areas like elaboration and correcting production received fair ratings. The evaluation concluded LEO 3 was good overall but identified opportunities for improvement and suggested further empirical evaluation.

1. Joan Jamieson’s
Putting Principles
into Practice
an article summarized by:
SEFUL BAHRI
16705261003
Yogyakata State University student

2.  This article is written by Joan Jamieson, Carol A
Chappele, and Sherry Preiss
Purpose of the study
To illustrate how a set of principles
informed a series of processes in the
evaluation of Longman English Online, a
commercial CALL product for teaching
intermediate English as a second or foreign
language (ESL/EFL).

3.  CALL Product to Evaluate:
 Longman English Online 3 (LEO 3). it’s the
intermediate level of the video-based multimedia,
integrated language skills program.
 The units not only contain listening comprehension
practice, but also include practice with grammar,
vocabulary, pronunciation, speaking, reading,
research on the web, and writing.
 Quizz in every unit, longer tests at the end of every
four units (“module test”), and one at the end of all
twelve units (“the end –of-level test).

4.  Nature of the evaluation :
 The evaluation was focused on the materials
rather than on the tasks that the leaners actually
carried out as they were using the materials
 It was judgmental evaluation (: based on the
logical analysis of a CALL activity), NOT empirical
evaluation (: based on the quantitative or
qualitative analysis of a CALL activity through
observed data which are summarized by the
evaluator)

5. CALL Evaluation Principles and their associated
criteria & vriables
Principles Criteria Variables Examples
1. Language
Learning
potential should
be the central
concern when
developing &
evaluating
* to indicate CALL
quality. (Chapele.
2001)
Enhanced
input
Input saliece,
modification,
elaboration
Text
highlighted on
the screen
interaction Between people,
between a person &
computer, within a
person
Learner-learner
communication
tasks, grammar
help available
production Planning, correcting
production, using
help
Taking notes
suggested,
model for
constructed
responses

6. Principle Criteria Variables Examples of
Operationalized
variables
2. CALL should
be evaluated in
terms of the
quality
assessment in
addition to
instruction
*to indicate the
usefulness of CALL
test. (Bachmer &
Palmer)
Authenticity Domain
Sampling
Correspondence
between tasks on
the test and in
courseware
Construct
Validity
Score
interpretatio
n
Meaningfulness of
scores

7. Principle Criteria Variables Examples of
Operationalize
d variables
3. The criteria to
conduct the analysis
should be drawn
from theory and
practice in SLA &
Lang. Assessment
4. Judgemental
evaluation through
defined variables is
valable for
examining materials
in a way that can
improve future
design

8.  All of the previously mentioned criteria
were examined, to find whether or not
they had been operationalized in LEO 3

9.  These criteria were investigated as they have a
relatively strong theoretical, pedagogical,
and empirical base:
 Theory and research on SLA converge to
suggest that the likelihood of learners acquiring
linguistic input increases if their attention is
drawn to linguistic features, using techniques
such as marking salience, modification, or
elaboration. (Schmidt, 1990; Sharwood Smith,
1993; Robinson, 1995; Skehan, 1998).
 Benefits and types of interaction for langugage
learning (Ellis, 1999; Kol & Schcolnik, 2000;
Lantolf & Appel, 1994, Long, 1996, Pica, 1994)

10. Swain (1985): From the cognitive
perspective, the benefits of producing
language may be enhanced when learners
have the opportunity to plan before
speaking or writing and to correct
linguistic output, which can be prompted
by feedback from others or from self-
evaluation.
From sociocognitive perspective, Swain
(1998) stated that the help from the
interlocutor allows for production
beyond what the learner can accomplish
alone.

11. Authenticity is explained as the degree
of correspondence between the
characteristics of a test-task and the
features of a task in “target language
use” (Bachman & Palmer, 1996).
Achievement test is considered authentic
in terms of the degree to which its tasks &
directions mirror the content & skills in an
insructional unit (:domain sampling)
Construct Validity dals with the
appropriateness and meaningfulness of
the interpretations which are made on the
basis of a test score.

12.  Summary of Judgemental Evaluation of LEO 3
Area Criteria Variables Operationai
zations
Evidence Judgem
ent
Languag
e
Larning
Potential
Enanced
input
salience Font
size/type/colo
r,
highlighting,
repetition
Font
size/highlig
hting
graphics,
animation,
audio
repetition
Good
modification Simplification
, repetition,
clarification,
L1 transl., L2
definitions,
image/video
Images/vid
eo,
transcripts,
L2
definitions
Good
elaboration Adding
grammatical
phrases or
clauses to
texts
Adding
grammatica
l phrases
Fair

13. Language
Learning
Potential
Interac
tion
between
people
Amount/ty
pes of co-
constructe
d and
attention
to form
Meaningf
ul tasks
Good
People and
computer
Potential of
providing
enhanced
input,
obtaining
help using
the
language
Program
correction
of
learner’s
explanatio
ns, culture
notes
Good
an inner
voice
choices

14. Language
Learning
Potential
Productio
n
planning Instructions
to plan,
directed
activities,
knowledge of
expectations
Directions
to plan,
pre-writing
exercises
Good
correcting Amount of self-
correction,
amount/type of
signals from
computer or
another person
Self-
correction of
speaking/
pronunciatio
n teacher
Fair
help Accessing
dictionary,
grammar,
models
Glossaries,
transcripts,
grammar
help, models
of speaking
and writing
Good

15. CALL
assessment
Athenticity
Construct
validity
domain
sampling
Correspon
dence
between
tasks on
tet & in
coursewar
e
Table of
specifica
tions,
direction
s, texts,
item
types
Fair
score
interpretati
on
Appropriate,
meaningful
Reporting
scheme
Good

16. CONCLUSION
based on judgments on the criteria of
language learning potential as well as
authenticity and construct validity of
assessment, LEO 3 received a good
evaluation.
This evaluation also points to areas in
which LEO 3 might be improved in the
future and it suggests that LEO 3 might
be worth pursuing further through the
addition of an empirical evaluation.

17. === end of the slides ===