Chapter 10 Designing and Conducting Formative Evaluations
1. CHAPTER 10
Designing and Conducting Formative
Evaluations
Carolyn Jenkins-Haigler
2. BACKGROUND
A formative evaluation, evidence of an instructional program’s worth is
gathered for use in making decisions about how to revise the program while it
is being developed. This is why it is called "formative" evaluation, because the
instruction is in its developmental stages and is not yet "grown up". The idea
is to find out if your newly developed course works at teaching the objectives
you need to teach to the learners who need to learn them, before you present it
to your target audience. In any given formative evaluation, you can find out
how to make your instruction more:
Effective
Efficient
Interesting/Motivating
Usable
Acceptable
You do this by carrying out procedures that will provide you with evidence
as to the effectiveness of your instruction. The emphasis is on collecting data
and revising the instruction.
3. OBJECTIVES
Describe the purposes for and various stages of
formative evaluation of instructor-developed materials,
instructor- selected materials, and instructor- presented
instruction.
Describe the instruments used in a formative
evaluation.
Develop an appropriate formative evaluation plan and
construct instruments for a set of instructional
materials or an instructor presentation.
Collect data according to a formative evaluation plan
for a given set of instructional materials or instructor
presentation.
4. THE CONCEPT OF
FORMATIVE
EVALUATION
Definition
The collection of data and information during
the development of instruction that can be used
to improve the effectiveness of the instruction.
Purpose
To obtain data that can be used to revise the
instruction to make it more efficient and
effective.
5. THE ROLE OF SUBJECT-MATTER,
LEARNING, AND
LEARNER SPECIALISTS
It’s important to have the instruction reviewed by
specialists.
SME may be able to comment on the accuracy and
currency of the instruction.
Learning specialist may be able to critique your
instruction related to what is known about enhancing
that particular type of learning
Learner specialist may be able to provide insights into the
appropriateness of the material for the eventual
performance context.
6. THE THREE PHASES
OF FORMATIVE
EVALUATION
I. One-to-One Evaluation
II. Small-Group Evaluation
III. Field Trial
7. ONE-TO-ONE
EVALUATION
Criteria
Selecting Learners
Data Collection
Procedures
Assessments and
Questionnaires
Learning Time
Data Interpretation
Outcomes
Purpose
To identify and remove the most
obvious errors in the instruction
To obtain initial performance
indications and reactions to the
content by learners
Criteria
Clarity
Impact
Feasibility
8. CRITERIA
During the development of the instructional strategy and the
instruction itself, designers and developers make a myriad of
translations and decisions that link the content, learners, instructional
format, and instructional setting.
The one- to- one trials provide designers with their first glimpse of the
viability of these links and translations from the learners’ perspective.
The three main criteria and the decisions de-signers will make during
the evaluation are as follows:
1. Clarity: Is the message, or what is being presented, clear to
individual target learners?
2. Impact: What is the impact of the instruction on individual
learner’s attitudes and achievement of the objectives and goals?
3. Feasibility: How feasible is the instruction given the available
resources ( time/ context)?
9. SELECTING
LEARNERS
One of the most critical decisions by the designer in the
formative evaluation is the selection of learners to
participate in the study.
This is not an experiment; there is no need for random
selection of large numbers of learners.
Actually, the designer wants to select a few learners
who represent the range of ability in the group because
prior learning or ability is usually one of the major
determiners of ability to learn new skills and
information.
10. DATA COLLECTION
The first category, message, relates to how clear the basic message is
to the learner determined by such factors as vocabulary, sentence
complexity, and message structures. Regardless of whether the
learner reads, hears, or sees the message, he or she must be able to
follow it.
The second category, links, refers to how the basic message is tailored
for the learner, including contexts, examples, analogies, illustrations,
demonstrations, and so forth. When these links are also unfamiliar to
the learner, the basic message will undoubtedly be more complex.
The third area, procedures, refers to characteristics of the instruction
such as the sequence, the size of segment presented, the transition
between segments, the pace, and the variation built into the
presentation. The clarity of instruction may change for the learner
when any one of these elements is inappropriate for her or him.
11. PROCEDURES
The typical procedure in a one- to- one evaluation is to
explain to the learner that a new set of instructional
materials has been designed and that you would like his or
her reaction to them.
You should say that any mistakes that learners might make
are probably due to deficiencies in the material and not
theirs.
Encourage the learners to be relaxed and to talk about the
materials.
You should have the learners not only go through the
instructional materials but also have them take the test( s)
provided with the materials.
12. ASSESSMENTS AND
QUESTIONAIRES
After the students in the one- to- one trials have
completed the instruction, they should review the
posttest and attitude questionnaire in the same
fashion.
After each item or step in the assessment, ask the
learners why they made the particular responses that
they did.
This will help you spot not only mistakes but also the
reasons for the mistakes, which can be quite helpful
during the re-vision process.
13. LEARNING TIME
One design interest during one- to- one evaluation is
determining the amount of time required for learners to
complete instruction, which is a very rough estimate,
because of the interaction between the learner and the
designer.
You can attempt to subtract a certain percentage of the
time from the total time, but experience has indicated
that such estimates can be quite inaccurate.
14. DATA
INTERPRETATION
The information on the clarity of instruction,
impact on learner, and feasibility of instruction
needs to be summarized and focused.
Particular aspects of the instruction found to be
weak can then be reconsidered in order to plan
revisions likely to improve the instruction for
similar learners.
15. OUTCOMES
The outcomes of one- to- one trials are instruction that
1) contains appropriate vocabulary, language
complexity, examples, and illustrations for the
participating learner;
( 2) either yields reasonable learner attitudes and
achievement or is revised with the objective of
improving learner attitudes or performance during
sub-sequent trials; and
( 3) appears feasible for use with the available
learners, resources, and setting. The instruction can
be refined further using small group trials.
16. FIELD TRIAL
Purpose
To determine whether the
changes/revisions in the
instruction made after the
small group stage were
effective.
To see whether the
instruction can be used in
the context for which it
was intended.
In the final stage of
formative evaluation
the instructor attempts
to use a learning con-text
that closely
resembles the intended
context for the ultimate
use of the instructional
materials.
One purpose of this
final stage of formative
evaluation is to
determine whether the
changes in the
instruction made after
the small group stage
17. SMALL-GROUP
EVALUATION
Purposes
To determine the effectiveness of changes made following
the one-to-one evaluation.
To identify any remaining learning problems that learners
may have.
To determine whether learners can use the instruction
without interacting with the instructor.
18. FORMATIVE
EVALUATION
To determine Weakness(es) in the Instruction
Focusing the design only on the goals and objectives of
the instruction would be too limited.
Data on learners’ achievement of goals and objectives
would be insufficient, though important, because these
data will only provide information about where errors
occur rather than why they occur.
19. FORMATIVE
EVALUATION
HAS SIX STAGES Design Review
Expert Review
One-To-One
Small Group
Field Trials
Ongoing Evaluation
20. DESIGN REVIEW
Does the instructional goal match the problem
identified in the needs assessment?
Does the learner & environmental analysis match the
audience?
Does the task analysis include all the prerequisite
skills?
Are the test items reliable and valid, and do they
match the objectives?
21. EXPERT REVIEW
Is the content accurate & up-to-date?
Does it present a consistent perspective?
Are examples, practice exercises, & feedback realistic
& accurate?
Is the pedagogy consistent with current instructional
theory?
Is the instruction appropriate to the audience?
22. ONE-TO-ONE
REVIEW
Is the message clear?
What is the impact on:
learner attitudes
achievement of objectives & goals
Feasibility of training
23. SMALL GROUP
REVIEW
Look for the effects caused by the changes
made in the one-to-one review
Identify any remaining learning problems
24. FIELD TRIAL
REVIEWS
Look for effects in changes made in small group
Can the instruction be used in the context in which it
was intended
25. ONGOING
EVALUATION
Project Size
Life span of content
Audiences change
One-To-One
Small Group Tryouts
Field Trials
26. LEARNER
EVALUATION
Do learners understand the instruction?
Do they know what to do during the practice &
the tests?
Can they interpret graphics in the text?
Can they read all the material?
How much time does it take?
28. SUMMARY
Formative evaluation of instructional materials is conducted to determine the
effectiveness of the materials and to revise them in areas where they are
ineffective. Formative evaluations should be conducted on newly developed
materials as well as existing materials that are selected based on the
instructional strategy. Evaluations are necessary for both mediated and
instructor presented materials. The evaluations should be designed to
produce data to pinpoint specific areas where the instruction is faulty and to
suggest how it should be revised. An iterative process of formative evaluation
containing at least three cycles of data collection, analysis, and revision is
recommended. Each cycle focuses on different aspects of quality. The first
cycle, one- to-one evaluation, is conducted to pinpoint gross errors in the
materials. These errors typically relate to both the clarity of vocabulary,
concepts, and examples used, and the motivational value of all five
components of the instructional materials. Evaluations can also be conducted
with content experts and individuals familiar with the characteristics of target
learners. One- to- one evaluations must be conducted with representatives of
the target population. An interactive interview process is used so the
evaluator can learn what was wrong with the materials and why it was
