Formative evaluation is used by instructional designers to improve instructional materials. There are three phases: 1) one-to-one evaluation identifies major errors; 2) small group evaluation further refines materials; 3) field trials test materials in the actual use context. Data is collected on clarity, impact, and feasibility. Designers evaluate message, links between concepts, and procedures through learner reactions, test performance, and comments. The goal is to improve relevance, understandability, and manageability of materials for the target learner population.

1. CHAPTER 11: DESIGNING &
CONDUCTING FORMATIVE
EVALUATIONS
Jordan Amerson

2. Objectives
■ Describe the purposes for and various stages of formative evaluation of instructor-
developed materials, instructor-selected materials, and instructor-presented
instruction
■ Describe the instruments used in a formative evaluation
■ Develop an appropriate formative evaluation plan and construct instruments for a set
of instructional materials or an instructor presentation
■ Collect data according to a formative evaluation plan for a given set of instructional
materials or instructor presentation

3. Concepts
■ Formative evaluation – process designers use to obtain data for revising their
instruction to make it more efficient and effective, its emphasis is on the collection
and analysis of data and the revision of the instruction
■ Summative evaluation – evaluation designed and used after an instructional program
has been implementedThe purpose is to make decisions concerning whether the
instruction works as intended in the performance context, and whether progress is
being made in ameliorating the performance problems that prompted the
instructional design and development effort.

4. 3 basic phases of formative evaluation
■ 1. one-to-one or clinical evaluation – designer works with individual learners to obtain
data to revise the materials
■ 2. small-group evaluation – referring to the use of small number of tryout students
who study an instructional program without intervention from the designer and are
tested to assess the effectiveness of the instruction
■ 3. field trial – referring to the evaluation of the program or product in the setting in
which it is intended to be used

5. Purpose of formative evaluation
■ The purpose of formation evaluation is to pinpoint specific error sin the materials in
order to correct them, the evaluation design – including instruments, procedures, and
personnel – must yield information about the location of and the reasons for any
problems

6. 5 areas of questions directly related to the decision you
made while developing the materials are appropriate for all
materials:
■ 1. Are the materials appropriate for the type of learning outcome? Specific
prescriptions for the development of materials were made based on whether the
objectives were intellectual or motor skills, attitudes, or verbal information.
■ 2. Do the materials include adequate instruction on the subordinate skills, and are
these skills sequenced and clustered logically?The best evaluator for this area of
questions is an expert in the content area.
■ 3. Are the materials clear and readily understood by representative members of the
target group? Obviously, only members of the target group can answer these
questions.

7. Contd.
■ 4.What is the motivational value of the materials? Do learners find the materials
relevant to their needs and interests? Are they confident as they work through the
materials?
■ 5. Can the materials be managed efficiently in the manner they are mediated? Both
target learners and instructors are appropriate to answer these questions.

8. Types of data to collect:
■ Reactions of the subject-matter expert
■ Reactions of a manager or supervisor who has observed the learner using the skills in
the performance context
■ Test data collected on entry skills tests, pretests, and posttests

9. Contd.
■ Comments or notations made by learners to you or marked on the instructional
materials about difficulties encountered at points in the materials
■ Data collected on attitude questionnaires or debriefing comments in which learners
reveal their overall reactions to the instruction and their perceptions of where
difficulties lie with the materials and the instructional procedures in general
■ The time required for learners to complete various components of the instruction

10. Role of subject- matter, learning, and learner
specialist in formative evaluation
■ A subject-matter expert- type of reviewer outside the project who has special
expertise in the content area of the instruction, should comment on the accuracy and
currency of the instruction.
■ Target population specialist – someone who can look at the instruction through the
target population’s eyes and react

11. One-to-one evaluation with learners
■ One-to-one evaluation is to identify and remove the most obvious errors in the
instruction and to obtain initial performance indications and reactions to the content
by learners.
■ During this stage of direct interaction between the designer and individual learners,
the designer works individually with three or more learners who are representative of
the target population.

12. The three main criteria and the decisions designers make
during the evaluation are as follows:
■ 1. clarity – is the message, or what is being presented, clear to individual target
learners?
■ 2. impact – what is the impact of the instruction or individual learner’s attitudes and
achievement of the objectives and goals?
■ 3. feasibility – how feasible is the instruction given the available resources
(time/context)?

13. Selecting learners
■ One of the most critical decisions by the designer in the formative evaluation is the
selection of learners to participate in the study
■ Selects at least one learner from the target population who is above average in the
ability, one who is average, and one who is below average.
■ The designer then works on an individual basis with each learner

14. Data collection
■ 3 main categories of illuminating information – message, links, and procedures.
■ Message- related to how clear the basic message is to the learner, determined by such
factors as vocabulary, sentence complexity, and message structures.
■ Category links- refers to how the basic message is tailored for the learner, including
contexts, examples, analogies, illustrations, and demonstrations
■ Procedures – refers to characteristics of the instruction, such as the sequence, the size
of segment presented, the transition between segments, the pace, and the variation
built into the presentation

15. The evaluator must determine whether the
learner perceives the instruction as being:
■ 1. personally relevant to her or him
■ 2. accomplishable with reasonable effort
■ 3. interesting and satisfying to experience

16. Examples of questions of interest include
the following:
■ 1. How should the maturity, independence, and motivation of the learner influence the
general amount of time required to complete the instruction?
■ 2. can learners such as this one operate or easily learn to operate any specialized
equipment required?
■ 3. Is the leaner comfortable in the environment?
■ 4. Is the cost of delivering this instruction reasonable given the time requirements?

17. One- to –one formative evaluation
procedures
■ Materials for the one-to-one trials
■ Participants and instructions
■ Pre instrutional materials
■ Content presentations
■ Learner participation
■ Assessments

18. Small group formative evaluation
procedures
■ Materials for the small-group trial
■ Participants and instructions
■ Pre instructional materials
■ Content presentation and practice
■ Assessments

19. Summary
■ Formative evaluation
■ 1. one-to-one evaluation – conducted to pinpoint gross errors in the materials
■ 2. small group evaluation – follows the correct of major errors identified in the
instruction
■ 3. field trial – conducted following refinement of the materials based on the small-
group evaluation

20. Contact
■ JordanAmerson
– Jordan.Amerson@aol.com