The document discusses the process of formative evaluation of instructional materials and instructor-led instruction. It describes conducting evaluations in three phases - one-to-one, small group, and field trial - to identify issues, improve the instruction, and determine effectiveness. The goal is to evaluate and revise materials based on learner performance and feedback before full implementation, in order to ensure the instruction is clear, impactful and feasible for its intended audience. Specialists may provide input during development, and the evaluations involve collecting various data on learner comprehension, reactions, and skill demonstration.

1. DESIGNING AND
CONDUCTING FORMATIVE
EVALUATIONS
TERRELL MCCALL
10.13.19

2. BACKGROUND
• In the early 1970’s, instructional materials were not very effective. Initial
drafts of materials would have been put into production for to the target
population. Studies have shown that thousands of the instructional materials
sold in the United States each year have not been evaluated with learners and
revised prior to distribution. Today, the idea of formative evaluation is in the
use in a variety of settings, and in these applications the idea of evaluating
learners’ performance and using resulting data to improve the instruction and
curriculum.

3. OBJECTIVES
• Describe the purpose for the various stages of formative evaluation of
instructor-developed materials, instructor-selected materials, and instructor-
presented instruction
• Describe the instruments used in a formative evaluation
• Develop an appropriate formative evaluation plan and construct instruments
for a set of instructional materials or an instructor presentation
• Collect data according to a formative evaluation plan for a given et of
instructional materials or instructor presentation.

4. CONCEPTS
• There are three basic phases to
formative evaluation: one-to-one
clinical, small group, field trial.
• The three stages are typically
preceded by the review of
instruction by interested specialists
who are not directly involved in the
instructional development project,
but who have relevant expertise.

5. FORMATIVE EVALUATION DESIGNS
• Focusing the design only on goals and objectives of the instruction is too
limited.
• One way the instructional strategy can be used to aid the design of the
formative evaluation is to create a matrix that lists the components of the
instructional strategy among one side and the major categories questions
about the instruction along the other.

6. ROLE OF SUBJECT-MATTER, LEARNING, AND LEARNER
SPECIALISTS IN FORMATIVE EVALUATION
• A subject-matter expert should comment on the accuracy and currency of
instruction.
• It is helpful to share the first draft of the instruction with a person who is
familiar with the target population.
• The designer is not obligated to use the suggestions of the specialists.

7. ONE-TO-ONE EVALUATION
• The purpose of the one-to-one evaluation stage is to identify and remove the
most obvious errors in the instruction and to obtain initial performance
indications and reactions to the content by learners.
• During the development of the instructional strategy and the instruction itself,
designers and developers make a myriad of translations and decisions that
link the content, learners, instructional format, and instructional setting.

8. CONTINUED
• ‘Three main criteria: Clarity, Impact,
and Feasibility
• The one-to-one trials help verify
whether the designers’ and
developer’s hunches were correct,
or reflected misconceptions of the
target group.

9. SELECTING LEARNERS
• One of the most critical decisions by the designer in the formative evaluation
is the selection of learners to participate in the study.
• The designer works on an individual basis with each learner. The designer
should be aware of the target population to the work in a one-to-one mode,
although three is usually sufficient.
• The designer must also make this decision for each particular instructional
design situation.

10. DATA COLLECTION
• The three main criteria and the decisions to be made during one-to-one trials
help evaluators focus on the kinds of information that would be useful.
• The lists in each criterion category are intended to be illustrative rather than
exhaustive, because the degree of relevance of each kind of information may
differ by learner maturity, instructional content, and delivery method.

11. SMALL-GROUP EVALUATION
• Two primary purposes for small group
evaluation: determining effectiveness of
changes made following the one-to-one
evaluation and identify any remaining
learning problems that learners may have;
and second, determine whether learners
can use the instruction without interacting
with the instructor.
• Typical measures used to evaluate
instructional effectiveness include learner
performance scores on pretests and
postests.
• The selection of learners to participate in
your small-group trial is important.

12. FIELD TRIAL
• One purpose of the final stage is to determine whether the changes in the
instruction can be used in the context for which for which it was intended.
• The field trial is much like the final dress rehearsal in theater, because the
instruction is polished and delivered in a manner that is close ad possible to
the final format.
• The use of multiple tryout sites may be necessary if such sites vary a great
deal.

13. FORMATIVE EVALUATION OF SELECTED MATERIALS
• The types of editorial and content changes made as result of one-to-one and
small group evaluations are typically not used with existing materials.
• The purpose of the expert judgement phase is to determine whether
currently used or other candidate instruction has the potential for meeting an
organization's defined instructional needs.

14. INSTRUCTOR-LED INSTRUCTION
• Instructor-led training is any kind of training that
occurs in a training room, typically in an office,
classroom, or conference room. This form of
training can have one or more instructors; and they
teach skills or material to another person or group
through lectures, presentations, demonstrations,
and discussions.
• Teams may also learn better with instructor-led
training, because they can share ideas, work in
groups, and debate with their peers. It's also useful
for bonding, team building, and team problem
solving. All of these means that instructor-led
training can have greater long-term benefits than
one-on-one or online training.

15. SMALL GROUP FORMATIVE EVALUATION
PROCEDURES
• If Instruction is being used with the target population without the benefit of
any formative evaluation, it is still possible to use that opportunity to gather
information useful for revising the instruction.
• The general principle for the designer is that formative evaluations are always
conducted; it is just a question of when, where and how.

16. CONCERNS ABOUT LEARNERS
• Learners who do not have the entry skills should also be included in a
formative evaluation.
• If, however, learners without the entry skills are successful with the
instruction, then you must seriously reconsider the validity of the entry skills
you have identified.

17. CONCERNS INFLUENCING FORMATIVE EVALUATION
• The formative evaluation component distinguishes the instructional design
process from a theosophical approach.
• For proper evaluation of materials, ensure that any technical equipment is
operating effectively. Data from learners were invalid, and the instructor
learned little more than that the equipment must operate effectively to try
out materials.

18. PROBLEM SOLVING DURING INSTRUCTIONAL DESIGN
• The designer is often faced with questions that can be best answered with
data with learners.
• The designer must develop two versions of the instruction to use in small-
group evaluation. The purpose is to make a decision about what to do with a
particular unit of instruction.
• The designer could collect enough data in the formative research about the
illustrations to make at least a tentative decision about their continued use of
instruction.

19. SUMMARY
• Formative evaluation instructional materials is conducted to determine the
effectiveness of the materials and revise them in areas where they are
ineffective. Formative evaluations should be conducted on newly developed
materials as well as existing materials that are selected based on the
instructional strategy.
• Following formative evaluation and materials revision, organizations using the
instruction often want an evaluation conducted to determine whether the
skills learned are being used on the job and with what effect.

20. CONTACT
• Email: tmccall4701@myasu.alasu.edu
• LinkedIn: Terrell McCall