Designing and Conducting Summative Evaluations

1. Tieka Wilkins
Chapter 12
Designing and Conductive Summative Evalautions

2. Background

3. Objective
• Designers need to know whether the skills learned in
instruction are actually transferred to the performance
setting.
• If the skills learned are transferred, they need to know
whether using them solves the original problem. If not,
and without other benefits to the organization
(unanticipated), the utility of the instruction should be
questioned.

4. Two phases of summative
evaluation
• Expert judgment :determine whether currently used
instruction or other candidate instruction has the potential
for meeting an organization's defined instructional needs
• Field trial document the effectiveness of promising
instruction with target group members in the intended
• setting.

5. Expert Judgment Phase of
Summative Evaluation
1. Congruence Analysis
2. Content Analysis
3. Design Analysis
4. Feasibility analysis
5. Current User Analysis

6. Congruence Analysis :
• To perform the congruence analysis, you should first obtain
a clear description of the organization's needs, which
includes an accurate description of the entry skills and
characteristics of the target learners. After obtaining this
information, you should locate instructional materials that
have potential for meeting the organization's needs.

7. Content Analysis
• One strategy would be to provide the experts with copies of all
candidate materials and ask them to judge the accuracy and
completeness of the materials for the organization's stated
goals.
• A better, more cost-effective strategy would be to work with
the expert(s) to produce an instructional analysis of the stated
goal.
• The document the expert(s) produces should include both the
goal analysis and the subordinate skills analysis.
• A framework that identifies and sequences the main steps and
subordinate skills in the goal would be a valuable standard
against which you can evaluate the accuracy and completeness
of any candidate materials.

8. Design Analysis
• Evaluate the adequacy of the components of the
instructional strategy included in the candidate materials.
• Although the basic components of the strategy do not
change, you may want to adopt criteria related to each
component based on the type of learning outcomes
addressed in the materials.
• The evaluator's response format can also be expected to
vary based on the nature of the instruction.

9. Feasibility Analysis
• The third area of questions about the instructional materials relates
to the utility of the candidate materials.
• For each set, you should consider such factors as the availability of a
learner guide or syllabus and an instructor's manual.
• Factors related to the durability of the materials are another
consideration.
• Another is any special resources, such as instructor capabilities,
equipment, or environments (e.g., learning centers) that are required.
• A utility concern is whether the materials require group or
individual pacing.

10. Current User Analysis
• One other analysis that you may wish to include in your
design is to seek additional information about the
candidate materials from organizations that are
experienced in using them.
• The names of current users can often be obtained from
the publishers of the materials.
• Another type of information relates to the instructor's
perceptions of the materials.

11. The Field Trial Phase Types:
1. Outcome Analysis
2. Management Analysis

12. Outcomes Analysis
Impact on Learners – are learners achieving satisfactory
results?
 Impact on Job – are learners able to make connections
between what they are learning and how they can apply
to their job? Applicable?
Impact on Organization – are learner’s attitudes and
behaviors helping the organization achieve a positive
difference?

13. Management Analysis
Are instructor/manager attitudes satisfactory?
Is implementation doable? Realistic?
Are costs (time, personnel, equipment and resources)
realistic?

14. Phases
• Planning for the evaluation
• Preparing for the implementation
• Implementing instruction and collecting data
• Summarizing and analyzing data
• Reporting results

15. REPORTING RESULTS
Prepare a report of the summative evaluation
findings that includes
Summary
Background
Description of Evaluation Study
Results
Discussion
Conclusion and Recommendations

17. • The purpose of formative evaluation
is to improve instruction by getting
data for revisions.
• The purpose of summative evaluation
is to prove the worth of the
instruction, given that it will not
revised.

19. • Summative evaluations are conducted to make decisions
about whether to maintain or adopt instruction. The
primary evaluator in a summative evaluation is rarely the
designer or developer of the instruction; the evaluator is
frequently unfamiliar with the materials, the organization
requesting the evaluation, or the setting in which the
materials are evaluated. Such evaluators are referred to as
external evaluators; these evaluators are preferred for
summative evaluations because they have no personal
investment in the instruction and are likely to be more
objective about the strengths and weaknesses of the
instruction.

20. Tieka Wilkins
• sewheatt@aol.com