Summative evaluation is defined as the design of evaluation studies and collection of data to verify the effectiveness of instructional materials with target learners. It has two main phases: the expert judgment phase and the field trial phase. The expert judgment phase involves analyses to determine if materials have potential to meet organizational needs, including congruence with goals, content accuracy, and design adequacy. The field trial phase tests effectiveness with target learners and examines impact on learners, jobs, and the organization. Summative evaluation is used to make "go-no-go" decisions about keeping or replacing instructional materials and typically involves external evaluators providing objective assessment.

1. Designing and ConductingDesigning and Conducting
Summative EvaluationsSummative Evaluations
Larry F. Cobb

2. ObjectivesObjectives
Define the purpose of summative
evaluation
Describe the two phases of
summative evaluation and the
decisions resulting from each phase
Design a summative evaluation to
examine organizational benefits of
instruction they have implemented
Contrast Formative and summative
evaluation by purpose and design

3. What’s the Difference?What’s the Difference?

4. What is Summative EvaluationWhat is Summative Evaluation
Summative evaluation is defined as
the design of evaluation studies and
the collection of data to verify the
effectiveness of instructional materials
with target learners.

5. The Purpose of SummativeThe Purpose of Summative
EvaluationEvaluation
Make “go-no-go” decisions
• Keep current materials?
• Look for something better suited to meet
organization’s specific instruction needs?
• Summative evaluations are used to
judge the impact of a plan of
instruction on the organization’s
initial problem.

6. Evaluators – Why they should beEvaluators – Why they should be
ExternalExternal
The primary evaluator in a summative
evaluation is rarely the designer or
developer of the instruction.
Evaluator is often unfamiliar with the
materials, the organization requesting
the evaluation, or the setting in which
the materials are evaluated.
Preferred because they have no
personal investment and will likely be
more objective

7. Two Main Phases of SummativeTwo Main Phases of Summative
EvaluationEvaluation
1.Expert Judgment Phase
2. Field Trial Phase

8. Expert Judgment PhaseExpert Judgment Phase
Purpose:
Do the materials have the potential for
meeting this organization’s needs?
There are several activities that decide whether
the candidate instruction is promising.
1. Congruence Analysis
2. Content Analysis
3. Design Analysis
4. Utility and Feasibility Analysis
5. Current User Analysis

9. Expert Judgment PhaseExpert Judgment Phase
Congruence Analysis
Analyzing the congruence among
1.An organization’s state needs and goals and
those addressed in candidate instruction
2.An organization’s target learners’ entry skills
and characteristics and those for which candidate
materials are intended
3.An organization’s resources and those required
for obtaining and implementing candidate
instruction

10. Expert Judgment PhaseExpert Judgment Phase
Content Analysis
During this activity, an identified expert is used to
judge material for accuracy and completeness to
determine if they are inline with the organization’s
stated goals. An instructional analysis of the
stated goal is a very cost effective method.

11. Expert Judgment PhaseExpert Judgment Phase
Design Analysis
The design analysis is an evaluation of the
adequacy of the components of the instructional
strategy included in the candidate material.
Checklist are great during this activity.

12. Expert Judgment PhaseExpert Judgment Phase
Utility and Feasibility Analysis
Factors such as the availability of a learner guide
or syllabus and an instructor’s manual are taking
into consideration during this activity. This is also
the time when you get information for the people
that determine that the evaluation was necessary.

13. Expert Judgment PhaseExpert Judgment Phase
Current User Analysis
This final analysis seeks to get information about
the candidate material from the organizations
that are experienced in using them. The names
of current users can often be obtained from the
publishers of the materials.

14. Steps for Conducting ExpertSteps for Conducting Expert
Judgment Phase of SummativeJudgment Phase of Summative
EvaluationEvaluation

15. Field Trial PhaseField Trial Phase
Purpose:
Are the materials effective
with target learners in the
prescribed setting?
Outcomes Analysis:
1.Impact on Learners
2.Impact on Job
3.Impact on Organization

16. Field Trial Phase (continued)Field Trial Phase (continued)
Management Analysis:
1.Are instructor and manager attitudes
satisfactory?
2.Are recommended implementation
procedures feasible?
3.Are costs related to time, personnel,
equipment, and resources reasonable?

17. Steps for Conducting Field TrialSteps for Conducting Field Trial
Phase of Summative EvaluationPhase of Summative Evaluation

18. Comparison of Formative andComparison of Formative and
Summative Evaluation (Table 12.3)Summative Evaluation (Table 12.3)
Formative Evaluation Summative Evaluation
Purpose
Locate weaknesses in
instructor in order to
revise it
Document strengths and
weaknesses in instruction
in order to decide whether
to maintain or adopt it
Phases or Stages
On-to-one
Small group
Field Trial
Expert judgment
Field trial
Instructional
Development History
Systematically designed
in-house and tailored to
the needs of the
organization
Produced in-house or
elsewhere not necessarily
following a systems
approach
Materials One set of materials
One set of materials or
several competing sets
Position of Evaluator
Member of design and
development team
Typically an external
evaluator
Outcomes
A prescription for revising
instruction
A report documenting the
design, procedures,
results, recommendations,
and rationale

19. Change AgentChange Agent
As a change agent, is very important for
us to constantly evaluate the
instructional strategy ensure that is
staying the course with what the
organization has in mind while being
usable and feasible for the
organization. The summative
evaluation does just that.

20. Larry F. CobbLarry F. Cobb
lfcobb@alasu.edu