This document discusses summative evaluation, which determines whether instruction achieves its intended outcomes. Summative evaluation has two phases: expert judgment and impact analysis. The expert judgment phase involves analyses of instructional congruence with organizational needs, content, design, transfer feasibility, and existing materials. The impact phase focuses on whether skills transfer to the workplace by establishing criteria, selecting respondents, planning studies, analyzing data, and reporting results. Formative evaluation identifies weaknesses for revision, while summative evaluation documents skill transfer and makes recommendations based on expert review and workplace data. The goal of summative evaluation is to help organizations meet their goals through continued learning and improvement.

1. DESIGNING AND
CONDUCTING SUMMATIVE
EVALUATIONS
By Kobie Jones

2. BACKGROUND
Summative evaluation is the process of collecting data and
information to make decisions about whether the instruction actually
works as intended in the performance context.
The need for summative evaluation became apparent several
decades ago, when advocates for school curriculums claimed that
they were better than their competitors.
The ultimate question of evaluation is not “which is better”?, but did
the innovative instruction solve the problem?

3. OBJECTIVES
Described the purpose of summative evaluation.
Described the two phases of summative evaluation and the decisions
resulting from each phase.
Design an expert judgment phase of summative evaluation.
Design an impact phases of summative evaluation.
Contrast formative and summative evaluation by purpose and
design.

4. FACTORS TO CONSIDER IN
EVALUATING TRANSFER OF SKILLS
FROM TRAINING TO THE
WORKPLACEInstructional
Characteristics
Person Characteristics Work Environment
Characteristics
Pre Instruction Cognitive ability Supervisors
Motivation Self-efficiency Positive feedback for
worker
Clear Understanding of
knowledge and skills
required for job
Motivation Involvement in training
Appropriate difficulty level
for ability
Perceptions of relevance
for immediate needs,
confidence, and personal
satisfaction for learning
and performing
Discuss skills among
employees
Links to prior knowledge
and job needs
Expectations Consequences for correct
use

5. EXPERT JUDGMENT PHASE OF
SUMMATIVE EVALUATION
The types of analysis:
1. Congruence analysis
2. Content analysis
3. Design analysis
4. Transfer feasibility analysis
5. Existing Materials Analysis

6. CONGRUENCE ANALYSIS
To conduct congruence analysis, you need the organizations
strategic plans, their current goals and objectives, and their stated
needs for the training.
You can then determine what action to take by examining the
congruence between the organizations needs and the instructional
materials.

7. ORGANIZATION’S NEEDS
The organizations needs will be analyzed to see if they meet the
expectations of the instructional materials.
The closer the instructional material is with the organizational needs
the easier it is for the managers and employees to implement
changes.

8. CONTENT ANALYSIS
Provide experts with copies of all materials and have them judge the
accuracy, currency, and completeness of the materials for the
organizations stated goals.
Obtain design documents from the group that produced the
instruction and use them as a standard against which to evaluate the
accuracy and completeness of the instructional materials.

9. DESIGN ANALYSIS
The evaluator should judge the adequacy of the components of the
instructional strategy included in the materials.
The materials need to be adequate enough for the given learners’
needs, so you will have to find out learner characteristics in order to
make this determination.

10. TRANSFER FEASIBILITY ANALYSIS
The fourth area of questions about the instructional materials relates
to the potential for transferability of knowledge and skills from the
learning context to the jobsite.
Critical aspects of the job need to be adequately simulated in the
learning context.
Supervisor capabilities, equipment, or environments, are also
important to examine

11. EXISTING MATERIALS ANALYSIS
The rapid expansion of the e-learning made organizations scramble
for quality instruction, so the evaluators will look at existing material
and see if it correlates with the organization needs.

12. IMPACT PHASE OF SUMMATIVE
EVALUATION
The second phase of summative evaluation is Impact analysis,
sometimes called outcomes analysis includes:
 focusing the impact study
Establishing criteria and data needs
Selecting respondents
Planning study procedures
Summarizing and analyzing data
Reporting results
Negotiating resources

13. FOCUSING THE
IMPACT STUDY
The first planning activity is to
center your study in the
workplace.
Review organization goals, their
defined needs, and their
relationships to the specific goals
for the instruction and to their
employees who participated in
the instruction.
Establishing if the learner has
transferred the skills learned
during the instruction.
Establishing
criteria and data
needs

14. SELECTING THE
RESPONDENTS
The nature of information you
need and the particular questions
assist you in planning the types
and number of persons who are
included in your study.
Persons who are typically
included are: learners, peers,
supervisors, managers, and
sometimes customers
In selecting the most
appropriate procedures for
collecting evidence of training
impact, you should consider
when, where, and how to collect
the information.
PLANNING STUDY
PROCEDURES

15. SUMMARIZING AND ANALYZING
DATA
Data analysis procedures should be straightforward and descriptive.
Data should be summarized within study questions for easy
interpretation, and the summaries can include item frequency counts
or content analysis of respondents’ comments
Fancy statistics will confuse those who do not understand.

16. REPORTING RESULTS
If you include both the expert judgement and impact analysis
phases, then both should be documented in the report.
For each one you should describe the general purpose, the specific
questions, the design and procedures, the results, and your
recommendations and rationale.
The rationale for your recommendations should be anchored in the
data you present in the results section

17. COMPARISON OF FORMATIVE AND
SUMMATIVE EVALUATIONS
Formative and summative evaluations differ in several aspects.
The first difference is related to the purpose for conducting each
type of evaluation.
The second difference involves the stages of the evaluations.

18. COMPARISON OF FORMATIVE AND
SUMMATIVE EVALUATIONS
FORMATIVE EVALUATION
Purpose: Locate weakness in
instruction in order to revise it.
Phases or Stages: One-to-one, Small
group, Impact evaluation
Instructional Development History:
Systematically designed in-house
and tailored to the needs of the
organization
Position of Evaluator: Member of
design and development team
Outcomes: A prescription for
revising instruction
SUMMATIVE EVALUATION
Purpose: Document the degree to
which skills learned during
instruction transferred to the
jobsite.
Phases or Stages: Expert judgment,
Impact analysis
Instructional Development History:
Produced in-house or elsewhere not
necessarily a systems approach
Position of Evaluator: External
evaluator
Outcomes: A report documenting
the soundness of the instruction

19. SUMMARY
As a lifelong learner it is my job to make continued advances in my
professional life. With summative evaluation I have learned throughout the years
ways that will continually push organizations towards meeting their goals. Taking
information from the data collected we were able to identify some of the problems
that prohibited a higher success rate. It is my job as a lifelong learner to continue to
learn from each opportunity and help organizations succeed.

20. Kobie Jones