This document discusses summative evaluation, which is used to make decisions about continuing the use of instructional materials. It describes the expert judgement and field trial phases of summative evaluation. The expert judgement phase involves analyzing the materials' congruence with organizational needs, content accuracy, instructional design, and feasibility of implementation. The field trial phase tests the materials' effectiveness with target learners and collects data on learner outcomes, instructor attitudes, and cost management. Summative evaluation aims to verify materials' effectiveness and impacts on learners, jobs, and the organization to determine whether the materials should be adopted or maintained.

1. DESIGNING AND CONDUCTING
SUMMATIVE EVALUATIONS
Angel Jones

2. BACKGROUND
• Formative Evaluation is used to improve the effectiveness of others.
• Summative Evaluation is used to make decision about the continuous
use of instruction.
• Did the instruction solve the problem
• Is the instruction effective in solving the performance problem.
• Relationship between the instruction of interest and needs of the
organization.
• Field trial is similar to the third phase of formative evaluartion but
conducted to determine whether is produces the desired results for
the decision maker.

3. OBJECTIVES
• I can describe the purpose of summative evaluation
• I can describe the two phases of summative evaluation and the
decisions resulting from each phase.
• I can design a summative evaluation for comparing alternative
sets of candidate instructional materials
• I can contrast formative and summative evaluation by purpose
and design.

4. WHAT IS SUMMATIVE EVALUATION
The design of evaluation studies and
the collection of data to verify the
effectiveness of instructional
materials with target learners.

5. SUMMATIVE EVALUATION EXPERT JUDGEMENT
PHASE
Overall Decisions
Do the materials have the potential for meeting the origination's needs?
Specific Decisions
Congruence Analysis: Are the needs and goals of the organization congruent
with those in the instructions?
Content Analysis: Are the materials complete, accurate and current?
Design Analysis: Are the principles of learning, instructions and motivation
clearly evident in the materials?
Feasibility Analysis: Are the materials convenient, durable, cost-effective, and
satisfactory for current users?

6. SUMMATIVE EVALUATION FIELD TRIAL PHASE
Overall Decisions
Are the materials effective with target learners in the prescribed setting?
Specific Decisions
Outcomes Analysis
Impact on Learners: Are the achievement and motivation levels of learners
satisfactory following instructions?
Impact on Job: Are learners able to transfer the information, skills and
attitudes from instructional setting to the job setting or to subsequent units of
related instruction?
Impact on Organization: Are learners’ changed behaviors making positive
differences in the achievement of the organization’s mission and goals.

7. SUMMATIVE EVALUATION FIELD TRIAL PHASE
Management Analysis
1. Are instructor and manager attitudes satisfactory?
2. Are recommended implementation proceduers feasible?
3. Are costs related to time, personnel, equipment, and resources
reasonable?

8. EXPERT JUDGEMENT PHASE OF
SUMMATIVE EVALUATION

9. CONGRUENCE ANALYSIS
• Obtain a clear description of the organization’s need
• Locate instructional materials that have potential for meeting
the organization’s needs.
 Obtain a clear description of the goals and objectives of the instruction
and the target audience for which it is intended.

10. RESOURCES
• Compare the organizations need vs. needs addressed in the
materials.
• Compare the target group vs. target group for the materials
• Compare the resources vs. requirements for implementing
instructions.

11. QUESTIONS RELATED TO THE DESIGN OF
QUALITY MATERIALS
Questions related to the design of quality materials.
1. Are the materials and assessments accurate and complete?
2. Is the instructional strategy adequate for the anticipated types
of learning outcomes?
3. Can the materials be used effectively?
4. Are current users of the materials satisfied?

12. CONTENT ANALYSIS
• Content expert may be necessary if you are not a content expert
in the materials you evaluate.
Strategies for content expert.
• Provide the expert with copies of all candidate materials to be
judged for accuracy and completeness of the goals.

13. DESIGN ANALYSIS
• Evaluate the adequacy of the components of the instructional
strategy included in the candidate materials.
• Checklist is the most thorough and time saving approach
• List the components of the instructional strategy in the far left
column and use the remaining columns for recording
information.
• Adopt criteria related to each component based on the type of
learning outcomes.

14. UTILITY AND FEASIBILITY ANALYSIS
• Available learners guide or syllabus
• Durability of material
• Special resources such as instructor capabilities, equipment or
environments that are required.
• Interviewing the persons in the orgainzations who request the
evaluation may help with the design.

15. SEQUENCE OF STAGES OF ANALYSIS DURING THE
EXPERT JUDGEMENT PHASE OF SUMMATIVE
EVALUATION
• Perform congruence analysis for organization and instruction.
• Perform content analysis on congruent instruction
• Perform design analysis on congruent, accurate, complete
instruction.
• Perform feasibility analysis on well designed instruction
• Perform current user analysis
• Conduct field trials for the materials judged as most promising
dureing the export judgement phase of summative evaluation.

16. FIELD TRIAL PHASE OF SUMMATIVE
EVALUATION

17. CURRENT USER ANALYSIS
• Data about the target learners in the other setting.
Entry skills; motivation for studying; pretest and posttest
performance level; attitudes about the materials.
• Instructor’s perceptions of the materials
Are the materials easy to use; problems in implementing the
materials; resources required; planning on continue using
materials

18. PLANNING
• Design Evaluation
• Outcomes Analysis
Describe resources, facilities, equipment needed
Describe ideal entry skills/characteristrics of target group
Describe number of groups and individuals needed
• Management Analysis
Describe skills/capabilities of instructiors or managers
Describe number of instructors needed
Plan and develop any training needed for instructors/managers.

19. PREPARING
• Obtain instruments; Set schedule for instructions and testing;
Create/modify syllabus
• Obtain resources, facilities, equipment
• Select sample; Verify entry skills; Schedule learners
• Select instructors; Verify skills; Schedule instructors
• Provide training for instructors.

20. IMPLEMENTING/COLLECTING DATA
• Adequate?
• Learner performance?;Learner attitudes?;Learner performance
in job context
• Validity of implementation? Modifications?
• Training effective?

21. SUMMARIZING AND ANALYZING DATA
• Describe problems by resource facilities, and equipment
• Item by objective analysis for group and individual; Cross test
summary by group and individua by objective; Attitude
summary
• Describe problems by instructor by objective
• Describe implementation problems by objective and activity

22. REPORTING RESULTS
• Describe limitations of design.
• Recommendation and rationale
• Explanation, recommendations, rationale for learner, job and
organization impact.
• Recommendations and rationale
• Recommendation and rationale

23. COMPARISON OF FORMATIVE AND SUMMATIVE
EVALUATION
Formative Evaluation Summative Evaluation
Purpose Locate weaknesses in instruction in
order to revise it
Document strengths and weakness in
instruction in order to decide
whether to maintain or adopt it
Phase or Stages One to One
Small group
Field Trial
Expert judgment
Field trial
Instructional Development
History
Systematically designed in-house and
tailored to the needs of the
organization
Produced in house or elsewhere not
necessarily following a systems
approach
Materials One set of materials One set of materials or several
competing sets
Position of Evaluator Member of design and development
team
Typically an external evaluator
Outcomes A prescription for revision
instructions
A report documenting the design,
procedures, results,
recommendation, and rationale.

24. SUMMATIVE EVALUATION CHANGE AGENT
As a change agent , I conduct evaluation to make decisions about
whether to maintain or adopt instructions. I will collaborate with
the curiculm team to develop evaluation that will verify the
effectiveness of the instruction material an target learners. As a
change agent I will improve the effectiveness of my instruction
with the data summary of the summative evaluation.

25. THE END
By: Angel Jones