Summative evaluation involves collecting and analyzing data after implementation to provide decision makers information on the effectiveness and efficiency of instruction. It determines whether learner objectives were achieved and costs. There are two approaches: objectivism uses empirical data while subjectivism employs qualitative methods like interviews. Both have limitations if used alone. The designer should not conduct the first summative evaluation due to potential bias. The evaluation report summarizes the needs assessment, study design, results, and conclusions to help guide recommendations.

1. Summative Evaluation The Evaluation after implementation

2. Involves _______ data Collecting Analyzing Summarizing

3. For the purpose of Giving decision makers information on the effectiveness and efficiency of instruction

4. Effectiveness of Content Instruction solve the problem? Criterion created prior to evaluation? Was the criterion established in conjunction with the needs assessment?

5. Specifically Did learners achieve the objectives? Learners feeling about instruction? What were the costs? How much time did it take? Was instruction implemented as designed? What unexpected outcomes?

6. Alternative Approaches to Summative Evaluation Objectivism Subjectivism

7. Objectivism Based on empiricism Answering questions on the bases of observed data Goal based and replicable, uses the scientific method

8. Subjectivism Employs expert judgment Includes qualitative methods observation and interviews evaluate content “Goal Free” evaluators haven’t a clue about the goals

9. Objectivism (limitations) Examine only a limited number of factors May miss critical effects

10. Subjectivism (limitations) Are not replicable Biased by idiosyncratic experiences, perspectives, or the people who do the evaluation May miss critical effects

11. Designer Role in Summative Evaluation? Somewhat controversial

12. Timing of Summative Evaluation? Not in the first cycle

13. Summary Diagram Formative Design Reviews Expert Reviews One-to-one Eval. Small Group Eval. Field Trials Ongoing Eval. Summative Determine Goals of the Evaluation Select Orientation Select Design Design or Select Evaluation Measures Collect Data Analyze Data Report Results

14. Goals of the Evaluation What decisions must be made? What are the best questions? How practical is it to gather data? Who wants the answer to a question? How much uncertainty?

15. Orientation of Evaluation Goal-based or goal-free A middle ground? Quantitative or qualitative appropriate? Experimental or naturalistic approach?

16. Select Design of Evaluation Describes what data to collect When the data will be collected And under what conditions Issues to consider: How much confidence must we have that the instruction caused the learning? (internal validity) How important is the generalizability? (external validity) How much control do we have over the instructional situation?

17. Design or Select Evaluation Measures Payoff outcomes Is the problem solved? Costs avoided Increased outputs Improved quality Improved efficiency

18. Design or Select Evaluation Measures (2) Learning Outcomes Use instrument you’ve already developed for the summative evaluation But measure the entire program

19. Design or Select Evaluation Measures (3) Attitudes Rarely the primary payoff goals Ask about learner attitudes toward learning instructional materials subject matter Indices of appeal attention, likeableness, interest, relevance, familiarity, credibility, acceptability, and excitement

20. Design or Select Evaluation Measures (4) Level of Implementation degree to which the instruction was implemented Costs Cost-feasibility Cost-effectiveness

21. Alternative Designs Instruction then posttest Pretest then instruction then posttest

22. The Report Summary Background Needs assessment, audience, context, program description Description of evaluation study Purpose of evaluation, evaluation of the design, outcomes measured, implementation measures, cost-effectiveness info., analysis of unintentional outcomes

23. The Report (continued) Results Outcomes, implementation, cost-effectiveness info., unintentional outcomes Discussion causal relationship between program & results Limitation of study Conclusion & Recommendations

24. Summary Summative evaluation is after implementation Limitations of subjective and objective evaluation What to include in the report