Summative Evaluation


Published on

Created by John Hollenbeck, Ph.D.

Published in: Education
  • Be the first to comment

Summative Evaluation

  1. 1. Summative Evaluation The Evaluation after implementation
  2. 2. Involves _______ data Collecting Analyzing Summarizing
  3. 3. For the purpose of Giving decision makers information on the effectiveness and efficiency of instruction
  4. 4. Effectiveness of Content <ul><li>Instruction solve the problem? </li></ul><ul><li>Criterion created prior to evaluation? </li></ul><ul><li>Was the criterion established in conjunction with the needs assessment? </li></ul>
  5. 5. Specifically <ul><li>Did learners achieve the objectives? </li></ul><ul><li>Learners feeling about instruction? </li></ul><ul><li>What were the costs? </li></ul><ul><li>How much time did it take? </li></ul><ul><li>Was instruction implemented as designed? </li></ul><ul><li>What unexpected outcomes? </li></ul>
  6. 6. Alternative Approaches to Summative Evaluation Objectivism Subjectivism
  7. 7. Objectivism <ul><li>Based on empiricism </li></ul><ul><ul><li>Answering questions on the bases of observed data </li></ul></ul><ul><ul><li>Goal based and replicable, uses the scientific method </li></ul></ul>
  8. 8. Subjectivism <ul><li>Employs expert judgment </li></ul><ul><li>Includes qualitative methods </li></ul><ul><ul><li>observation and interviews </li></ul></ul><ul><ul><li>evaluate content </li></ul></ul><ul><li>“Goal Free” </li></ul><ul><ul><li>evaluators haven’t a clue about the goals </li></ul></ul>
  9. 9. Objectivism (limitations) <ul><li>Examine only a limited number of factors </li></ul><ul><li>May miss critical effects </li></ul>
  10. 10. Subjectivism (limitations) <ul><li>Are not replicable </li></ul><ul><li>Biased by idiosyncratic experiences, perspectives, or the people who do the evaluation </li></ul><ul><li>May miss critical effects </li></ul>
  11. 11. Designer Role in Summative Evaluation? Somewhat controversial
  12. 12. Timing of Summative Evaluation? Not in the first cycle
  13. 13. Summary Diagram <ul><li>Formative </li></ul><ul><ul><li>Design Reviews </li></ul></ul><ul><ul><li>Expert Reviews </li></ul></ul><ul><ul><li>One-to-one Eval. </li></ul></ul><ul><ul><li>Small Group Eval. </li></ul></ul><ul><ul><li>Field Trials </li></ul></ul><ul><ul><li>Ongoing Eval. </li></ul></ul><ul><li>Summative </li></ul><ul><ul><li>Determine Goals of the Evaluation </li></ul></ul><ul><ul><li>Select Orientation </li></ul></ul><ul><ul><li>Select Design </li></ul></ul><ul><ul><li>Design or Select Evaluation Measures </li></ul></ul><ul><ul><li>Collect Data </li></ul></ul><ul><ul><li>Analyze Data </li></ul></ul><ul><ul><li>Report Results </li></ul></ul>
  14. 14. Goals of the Evaluation <ul><li>What decisions must be made? </li></ul><ul><li>What are the best questions? </li></ul><ul><li>How practical is it to gather data? </li></ul><ul><li>Who wants the answer to a question? </li></ul><ul><li>How much uncertainty? </li></ul>
  15. 15. Orientation of Evaluation <ul><li>Goal-based or goal-free </li></ul><ul><li>A middle ground? </li></ul><ul><li>Quantitative or qualitative appropriate? </li></ul><ul><li>Experimental or naturalistic approach? </li></ul>
  16. 16. Select Design of Evaluation <ul><li>Describes what data to collect </li></ul><ul><li>When the data will be collected </li></ul><ul><li>And under what conditions </li></ul><ul><li>Issues to consider: </li></ul><ul><ul><li>How much confidence must we have that the instruction caused the learning? (internal validity) </li></ul></ul><ul><ul><li>How important is the generalizability? (external validity) </li></ul></ul><ul><ul><li>How much control do we have over the instructional situation? </li></ul></ul>
  17. 17. Design or Select Evaluation Measures <ul><li>Payoff outcomes </li></ul><ul><ul><li>Is the problem solved? </li></ul></ul><ul><ul><ul><li>Costs avoided </li></ul></ul></ul><ul><ul><ul><li>Increased outputs </li></ul></ul></ul><ul><ul><ul><li>Improved quality </li></ul></ul></ul><ul><ul><ul><li>Improved efficiency </li></ul></ul></ul>
  18. 18. Design or Select Evaluation Measures (2) <ul><li>Learning Outcomes </li></ul><ul><ul><li>Use instrument you’ve already developed for the summative evaluation </li></ul></ul><ul><ul><li>But measure the entire program </li></ul></ul>
  19. 19. Design or Select Evaluation Measures (3) <ul><li>Attitudes </li></ul><ul><ul><li>Rarely the primary payoff goals </li></ul></ul><ul><ul><li>Ask about learner attitudes toward </li></ul></ul><ul><ul><ul><li>learning </li></ul></ul></ul><ul><ul><ul><li>instructional materials </li></ul></ul></ul><ul><ul><ul><li>subject matter </li></ul></ul></ul><ul><ul><li>Indices of appeal </li></ul></ul><ul><ul><ul><li>attention, likeableness, interest, relevance, familiarity, credibility, acceptability, and excitement </li></ul></ul></ul>
  20. 20. Design or Select Evaluation Measures (4) <ul><li>Level of Implementation </li></ul><ul><ul><li>degree to which the instruction was implemented </li></ul></ul><ul><li>Costs </li></ul><ul><ul><li>Cost-feasibility </li></ul></ul><ul><ul><li>Cost-effectiveness </li></ul></ul>
  21. 21. Alternative Designs <ul><li>Instruction then posttest </li></ul><ul><li>Pretest then instruction then posttest </li></ul>
  22. 22. The Report <ul><li>Summary </li></ul><ul><li>Background </li></ul><ul><ul><li>Needs assessment, audience, context, program description </li></ul></ul><ul><li>Description of evaluation study </li></ul><ul><ul><li>Purpose of evaluation, evaluation of the design, outcomes measured, implementation measures, cost-effectiveness info., analysis of unintentional outcomes </li></ul></ul>
  23. 23. The Report (continued) <ul><li>Results </li></ul><ul><ul><li>Outcomes, implementation, cost-effectiveness info., unintentional outcomes </li></ul></ul><ul><li>Discussion </li></ul><ul><ul><li>causal relationship between program & results </li></ul></ul><ul><ul><li>Limitation of study </li></ul></ul><ul><li>Conclusion & Recommendations </li></ul>
  24. 24. Summary <ul><li>Summative evaluation is after implementation </li></ul><ul><li>Limitations of subjective and objective evaluation </li></ul><ul><li>What to include in the report </li></ul>