Your SlideShare is downloading. ×
The Evaluation Checklist
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

The Evaluation Checklist

13,099

Published on

Advantages and disadvantages of evaluation checklists and how to use them to improve evaluation practice. Presented at USF Center for Research, Evaluation, Assessment, and Measurement.

Advantages and disadvantages of evaluation checklists and how to use them to improve evaluation practice. Presented at USF Center for Research, Evaluation, Assessment, and Measurement.

2 Comments
1 Like
Statistics
Notes
No Downloads
Views
Total Views
13,099
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
108
Comments
2
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Mnemonic deviceInherently systematicProvides guidance for collection of evidenceHighly relevant for evaluative purposesNOT to be thought of as measurement device
  • No statistically significant difference found among groupsInter-item correlation ranged from .79 to .96
  • Transcript

    • 1. The Evaluation Checklist
      Wes Martz, Ph.D.
    • 2. A mnemonic device that consists of a list of activities, items, and criteria used to perform a certain task.
      2
    • 3. Guidance for the collection of relevant evidence used to determine the merit, worth, or significance of an evaluand.
      3
    • 4. Existing Evaluation Checklists
      Evaluation management
      Evaluation models
      Evaluation values and criteria
      Metaevaluation
      Evaluation capacity building
      Checklist creation
      4
      http://www.wmich.edu/evalctr/checklists
    • 5. Checklist Organizer
      5
    • 6. Good Checklists
      6
    • 7. Criteria to Evaluate Checklists
      Applicability to full range of intended uses
      Clarity
      Comprehensiveness
      Concreteness
      Ease of use
      Fairness
      Parsimony
      Pertinence to the content area
      7
      Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf
    • 8. Checklist Advantages
      Consolidate vast knowledge in a parsimonious manner
      Improve task performance
      Reduce influence of halo and Rorschach effects
      Reduce resource use
      Improve memory recall
      Set out minimum necessary steps in a process
      8
      Scriven, M. (2005). Checklists. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 53-59). Thousand Oaks, CA: Sage.
    • 9. Checklist Disadvantages
      Evaluation myopia
      Inappropriate use
      Fatigue resulting from overuse
      Unnecessary complexity decreases reliability
      Burdensome process delays completing evaluation
      9
      Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005
    • 10. Organizational Effectiveness Evaluation Checklist (OEC)
      Developing and Validating an Evaluation Checklist: A Case Example
      10
    • 11. OEC Overview
      Organizational evaluation process framework
      Iterative, explicit, weakly sequential
      Six steps, 29 checkpoints
      Criteria of merit checklist
      12 universal criteria of merit
      84 suggested measures
      11
    • 12. OEC Overview
      12
    • 13. Criteria to Evaluate the OEC
      Applicability to full range of intended uses
      Clarity
      Comprehensiveness
      Concreteness
      Ease of use
      Fairness
      Parsimony
      Pertinence to the content area
      13
    • 14. OEC Validation Process
      Phase 1: Expert panel review
      Critical feedback survey
      Written comments made on checklist
      Phase 2: Field test
      Single-case study
      Semi-structured interview
      14
    • 15. Expert Panel Overview
      Study participants
      Subject matter experts (organizational and evaluation theorists)
      Targeted users (professional evaluators, organizational consultants, managers)
      Review OEC for providing critical feedback
      Identify strengths and weaknesses
      Complete the critical feedback survey
      Write comments directly on the checklist
      15
    • 16. Expert Panel Data Analyses
      Critical feedback survey
      Descriptive statistics
      Parametric and nonparametric analysis of variance
      Written comments on checklist
      Hermeneutic interpretation
      Thematic analysis to cluster and impose meaning
      Triangulation across participants to corroborate or falsify the imposed categories
      16
    • 17. Key Findings
      Pertinent to content area
      Clear
      Fair
      Sound theory
      Parsimony and ease of use were identified as areas to address
      17
      • Content relevance
      • 18. Representativeness
      • 19. Substantive validity
    • Field Test Overview
      Conducted evaluation using revised OEC
      Strictly followed the OEC to ensure fidelity
      Post-evalsemi-structured client interview
      A formative metaevaluation to detect and correct deficiencies in the process
      18
    • 20. Observations from Field Test
      Structured format minimized “scope-creep”
      Identified several areas to clarify in OEC
      Reinforced need for multiple measures,transparency in standards
      Minimal disruption to the organization
      19
    • 21. Validity Study Assessment
      Strengths
      Relatively quick validation process
      Based on relevant evaluative criteria
      Features a real-world application
      Weaknesses
      Single-case field study
      Selection of the case study
      Selection of the expert panel members
      20
      Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005
    • 22. Lessons Learned
      Checklist development should address unique attributes of the evaluand
      Sampling frame is critical
      Checklist validation should be grounded in theory, practice, and use
      Mixed method approach provides increased confidence in validation conclusions
      All checklists are a “work-in-process”
      21
    • 23. Additional Checklist Resources
      Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009).
      Scriven, M. (2007). The logic and methodology of checklists. Available at http://www.wmich.edu/evalctr/checklists/papers/logic&methodology_dec07.pdf.
      Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf.
      Stufflebeam, D. L. (2001). Evaluation checklists: Practical tools for guiding and judging evaluations. American Journal of Evaluation, 22, 71–79.
      22
      http://www.wmich.edu/evalctr/checklists
    • 24. The Evaluation Checklist
      martz@EvaluativeOrganization.com
      @wesmartz

    ×