• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
The Evaluation Checklist
 

The Evaluation Checklist

on

  • 12,785 views

Advantages and disadvantages of evaluation checklists and how to use them to improve evaluation practice. Presented at USF Center for Research, Evaluation, Assessment, and Measurement.

Advantages and disadvantages of evaluation checklists and how to use them to improve evaluation practice. Presented at USF Center for Research, Evaluation, Assessment, and Measurement.

Statistics

Views

Total Views
12,785
Views on SlideShare
12,772
Embed Views
13

Actions

Likes
1
Downloads
82
Comments
2

4 Embeds 13

http://www.slideshare.net 9
http://www.linkedin.com 2
http://www.docseek.net 1
http://www.docshut.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Mnemonic deviceInherently systematicProvides guidance for collection of evidenceHighly relevant for evaluative purposesNOT to be thought of as measurement device
  • No statistically significant difference found among groupsInter-item correlation ranged from .79 to .96

The Evaluation Checklist The Evaluation Checklist Presentation Transcript

  • The Evaluation Checklist
    Wes Martz, Ph.D.
  • A mnemonic device that consists of a list of activities, items, and criteria used to perform a certain task.
    2
  • Guidance for the collection of relevant evidence used to determine the merit, worth, or significance of an evaluand.
    3
  • Existing Evaluation Checklists
    Evaluation management
    Evaluation models
    Evaluation values and criteria
    Metaevaluation
    Evaluation capacity building
    Checklist creation
    4
    http://www.wmich.edu/evalctr/checklists
  • Checklist Organizer
    5
  • Good Checklists
    6
  • Criteria to Evaluate Checklists
    Applicability to full range of intended uses
    Clarity
    Comprehensiveness
    Concreteness
    Ease of use
    Fairness
    Parsimony
    Pertinence to the content area
    7
    Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf
  • Checklist Advantages
    Consolidate vast knowledge in a parsimonious manner
    Improve task performance
    Reduce influence of halo and Rorschach effects
    Reduce resource use
    Improve memory recall
    Set out minimum necessary steps in a process
    8
    Scriven, M. (2005). Checklists. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 53-59). Thousand Oaks, CA: Sage.
  • Checklist Disadvantages
    Evaluation myopia
    Inappropriate use
    Fatigue resulting from overuse
    Unnecessary complexity decreases reliability
    Burdensome process delays completing evaluation
    9
    Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005
  • Organizational Effectiveness Evaluation Checklist (OEC)
    Developing and Validating an Evaluation Checklist: A Case Example
    10
  • OEC Overview
    Organizational evaluation process framework
    Iterative, explicit, weakly sequential
    Six steps, 29 checkpoints
    Criteria of merit checklist
    12 universal criteria of merit
    84 suggested measures
    11
  • OEC Overview
    12
  • Criteria to Evaluate the OEC
    Applicability to full range of intended uses
    Clarity
    Comprehensiveness
    Concreteness
    Ease of use
    Fairness
    Parsimony
    Pertinence to the content area
    13
  • OEC Validation Process
    Phase 1: Expert panel review
    Critical feedback survey
    Written comments made on checklist
    Phase 2: Field test
    Single-case study
    Semi-structured interview
    14
  • Expert Panel Overview
    Study participants
    Subject matter experts (organizational and evaluation theorists)
    Targeted users (professional evaluators, organizational consultants, managers)
    Review OEC for providing critical feedback
    Identify strengths and weaknesses
    Complete the critical feedback survey
    Write comments directly on the checklist
    15
  • Expert Panel Data Analyses
    Critical feedback survey
    Descriptive statistics
    Parametric and nonparametric analysis of variance
    Written comments on checklist
    Hermeneutic interpretation
    Thematic analysis to cluster and impose meaning
    Triangulation across participants to corroborate or falsify the imposed categories
    16
  • Key Findings
    Pertinent to content area
    Clear
    Fair
    Sound theory
    Parsimony and ease of use were identified as areas to address
    17
    • Content relevance
    • Representativeness
    • Substantive validity
  • Field Test Overview
    Conducted evaluation using revised OEC
    Strictly followed the OEC to ensure fidelity
    Post-evalsemi-structured client interview
    A formative metaevaluation to detect and correct deficiencies in the process
    18
  • Observations from Field Test
    Structured format minimized “scope-creep”
    Identified several areas to clarify in OEC
    Reinforced need for multiple measures,transparency in standards
    Minimal disruption to the organization
    19
  • Validity Study Assessment
    Strengths
    Relatively quick validation process
    Based on relevant evaluative criteria
    Features a real-world application
    Weaknesses
    Single-case field study
    Selection of the case study
    Selection of the expert panel members
    20
    Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005
  • Lessons Learned
    Checklist development should address unique attributes of the evaluand
    Sampling frame is critical
    Checklist validation should be grounded in theory, practice, and use
    Mixed method approach provides increased confidence in validation conclusions
    All checklists are a “work-in-process”
    21
  • Additional Checklist Resources
    Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009).
    Scriven, M. (2007). The logic and methodology of checklists. Available at http://www.wmich.edu/evalctr/checklists/papers/logic&methodology_dec07.pdf.
    Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf.
    Stufflebeam, D. L. (2001). Evaluation checklists: Practical tools for guiding and judging evaluations. American Journal of Evaluation, 22, 71–79.
    22
    http://www.wmich.edu/evalctr/checklists
  • The Evaluation Checklist
    martz@EvaluativeOrganization.com
    @wesmartz