This document discusses program evaluation, outlining key concepts and approaches. It describes the purposes of program evaluation as determining if objectives are met and improving decision making. Formative and summative evaluations are explained, with formative used for ongoing improvement and summative to determine effects. Both quantitative and qualitative methods are appropriate, including experimental, quasi-experimental and non-experimental designs. Stakeholder involvement, utilization of results, and addressing ethical considerations are important aspects of program evaluation.

1. Sheryl Ryan, PhD, OTR/L

2. Week 5

3.  Program evaluation:
 Definition
 Purposes
 Describe several approaches
 Discuss the process
 Describe the differences between experimental, quasi-experimental, and non-
experimental evaluation designs.
 Discuss the appropriate use of qualitative methods in program evaluation.
 Identify the uses of evaluation results.
 Discuss the importance of disseminating the results of program development and
evaluation.
 Identify the ethical considerations in designing and conducting evaluation research.

4. Quiz 1
Vocabulary database
2 Quotes/2 Questions
Case: Ocean Therapy
Program Evaluation
Photo: http://www.surfersway.org/

5.  Appreciative inquiry approach
 Conceptual, instrumental, and process
use
 Contingency perspective
 Efficiency evaluations
 Impact evaluations
 Outcome evaluations
 Process evaluations
 Experimental, non-experimental, and
quasi-experimental designs
 Formative and Summative evaluation
 Logic models
 Managerial approach
 Objectives approach
 Participatory approach
 Program efficiency evaluation
 Program evaluation
 Qualitative
 Quantitative
 Stakeholders
 4D Model
 Indicators

6. Chapter 6

7. Carly Rogers:
Surfing – Infinite
Possibilities to
Heal
https://www.youtube.com/watch?v=
Wfb8tHn8Xv4
http://jimmymillerfoundation.org/ocean-therapy/

8. Collect data and
information
Use it to inform future
action
“Evaluators rely on stakeholders to
help develop the essential questions
that will be answered in the
evaluation” (Scaffa and Reitz, 2014:97)

9.  Evaluation is vital to helping design
and shape your practice
 Evaluation of your program should be
one of the first things you consider
during the development stage
 Evaluation should be continuous,
throughout the life of your program
https://www.opendemocracy.net/openglobalrights/evaluation-and-human-rights

10. :
Formative or Process Evaluations
 Determines whether program
activities have been implemented as
intended and resulted in certain
outputs. The results are used for
program improvement.
 Theoretical framework
 Design
 Activities
 Operation
Summative
 When the results are used to
determine effects or
continue/discontinue the program.
 Outcomes
 Impact
 Effectiveness/Efficiency
https://education.uky.edu/evaluationcenter/

11. Needs assessment
Program theory
Program implementation
Program outcome – Long-term effects
Program impact – Immediate effects
Program efficiency – Cost/benefit analysis justifies funding
https://www.isasurf.org/first-ever-isa-world-adaptive-surfing-championship-set-to-be-held-in-la-jolla-california-september-24-27/

12. Objectives Approach
 Collect data by previously agreed
upon methods to determine whether
goals and objects are being met.
 Relatively easy to quantify, but
limited to evaluating program
objectives only.
 Rarely used in isolation.
Managerial Approach
 Goal is to provide useful information
to managers to improve their decision
making.
 Evaluations look at how the program
operates as well as goals.
 Value of a program is determined by
managers and is reflected in decisions
they make based on evaluation
results.

13. Participatory Approach
 A formative/process approach
 Qualitative
 Considers the needs of all stakeholder
groups rather than just managers –
360* approach.
 Determines critical information
needed by each of stakeholder groups
to develop program improvement
plans.
Utilization-Focused Approach
 A process for including “users” in the
design of the evaluation process.
 Shifts attention from the program to
be evaluated and focuses on those
who will use the program evaluation
results.
 Highly situation and context specific.
 Mixed qualitative and quantitative
methods.

14. Appreciative Inquiry Approach
 4D Model
 Discovery – appreciate what is good
 Dream – envision what might be
 Design – consider what should be
 Destiny – implementing change
AKA Transtheoretical Model of Health
Behavior Change
 5 Principles
 Constructivist – multiple realities exist
 Simultaneity – inquiry is intervention
 Poetic – programs narrating own stories
and can change directions at any time
 Anticipatory – imagination of program
guides its current actions
 Positive – focus on positive experiences
increases motivation, inspiration, and
engagement
http://dancumberworth.co.uk/practice/

15. Develop evaluation questions
Determine
data needs
Choose evaluation
methods
Utilize evaluation
results
Identify stakeholders
http://www.theintrinsicvalue.com/tag/picture-of-a-airplane

16.  Determine OT Related Data Needs  Occupational performance
 Adaptation
 Health and wellness
 Participation
 Prevention
 Quality of life
 Role competence
 Self-advocacy
 Occupational justice
https://www.amazon.com/Occupational-Therapy-Practice-Framework-Process/dp/1569003610

17. Choosing Evaluation Methods
 Quantitative Designs
 Non-experimental
 Cross-sectional, cohort
 Quasi-experimental
 Compare two groups, one receiving
intervention
 Experimental
 Randomized controlled trial
 Qualitative Designs
 Provides an in-depth perspective on
phenomena that are not easily
quantifiable.
 Thick description, naturalistic,
experience oriented
 Interviews, focus groups, observation,
document review

18.  Instrumental use – to inform future action
 Conceptual use – impact on decision makers’
thought processes
 Process use – cognitive and behavioral changes result
from participating in the process
 Symbolic use – for political gain
COMMUNICATION!

19.  Beneficence
 Select appropriate evaluation approach
and outcome measures
 IRB approval
 Using current assessments, following
copyright laws
 Nonmaleficence
 Train evaluators and staff to ensure
competency
 Autonomy/Confidentiality
 Secure evaluation materials and data
 Compliance with evaluation protocol
and confidentiality as needed
 Procedural Justice
 IRB approval
 Veracity
 Report accurate results to participants
and stakeholders in a timely manner
 Fidelity
 Avoid conflicts of interest