Program evaluation in the field of health professions education

1. Programme
Evaluation
Hirotaka Onishi MD, PhD, MHPE
Dept of International Cooperation for Medical Education
International Research Center for Medical Education
Graduate School of Medicine, University of Tokyo

2. Programme
 Context of programme evaluation
 General activities for people to achieve
a certain goal
 Context of policy evaluation
 Aggregated measures and projects to
achieve a certain outcome or goal

3. Definition of Program Evaluation
 Scriven (1967) and Scriven (1991), Carol Weiss (1998),
Melvin Mark, Gary Henry, and George Julnes (2000),
Robert Stake (2000), Daniel Stufflebeam (2001), and
Thomas Schwandt (2008).
 a systematic process of gathering
evidence to inform judgments of
whether a program, product, policy, or
system is meeting is goals and how it
can be improved to better do so.
 Value in the programme seems more
emphasized than before.

4. What is Evaluated?
 Processes
 operations
 functions
 Outcomes, Results
 effectiveness
 outputs
 impacts
 sustainability

5. Objectives
 Formative evaluation
 To develop and improve the programme
 Summative evaluation
 For decision-making and accountability
 Evaluation for valuing
 Learning effect by participating in
evaluation process
 Evaluation for advertisement

6. Life Curve of a Programme
and Change in Objective
Length of a Programme
Formative Evaluation
Summative Evaluation
Relativeimportance

7. Internal vs External
 External evaluation/analysis
 Neutral from internal politics
 Difficult to access some information
 Internal evaluation/analysis
 Rich in information
 Evaluation as a learning process
 Important aspect is sometimes
ignored or concealed

8. Kirkpatrick’s 4 Levels for
Training Programmes
Benefit
to patients
Organisational
change
Behaviour
change
Acquisition of
knowledge and skills
Modification of attitudes
and perceptions
Learners’ reactions
Level 4
Level 3
Level 2
Level 1

9. Relationship Among Design,
Conduct, and Reasoning
Design Implement
Research
Question Study Plan Actual Study
Intended
Sample
Findings in
the Study
Truth in the
Study
Truth in the
Universe
Infer Infer
Internal Validity
Phenomena of
Interest
Intended
Variables
Actual
Measurements
External Validity
Actual
Subjects
Target
Population
Error Error
(Hurry SB, et al. Designing Clinical Research, 2007)

10. Research Design
 Post-test only vs Pre-Post-test
 Improvement is measured by pre-post
 Whether control or not?
 Control group: quasi-experimental
 Whether randomized or not?
 Randomization: true experimental

11. Common Evaluation
Designs
X: Educational intervention, O: Observation or measurement,
E: Experimental group, C: Control group, R: Random allocation
Pre
experi-
mental
Post-test only design X - - O
Pre-post-test design O1 - - X - - O2
Quasi-
experi-
mental
Nonrandomised controlled
pre-post-test design
E O1 - - X - - O2
C O1 - - - - - O2
True
experi-
mental
Nonrandomised
post-test only design
E X - - - O1
R
C - - - O1
Nonrandomised controlled
pre-post-test design
E O1 - - X - - O2
R
C O1 - - - - - O2

12. Weaknesses of
Pre-experimental Design
 Ex: Try to confirm effectiveness of a class
 O1 - - X - - O2： Pre-post design
 O1 : 55%, O2 : 70% (1 mo later)
 Can we conclude the class was effective?
 Threats for internal validity to be considered:
 History: External events may influence
 Maturation: Subjects may have unrelated growth
 Testing: 1st test may improve the 2nd test result
 Instrumentation: Unreliable instrument, etc.
 …

13. Design for Quantitative Data
Analysis
 If you have randomised control group
(without any educational intervention), is it
ethically allowed?
 Is the control group demographically or
psychologically equivalent with
experimental group?
 If a programme implemented nationwide,
you cannot have a control group

14. Better Quasiexperimental Design
with Unequivalent Control Group
E O1 O2 X O3
C O1 O2 O3
O1 O2 O3
E
C

15. Better Quasiexperimental Design
Without Control Group
O1A O1B X O2A O2B

16. Data
Quantitative
 Hypothesis testing
 More standardised,
objective/reliable,an
d generalisable
results
 Easier to rank-order
Qualitative
 Naturalistic
 Thick description
 Easier to focus on
social process
 Multiple data
collection methods
Both data are important for evaluation

17. Compression of Information
Q: You were satisfied with the lecture.
1) Definitely 2) Yes 3) Somewhat 4) NoPneumonia
Etiology
S. pneumoniae
- most
common
H. influenzae
K. pneumoniae
Her lecture is well organised
and easy to understand.
1)
Her lecture is understandable,
but sometimes too simple.
2)
Lecture note is OK, but I
hate her artificial smile.
3)
Her clothes is too old-
fashioned.
4)

18. Data Used for Evaluation
Subjective Objective
Qualitative
Focus groups
Interviews
Analysis of tasks
successfully
completed in an
OSCE station
Quantitative Surveys
Trends in examination
scores

19. Program Evaluation
 Scriven (1967)
 Judging the worth or merit of something
 Quantitative methodology was
emphasised
 Scriven(1991)
 Valuing the programme by clarified and
justified criteria

20. Difference between
Research and Evaluation
Research Evaluation
Focus for inquiry Consideration Decision-making
Generalisability
of results
High Low
Criteria for
importance
Truth Value

21. Dimension of Different
Evaluation Approaches
(Worthen, Sanders, & Fitzpatrick)
Objectives-
oriented
Management-
oriented
Consumer-
oriented
Expertise-
oriented
Participants-
oriented
Utilitarian
Evaluation
Intuitional-
pluralist
Evaluation
• Stake’s
Countenance
Model
• Formal/informal
Professional
Review
(accreditation etc.)
• Ad Hoc Panel/
individual Review
• Scriven’s model
(goal free
evaluation)
• Tylerian
evaluation
Approach
• Logical method
------------------------
• Stufflebeam’s
CIPP evaluation
Model

22. Management-oriented Evaluation
~CIPP Model~ (Stufflebeam, 2003)
 Context：Evaluate plans, needs, etc.
 Input：Evaluate systems, resources, etc.
 Process：Evaluate implementation:
Issues and improved points
 Product：Evaluate outcomes. Review
and update the initial plan.

23. Logic Model
 Also called as logical framework, project cycle
(Kellogg Foundation, 2001)
 Purpose: To assess the causal relationships
between the elements of the program
 Evaluation for international assistance projects
are conducted by this framework
input activity output outcome impact

24. Five Evaluation Criteria for
International Assistance (DAC in OECD)
 Relevance
 Effectiveness
 Efficiency
 Impact
 Sustainability

25. 25
PCM Evaluation Model
Overall
Goal
Rele-
vance
Impact
Sustain-
ability
Project
Purpose
Effective-
nessOutputs
Efficiency
Inputs

26. Relevance
 Key question: Are Project Purpose and
Overall Goal meaningful as the objective of
the Project?
Check Points:
 Country’s policies/needs
 Donor’s policies/needs
 Local people’s or target group’s needs
 Local conditions such as economic conditions,
resources, etc.

27. Effectiveness
 Key question: To what extent has Project Purpose been
achieved, and how much contribution have Outputs
made to the achievement of Project Purpose?
Check Points:
 Achievement of Project Purpose
 Relationships between Outputs and Project Purpose
 Factors that have contributed to the achievement of
Project Purpose
 Factors that have hindered the achievement of Project
Purpose
 Important Assumptions

28. Efficiency
 Key question: What ratio of Inputs was
converted into Outputs? How efficiently have
the Inputs been used?
Check Points:
 Achievement of Outputs
 Quality, quantity and appropriateness of the
means, methods, and timings of Inputs
 Situations of Input utilization

29. Impacts
 Key question: What positive and negative,
and direct and indirect effects of the
implementation of the Project are happening?
Check Points:
 Possibility of Overall Goal to be achieved
 Relationships between Project Purpose and Overall Goal
 Positive or negative effects of the Project from the cross- cutting
viewpoints?
 Influence on areas outside the project site
 Changes in the life of the target group
 Factors that have influenced the project activities, management
system etc.

30. Sustainability
 Will the positive effects from the
implementation of the project continue after
the external assistance is terminated?
Check Points:
 Continuity of the implementation of Activities
 Organizational sustainability
 Financial sustainability
 Technological sustainability

31. Narrative Summary
Overall Goal The new medical education system is recognized as the
standardized model in Afghanistan
Project Purpose The new medical education system of GP training model
is implemented in KMU
Outputs 1. Capacity for GP training improved
2. Function of EDC reinforced
Activities 1-1 Conduct FD
1-2 Create materials
1-3 Implement new teaching methods
1-4 Formulate an evaluation system
1-5 Conduct trainings
2-1 Develop a system to collect information and statistics
2-2 Publish newsletters on medical education
2-3 Develop EDC staff
2-4 Promote awareness
2-5 Plan, operate, manage the FD
PDM Example: Afghan Project

32. 32
Narrative Summary
Overall
Goal
The new medical education system is
recognized as the standardized model
in Afghanistan
Project
Purpose
The new medical education system of
GP training model is implemented in
KMU
Outputs 1. Capacity for GP training improved
2. Function of EDC reinforced
Activities 1-1 Conduct FD
1-2 Create materials
1-3 Implement new teaching methods
1-4 Formulate an evaluation system
1-5 Conduct trainings
2-1 Develop a system to collect
information and statistics
2-2 Publish newsletters on medical
education
2-3 Develop EDC staff
2-4 Promote awareness
2-5 Plan, operate, manage the FD
1. No big change in basic policy
2. Gov. recognizes that GP is
essential
1. Organizations and authorities
support undergraduate
education
2. Collaboration between
MOHE/KMU and MOPH
1. Trained and competent staff
not part of brain drain
2. Full-time faculty members and
office workers secured in EDC
Political stability secured
Preconditions
Important Assumptions