Evaluating a community project

1. L U K U M A Y , G . B S C N , M S C . P H E C
Community Evaluation

2. OBJECTIVES
 At the end of this session each student will be able
to;
 Briefly explain evaluation principles
 Explain the evaluation process
 Describe components of evaluation
 Describe the evaluation strategies


3. Evaluation
 It determines the worth or value of
something
 The community nurse evaluates the
responses of community to a health program
to measure progress that is being made
towards the program goals and objectives
 Evaluation is a final step in nursing process,
but it is linked to assessment which is the
first step

4. Evaluation cont………..
 Nursing practice is cyclic as well as dynamic,
and for community based interventions to
be timely and relevant, the community data
base, nursing diagnoses and health program
plans must be evaluated routinely.
 The effectiveness of community nursing
interventions depends on continuous
reassessment of the community`s health
and on appropriate revisions of planned
interventions.

5. Evaluation principles
 While working with a community as partner,
we also base our program evaluation on
principles,
1. Strengthen programs. Our goal is health
promotion and improving the self-reliance
of the community. Evaluation assists in
attaining this goal by providing an ongoing
and systematic process for assessing the
program, its impact, and its outcomes.

6. 2. Use multiple approaches. In addition to
multidisciplinary approaches, evaluation
methods may be numerous and varied. No
single approach is favored, but the method
chosen must be congruent with the purposes
of the program

7. 3. Design evaluation to address real issues.
Community-based programs, rooted in the
“real” community and based on an
assessment of that community, must contain
an evaluation to measure those criteria of
importance to the community

8. 4.Create a participatory process. Just as the
community members were part of
assessment, analysis, planning, and
implementation, so too they must be partners
in evaluation.

9. 5.Allow for flexibility. “Evaluation approaches
must not be rigid and prescriptive ,or it will be
difficult to document the incremental,
complex, and often subtle changes that occur”
(W. K. Kellogg Foundation, 1998, p. 3)

10. 6. Build capacity. The process of evaluation,
in addition to measuring outcomes, should
enhance the skills, knowledge, and attitudes
of those engaged in it. This includes both
professionals and nonprofessionals

11. The evaluation process
 Evaluation process –is the method of doing
an analysis on the effectiveness of the
implementation of the planned
interventions

12.  On introduction to evaluation program, the
evaluation process uses the three parts
model which involves
the process of implementing the program
the program’s impact, and
the outcome of the program

13. The process(formative evaluation)
 The formative evaluation involves the
evaluation of the formats of implementation,
answer the qn like are we doing what we
said we would do?, did we deliver the
program?
 It involves the evaluation of the following:
 Site response
 Recipient response

14.  Practitioner response
 Competencies of personnel
 This is done while the process of
implementation is going on

15. Impact (Summative-
Short-Term Outcome)
 Involves conducting evaluation at the end of
the program .Immediate effects of program
on, for example
 Knowledge, Attitudes, Perceptions, Skills
and beliefs
 To determine if factors that affect health
both within the individual and in the
environment have changed. For example,
did the person’s behavior change after an
intervention?

16. Outcome(long-term outcome)
 The evaluator focus on the outcome of the
intervention. Is there a change in the
problem after intervention? If no
improvement, then what was wrong?
 Considering specific evaluation strategies, it
is important to consider the “evaluability” of
the program and this is done by asking the
following questions

17.  Are the program activities stated in precise
words where concepts can be measured?
 Has the time frame for attaining the change
been included?
 Are the direction and magnitude of the
change included?
 Has the method of measuring the change
been included?

18.  Are the data that will be needed to measure
the objectives available at a reasonable cost?
 Are the program activities that are designed
to meet the objectives plausible?
 The evaluation process involves the cyclical
process of nursing care process that need
continued validation of data/ information
until the problem has been resolved

19. Components of evaluation
 Evaluation comprises of 5 components
1. Relevancy
2. Progress
3. Cost efficiency
4. Effectiveness
5. Outcome

20. Relevancy
 Relevancy determines the reasons for having
a program or set of activities.
 Questions of relevancy may be more
important for existing programs than for
new programs.
 Frequently, a program, such as a blood
pressure screening, is planned to meet an
expressed community need

21. Progress
 Are program activities following the
intended plan?
 Are appropriate staff and materials available
in the right quantity and at the right time to
implement the program activities?
 Are expected numbers of clients
participating in the scheduled program
activities?

22. Cost Efficiency
 What are the costs of a program? What are
its benefits?
 Are program benefits sufficient for the costs
incurred?
 Cost–efficiency evaluation measures the
relationship between the results (benefits) of
a program and the costs of presenting the
program (such as staff salary and materials).

23. Effectiveness( impact)
 Were program objectives met?
 Were the clients satisfied with the program?
Were program providers satisfied with the
activities and client involvement?
 Effectiveness focuses on formative
evaluation as well as the immediate, short-
term results

24. Outcome
 What are the long-term implications of the
program?
 As a result of the program, what changes in
behaviour can be expected in 6 weeks, 6
months, or 6 years?
 Effectiveness measures the immediate
results, whereas outcome evaluation
measures whether the program activities
changed the initial reason for the program.

25. Evaluation strategies
 Data collection and analysis
Program “evaluation can be defined as the
consistent, ongoing collection and analysis
of information for use in decision making”
(W. K. Kellogg Foundation, 1998).
As such, the choice of approach or method
to collect the information is an important
decision and needs to be agreed on by all
involved from the beginning.

26. Things to consider while choosing
data collection method to be used
1. What resources are available for the
evaluation tasks?
2. Is the method sensitive to the
respondents/participants of the program?
3. How credible will your evaluation be as a
result of this method?
4. What is the importance of the data to be
collected? To the overall program?

27. Data collection methods
 Case study
 Survey
 Experimental Design
 Monitoring (Process)
 Cost-Benefit and Cost-
Effectiveness Analyses

28. USEFULNESS OF CASE STUDY TO
EVALUATION
 Case study is the presentation and
consequent analysis of a person,
group or event.
 Case studies involve a particular method
of research.
 Rather than using large samples and
following a rigid protocol to examine a
limited number of variables, case study
methods involve an in-depth, longitudinal
examination of a single instance or events.

29. Cont…
 They provide a systematic way of looking at
events, collecting data, analyzing
information, and reporting the results.
 As a result the researcher may gain a
sharpened understanding of why the
instance happened as it did, and what might
become important to look at more
extensively in future research.
 Case studies lend themselves especially to
generating (rather than testing) hypotheses.

30. Usefulness of case study to
evaluation
 Help answer questions of relevance
 Questioning clients and health care providers
helps to explore perceptions of how well the
program is meeting its defined goals and to
ascertain problem areas and possible solutions
 Questions of progress can also be addressed
through case study method because the case
study provides an examination of the program
,much can be learned if program activities are
already in place.

31. Cont…………
 Determine aspects of effectiveness, such
as whether the aims of the program have
been met in the short run, it is very difficult
to measure long-term consequences unless
the case-study method is conducted over a
long period that allows a retrospective view
of the program

32. Point to note
 Cost efficiency is difficult to evaluate
using case study method because the case
study method is designed to look at only one
program, and to evaluate if the program
could have offered more economically, a
comparable program must exist.
 This method is not formatted to look at two
programs and compare them.

33. Surveys
 A survey is a method of collecting
information ,so it can be used to collect
evaluation information.
 Surveys are usually completed by :
 self-administered questionnaires
 or by personal interviews

34. Formulation of survey:
 Formulation of survey Can be either
descriptive or analytical.
 Surveys are formulated to describe
(descriptive surveys)
 or to analyze relationships (analytic
surveys).
 (Actually, most surveys can be used both to
describe and to analyze.)

35. Cont……….
• Surveys are usually performed for summative
(impact) evaluation.
• Did the program accomplish what it was proposed
to do? Was the program perceived as successful by
clients? By personnel? If the program was
considered successful, what parts were most
helpful? Least helpful? What should be changed?
Left unchanged?
• The questions asked by the survey are determined
by the initial list of questions about program
evaluation

36. Reliability and Validity
 Surveys that are used for program
evaluation must be concerned with the
reliability and validity of the information
collected.
 Reliability deals with the repeatability, or
reproducibility, of the data (i.e., if the same
questions were asked of the same people 1
week later, would the same responses be
recorded?).

37. Cont:
 Validity is the correctness of the
information. If questions are written to
evaluate knowledge, and the answers of the
respondents reflect behaviors, then the
questions are not valid because they do not
measure what they claim to measure.

38. Usefulness of survey to
Evaluation
 Surveys can be very valuable to answer
questions of relevance, or the need for
proposed or existing programs,
especially if the perceptions of clients,
providers, and management are solicited.
 surveys can measure progress.
 Cost efficiency, effectiveness, and outcome
are difficult to measure using a survey.

39. Cont:
• Although a survey can measure the
perceived efficiency of the program or ideas
on alternative ways of operating to make the
program more cost-efficient, these
perceptions are formed only in the context
of the existing program.

40. EXPERIMENTAL DESIGN
 Experimental Design is the method of
data collection which can provide an answer
to the crucial questions:
 Did the program make a difference?
 Are health behaviors, knowledge, and
attitudes changed as a result of the program
activities?
 Is the community healthier because of the
programs offered?

41.  However, the problem with experimental
studies in program evaluation is that they
require selective implementation, meaning that
people who participate are selected through a
process such as random assignment to a control
group and an experimental group.
 For many ethical, political, and community
health reasons, selective implementation is
difficult to complete and is sometimes
impossible.

42. Cont..
 Despite these problems, the experimental study
remains the best method to evaluate
summative effects (outcomes) of a program and
the only way to produce quantified information
on whether the program made a difference.
 The following designs are the most feasible and
appropriate to health care settings.
Pretest–Posttest One-Group Design.
Pretest–Posttest Two-Group Design

43. Pretest–Posttest One-Group Design
 Two observations are made,
- the first at Time 1 and
- the second at Time 2.
 The observation can be the prevalence of a
health state (e.g., the percentage of adults who
exercise regularly, the teenage pregnancy rate,
cases of child abuse, and so on), knowledge
scores, or other important community health
facts.
 Between Time 1 and Time 2, an experiment is
introduced.

44. Cont..
Pretest–Posttest One-Group Design
table. Time 1 Time 2
Experiment
al Group
Observation
1
Experiment Observation
2

45. PRETEST–POSTTEST TWO-
GROUP DESIGN
 The design has both an experimental group
and a control group.
 At Time 1, an observation is made of both
the experimental and control groups.
 Between Time 1 and Time 2, an experiment
is introduced with the experimental group.

46. Cont..
Pretest–Posttest One-Group Design
table. Time 1 Time 2
Experimental
Group
Observation
1
Experiment Observatio
n 2
Control Group Observation
1
Observatio
n 2

47. USEFULNESS TO EVALUATION
 An experimental design can yield data on
whether a program has produced the desired
outcomes when compared to the absence of
such a program .
or alternatively
 whether one program strategy has produced
better results with regard to the desired
outcomes than some other strategy.

48. MONITORING PROCESS
 Monitoring measures the difference between
the program plan and what has actually
happened
 Monitoring focuses on the sequence of
activities of the program, specifically, how
the program is to be implemented (the
activities), by whom (the personnel and
other resources), and when (the timing of
activities)

49. MONITORING CONT…..
 Monitoring is usually done with a chart, and,
although there are several different styles of
charts, all arrange activities in a sequence
and specify the time allotted to complete
each task

50. Monitoring Charts
 To construct a monitoring chart for your
program plan, information is needed on the
 inputs (resources necessary to carry out
the program such as personnel, equipment,
and finances),
 The process (the program activities, their
sequencing, and timing)
 outputs (the expected results of the
program, including immediate and long-
term health effects)
 It is helpful to make a list of inputs,
processes, and outputs.

51. USEFULNESS TO EVALUATION
 A monitoring chart measures progress and
can be used to evaluate whether a program
is on schedule and within budget.
 Perhaps no other evaluation method is as
perfectly suited to process evaluation as the
monitoring chart.

52. Cost-Benefit and Cost-Effectiveness
Analyses
 Much has been written and discussed about
the escalating cost of health, care services
and ways that cost can be reduced.
 Every program has a money cost both in
terms of the resources needed to offer the
program (e.g., personnel and equipment)
 and the money benefits to be gained from
improved health (such as increased worker
productivity

53.  Two of the most common methods of
analyzing the economic costs and benefits of
a program are
 cost–benefit analysis (CBA)
 and cost–effectiveness analysis (CEA).
 Both CBA and CEA are formal analytic
techniques that list all costs (direct and
indirect) and consequences (negative and
positive) of a particular program.

54. COST–BENEFIT ANALYSIS
CBA has the purposes of providing a basis for comparing
projects.
 It involves comparing the total expected cost of each
option against the total expected benefits, to see
whether the benefits outweigh the costs, and by how
much.

55. Cost benefit analysis
 In CBA, consequences or benefits of a
program are valued in terms of money(
shillings), this makes it possible to compare
different projects, because all measurement
is made in money
 Therefore, the worth of a project can be
judged by asking if shilling benefits exceed
shilling costs and, if so, by how much.

56. Cost-effectiveness analysis (CEA)
 Is a form of economic analysis that
compares the relative costs and outcomes
(effects) of two or more courses of action
 Cost-effectiveness analysis is often used in
the field of health services, where it may be
inappropriate to monetize health effect.

57.  CEA does not place a money value on either
the consequences or the costs of a project.
 A CEA can also be used if the costs of
alternative programs are the same or if only a
given amount of money exists and the
objective is to select the program with the
greatest benefits (not measured in shillings
terms

58.  The choice between CBA and CEA depends on
the type of questions and programs
considered.
 Neither technique is superior to the other Both
techniques can be used in planning for future
programs or as an evaluation strategy of
present or past programs

59. SUMMARY
 Several methods of evaluation discussed
 No one method will evaluate each
component equally well
 It is well to be knowledgeable about different
methods of evaluation
 And to discuss the benefits and limitations
to each with the community as the program
is planned and before program
implementation

60. References
1. Anderson, E.T. , McFarlane, J. ( 2011). Community
as partner: Theory and practice in nursing.
Philadelphia: Wolters Kluwer/ Lippincott Williams &
Wilkins.