SlideShare a Scribd company logo
1 of 35
EVALUATIONAS SOCIALINTERVENTION:AN
EMPIRICALSTUDYOF EMPOWERMENTEVALUATION
PRACTICEAND PRINCIPLE EFFECTS ON
PSYCHOLOGICALEMPOWERMENTAND SELF –
DETERMINATION OUTCOMES
Jeffrey Andrew Sheldon
Claremont Graduate University
2015
A dissertation submitted to the Faculty of Claremont Graduate University in partial
fulfillment of the requirements for the degree of Doctor of Philosophy in Psychology.
Overview
• Conceptual Primer
• Concerns About Empowerment Evaluation from Evaluation Theorists and Commentators
• The Need for Research on Empowerment Evaluation
• Research Questions
• Methodology and Analytical Strategy
• Survey Design
• Sampling Rationale and Procedures
• Participant Characteristics
• Results and Take-Aways for Research Questions 1 - 3
• Results and Implications for Practice and Policy from Research Question 4 ($)
• Strengths and Limitations of this Study
• Recommendations for Future Research on Empowerment Evaluation
• Conclusions
Empowerment Evaluation Practice, Principles, and Outcomes
2015
2
Conceptual Primer: EmpowermentTheory
• A long-term process individuals, organizations, and communities undertake for themselves rather than
something done to or for them (Lather, 1991, 1992; Claridge, 1996)
• Empowered outcomes are evidenced by individuals or aggregate bodies of individuals engaging in
behaviors that permit effective pursuit of planned social change: working with others, managing
resources, learning decision-making skills, influencing policy, accessing government, sharing
leadership, obtaining needed resources, being networked to others, etc. (Schulz et al., 1995; Zimmerman,
2000).
• Three interrelated components (Zimmerman, Israel, Schulz, & Checkoway, 1992).
• Interactional: Developing knowledge, skills, and strategies to become leaders/advocates for change.
• Intrapersonal: Appraisal/critical awareness of, and the motivation and agency to change one’s life
circumstances (i.e., self-efficacy).
• Behavioral: Individual-level actions requiring leadership that benefits individuals and communities by
influencing social policy.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
3
Conceptual Primer: Self-DeterminationTheory
• A general, organismic theory of human motivation organized around three sets of motivational processes—
intrinsic, extrinsic, and amotivational —and their relationship to self-determined behaviors - free will
and determination of one’s own fate or course of action without compulsion (Ryan & Connell, 1989).
• To be self-determined, experience well–being, and have optimal health (e.g., self-esteem, self-actualization,
vitality, positive affect, and work engagement) people have to satisfy three basic psychological needs
endowed by nature (Koestner, Ryan, Bernieri, & Holt, 1984):
• Competence: efficacious performance of behaviors based on positive feedback and communication.
• Autonomy: choice in how to perform a task is provided, the task is recognized as being important, and
feelings about the task are acknowledged.
• Relatedness: meaningful connections to others and a sense of belonging; the opposite is alienation and
inauthenticity.
• The collective research on SDT suggests that when basic psychological needs are supported, intrinsic
motivation and integrated extrinsic motivation are most likely to be evident.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
4
Conceptual Primer: Social Intervention
• An emancipatory process that identifies and eradicates factors contributing to disempowerment and
increases self-determination through a sense of connectedness to a larger social collective (Bennett, 1987;
Lennie, 2005; Narayan, 2005, 2005).
• Committed to a vision of social justice (Vanderplaat, 1995).
• Fundamental goal: “ensure individuals and groups have the power to influence the direction of their lives
and of their social institutions” on their own behalf (Bennett & Hallman, 1987, p. 93; Vanderplaat, 1995,
1998; Vanderplaat, Samson, & Raven, 2001).
• Characteristically participatory/a collective social activity that encourages person-to-person interaction
and mutual support rather than person-to-program interaction through which a collective identity emerges
that is communicatively secured and, in turn, makes possible the emergence of collective political agency
(Vanderplaat, 1995).
• An evaluation serving as a social intervention is a communicative bridge between the collective needs and
interests of a group and social policy formation (Vanderplaat, 1995).
Empowerment Evaluation Practice, Principles, and Outcomes
2015
5
Conceptual Primer: Empowerment Evaluation
• Derived from the ideological evaluation traditions, is by nature democratic, and facilitates examination of
important issues in an open, communicative, and collective forum; the evaluator and evaluation participants learn
from each other as equals.
• Comes under the umbrella of stakeholder-involved evaluation approaches (e.g., participatory, collaborative,
democratic, and utilization-focused) and resonates with many in the evaluation community (Miller & Campbell,
2006).
• Has an overt political agenda of changing power differentials (Mertens, 1999, 2003; Smith, 2007).
• A process that facilitates the development of control over the fate of people joined by their relationship to a social
program (Fetterman & Wandersman, 2005, 2007).
• Its intended outcome is increasing evaluation participants’ perceptions of their own empowerment and self-
determination, less so assessing program merit or worth (Bennett, 1987; Smith, 1999).
• The means by which empowerment and self-determination are fostered, illumination generated, and liberation
actualized is through the assessment of program merit or worth and subsequent plans for program improvement
(Fetterman, 1995).
• People are able to help themselves as they build evaluation capacity and a sense of community referred to as
communities of practice (Fetterman, 1996, 2001, 2002).
• Two purported advantages: self-sufficiency and liberation from traditional and preexisting expectations, roles, and
constraints imposed by others – both of which speak to psychological well-being (Fetterman, 1994).
Empowerment Evaluation Practice, Principles, and Outcomes
2015
6
Conceptual Primer: Empowerment Evaluation Cont’d…
• Distinguished from other stakeholder-involved evaluation approaches by 10 guiding principles derived from
Community Psychology and Participatory Action Research.
• Process (6): Community Ownership, Inclusiveness, Democratic Participation, Community Knowledge,
Evidence –Based Strategies, and Accountability.
• Outcome (4): Improvement, Organizational Learning, Social Justice, and Capacity Building.
• From its inception, Fetterman (1994, 1996) has recommended specific steps in empowerment evaluation
practice. Currently, there are three models (from earliest to most recent):
• Three-step – 1994
• Ten-step (a.k.a., Getting To Outcomes) - 2003
• Five –tool - 2010
• Each model has a significant degree of specificity, each incorporates Fetterman’s original steps, and each
provides the basis for contemporary empowerment evaluation practice (Fetterman & Wandersman,
2007).
Empowerment Evaluation Practice, Principles, and Outcomes
2015
7
ConcernsAbout Empowerment Evaluation
• Vague contingencies/prescriptives for actual practice (Cousins, 2005; Smith, 1999; Worthington, 1999).
• An absence of rigorous evidence to show it is empowering and leads to empowered outcomes (Patton,
2005; Worthington, 1999).
• Evidence of the effectiveness of empowerment evaluation is suggestive, not preponderant, and inconclusive
(Smith, 2007).
• Outcomes are difficult to operationalize and assess, and even if measurable, they may be unrealistic for
some smaller-scale or shorter-term projects (Campbell, Dorey, Naegeli, Grubstein et al., 2004; Patton,
1997).
• How to measure the success of an empowerment evaluation remains a challenge (Levin, 1996).
• Most practitioners have focused on whether the participating organizations can learn evaluation skills and
successfully develop, launch, and utilize evaluations of their programs rather than on measuring
empowerment and self – determination outcomes (Lennie, 2005).
Empowerment Evaluation Practice, Principles, and Outcomes
2015
8
The Need forResearch on Empowerment Evaluation
• The empowering and self – determining outcomes of empowerment evaluation, has been widely
acknowledged, although its study has often been superficial or ignored (Diaz – Puente, 2007; Diaz –
Puente, Yague, & Alfonso, 2008; Diez 2001; Floc’hlay and Plottu 1998; Gregory 2000; Potter 2005;
Saraceno 1999).
• Cousins (2005) noted that the quality of empowerment evaluation implementation resides in the extent to
which persuasive evidence can be provided to examine the claims it makes. Further, the concept of
empowerment evaluation “still remains a little bit fuzzy” which calls for stronger evidence of
empowerment evaluation success from actual studies (p. 207).
• According to Miller (2005) a thorough account of how obtaining evaluation skills is empowering, to whom
it is or ought to be empowering, and what role evaluators play in the process of shifting power is needed.
• Anderson (1996), Gore (1992), Lennie, (2002), and Lennie et al. (2003) have called for closer examination
of research and evaluations that claim to have produced empowerment, and by extension, self-
determination.
• Small number of studies (original research) on stake-holder involvement reflects a dearth of empirical
research on evaluation, specifically empowerment evaluation (Brandon & Fukunaga, 2014)
Empowerment Evaluation Practice, Principles, and Outcomes
2015
9
Research Questions: 3 Descriptive,1 Inferential
Empowerment Evaluation Practice, Principles, and Outcomes
2015
10
1. What was the extent to which evaluation practitioners adhered with fidelity to the three-step
model, the 10-step model, or the five-tool model?
2. What was the extent to which the 6 process principles (i.e., community ownership, inclusion,
democratic participation, community knowledge, evidence-based strategies, and accountability)
were reported to be in evidence during the evaluations reported on by evaluation practitioners?
3. What was the extent to which the 4 outcome principles (i.e., improvement, organizational learning,
social justice, and capacity building) were reported to have resulted from the evaluations reported
on by evaluation practitioners?
4. To what extent could variation in empowerment and self-determination be explained by:
a) The interaction between empowerment evaluation model fidelity and percentage of activities
implemented?
b) The interaction between empowerment evaluation model fidelity and percentage of activities
implemented during the early, mid, and late phases of the empowerment evaluations?
c) The process principles reported to be in evidence during the empowerment evaluations?
d) The outcome principles that resulted from the empowerment evaluations?
e) Empowerment evaluator demographic, academic, and professional characteristics (e.g.,
gender, number of years conducting empowerment evaluations, highest earned degree, etc.)?
Methodology
• 92 item, self–report, on–line survey consisting of Likert – type and open–ended items.
• Item development based on:
— Constructs found in the literature regarding the three known empowerment evaluation models and their
respective implementation steps and activities.
— The 10 principles of empowerment evaluation as outlined in the literature.
— The purported empowerment and self – determination outcomes for individuals and organizations
engaged in the process of an empowerment evaluation discussed in the literature.
— Constructs found in the social psychology literature regarding empowerment theory and self–
determination theory.
• Presented an early version to a graduate-level survey research methods course to deal with design flaws,
problems with wording and comprehension, issues in cognitive overload, and potential issues in internal
consistency through elimination of similar items.
• Worked with a university–based survey research expert on culling out items, refining item tags and response
choices, overall survey design, and on-line beta testing.
• Two university-based experts in empowerment evaluation each performed an independent content analysis.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
11
Survey Design
Empowerment Evaluation Practice, Principles, and Outcomes
2015
12
Section Items Scale(s) Developed Research Question(s)
Your evaluation activities 29 Empowerment Evaluation Models RQ 1, RQ 4
Process Principles RQ 2, RQ 4
Evaluation participant activities 12 Process Principles RQ 2, RQ 4
Changes you observed in individual values 5 Process Principles RQ 2, RQ 4
Outcome Principles RQ 3, RQ 4
Changes you observed in individual behaviors 16 Outcome Principles RQ 3, RQ 4
Individual Empowerment RQ 4
Individual Self-Determination RQ 4
Changes you observed within the organization 14 Outcome Principles RQ 3, RQ 4
Organizational Empowerment and Self-
Determination
RQ 4
Inclusiveness 1 Process Principles RQ 2, RQ 4
Evaluator accountability 2 Process Principles RQ 2, RQ 4
The evaluation model you used 1 -- --
About you: demographic information 12 -- RQ 4
AnalyticalStrategy
Unit of Analysis: The evaluation practitioner – creates the evaluation context conducive to development of
empowerment and self-determination and adheres to principles.
Descriptive statistics used to determine:
• The extent to which evaluation practitioners adhered with fidelity to the steps of the empowerment evaluation model
they implemented.
• The extent to which the process principles were reported to be in evidence during the evaluations reported on by
evaluation practitioners.
• The extent to which the outcome principles were reported to have resulted from the evaluations reported on by
evaluation practitioners.
Multiple regression techniques used to determine the extent to which variation in the four empowerment sub-
constructs, the individual empowerment construct, the three self – determination sub-constructs, the individual self-
determination construct, and the organizational empowerment and self – determination construct could be explained by:
• The interaction between empowerment evaluation model fidelity and percentage of activities implemented.
• The interaction between empowerment evaluation model fidelity and percentage of activities implemented during the
early, mid, and late phases of an empowerment evaluation.
• The process principles reported to be in evidence during the empowerment evaluations.
• The outcome principles that resulted from the empowerment evaluations.
• Empowerment evaluator demographic, academic, and professional characteristics (e.g., gender, number of years
conducting empowerment evaluations, highest earned degree, etc.) selected through a correlational analysis.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
13
SamplingRationale & Procedure
• Non – probabilistic (i.e. self – selected), purposive, and not dependent on the rationale of probability theory.
• Criterion for inclusion: evaluation practitioner had to have conducted an empowerment or empowering
evaluation in the past five years.
• Building the Sample:
• Initial sample built from the American Evaluation Association (AEA) Collaborative, Participatory, and
Empowerment Evaluation Topical Interest Group membership (with help from the moderator), extant
evaluation literature, and snowball sampling (n = 232).
• Researcher’s professional networks, global evaluation networks, global evaluation associations and
societies, AEA eLibrary, AEA365 blogs, and evaluation journals used to further identify potential study
participants.
• Names and contact information obtained through networks, etc. were cross-checked against the initial
sample to update contact information and eliminate duplicates.
• Upon reconciliation of names and contact information the sample included 615 evaluation practitioners.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
14
Recruitment
Empowerment Evaluation Practice, Principles, and Outcomes
2015
15
Groups Potential Number of Recipients
3 AEA eGroups (i.e., CPE, ICCE, OL-ECB) 1,416
4 Listserves (e.g., EvalTalk, SCRA, CommunityPsychUk, etc.) 7,800+
19 LinkedIn Groups (e.g., SAMEA, Evaluators Group, etc.) 83,531
5 Yahoo Groups (e.g., MandENews, XCeval, AfRea, etc.) 5,850
39 Facebook Communities/Interest Groups (e.g., InDEC, M & E, etc.) 57,810
Total* 156,407
*Note: The total is likely duplicated head-count as people often belong to multiple professional groups.
Data Collection and Response Rate
Data Collection
• Dillman’s (2000) four-step process guided implementation of the survey.
• Study officially launched one week after the recruitment phase ended by directly contacting potential study
participants through email, LinkedIn (In Mail), and Facebook messaging, and diffusely contacting potential
study participants via groups used in recruitment with the survey link resulting in 512 total respondents.
• From study launch to first reminder 112 respondents.
• From first to second reminder 140 additional respondents.
• From second to final reminder, 107 additional respondents.
• From final reminder to close of survey, 162 additional respondents.
Response Rate
• Difficult to calculate because both direct and diffuse recruitment methods were used.
• Response rate considered to be 25% as 131/521 provided usable data.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
16
Participant Characteristics
99 of 131 evaluation practitioners (76%) reported demographic, academic preparation, and professional characteristics:
• 58.3% English speaking.
• 59.8 % White, Non – Hispanic.
• 67 % Female.
• An average of 15 years in the field of evaluation.
• An average of 11 years conducting empowerment evaluations.
• 52% lived in a North American country (i.e., United States, Canada, or Mexico).
• 51.1% conducted empowerment evaluations in a North American country.
• 20% conducted empowerment evaluations in an African country.
• 48.5% held a Ph. D.
• 38.4% held a Master’s Degree.
• Academic degree fields: 46.8%s social sciences, 27.4% monitoring and evaluation, 17.9 % research, 12.8% public
health, and 12.6% other academic (e.g., international development).
• Professional content area of expertise: 21.1% monitoring, evaluation, and research, 12.6 % respectively development
and social justice, and 11.6 % respectively action and community – based research, health, and social science.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
17
Results: Research Question 1 – Model Fidelity
Empowerment Evaluation Practice, Principles, and Outcomes
2015
18
Fidelity to the Three-Step Model Frequency Valid Percent
No Fidelity 10 7.8
Fidelity 119 92.2
Total 129 100.0
Fidelity to the Ten-Step Model Frequency Valid Percent
No Fidelity 12 9.4
Fidelity 115 90.6
Total 127 100.0
Fidelity to the Five-Tool Model Frequency Valid Percent
No Fidelity 26 20.3
Fidelity 102 79.7
Total 128 100.0
Results: Research Question 2 – Process Principles in Evidence
Empowerment Evaluation Practice, Principles, and Outcomes
2015
19
Community Ownership Process Principle Frequency Valid Percent
No Evidence 8 6.3
Evidence 120 93.8
Total 128 100.0
Inclusiveness Process Principle Frequency Valid Percent
No Evidence 25 25.8
Evidence 72 74.2
Total 97 100.0
Democratic Participation Process Principle Frequency Valid Percent
No Evidence 20 17.5
Evidence 94 82.5
Total 114 100.0
Results: Research Question 2 Cont’d…
Empowerment Evaluation Practice, Principles, and Outcomes
2015
20
Community Knowledge Process Principle Frequency Valid Percent
No Evidence 24 20.7
Evidence 92 79.3
Total 116 100.0
Evidence-Based Strategies Process Principle Frequency Valid Percent
No Evidence 25 25.8
Evidence 72 74.2
Total 97 100.0
Accountability Process Principle Frequency Valid Percent
No Evidence 28 29.8
Evidence 66 70.2
Total 94 100.0
Results: Research Question 3 – Outcome Principles in Evidence
Empowerment Evaluation Practice, Principles, and Outcomes
2015
21
Improvement Outcome Principle Frequency Valid Percent
No Evidence 10 9.7
Evidence 95 90.5
Total 105 100.0
Organizational learning Outcome Principle Frequency Valid Percent
No Evidence 14 14.1
Evidence 85 85.9
Total 99 100.0
Results: Research Question 3 Cont’d…
Empowerment Evaluation Practice, Principles, and Outcomes
2015
22
Social Justice Outcome Principle Frequency Valid Percent
No Evidence 13 13.8
Evidence 81 86.2
Total 94 100.0
Capacity Building Outcome Principle Frequency Valid Percent
No Evidence 41 42.3
Evidence 56 57.7
Total 97 100.0
Research Questions 1 – 3Take-Aways
Research Question 1
• Based on the literature, I thought that the majority of evaluation practitioners would not adhere with fidelity
to the steps of the empowerment evaluation model they implemented. However, the average fidelity score
index revealed a majority of the study participants adhered with fidelity to the three-step, 10-step, and five-
tool models.
Research Question 2
• I thought that a majority of evaluation practitioners would report the readily observable process principles of
community ownership, inclusion, and democratic participation were evident to them, but the community
knowledge, evidence-based strategies, and accountability process principles were not.
• The average evidence score index indicated evidence of the community ownership, inclusion, and democratic
participation process principles. However, the average evidence score index also indicated evidence of the
community knowledge, evidence-based strategies, and accountability process principle.
Research Question 3
• I thought that a majority of evaluation practitioners would report the outcome principles of improvement and
capacity building were evident to them, but the more distal outcomes of organizational learning and social
justice outcome principles were not.
• The average evidence score index indicated evidence of the improvement and capacity building principles.
However, the average evidence score index also indicated evidence of the organizational learning and social
justice principles.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
23
Results: Research Question 4 – Reject orFail to Reject H0
Empowerment Evaluation Practice, Principles, and Outcomes
2015
24
Research Question 4.1: To what extent could variation in empowerment and self-determination be
explained by the interaction between empowerment evaluation model fidelity and percentage of activities
implemented?
H0 = The interaction between empowerment evaluation model fidelity and percentage of activities implemented
would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self –
determination, and organizational empowerment and self – determination.
In nine of 10 models I failed to reject the null hypothesis, but I rejected the null hypothesis in one of 10
models because the interaction between model fidelity and percentage of activities implemented was able to
explain variation in evaluation capacity; notably it was model fidelity that explained most of the variation.
Interaction: R Square = .081, F = 4.158, p = .019
Model Fidelity: Beta = .251, t = 2.493, p = .014
Results: Research Question 4 – Reject orFail to Reject H0
Empowerment Evaluation Practice, Principles, and Outcomes
2015
25
Research Question 4.2: To what extent could variation in empowerment and self-determination be
explained by the interaction between empowerment evaluation model fidelity and percentage of activities
implemented during the early, mid, or late phases of an empowerment evaluation?
H0 = The interaction between empowerment evaluation model fidelity and percentage of activities implemented
would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self –
determination, and organizational empowerment and self – determination during the early, mid, and late phases
of an empowerment evaluation.
In nine of 10 models I failed to reject the null hypothesis, but I rejected the null hypothesis in one of 10
models because the interaction between early evaluation model fidelity and percentage of activities
implemented, and mid-evaluation model fidelity and percentage of activities implemented was able to explain
variation in evaluation capacity; notably it was model fidelity during the mid-evaluation phase that explained
most of the variation.
Interaction: R Square = .097, F = 2.484, p = .049
Mid-evaluation Model Fidelity: Beta = .308, t = 2.076, p = .041
Results: Research Question 4 - Reject orFail to Reject H0 Cont’d…
Research Question 4.3: To what extent could variation in empowerment and self-determination be explained by
the process principles reported to be in evidence during the evaluations reported on by evaluation
practitioners?
H0 = The community ownership, inclusiveness, and democratic participation process principles would not be able to
explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and
organizational empowerment and self – determination.
In six of 10 models I failed to reject the null hypothesis because community ownership and democratic participation
were unable to explain variation, but I was able to reject the null hypothesis in four of 10 models because the
inclusiveness process principle was able to explain variation in evaluation knowledge, evaluation capacity,
individual empowerment, and evaluation competence.
Evaluation knowledge - Inclusiveness: R Square = .054, F = 4.068, p = .047; Beta = -.233, t = -2.017, p = .047
Evaluation Capacity - Inclusiveness: R Square = .082, F = 6.310, p = .014; Beta = -.286., t = -2.512, p = .014
Individual Empowerment - Inclusiveness: R Square = .057, F = 4.317, p = .041; Beta = -.238, t = -2.078, p = .041
Evaluation Competence - Inclusiveness: R Square = .059, F = 4.552, p = .036; Beta = -.244, t = -2.134, p = .036
Empowerment Evaluation Practice, Principles, and Outcomes
2015
26
Results: Research Question 4 - Reject orFail to Reject H0 Cont’d…
Research Question 4.4: To what extent could variation in empowerment and self-determination be
explained by the outcome principles that resulted from the evaluations reported on by evaluation
practitioners?
H0 = The improvement and capacity building outcome principles would not be able to explain variation in the
sub-constructs and constructs of individual empowerment and self – determination, and organizational
empowerment and self – determination.
In all 10 models I failed to reject the null hypothesis because evidence of the improvement and capacity
building outcome principles were unable to explain variation.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
27
Results: Research Question 4 - Reject orFail to Reject H0 Cont’d…
Research Question 4.5: To what extent could variation in empowerment and self-determination be explained
by empowerment evaluator demographic, academic, and professional characteristics?
H0 = The characteristics gender, professional expertise, and where empowerment evaluations were conducted
would be unable to explain variation in the sub-constructs and constructs of individual empowerment and self
– determination, and organizational empowerment and self – determination.
In nine of 10 models I failed to reject the null hypothesis because gender was unable to explain variation, but I
was able to reject the null hypothesis in one of 10 models because gender was able to explain variation in
individual empowerment.
Individual empowerment - Gender: Beta = -.293, t = -2.331, p = .024
In all 10 models I failed to reject the null hypothesis because professional expertise was unable to explain
variation.
In five of 10 models I failed to reject the null hypothesis because where empowerment evaluations were
conducted was unable to explain variation, but in the other five models I was able to reject the null hypothesis
because where empowerment evaluations were conducted was able to explain variation in evaluation knowledge,
individual empowerment, evaluation competence, evaluation autonomy, and feelings of relatedness to others.
Evaluation Knowledge - Work in an African Country: Beta = .627, t = 2.090, p = .041
Individual Empowerment - Work in an African Country: Beta = 699, t = 2.343, p = .023
Evaluation Competence - Work in an African Country: Beta = .715, t = 2.407, p = .019
Evaluation Autonomy - Work in an African Country: Beta = .630, t = 2.121, p = .038
Feelings of Relatedness to Others - Work in a Non-African Country: Beta = -.361, t = -2.895, p = .006;
- Work in an African Country: Beta =.645, t = 2.306, p = .025
Empowerment Evaluation Practice, Principles, and Outcomes
2015
28
Results: Research Question 4 - Reject orFail to Reject H0 Cont’d…
Although not tested against the null hypothesis, variation in evaluation knowledge, individual empowerment,
evaluation competence, and evaluation autonomy was explained by where evaluation practitioners lived,
specifically an African country.
• Evaluation Knowledge - Live in an African Country: Beta = -.675, t = -2.395, p = .020
• Individual Empowerment - Live in an African Country: Beta = -.787, t = -2.850, p = .006
• Evaluation Competence - Live in an African Country: Beta = -.634, t = -2.270, p = .027
• Evaluation Autonomy - Live in an African Country: Beta = -.684, t = -2.454, p = .017
Variance accounted for with all 10 demographic, academic, and professional characteristics in the model:
• Evaluation Knowledge: R Square = .283, F = 2.214, p = .030
• Individual Empowerment: R Square = .348, F = 2.829, p = .007
• Evaluation Competence: R Square = .297, F = 2.369, p = .020
• Evaluation Autonomy: R Square = .299, F = 2.390, p = .019
• Feelings of Relatedness to Others: R Square = .385, F = 3.188, p = .003
Empowerment Evaluation Practice, Principles, and Outcomes
2015
29
Research Question 4 - Implications forEvaluation Practice and Policy
If building evaluation capacity is important:
• Implement a high percentage of activities to ensure model fidelity especially during the mid-phase of an
empowerment evaluation.
• Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the
evaluation is included.
If building evaluation knowledge is important:
• Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the
evaluation is included.
• Attend to the community knowledge process principle so that everyone who is engaged in the evaluation can
use their collective wisdom to develop evaluation tools, evaluation procedures, interpreting data, etc.
If building evaluation competence is important:
• Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the
evaluation is included.
If individual empowerment is important:
• Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the
evaluation is included.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
30
Strengths of this Study:Addressing Concerns
• Determining which steps/activities of the three empowerment evaluation models have been applied in actual
evaluation practice.
• Operationalizing the steps of the three empowerment evaluation models as evaluation practitioner and
evaluation participant activities.
• Defining empowerment evaluation model fidelity, developing a model fidelity index, and measuring model
fidelity.
• Operationalizing the empowerment evaluation process and outcome principles as evaluation practitioner and
evaluation participant activities, developing an index for determining evidence of the principles, and
determining whether the principles were in evidence during and as a result of the empowerment evaluations
reported on by evaluation practitioners.
• Defining the sub-constructs and constructs of empowerment theory and self-determination theory for and
adapting them to empowerment evaluation.
• Explaining variation in empowerment and self-determination by empowerment evaluation model fidelity,
evidence of the empowerment evaluation process and outcome principles, and evaluation practitioners’
demographic, academic, and professional characteristics.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
31
Limitations Most Impactingthis Study
• Self - Selection
• Calls into question the extent to which the evaluation practitioners in the sample were equivalent to other
evaluation practitioners who conduct empowerment evaluations.
• Social Desirability
• Calls into question the accuracy of responses (see for example, model fidelity scores).
• Study participants claim and over-report activities which are socially desirable (Krumpal, 2013).
• Self-deception: A respondent believes a statement to be true of herself even though it is inaccurate.
• SEEPPO Reliability
• Although the SEEPPO appeared to have strong content validity there were not enough observations to
conduct an adequate reliability analysis.
• Further refinement and testing, including a factor analysis, to establish its reliability and predictive
validity is necessary.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
32
Recommendations forResearch on Empowerment Evaluation
• Develop and validate, or adapt existing empowerment and self-determination scales so evaluation
practitioners can measure the empowerment and self-determination of individuals and organizations with
whom they work; instruments should be able to discriminate between individual empowerment and self-
determination sub-constructs and organizational empowerment and self-determination sub-constructs.
• Measure the empowerment and self-determination of empowerment evaluation participants before, during,
and post-evaluation as well as the distal outcomes of empowerment evaluation (e.g., the extent to which
social policy was influenced towards favorable outcomes, the extent to which power differentials were
equalized, etc.)
• Create and test a theory of change model, including moderators and mediators, showing the likely pathways
from empowerment evaluation model implementation and the process and outcome principles to individual
and organizational empowerment and self-determination.
• Test whether there is a relationship between models and principles, the extent to which a relationship exists,
and whether model fidelity (along a continuum) or evidence of the principles (along a continuum) attenuates
that relationship.
• Explore the role of contextual variables in the development/enhancement of empowerment and self-
determination by comparing the outcomes of empowerment evaluations conducted in different countries.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
33
Conclusions
• To a limited extent individual empowerment and self-determination were the likely outcomes of the
empowerment evaluations reported on by this sample of 131 evaluation practitioners; model fidelity, the
inclusiveness and community knowledge process principles, and evaluation practitioner characteristics
appeared to have been able to explain variation.
• Given evidence of all 10 guiding principles it can be said that the evaluations reported on were likely to be
empowerment evaluations as defined by Fetterman and Wandersman.
• The evaluation practitioners in this study largely adhered with fidelity to the steps of the models they
implemented; no model was privileged over another as steps were chosen from across all three models
(96.2%, n = 126).
• Evaluation practitioner’s personal, academic, and professional characteristics did not fully factor into the
empowerment of individuals and organizations with whom they worked. However, evaluation knowledge,
individual empowerment and all three individual self-determination sub-constructs (i.e., competence,
autonomy, and relatedness) appeared to have been correlated (both + and -) with empowerment evaluations
conducted by evaluation practitioners who lived and worked in an African country, who were female, and
who conducted empowerment evaluations in another country.
• The SEEPPO was a successful first attempt at providing a practical, largely quantitative tool to empowerment
evaluators interested in assessing the process and outcomes of their work, less so actual empowerment and
self-determination.
Empowerment Evaluation Practice, Principles, and Outcomes
2015
34
Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides

More Related Content

What's hot

Evaluation assessment of OD - OD process - Organizational Change and Develop...
Evaluation assessment of OD - OD process -  Organizational Change and Develop...Evaluation assessment of OD - OD process -  Organizational Change and Develop...
Evaluation assessment of OD - OD process - Organizational Change and Develop...manumelwin
 
Functional theory of communication
Functional theory of communicationFunctional theory of communication
Functional theory of communicationMary Joy Atendido
 
Week2 rainey chapter_4
Week2 rainey chapter_4Week2 rainey chapter_4
Week2 rainey chapter_4mmzzmartinez
 
Functional Perspective On Group Decision Making
Functional Perspective On Group Decision MakingFunctional Perspective On Group Decision Making
Functional Perspective On Group Decision MakingArun Jacob
 
Organisational behaviour
Organisational behaviourOrganisational behaviour
Organisational behaviourAprameya joshi
 
William Allan Kritsonis, PhD - Dissertation Chair for Desiree Adair Skinner
William Allan Kritsonis, PhD - Dissertation Chair for Desiree Adair SkinnerWilliam Allan Kritsonis, PhD - Dissertation Chair for Desiree Adair Skinner
William Allan Kritsonis, PhD - Dissertation Chair for Desiree Adair Skinnerguestfa49ec
 
2 h. relation & intr-pers skill -ob-23
2  h. relation & intr-pers skill -ob-232  h. relation & intr-pers skill -ob-23
2 h. relation & intr-pers skill -ob-23Hitesh Rochwani
 
PPT Staffing decission
PPT Staffing decissionPPT Staffing decission
PPT Staffing decissionErrienMaulidia
 
K5111 General Functional Theory
K5111 General Functional TheoryK5111 General Functional Theory
K5111 General Functional TheoryGnaliny Yogeswaran
 
Not a community of practice? Creating safe spaces with multitudes of functions
Not a community of practice? Creating safe spaces with multitudes of functionsNot a community of practice? Creating safe spaces with multitudes of functions
Not a community of practice? Creating safe spaces with multitudes of functionsSEDA
 
Sabin SIF Social learning lecture XII.11
Sabin SIF Social learning lecture XII.11Sabin SIF Social learning lecture XII.11
Sabin SIF Social learning lecture XII.11Mike McQuestion
 
The Social Psychology Of Organizations
The Social Psychology Of OrganizationsThe Social Psychology Of Organizations
The Social Psychology Of Organizationsprince440
 
Origins of organisational behaviour theories
Origins of organisational behaviour theoriesOrigins of organisational behaviour theories
Origins of organisational behaviour theoriesRinu Kumar
 
Diagnosing behavioral problems and perception
Diagnosing behavioral problems and perceptionDiagnosing behavioral problems and perception
Diagnosing behavioral problems and perceptionEui Jung Hwang
 

What's hot (20)

Evaluation assessment of OD - OD process - Organizational Change and Develop...
Evaluation assessment of OD - OD process -  Organizational Change and Develop...Evaluation assessment of OD - OD process -  Organizational Change and Develop...
Evaluation assessment of OD - OD process - Organizational Change and Develop...
 
Functional theory of communication
Functional theory of communicationFunctional theory of communication
Functional theory of communication
 
Action research
Action researchAction research
Action research
 
Week2 rainey chapter_4
Week2 rainey chapter_4Week2 rainey chapter_4
Week2 rainey chapter_4
 
Self-Determination Theory and Learning
Self-Determination Theory and LearningSelf-Determination Theory and Learning
Self-Determination Theory and Learning
 
Organizational Behavior
Organizational BehaviorOrganizational Behavior
Organizational Behavior
 
Functional Perspective On Group Decision Making
Functional Perspective On Group Decision MakingFunctional Perspective On Group Decision Making
Functional Perspective On Group Decision Making
 
Motivation Report
Motivation ReportMotivation Report
Motivation Report
 
Organisational behaviour
Organisational behaviourOrganisational behaviour
Organisational behaviour
 
William Allan Kritsonis, PhD - Dissertation Chair for Desiree Adair Skinner
William Allan Kritsonis, PhD - Dissertation Chair for Desiree Adair SkinnerWilliam Allan Kritsonis, PhD - Dissertation Chair for Desiree Adair Skinner
William Allan Kritsonis, PhD - Dissertation Chair for Desiree Adair Skinner
 
Kissick Thesis 2015
Kissick Thesis 2015Kissick Thesis 2015
Kissick Thesis 2015
 
2 h. relation & intr-pers skill -ob-23
2  h. relation & intr-pers skill -ob-232  h. relation & intr-pers skill -ob-23
2 h. relation & intr-pers skill -ob-23
 
PPT Staffing decission
PPT Staffing decissionPPT Staffing decission
PPT Staffing decission
 
K5111 General Functional Theory
K5111 General Functional TheoryK5111 General Functional Theory
K5111 General Functional Theory
 
Not a community of practice? Creating safe spaces with multitudes of functions
Not a community of practice? Creating safe spaces with multitudes of functionsNot a community of practice? Creating safe spaces with multitudes of functions
Not a community of practice? Creating safe spaces with multitudes of functions
 
Sabin SIF Social learning lecture XII.11
Sabin SIF Social learning lecture XII.11Sabin SIF Social learning lecture XII.11
Sabin SIF Social learning lecture XII.11
 
The Social Psychology Of Organizations
The Social Psychology Of OrganizationsThe Social Psychology Of Organizations
The Social Psychology Of Organizations
 
Origins of organisational behaviour theories
Origins of organisational behaviour theoriesOrigins of organisational behaviour theories
Origins of organisational behaviour theories
 
Diagnosing behavioral problems and perception
Diagnosing behavioral problems and perceptionDiagnosing behavioral problems and perception
Diagnosing behavioral problems and perception
 
Poster
PosterPoster
Poster
 

Similar to Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides

Using realist evaluation with vulnerable young people and the services that s...
Using realist evaluation with vulnerable young people and the services that s...Using realist evaluation with vulnerable young people and the services that s...
Using realist evaluation with vulnerable young people and the services that s...BASPCAN
 
Presentation group 《5》sw with individual nd families.pptx
Presentation group 《5》sw with individual nd families.pptxPresentation group 《5》sw with individual nd families.pptx
Presentation group 《5》sw with individual nd families.pptxAyuub Bashiir Mohamed
 
Critical evaluation in social work
Critical evaluation in social workCritical evaluation in social work
Critical evaluation in social workMinna Kivipelto
 
I school evaluation symposium, irwin, april 2014 pauledit
I school evaluation symposium, irwin, april 2014 pauleditI school evaluation symposium, irwin, april 2014 pauledit
I school evaluation symposium, irwin, april 2014 pauleditStephen Abram
 
Organizational change and its approaches
Organizational change and its approachesOrganizational change and its approaches
Organizational change and its approachesAamir chouhan
 
ACTION RESEARCH -An Important Research Typeppt
ACTION RESEARCH -An Important Research TypepptACTION RESEARCH -An Important Research Typeppt
ACTION RESEARCH -An Important Research Typepptumarashaheen2
 
Stryker: Decision Making Process
Stryker: Decision Making ProcessStryker: Decision Making Process
Stryker: Decision Making ProcessIsabelleSchultz1
 
KEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSR
KEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSRKEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSR
KEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSRPublic Health Foundation of India (PHFI)
 
Organization Development
Organization DevelopmentOrganization Development
Organization DevelopmentAman Arora
 
Stages of Change & Reactance in Group Work
Stages of Change & Reactance in Group WorkStages of Change & Reactance in Group Work
Stages of Change & Reactance in Group WorkJane Gilgun
 
Developing a quality culture presentation [autosaved]
Developing a quality culture presentation [autosaved]Developing a quality culture presentation [autosaved]
Developing a quality culture presentation [autosaved]Yomna Motea
 
Presentation by Dr. David Guest on "Employee Engagement: A Path to Organisati...
Presentation by Dr. David Guest on "Employee Engagement: A Path to Organisati...Presentation by Dr. David Guest on "Employee Engagement: A Path to Organisati...
Presentation by Dr. David Guest on "Employee Engagement: A Path to Organisati...OECD Governance
 
Vicky Pelka's Training Session On Impact Evaluation
Vicky Pelka's Training Session On Impact EvaluationVicky Pelka's Training Session On Impact Evaluation
Vicky Pelka's Training Session On Impact EvaluationJosh Chandler
 
Oragnization development OD (INTRODUCTION)
Oragnization development OD (INTRODUCTION)Oragnization development OD (INTRODUCTION)
Oragnization development OD (INTRODUCTION)shagun jain
 

Similar to Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides (20)

Using realist evaluation with vulnerable young people and the services that s...
Using realist evaluation with vulnerable young people and the services that s...Using realist evaluation with vulnerable young people and the services that s...
Using realist evaluation with vulnerable young people and the services that s...
 
Presentation group 《5》sw with individual nd families.pptx
Presentation group 《5》sw with individual nd families.pptxPresentation group 《5》sw with individual nd families.pptx
Presentation group 《5》sw with individual nd families.pptx
 
Critical evaluation in social work
Critical evaluation in social workCritical evaluation in social work
Critical evaluation in social work
 
Chapter18
Chapter18Chapter18
Chapter18
 
I school evaluation symposium, irwin, april 2014 pauledit
I school evaluation symposium, irwin, april 2014 pauleditI school evaluation symposium, irwin, april 2014 pauledit
I school evaluation symposium, irwin, april 2014 pauledit
 
Qrm 210 unit 1
Qrm 210 unit 1Qrm 210 unit 1
Qrm 210 unit 1
 
Organizational change and its approaches
Organizational change and its approachesOrganizational change and its approaches
Organizational change and its approaches
 
Cet7034 unit 4
Cet7034 unit 4Cet7034 unit 4
Cet7034 unit 4
 
Strategic human resource practice implementation (research paper design)
Strategic human resource practice implementation (research paper design)Strategic human resource practice implementation (research paper design)
Strategic human resource practice implementation (research paper design)
 
ACTION RESEARCH -An Important Research Typeppt
ACTION RESEARCH -An Important Research TypepptACTION RESEARCH -An Important Research Typeppt
ACTION RESEARCH -An Important Research Typeppt
 
Stryker: Decision Making Process
Stryker: Decision Making ProcessStryker: Decision Making Process
Stryker: Decision Making Process
 
KEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSR
KEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSRKEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSR
KEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSR
 
Criteria in social research
Criteria in social researchCriteria in social research
Criteria in social research
 
Organization Development
Organization DevelopmentOrganization Development
Organization Development
 
Stages of Change & Reactance in Group Work
Stages of Change & Reactance in Group WorkStages of Change & Reactance in Group Work
Stages of Change & Reactance in Group Work
 
Developing a quality culture presentation [autosaved]
Developing a quality culture presentation [autosaved]Developing a quality culture presentation [autosaved]
Developing a quality culture presentation [autosaved]
 
Presentation by Dr. David Guest on "Employee Engagement: A Path to Organisati...
Presentation by Dr. David Guest on "Employee Engagement: A Path to Organisati...Presentation by Dr. David Guest on "Employee Engagement: A Path to Organisati...
Presentation by Dr. David Guest on "Employee Engagement: A Path to Organisati...
 
Vicky Pelka's Training Session On Impact Evaluation
Vicky Pelka's Training Session On Impact EvaluationVicky Pelka's Training Session On Impact Evaluation
Vicky Pelka's Training Session On Impact Evaluation
 
Writing Effective Feedback
Writing Effective FeedbackWriting Effective Feedback
Writing Effective Feedback
 
Oragnization development OD (INTRODUCTION)
Oragnization development OD (INTRODUCTION)Oragnization development OD (INTRODUCTION)
Oragnization development OD (INTRODUCTION)
 

More from Jeffrey Andrew Sheldon, M.A., Ed.M.

Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019Jeffrey Andrew Sheldon, M.A., Ed.M.
 
Lecture 9 strategic planning for iranian ng os - dr. jeff sheldon - march 2019
Lecture 9   strategic planning for iranian ng os - dr. jeff sheldon - march 2019Lecture 9   strategic planning for iranian ng os - dr. jeff sheldon - march 2019
Lecture 9 strategic planning for iranian ng os - dr. jeff sheldon - march 2019Jeffrey Andrew Sheldon, M.A., Ed.M.
 
Developing Leadership Capacity to Influence the Self-Evaluating Organization
Developing Leadership Capacity to Influence the Self-Evaluating Organization Developing Leadership Capacity to Influence the Self-Evaluating Organization
Developing Leadership Capacity to Influence the Self-Evaluating Organization Jeffrey Andrew Sheldon, M.A., Ed.M.
 
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016Jeffrey Andrew Sheldon, M.A., Ed.M.
 
Geita District Mwanza Region Northern Tanzania April 2007 J Sheldon
Geita District Mwanza Region Northern Tanzania April 2007 J SheldonGeita District Mwanza Region Northern Tanzania April 2007 J Sheldon
Geita District Mwanza Region Northern Tanzania April 2007 J SheldonJeffrey Andrew Sheldon, M.A., Ed.M.
 
Democratic Republic Of Congo Education System Transformation December 2006 J ...
Democratic Republic Of Congo Education System Transformation December 2006 J ...Democratic Republic Of Congo Education System Transformation December 2006 J ...
Democratic Republic Of Congo Education System Transformation December 2006 J ...Jeffrey Andrew Sheldon, M.A., Ed.M.
 

More from Jeffrey Andrew Sheldon, M.A., Ed.M. (14)

Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
Operational planning for iranian NGOs - Dr. Jeff Sheldon, March 2019
 
Lecture 9 strategic planning for iranian ng os - dr. jeff sheldon - march 2019
Lecture 9   strategic planning for iranian ng os - dr. jeff sheldon - march 2019Lecture 9   strategic planning for iranian ng os - dr. jeff sheldon - march 2019
Lecture 9 strategic planning for iranian ng os - dr. jeff sheldon - march 2019
 
Developing Leadership Capacity to Influence the Self-Evaluating Organization
Developing Leadership Capacity to Influence the Self-Evaluating Organization Developing Leadership Capacity to Influence the Self-Evaluating Organization
Developing Leadership Capacity to Influence the Self-Evaluating Organization
 
Final Products Report
Final Products ReportFinal Products Report
Final Products Report
 
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
Virginia G. Piper Trust Presentation - Dr. Jeff Sheldon - January 4, 2016
 
Program Theory Lecture University of San Diego October 2006
Program Theory Lecture  University of San Diego October 2006Program Theory Lecture  University of San Diego October 2006
Program Theory Lecture University of San Diego October 2006
 
Operacion Pedro Pan December 2006 J Sheldon
Operacion Pedro Pan December 2006 J SheldonOperacion Pedro Pan December 2006 J Sheldon
Operacion Pedro Pan December 2006 J Sheldon
 
Mc Cord Hosptial Final Products Report August 2005 J Sheldon
Mc Cord Hosptial Final Products Report August 2005 J SheldonMc Cord Hosptial Final Products Report August 2005 J Sheldon
Mc Cord Hosptial Final Products Report August 2005 J Sheldon
 
Geita District Mwanza Region Northern Tanzania April 2007 J Sheldon
Geita District Mwanza Region Northern Tanzania April 2007 J SheldonGeita District Mwanza Region Northern Tanzania April 2007 J Sheldon
Geita District Mwanza Region Northern Tanzania April 2007 J Sheldon
 
Democratic Republic Of Congo Education System Transformation December 2006 J ...
Democratic Republic Of Congo Education System Transformation December 2006 J ...Democratic Republic Of Congo Education System Transformation December 2006 J ...
Democratic Republic Of Congo Education System Transformation December 2006 J ...
 
Ces Conference Presentation June 2006 J Sheldon
Ces Conference Presentation June 2006 J SheldonCes Conference Presentation June 2006 J Sheldon
Ces Conference Presentation June 2006 J Sheldon
 
Aea Conference Presentation November 2007 J Sheldon
Aea Conference Presentation November 2007 J SheldonAea Conference Presentation November 2007 J Sheldon
Aea Conference Presentation November 2007 J Sheldon
 
Aea Conference Presentation November 2007 J Sheldon
Aea Conference Presentation November 2007 J SheldonAea Conference Presentation November 2007 J Sheldon
Aea Conference Presentation November 2007 J Sheldon
 
Aea Ces Conference Poster October 2005 J Sheldon
Aea Ces Conference Poster October 2005 J SheldonAea Ces Conference Poster October 2005 J Sheldon
Aea Ces Conference Poster October 2005 J Sheldon
 

Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides

  • 1. EVALUATIONAS SOCIALINTERVENTION:AN EMPIRICALSTUDYOF EMPOWERMENTEVALUATION PRACTICEAND PRINCIPLE EFFECTS ON PSYCHOLOGICALEMPOWERMENTAND SELF – DETERMINATION OUTCOMES Jeffrey Andrew Sheldon Claremont Graduate University 2015 A dissertation submitted to the Faculty of Claremont Graduate University in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Psychology.
  • 2. Overview • Conceptual Primer • Concerns About Empowerment Evaluation from Evaluation Theorists and Commentators • The Need for Research on Empowerment Evaluation • Research Questions • Methodology and Analytical Strategy • Survey Design • Sampling Rationale and Procedures • Participant Characteristics • Results and Take-Aways for Research Questions 1 - 3 • Results and Implications for Practice and Policy from Research Question 4 ($) • Strengths and Limitations of this Study • Recommendations for Future Research on Empowerment Evaluation • Conclusions Empowerment Evaluation Practice, Principles, and Outcomes 2015 2
  • 3. Conceptual Primer: EmpowermentTheory • A long-term process individuals, organizations, and communities undertake for themselves rather than something done to or for them (Lather, 1991, 1992; Claridge, 1996) • Empowered outcomes are evidenced by individuals or aggregate bodies of individuals engaging in behaviors that permit effective pursuit of planned social change: working with others, managing resources, learning decision-making skills, influencing policy, accessing government, sharing leadership, obtaining needed resources, being networked to others, etc. (Schulz et al., 1995; Zimmerman, 2000). • Three interrelated components (Zimmerman, Israel, Schulz, & Checkoway, 1992). • Interactional: Developing knowledge, skills, and strategies to become leaders/advocates for change. • Intrapersonal: Appraisal/critical awareness of, and the motivation and agency to change one’s life circumstances (i.e., self-efficacy). • Behavioral: Individual-level actions requiring leadership that benefits individuals and communities by influencing social policy. Empowerment Evaluation Practice, Principles, and Outcomes 2015 3
  • 4. Conceptual Primer: Self-DeterminationTheory • A general, organismic theory of human motivation organized around three sets of motivational processes— intrinsic, extrinsic, and amotivational —and their relationship to self-determined behaviors - free will and determination of one’s own fate or course of action without compulsion (Ryan & Connell, 1989). • To be self-determined, experience well–being, and have optimal health (e.g., self-esteem, self-actualization, vitality, positive affect, and work engagement) people have to satisfy three basic psychological needs endowed by nature (Koestner, Ryan, Bernieri, & Holt, 1984): • Competence: efficacious performance of behaviors based on positive feedback and communication. • Autonomy: choice in how to perform a task is provided, the task is recognized as being important, and feelings about the task are acknowledged. • Relatedness: meaningful connections to others and a sense of belonging; the opposite is alienation and inauthenticity. • The collective research on SDT suggests that when basic psychological needs are supported, intrinsic motivation and integrated extrinsic motivation are most likely to be evident. Empowerment Evaluation Practice, Principles, and Outcomes 2015 4
  • 5. Conceptual Primer: Social Intervention • An emancipatory process that identifies and eradicates factors contributing to disempowerment and increases self-determination through a sense of connectedness to a larger social collective (Bennett, 1987; Lennie, 2005; Narayan, 2005, 2005). • Committed to a vision of social justice (Vanderplaat, 1995). • Fundamental goal: “ensure individuals and groups have the power to influence the direction of their lives and of their social institutions” on their own behalf (Bennett & Hallman, 1987, p. 93; Vanderplaat, 1995, 1998; Vanderplaat, Samson, & Raven, 2001). • Characteristically participatory/a collective social activity that encourages person-to-person interaction and mutual support rather than person-to-program interaction through which a collective identity emerges that is communicatively secured and, in turn, makes possible the emergence of collective political agency (Vanderplaat, 1995). • An evaluation serving as a social intervention is a communicative bridge between the collective needs and interests of a group and social policy formation (Vanderplaat, 1995). Empowerment Evaluation Practice, Principles, and Outcomes 2015 5
  • 6. Conceptual Primer: Empowerment Evaluation • Derived from the ideological evaluation traditions, is by nature democratic, and facilitates examination of important issues in an open, communicative, and collective forum; the evaluator and evaluation participants learn from each other as equals. • Comes under the umbrella of stakeholder-involved evaluation approaches (e.g., participatory, collaborative, democratic, and utilization-focused) and resonates with many in the evaluation community (Miller & Campbell, 2006). • Has an overt political agenda of changing power differentials (Mertens, 1999, 2003; Smith, 2007). • A process that facilitates the development of control over the fate of people joined by their relationship to a social program (Fetterman & Wandersman, 2005, 2007). • Its intended outcome is increasing evaluation participants’ perceptions of their own empowerment and self- determination, less so assessing program merit or worth (Bennett, 1987; Smith, 1999). • The means by which empowerment and self-determination are fostered, illumination generated, and liberation actualized is through the assessment of program merit or worth and subsequent plans for program improvement (Fetterman, 1995). • People are able to help themselves as they build evaluation capacity and a sense of community referred to as communities of practice (Fetterman, 1996, 2001, 2002). • Two purported advantages: self-sufficiency and liberation from traditional and preexisting expectations, roles, and constraints imposed by others – both of which speak to psychological well-being (Fetterman, 1994). Empowerment Evaluation Practice, Principles, and Outcomes 2015 6
  • 7. Conceptual Primer: Empowerment Evaluation Cont’d… • Distinguished from other stakeholder-involved evaluation approaches by 10 guiding principles derived from Community Psychology and Participatory Action Research. • Process (6): Community Ownership, Inclusiveness, Democratic Participation, Community Knowledge, Evidence –Based Strategies, and Accountability. • Outcome (4): Improvement, Organizational Learning, Social Justice, and Capacity Building. • From its inception, Fetterman (1994, 1996) has recommended specific steps in empowerment evaluation practice. Currently, there are three models (from earliest to most recent): • Three-step – 1994 • Ten-step (a.k.a., Getting To Outcomes) - 2003 • Five –tool - 2010 • Each model has a significant degree of specificity, each incorporates Fetterman’s original steps, and each provides the basis for contemporary empowerment evaluation practice (Fetterman & Wandersman, 2007). Empowerment Evaluation Practice, Principles, and Outcomes 2015 7
  • 8. ConcernsAbout Empowerment Evaluation • Vague contingencies/prescriptives for actual practice (Cousins, 2005; Smith, 1999; Worthington, 1999). • An absence of rigorous evidence to show it is empowering and leads to empowered outcomes (Patton, 2005; Worthington, 1999). • Evidence of the effectiveness of empowerment evaluation is suggestive, not preponderant, and inconclusive (Smith, 2007). • Outcomes are difficult to operationalize and assess, and even if measurable, they may be unrealistic for some smaller-scale or shorter-term projects (Campbell, Dorey, Naegeli, Grubstein et al., 2004; Patton, 1997). • How to measure the success of an empowerment evaluation remains a challenge (Levin, 1996). • Most practitioners have focused on whether the participating organizations can learn evaluation skills and successfully develop, launch, and utilize evaluations of their programs rather than on measuring empowerment and self – determination outcomes (Lennie, 2005). Empowerment Evaluation Practice, Principles, and Outcomes 2015 8
  • 9. The Need forResearch on Empowerment Evaluation • The empowering and self – determining outcomes of empowerment evaluation, has been widely acknowledged, although its study has often been superficial or ignored (Diaz – Puente, 2007; Diaz – Puente, Yague, & Alfonso, 2008; Diez 2001; Floc’hlay and Plottu 1998; Gregory 2000; Potter 2005; Saraceno 1999). • Cousins (2005) noted that the quality of empowerment evaluation implementation resides in the extent to which persuasive evidence can be provided to examine the claims it makes. Further, the concept of empowerment evaluation “still remains a little bit fuzzy” which calls for stronger evidence of empowerment evaluation success from actual studies (p. 207). • According to Miller (2005) a thorough account of how obtaining evaluation skills is empowering, to whom it is or ought to be empowering, and what role evaluators play in the process of shifting power is needed. • Anderson (1996), Gore (1992), Lennie, (2002), and Lennie et al. (2003) have called for closer examination of research and evaluations that claim to have produced empowerment, and by extension, self- determination. • Small number of studies (original research) on stake-holder involvement reflects a dearth of empirical research on evaluation, specifically empowerment evaluation (Brandon & Fukunaga, 2014) Empowerment Evaluation Practice, Principles, and Outcomes 2015 9
  • 10. Research Questions: 3 Descriptive,1 Inferential Empowerment Evaluation Practice, Principles, and Outcomes 2015 10 1. What was the extent to which evaluation practitioners adhered with fidelity to the three-step model, the 10-step model, or the five-tool model? 2. What was the extent to which the 6 process principles (i.e., community ownership, inclusion, democratic participation, community knowledge, evidence-based strategies, and accountability) were reported to be in evidence during the evaluations reported on by evaluation practitioners? 3. What was the extent to which the 4 outcome principles (i.e., improvement, organizational learning, social justice, and capacity building) were reported to have resulted from the evaluations reported on by evaluation practitioners? 4. To what extent could variation in empowerment and self-determination be explained by: a) The interaction between empowerment evaluation model fidelity and percentage of activities implemented? b) The interaction between empowerment evaluation model fidelity and percentage of activities implemented during the early, mid, and late phases of the empowerment evaluations? c) The process principles reported to be in evidence during the empowerment evaluations? d) The outcome principles that resulted from the empowerment evaluations? e) Empowerment evaluator demographic, academic, and professional characteristics (e.g., gender, number of years conducting empowerment evaluations, highest earned degree, etc.)?
  • 11. Methodology • 92 item, self–report, on–line survey consisting of Likert – type and open–ended items. • Item development based on: — Constructs found in the literature regarding the three known empowerment evaluation models and their respective implementation steps and activities. — The 10 principles of empowerment evaluation as outlined in the literature. — The purported empowerment and self – determination outcomes for individuals and organizations engaged in the process of an empowerment evaluation discussed in the literature. — Constructs found in the social psychology literature regarding empowerment theory and self– determination theory. • Presented an early version to a graduate-level survey research methods course to deal with design flaws, problems with wording and comprehension, issues in cognitive overload, and potential issues in internal consistency through elimination of similar items. • Worked with a university–based survey research expert on culling out items, refining item tags and response choices, overall survey design, and on-line beta testing. • Two university-based experts in empowerment evaluation each performed an independent content analysis. Empowerment Evaluation Practice, Principles, and Outcomes 2015 11
  • 12. Survey Design Empowerment Evaluation Practice, Principles, and Outcomes 2015 12 Section Items Scale(s) Developed Research Question(s) Your evaluation activities 29 Empowerment Evaluation Models RQ 1, RQ 4 Process Principles RQ 2, RQ 4 Evaluation participant activities 12 Process Principles RQ 2, RQ 4 Changes you observed in individual values 5 Process Principles RQ 2, RQ 4 Outcome Principles RQ 3, RQ 4 Changes you observed in individual behaviors 16 Outcome Principles RQ 3, RQ 4 Individual Empowerment RQ 4 Individual Self-Determination RQ 4 Changes you observed within the organization 14 Outcome Principles RQ 3, RQ 4 Organizational Empowerment and Self- Determination RQ 4 Inclusiveness 1 Process Principles RQ 2, RQ 4 Evaluator accountability 2 Process Principles RQ 2, RQ 4 The evaluation model you used 1 -- -- About you: demographic information 12 -- RQ 4
  • 13. AnalyticalStrategy Unit of Analysis: The evaluation practitioner – creates the evaluation context conducive to development of empowerment and self-determination and adheres to principles. Descriptive statistics used to determine: • The extent to which evaluation practitioners adhered with fidelity to the steps of the empowerment evaluation model they implemented. • The extent to which the process principles were reported to be in evidence during the evaluations reported on by evaluation practitioners. • The extent to which the outcome principles were reported to have resulted from the evaluations reported on by evaluation practitioners. Multiple regression techniques used to determine the extent to which variation in the four empowerment sub- constructs, the individual empowerment construct, the three self – determination sub-constructs, the individual self- determination construct, and the organizational empowerment and self – determination construct could be explained by: • The interaction between empowerment evaluation model fidelity and percentage of activities implemented. • The interaction between empowerment evaluation model fidelity and percentage of activities implemented during the early, mid, and late phases of an empowerment evaluation. • The process principles reported to be in evidence during the empowerment evaluations. • The outcome principles that resulted from the empowerment evaluations. • Empowerment evaluator demographic, academic, and professional characteristics (e.g., gender, number of years conducting empowerment evaluations, highest earned degree, etc.) selected through a correlational analysis. Empowerment Evaluation Practice, Principles, and Outcomes 2015 13
  • 14. SamplingRationale & Procedure • Non – probabilistic (i.e. self – selected), purposive, and not dependent on the rationale of probability theory. • Criterion for inclusion: evaluation practitioner had to have conducted an empowerment or empowering evaluation in the past five years. • Building the Sample: • Initial sample built from the American Evaluation Association (AEA) Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group membership (with help from the moderator), extant evaluation literature, and snowball sampling (n = 232). • Researcher’s professional networks, global evaluation networks, global evaluation associations and societies, AEA eLibrary, AEA365 blogs, and evaluation journals used to further identify potential study participants. • Names and contact information obtained through networks, etc. were cross-checked against the initial sample to update contact information and eliminate duplicates. • Upon reconciliation of names and contact information the sample included 615 evaluation practitioners. Empowerment Evaluation Practice, Principles, and Outcomes 2015 14
  • 15. Recruitment Empowerment Evaluation Practice, Principles, and Outcomes 2015 15 Groups Potential Number of Recipients 3 AEA eGroups (i.e., CPE, ICCE, OL-ECB) 1,416 4 Listserves (e.g., EvalTalk, SCRA, CommunityPsychUk, etc.) 7,800+ 19 LinkedIn Groups (e.g., SAMEA, Evaluators Group, etc.) 83,531 5 Yahoo Groups (e.g., MandENews, XCeval, AfRea, etc.) 5,850 39 Facebook Communities/Interest Groups (e.g., InDEC, M & E, etc.) 57,810 Total* 156,407 *Note: The total is likely duplicated head-count as people often belong to multiple professional groups.
  • 16. Data Collection and Response Rate Data Collection • Dillman’s (2000) four-step process guided implementation of the survey. • Study officially launched one week after the recruitment phase ended by directly contacting potential study participants through email, LinkedIn (In Mail), and Facebook messaging, and diffusely contacting potential study participants via groups used in recruitment with the survey link resulting in 512 total respondents. • From study launch to first reminder 112 respondents. • From first to second reminder 140 additional respondents. • From second to final reminder, 107 additional respondents. • From final reminder to close of survey, 162 additional respondents. Response Rate • Difficult to calculate because both direct and diffuse recruitment methods were used. • Response rate considered to be 25% as 131/521 provided usable data. Empowerment Evaluation Practice, Principles, and Outcomes 2015 16
  • 17. Participant Characteristics 99 of 131 evaluation practitioners (76%) reported demographic, academic preparation, and professional characteristics: • 58.3% English speaking. • 59.8 % White, Non – Hispanic. • 67 % Female. • An average of 15 years in the field of evaluation. • An average of 11 years conducting empowerment evaluations. • 52% lived in a North American country (i.e., United States, Canada, or Mexico). • 51.1% conducted empowerment evaluations in a North American country. • 20% conducted empowerment evaluations in an African country. • 48.5% held a Ph. D. • 38.4% held a Master’s Degree. • Academic degree fields: 46.8%s social sciences, 27.4% monitoring and evaluation, 17.9 % research, 12.8% public health, and 12.6% other academic (e.g., international development). • Professional content area of expertise: 21.1% monitoring, evaluation, and research, 12.6 % respectively development and social justice, and 11.6 % respectively action and community – based research, health, and social science. Empowerment Evaluation Practice, Principles, and Outcomes 2015 17
  • 18. Results: Research Question 1 – Model Fidelity Empowerment Evaluation Practice, Principles, and Outcomes 2015 18 Fidelity to the Three-Step Model Frequency Valid Percent No Fidelity 10 7.8 Fidelity 119 92.2 Total 129 100.0 Fidelity to the Ten-Step Model Frequency Valid Percent No Fidelity 12 9.4 Fidelity 115 90.6 Total 127 100.0 Fidelity to the Five-Tool Model Frequency Valid Percent No Fidelity 26 20.3 Fidelity 102 79.7 Total 128 100.0
  • 19. Results: Research Question 2 – Process Principles in Evidence Empowerment Evaluation Practice, Principles, and Outcomes 2015 19 Community Ownership Process Principle Frequency Valid Percent No Evidence 8 6.3 Evidence 120 93.8 Total 128 100.0 Inclusiveness Process Principle Frequency Valid Percent No Evidence 25 25.8 Evidence 72 74.2 Total 97 100.0 Democratic Participation Process Principle Frequency Valid Percent No Evidence 20 17.5 Evidence 94 82.5 Total 114 100.0
  • 20. Results: Research Question 2 Cont’d… Empowerment Evaluation Practice, Principles, and Outcomes 2015 20 Community Knowledge Process Principle Frequency Valid Percent No Evidence 24 20.7 Evidence 92 79.3 Total 116 100.0 Evidence-Based Strategies Process Principle Frequency Valid Percent No Evidence 25 25.8 Evidence 72 74.2 Total 97 100.0 Accountability Process Principle Frequency Valid Percent No Evidence 28 29.8 Evidence 66 70.2 Total 94 100.0
  • 21. Results: Research Question 3 – Outcome Principles in Evidence Empowerment Evaluation Practice, Principles, and Outcomes 2015 21 Improvement Outcome Principle Frequency Valid Percent No Evidence 10 9.7 Evidence 95 90.5 Total 105 100.0 Organizational learning Outcome Principle Frequency Valid Percent No Evidence 14 14.1 Evidence 85 85.9 Total 99 100.0
  • 22. Results: Research Question 3 Cont’d… Empowerment Evaluation Practice, Principles, and Outcomes 2015 22 Social Justice Outcome Principle Frequency Valid Percent No Evidence 13 13.8 Evidence 81 86.2 Total 94 100.0 Capacity Building Outcome Principle Frequency Valid Percent No Evidence 41 42.3 Evidence 56 57.7 Total 97 100.0
  • 23. Research Questions 1 – 3Take-Aways Research Question 1 • Based on the literature, I thought that the majority of evaluation practitioners would not adhere with fidelity to the steps of the empowerment evaluation model they implemented. However, the average fidelity score index revealed a majority of the study participants adhered with fidelity to the three-step, 10-step, and five- tool models. Research Question 2 • I thought that a majority of evaluation practitioners would report the readily observable process principles of community ownership, inclusion, and democratic participation were evident to them, but the community knowledge, evidence-based strategies, and accountability process principles were not. • The average evidence score index indicated evidence of the community ownership, inclusion, and democratic participation process principles. However, the average evidence score index also indicated evidence of the community knowledge, evidence-based strategies, and accountability process principle. Research Question 3 • I thought that a majority of evaluation practitioners would report the outcome principles of improvement and capacity building were evident to them, but the more distal outcomes of organizational learning and social justice outcome principles were not. • The average evidence score index indicated evidence of the improvement and capacity building principles. However, the average evidence score index also indicated evidence of the organizational learning and social justice principles. Empowerment Evaluation Practice, Principles, and Outcomes 2015 23
  • 24. Results: Research Question 4 – Reject orFail to Reject H0 Empowerment Evaluation Practice, Principles, and Outcomes 2015 24 Research Question 4.1: To what extent could variation in empowerment and self-determination be explained by the interaction between empowerment evaluation model fidelity and percentage of activities implemented? H0 = The interaction between empowerment evaluation model fidelity and percentage of activities implemented would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination. In nine of 10 models I failed to reject the null hypothesis, but I rejected the null hypothesis in one of 10 models because the interaction between model fidelity and percentage of activities implemented was able to explain variation in evaluation capacity; notably it was model fidelity that explained most of the variation. Interaction: R Square = .081, F = 4.158, p = .019 Model Fidelity: Beta = .251, t = 2.493, p = .014
  • 25. Results: Research Question 4 – Reject orFail to Reject H0 Empowerment Evaluation Practice, Principles, and Outcomes 2015 25 Research Question 4.2: To what extent could variation in empowerment and self-determination be explained by the interaction between empowerment evaluation model fidelity and percentage of activities implemented during the early, mid, or late phases of an empowerment evaluation? H0 = The interaction between empowerment evaluation model fidelity and percentage of activities implemented would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination during the early, mid, and late phases of an empowerment evaluation. In nine of 10 models I failed to reject the null hypothesis, but I rejected the null hypothesis in one of 10 models because the interaction between early evaluation model fidelity and percentage of activities implemented, and mid-evaluation model fidelity and percentage of activities implemented was able to explain variation in evaluation capacity; notably it was model fidelity during the mid-evaluation phase that explained most of the variation. Interaction: R Square = .097, F = 2.484, p = .049 Mid-evaluation Model Fidelity: Beta = .308, t = 2.076, p = .041
  • 26. Results: Research Question 4 - Reject orFail to Reject H0 Cont’d… Research Question 4.3: To what extent could variation in empowerment and self-determination be explained by the process principles reported to be in evidence during the evaluations reported on by evaluation practitioners? H0 = The community ownership, inclusiveness, and democratic participation process principles would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination. In six of 10 models I failed to reject the null hypothesis because community ownership and democratic participation were unable to explain variation, but I was able to reject the null hypothesis in four of 10 models because the inclusiveness process principle was able to explain variation in evaluation knowledge, evaluation capacity, individual empowerment, and evaluation competence. Evaluation knowledge - Inclusiveness: R Square = .054, F = 4.068, p = .047; Beta = -.233, t = -2.017, p = .047 Evaluation Capacity - Inclusiveness: R Square = .082, F = 6.310, p = .014; Beta = -.286., t = -2.512, p = .014 Individual Empowerment - Inclusiveness: R Square = .057, F = 4.317, p = .041; Beta = -.238, t = -2.078, p = .041 Evaluation Competence - Inclusiveness: R Square = .059, F = 4.552, p = .036; Beta = -.244, t = -2.134, p = .036 Empowerment Evaluation Practice, Principles, and Outcomes 2015 26
  • 27. Results: Research Question 4 - Reject orFail to Reject H0 Cont’d… Research Question 4.4: To what extent could variation in empowerment and self-determination be explained by the outcome principles that resulted from the evaluations reported on by evaluation practitioners? H0 = The improvement and capacity building outcome principles would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination. In all 10 models I failed to reject the null hypothesis because evidence of the improvement and capacity building outcome principles were unable to explain variation. Empowerment Evaluation Practice, Principles, and Outcomes 2015 27
  • 28. Results: Research Question 4 - Reject orFail to Reject H0 Cont’d… Research Question 4.5: To what extent could variation in empowerment and self-determination be explained by empowerment evaluator demographic, academic, and professional characteristics? H0 = The characteristics gender, professional expertise, and where empowerment evaluations were conducted would be unable to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination. In nine of 10 models I failed to reject the null hypothesis because gender was unable to explain variation, but I was able to reject the null hypothesis in one of 10 models because gender was able to explain variation in individual empowerment. Individual empowerment - Gender: Beta = -.293, t = -2.331, p = .024 In all 10 models I failed to reject the null hypothesis because professional expertise was unable to explain variation. In five of 10 models I failed to reject the null hypothesis because where empowerment evaluations were conducted was unable to explain variation, but in the other five models I was able to reject the null hypothesis because where empowerment evaluations were conducted was able to explain variation in evaluation knowledge, individual empowerment, evaluation competence, evaluation autonomy, and feelings of relatedness to others. Evaluation Knowledge - Work in an African Country: Beta = .627, t = 2.090, p = .041 Individual Empowerment - Work in an African Country: Beta = 699, t = 2.343, p = .023 Evaluation Competence - Work in an African Country: Beta = .715, t = 2.407, p = .019 Evaluation Autonomy - Work in an African Country: Beta = .630, t = 2.121, p = .038 Feelings of Relatedness to Others - Work in a Non-African Country: Beta = -.361, t = -2.895, p = .006; - Work in an African Country: Beta =.645, t = 2.306, p = .025 Empowerment Evaluation Practice, Principles, and Outcomes 2015 28
  • 29. Results: Research Question 4 - Reject orFail to Reject H0 Cont’d… Although not tested against the null hypothesis, variation in evaluation knowledge, individual empowerment, evaluation competence, and evaluation autonomy was explained by where evaluation practitioners lived, specifically an African country. • Evaluation Knowledge - Live in an African Country: Beta = -.675, t = -2.395, p = .020 • Individual Empowerment - Live in an African Country: Beta = -.787, t = -2.850, p = .006 • Evaluation Competence - Live in an African Country: Beta = -.634, t = -2.270, p = .027 • Evaluation Autonomy - Live in an African Country: Beta = -.684, t = -2.454, p = .017 Variance accounted for with all 10 demographic, academic, and professional characteristics in the model: • Evaluation Knowledge: R Square = .283, F = 2.214, p = .030 • Individual Empowerment: R Square = .348, F = 2.829, p = .007 • Evaluation Competence: R Square = .297, F = 2.369, p = .020 • Evaluation Autonomy: R Square = .299, F = 2.390, p = .019 • Feelings of Relatedness to Others: R Square = .385, F = 3.188, p = .003 Empowerment Evaluation Practice, Principles, and Outcomes 2015 29
  • 30. Research Question 4 - Implications forEvaluation Practice and Policy If building evaluation capacity is important: • Implement a high percentage of activities to ensure model fidelity especially during the mid-phase of an empowerment evaluation. • Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the evaluation is included. If building evaluation knowledge is important: • Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the evaluation is included. • Attend to the community knowledge process principle so that everyone who is engaged in the evaluation can use their collective wisdom to develop evaluation tools, evaluation procedures, interpreting data, etc. If building evaluation competence is important: • Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the evaluation is included. If individual empowerment is important: • Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the evaluation is included. Empowerment Evaluation Practice, Principles, and Outcomes 2015 30
  • 31. Strengths of this Study:Addressing Concerns • Determining which steps/activities of the three empowerment evaluation models have been applied in actual evaluation practice. • Operationalizing the steps of the three empowerment evaluation models as evaluation practitioner and evaluation participant activities. • Defining empowerment evaluation model fidelity, developing a model fidelity index, and measuring model fidelity. • Operationalizing the empowerment evaluation process and outcome principles as evaluation practitioner and evaluation participant activities, developing an index for determining evidence of the principles, and determining whether the principles were in evidence during and as a result of the empowerment evaluations reported on by evaluation practitioners. • Defining the sub-constructs and constructs of empowerment theory and self-determination theory for and adapting them to empowerment evaluation. • Explaining variation in empowerment and self-determination by empowerment evaluation model fidelity, evidence of the empowerment evaluation process and outcome principles, and evaluation practitioners’ demographic, academic, and professional characteristics. Empowerment Evaluation Practice, Principles, and Outcomes 2015 31
  • 32. Limitations Most Impactingthis Study • Self - Selection • Calls into question the extent to which the evaluation practitioners in the sample were equivalent to other evaluation practitioners who conduct empowerment evaluations. • Social Desirability • Calls into question the accuracy of responses (see for example, model fidelity scores). • Study participants claim and over-report activities which are socially desirable (Krumpal, 2013). • Self-deception: A respondent believes a statement to be true of herself even though it is inaccurate. • SEEPPO Reliability • Although the SEEPPO appeared to have strong content validity there were not enough observations to conduct an adequate reliability analysis. • Further refinement and testing, including a factor analysis, to establish its reliability and predictive validity is necessary. Empowerment Evaluation Practice, Principles, and Outcomes 2015 32
  • 33. Recommendations forResearch on Empowerment Evaluation • Develop and validate, or adapt existing empowerment and self-determination scales so evaluation practitioners can measure the empowerment and self-determination of individuals and organizations with whom they work; instruments should be able to discriminate between individual empowerment and self- determination sub-constructs and organizational empowerment and self-determination sub-constructs. • Measure the empowerment and self-determination of empowerment evaluation participants before, during, and post-evaluation as well as the distal outcomes of empowerment evaluation (e.g., the extent to which social policy was influenced towards favorable outcomes, the extent to which power differentials were equalized, etc.) • Create and test a theory of change model, including moderators and mediators, showing the likely pathways from empowerment evaluation model implementation and the process and outcome principles to individual and organizational empowerment and self-determination. • Test whether there is a relationship between models and principles, the extent to which a relationship exists, and whether model fidelity (along a continuum) or evidence of the principles (along a continuum) attenuates that relationship. • Explore the role of contextual variables in the development/enhancement of empowerment and self- determination by comparing the outcomes of empowerment evaluations conducted in different countries. Empowerment Evaluation Practice, Principles, and Outcomes 2015 33
  • 34. Conclusions • To a limited extent individual empowerment and self-determination were the likely outcomes of the empowerment evaluations reported on by this sample of 131 evaluation practitioners; model fidelity, the inclusiveness and community knowledge process principles, and evaluation practitioner characteristics appeared to have been able to explain variation. • Given evidence of all 10 guiding principles it can be said that the evaluations reported on were likely to be empowerment evaluations as defined by Fetterman and Wandersman. • The evaluation practitioners in this study largely adhered with fidelity to the steps of the models they implemented; no model was privileged over another as steps were chosen from across all three models (96.2%, n = 126). • Evaluation practitioner’s personal, academic, and professional characteristics did not fully factor into the empowerment of individuals and organizations with whom they worked. However, evaluation knowledge, individual empowerment and all three individual self-determination sub-constructs (i.e., competence, autonomy, and relatedness) appeared to have been correlated (both + and -) with empowerment evaluations conducted by evaluation practitioners who lived and worked in an African country, who were female, and who conducted empowerment evaluations in another country. • The SEEPPO was a successful first attempt at providing a practical, largely quantitative tool to empowerment evaluators interested in assessing the process and outcomes of their work, less so actual empowerment and self-determination. Empowerment Evaluation Practice, Principles, and Outcomes 2015 34