1. ““Stuck Somewhere in theStuck Somewhere in the
Middle”Middle”
Anthropologists as Evaluators:Anthropologists as Evaluators:
Mediating Between Community-Based HealthMediating Between Community-Based Health
Projects and the StateProjects and the State
Panel: “Studying Up: Exploring the Exotic World of Elites”Panel: “Studying Up: Exploring the Exotic World of Elites”
2006 SfAA Annual Meeting2006 SfAA Annual Meeting
Beverly Hill & Sharon WatsonBeverly Hill & Sharon Watson
The University of South FloridaThe University of South Florida

2. IntroductionIntroduction
► In this presentation, we address Laura Nader’sIn this presentation, we address Laura Nader’s
concepts of “controlling processes” andconcepts of “controlling processes” and
“studying up” in order to explore:“studying up” in order to explore:
 Evaluation as a tool for empowermentEvaluation as a tool for empowerment
 Negotiating the expert-client relationshipNegotiating the expert-client relationship
 Adapting expertise to accommodate democraticAdapting expertise to accommodate democratic
organizational practicesorganizational practices
Goal: To better inform our practice as appliedGoal: To better inform our practice as applied
anthropologists serving as evaluators.anthropologists serving as evaluators.

3. Description of ProjectDescription of Project
► Two-year project, collaborative between aTwo-year project, collaborative between a
private communications firm and a hospital-private communications firm and a hospital-
based research centerbased research center
► Funded by the state, our clientFunded by the state, our client
► Evaluation teams represented in threeEvaluation teams represented in three
regions, mostly university-basedregions, mostly university-based
► Community based organizations with healthCommunity based organizations with health
promotion projects that served populationspromotion projects that served populations
where ethnic and racial disparities existwhere ethnic and racial disparities exist

4. The Big PictureThe Big Picture
General Population including minority
Groups suffering from health disparity
Legislation passed to reduce health
Disparities in six target areas
Appropriate $5 million
Department of Health
Recipient of $5 million
Office of Minority
Health
“Client”
Office of Minority
Health
“Client”
Advisory
Committee
Evaluation Team
“Experts”
Directors
Field Visit Teams
“Grantees” Organizations
Implementing Programs to address racial
And ethnic health disparities
“the other client”
CBOs DH NGO Community Centers

5. Evaluation RoleEvaluation Role
““Our roles are defined by our contract with the DOHOur roles are defined by our contract with the DOH
of the State of Florida to achieve two overall linesof the State of Florida to achieve two overall lines
of action:of action: formative and summative evaluationformative and summative evaluation..
Briefly stated, the formative evaluation is designedBriefly stated, the formative evaluation is designed
to help you (the projects) have success in yourto help you (the projects) have success in your
future efforts to improve your community’s health.future efforts to improve your community’s health.
Within the formative evaluation, we will assessWithin the formative evaluation, we will assess
your program’s strengths and challenges and offeryour program’s strengths and challenges and offer
suggestions for improvement.suggestions for improvement. Our efforts willOur efforts will
focus on helping you improve how you measurefocus on helping you improve how you measure
and demonstrate your performanceand demonstrate your performance.”.”

6. Evaluation Role (cont’d)Evaluation Role (cont’d)
““ On the other hand, the summative evaluation is what you might considerOn the other hand, the summative evaluation is what you might consider
a more traditional form of evaluation. Within the summative evaluation,a more traditional form of evaluation. Within the summative evaluation,
we will examine whether or notwe will examine whether or not to what extent your program isto what extent your program is
meeting it’s goals and how much value it is providing tomeeting it’s goals and how much value it is providing to
your communityyour community .”.”
““ In both of these lines of action, we will be fair and equitable. EvenIn both of these lines of action, we will be fair and equitable. Even
though our assistance will be based on your challenges and requests,though our assistance will be based on your challenges and requests,
most communications will be standardized through the use of protocolmost communications will be standardized through the use of protocol
developed by our entire team. Moreover, we will establish and usedeveloped by our entire team. Moreover, we will establish and use
consistent standards of evaluation across sites and we will report ourconsistent standards of evaluation across sites and we will report our
findings to the State using the most objective and scientifically rigorousfindings to the State using the most objective and scientifically rigorous
methods that can be accomplished within the constraints we all face.methods that can be accomplished within the constraints we all face.
However, we will also attempt to understand your specificHowever, we will also attempt to understand your specific
challenges well enough to permit sensitive and respectfulchallenges well enough to permit sensitive and respectful
communications to the Statecommunications to the State .”.”

7. Role of evaluatorsRole of evaluators
Year oneYear one
► BoundariesBoundaries
► AuthorityAuthority
► RoleRole
 Participatory, objectiveParticipatory, objective
researcher, advocacy,researcher, advocacy,
coachcoach
► Task: to measure andTask: to measure and
evaluate performanceevaluate performance
Year twoYear two
► On-line data systemOn-line data system
was emphasized aswas emphasized as
the primary tool forthe primary tool for
measurement andmeasurement and
evaluation.evaluation.
► Technicians/teachersTechnicians/teachers
of new system, inof new system, in
addition to the firstaddition to the first
year’s tasks.year’s tasks.

8. Adapting expertise to accommodateAdapting expertise to accommodate
democratic organizational practicesdemocratic organizational practices
► Flow of power:Flow of power:
 It is acquired, used, maintained, and lostIt is acquired, used, maintained, and lost
 It is redistributed by professionsIt is redistributed by professions
 It involves embedded microprocesses, which challengeIt involves embedded microprocesses, which challenge
our ability to study upour ability to study up
 It can be analyzed through models of harmony viaIt can be analyzed through models of harmony via
mediation vs. the quest for justice (Nader 1997)mediation vs. the quest for justice (Nader 1997)
Q: Are we, as evaluators, complicit in the system’sQ: Are we, as evaluators, complicit in the system’s
perpetuation of hegemony, rather than acting as agentsperpetuation of hegemony, rather than acting as agents
of change in it’s reversal?of change in it’s reversal?

9. Negotiating the expert-clientNegotiating the expert-client
relationshiprelationship
►Fundamental question: Is it possible toFundamental question: Is it possible to
restructure the largely undemocratic expert-restructure the largely undemocratic expert-
client relationship?client relationship?
►Does autonomy have a place in theDoes autonomy have a place in the
evaluation process?evaluation process?
 Are the decisions made by those of the CBOsAre the decisions made by those of the CBOs
voluntary, or are they disguised as free will?voluntary, or are they disguised as free will?
 Example: MCHD project vs. LHANC projectExample: MCHD project vs. LHANC project

10. Evaluation as empowerment?Evaluation as empowerment?
►Empowerment evaluation is conceptualizedEmpowerment evaluation is conceptualized
by David Fetterman (1997) and describedby David Fetterman (1997) and described
as, “the use of evaluation concepts,as, “the use of evaluation concepts,
techniques, and findings to fostertechniques, and findings to foster
improvement and self-determination.”improvement and self-determination.”
►Harmony law model (“ethic of treatment”Harmony law model (“ethic of treatment”
approach) vs. the explicit quest for justiceapproach) vs. the explicit quest for justice
(“ethic of right and wrong”)(“ethic of right and wrong”)

11. Ethical ConsiderationsEthical Considerations
► American EvaluationAmerican Evaluation
Association guidingAssociation guiding
principles:principles:
 Systematic inquirySystematic inquiry
 CompetenceCompetence
 Integrity/honestyIntegrity/honesty
 Respect for peopleRespect for people
 Responsibilities forResponsibilities for
general & public welfaregeneral & public welfare
► Constraints within theConstraints within the
boundaries of ourboundaries of our
authorization arose,authorization arose,
limiting our ability tolimiting our ability to
serve as effectiveserve as effective
mediators.mediators.

12. Ethical Considerations (cont’d)Ethical Considerations (cont’d)
► Human agency (autonomy vs. conformity)Human agency (autonomy vs. conformity)
 Example: BP project, accepting funding but notExample: BP project, accepting funding but not
reporting.reporting.
► Lack of mechanism to ensureLack of mechanism to ensure
truthfulness/accuracy of data reportedtruthfulness/accuracy of data reported
 Example: SWP reporting numbers vs. health outcomes.Example: SWP reporting numbers vs. health outcomes.
► Relationships between CBOs and contractRelationships between CBOs and contract
managers—not equal across all projects.managers—not equal across all projects.

13. Revisit The Big PictureRevisit The Big Picture
General Population including minority
Groups suffering from health disparity
Legislation passed to reduce health
Disparities in six target areas
Appropriate $5 million
Department of Health
Recipient of $5 million
Office of Minority
Health
“Client”
Office of Minority
Health
“Client”
Advisory
Committee
Evaluation Team
“Experts”
Directors
Field Visit Teams
“Grantees” Organizations
Implementing Programs to address racial
And ethnic health disparities
“the other client”
CBOs DH NGO Community Centers
Contract Managers

14. SummarySummary
►Why should we employ studying up toWhy should we employ studying up to
determine how individuals/groups aredetermine how individuals/groups are
influenced and persuaded to participate ininfluenced and persuaded to participate in
their own domination or, alternatively, totheir own domination or, alternatively, to
resist it?resist it?

15. Summary (cont’d)Summary (cont’d)
► Mechanisms for ensuring that “studying up” will result inMechanisms for ensuring that “studying up” will result in
more equitable outcomes:more equitable outcomes:
 Create space for the reversal of top-down power flowsCreate space for the reversal of top-down power flows
in order to maintain accountability by those in positionsin order to maintain accountability by those in positions
of power, and to engage social actors at every level ofof power, and to engage social actors at every level of
the evaluation process.the evaluation process.
 Utilize our participation in evaluation roles to addressUtilize our participation in evaluation roles to address
decision makers in their forum.decision makers in their forum.
 Move away from the “harmony law model,” whichMove away from the “harmony law model,” which
encapsulates coercive compromise and consensus as aencapsulates coercive compromise and consensus as a
form of behavior modification, toward a model of justiceform of behavior modification, toward a model of justice
that is characterized by an ethic of right and wrong vs.that is characterized by an ethic of right and wrong vs.
an ethic of treatment.an ethic of treatment.