This document provides an overview of project evaluation. It discusses why evaluations are conducted (to provide feedback and determine if objectives were met), who is typically involved in the evaluation process (e.g. project directors, evaluators), and how to design and carry out an evaluation (e.g. creating an evaluation plan with goals, indicators, and benchmarks). The document also covers common evaluation methods like interviews, surveys, documentation review and observations. Overall, the document aims to demystify the evaluation process and provide guidance on the key questions of why, who, where, when and how to approach project evaluation.

1. Taking the Mystery out ofTaking the Mystery out of
Project EvaluationProject Evaluation
Marjorie Dennis, UMass LowellMarjorie Dennis, UMass Lowell
Lisa Glickstein PhDLisa Glickstein PhD
Andover Public SchoolsAndover Public Schools

2. IntroductionsIntroductions
 Marjorie Dennis: NSF GK-12, RET; US EDMarjorie Dennis: NSF GK-12, RET; US ED
TAH, PT3; State ITQ, STEM PipelineTAH, PT3; State ITQ, STEM Pipeline
 Lisa Glickstein: US ED TAH, PEP; StateLisa Glickstein: US ED TAH, PEP; State
STEM PipelineSTEM Pipeline
 Participant ExperiencesParticipant Experiences

3. Googlisms!Googlisms!
 Evaluation is…Evaluation is…
 …”…”used to manage a project to meetused to manage a project to meet
objectives.”objectives.”
 …”…”the process of measuring the quality ofthe process of measuring the quality of
performance against standards toperformance against standards to
determine if the standards have beendetermine if the standards have been
met.”met.”
 …”…”a systematic inquiry designed toa systematic inquiry designed to
provide information to decision makers.”provide information to decision makers.”

4. Session OverviewSession Overview
 WhyWhy do you evaluate?do you evaluate?
 WhoWho chooses the evaluator?chooses the evaluator?
 WhereWhere do you find an evaluator?do you find an evaluator?
 WhenWhen do you start working with thedo you start working with the
evaluator?evaluator?
 HowHow do you design and carry out ando you design and carry out an
evaluation?evaluation?
 WhatWhat do you do if something goes wrong?do you do if something goes wrong?

5. WHY?WHY?
 Because you have to – federal or state lawBecause you have to – federal or state law
 To get feedback during the projectTo get feedback during the project

Formative evaluationFormative evaluation
 Because you want to answer the bigBecause you want to answer the big
question:question: DID IT WORK?DID IT WORK?

Summative evaluationSummative evaluation

6. ““Research seeks toResearch seeks to proveprove,,
evaluation seeks toevaluation seeks to improveimprove…”…”
M.Q. PattonM.Q. Patton
(former President of the American Evaluation Association)(former President of the American Evaluation Association)
Adapted from: Maberry Consulting Group

7. WHO?WHO?
 Who is a good evaluator?Who is a good evaluator?
 Who selects the evaluator?Who selects the evaluator?
 Who decides contract terms?Who decides contract terms?
 Who will the evaluator report to?Who will the evaluator report to?
 Who will the evaluator work with?Who will the evaluator work with?
 Who owns the evaluation plan, data andWho owns the evaluation plan, data and
reports?reports?

8. Internal versus External EvaluationInternal versus External Evaluation
 Program guidelinesProgram guidelines
 CostCost
 AvailabilityAvailability
 Existing RelationshipsExisting Relationships

9. Four Steps to Select an EvaluatorFour Steps to Select an Evaluator
1.1. Decide who selects the evaluatorDecide who selects the evaluator

Project Director or Advisory BoardProject Director or Advisory Board

Head of organizationHead of organization

Grant or Business Office (lowest bidder)Grant or Business Office (lowest bidder)

Grant-makerGrant-maker
1.1. Contact potential evaluators and solicitContact potential evaluators and solicit
proposalsproposals

10. Four Steps (cont’d)Four Steps (cont’d)
3.3. Choose selection criteria (see Notes)Choose selection criteria (see Notes)

KnowledgeKnowledge

Experience – overall, and project-specificExperience – overall, and project-specific

CostCost

ReferencesReferences

Communication skillsCommunication skills
3.3. Evaluate proposals and selectEvaluate proposals and select

11. Intellectual PropertyIntellectual Property
 The funder paid for the evaluation and canThe funder paid for the evaluation and can
request data or reports at any timerequest data or reports at any time
 The fiscal agent/sponsor retains the hardThe fiscal agent/sponsor retains the hard
and electronic copies of all evaluationand electronic copies of all evaluation
materialsmaterials
 Evaluation plan, rubrics, data and reportsEvaluation plan, rubrics, data and reports
all belong to the project (not to theall belong to the project (not to the
evaluator)evaluator)

12. WHEREWHERE??
 WhereWhere do you find an evaluator?do you find an evaluator?

In houseIn house

ExternalExternal
 WhereWhere does one learn to be an evaluator?does one learn to be an evaluator?
 WhereWhere are evaluators certified?are evaluators certified?
 WhereWhere do external evaluators work?do external evaluators work?

13. Professional EvaluatorsProfessional Evaluators
 American Evaluation Association (American Evaluation Association (
http://www.eval.org/http://www.eval.org/))
 Graduate & Certificate Programs (Graduate & Certificate Programs (
http://www.eval.org/p/cm/ld/fid=43http://www.eval.org/p/cm/ld/fid=43))
 There is no single recognized credentialThere is no single recognized credential
 Consulting or in a university setting (Consulting or in a university setting (
http://www.eval.org/p/cm/ld/fid=108http://www.eval.org/p/cm/ld/fid=108))

14. WHEN?WHEN?
 WhenWhen do you select the evaluator?do you select the evaluator?
 WhenWhen do you work out terms of hire?do you work out terms of hire?
 WhenWhen do you have the first projectdo you have the first project
meeting?meeting?
 WhenWhen and how often will you meet?and how often will you meet?
 WhenWhen do you expect reports?do you expect reports?

15. Initial ContactInitial Contact
 Involving the evaluator in project planning, orInvolving the evaluator in project planning, or
planning evaluation during (not after) projectplanning evaluation during (not after) project
design is a good ideadesign is a good idea
 An evaluator can prepare an evaluation plan forAn evaluator can prepare an evaluation plan for
a proposal (usually at no cost)a proposal (usually at no cost)
 Sometimes it wins points to name an evaluatorSometimes it wins points to name an evaluator
in the grant proposalin the grant proposal
 Be aware of bid laws in your stateBe aware of bid laws in your state
 Post-award some changes to the evaluation planPost-award some changes to the evaluation plan
are okayare okay

16. Contract and TermsContract and Terms
 Scope of services – list of activities andScope of services – list of activities and
work products (see Notes)work products (see Notes)
 Cost of entire contract, or hourly rate andCost of entire contract, or hourly rate and
estimated total (8-15% of total grant is toestimated total (8-15% of total grant is to
be expected)be expected)
 Automatic termination if grant ends,Automatic termination if grant ends,
options to terminate for cause or no causeoptions to terminate for cause or no cause
 Protections for organization from failuresProtections for organization from failures
or omissions on part of evaluatoror omissions on part of evaluator

17. HOW?HOW?
 How do you design and carry out anHow do you design and carry out an
evaluation?evaluation?

Create evaluation planCreate evaluation plan

Select indicators and benchmarksSelect indicators and benchmarks
 How do you collect and analyze data?How do you collect and analyze data?

Analyze dataAnalyze data

Write reportsWrite reports

18. Evaluation PlanEvaluation Plan
 Project’s goals, activities and timeline includeProject’s goals, activities and timeline include
data collectiondata collection
 Goal = ultimate change project intends to make inGoal = ultimate change project intends to make in
a condition (e.g. poverty)a condition (e.g. poverty)
 Indicators = changes in attitudes, knowledge andIndicators = changes in attitudes, knowledge and
behaviors that show progress towards meetingbehaviors that show progress towards meeting
goalgoal
 Benchmarks = amount of change expected in aBenchmarks = amount of change expected in a
given project yeargiven project year

19. Mini-EvaluationMini-Evaluation
 Working alone or in groups, fill out theWorking alone or in groups, fill out the
sample evaluation form for your programsample evaluation form for your program
 Focus first on what you are certain of –Focus first on what you are certain of –
include details where knowninclude details where known
 Focus next on what you don’t know – howFocus next on what you don’t know – how
would you measure or find out? Whowould you measure or find out? Who
would you ask and how?would you ask and how?
 Can you make recommendations?Can you make recommendations?

20. Research vs EvaluationResearch vs Evaluation
 Production of generalizableProduction of generalizable
knowledge is the goalknowledge is the goal
 Researcher-derivedResearcher-derived
questions (hypothesis)questions (hypothesis)
 Paradigm stance (how orParadigm stance (how or
why does it work?)why does it work?)
 Setting designed to testSetting designed to test
hypothesis (controlled)hypothesis (controlled)
 Clearer role (principalClearer role (principal
investigator)investigator)
 PublishedPublished
 Clearer allegianceClearer allegiance
 May produce generalizableMay produce generalizable
knowledgeknowledge
 Program- or funder-derivedProgram- or funder-derived
questionsquestions
 Judgmental quality (does itJudgmental quality (does it
work?)work?)
 Action setting to provideAction setting to provide
direct servicesdirect services
 Role conflicts (projectRole conflicts (project
director AND evaluator)director AND evaluator)
 Often unpublishedOften unpublished
 Multiple allegiancesMultiple allegiances
Adapted from: Maberry Consulting Group

21. Formative EvaluationFormative Evaluation
 Project proceeding as planned?Project proceeding as planned?
 Activities, recruitment, data collection takingActivities, recruitment, data collection taking
place?place?
 All partners working together?All partners working together?
 How can or should project get back on track?How can or should project get back on track?
 Is project getting closer to its anticipated goals?Is project getting closer to its anticipated goals?

22. Summative EvaluationSummative Evaluation
 Did the project meet its goals or not?Did the project meet its goals or not?
 Did the participants benefit?Did the participants benefit?
 To what extent did they benefit?To what extent did they benefit?
 Can you prove that activities caused theCan you prove that activities caused the
benefits?benefits?
 What activities were most effective or cost-What activities were most effective or cost-
effective?effective?
 Is the project sustainable?Is the project sustainable?
 Is the project replicable?Is the project replicable?

23. Short vs Long-term OutcomesShort vs Long-term Outcomes
OUTCOMES
What results for individuals, families, communities..…
SHORT
Learning
Changes in:
• Awareness
• Knowledge
• Attitudes
• Skills
• Opinion
• Aspirations
• Motivation
• Behavioral intent
MEDIUM
Action
Changes in:
•Behavior
•Decision-making
•Policies
•Social action
LONG-TERM
Conditions
Changes in:
•Well-being
•Health
•Economy
•Civics
•Environment
Adapted from: University of Wisconsin – Logic Models

24. The Four StandardsThe Four Standards
 UtilityUtility: Who needs the information and: Who needs the information and
what information do they need?what information do they need?
 FeasibilityFeasibility: How much money, time, and: How much money, time, and
effort can we put into this?effort can we put into this?
 Propriety:Propriety: What steps need to be takenWhat steps need to be taken
for the evaluation to be ethical?for the evaluation to be ethical?
 Accuracy:Accuracy: What design will lead toWhat design will lead to
accurate information?accurate information?
Adapted from: Maberry Consulting Group

25. Six methods to collect information and data…Six methods to collect information and data…
Method Overall Purpose Advantages Challenges
1. Questionnaires,
Surveys,
and Checklists
To quickly and/or
easily get a lot of
information from
people in a non-
threatening way
• Can be completed
anonymously
• Inexpensive to
administer
• Easy to compare
and analyze
• Administer to
many people
• Can get lots of
data
• Many sample
questionnaires
• Might not get
careful feedback
• Wording can bias
responses
• Are impersonal
• In surveys, may
need sampling
expert
• Doesn't get the full
story
Adapted from: Maberry Consulting Group

26. Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
2. Interviews To fully understand
participant's
impressions or
experiences, or to
learn more about their
answers to
questionnaires
• Collects a full
range and depth of
information
• Develops
relationship with
the participant
• Can be flexible
and responsive
• May take too long
• Can be hard to
analyze and
compare
• Can be costly
• Interviewer can
bias responses
Adapted from: Maberry Consulting Group

27. Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
3. Documentation
review
To get an impression of
how program operates
without interrupting the
program; review of
applications, finances,
memos, minutes,
participant work
products, and so on
• Collect
comprehensive and
historical
information
• Doesn't interrupt
program or
participant’s routine
• Information already
exists
• Few biases about
information
• Often time-
consuming
• Information may be
incomplete
• Need to be quite
clear about what you
are looking for
• Not flexible, as data
restricted to what
already exists
Adapted from: Maberry Consulting Group

28. Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
4. Observation To gather accurate
information about
how a program
actually operates,
particularly about
processes
• View operations of
a program as they
are actually
occurring
• Can adapt to
events as they
occur
• Difficult to
interpret observed
behaviors
• Complex to
categorize
observations
• Can influence
behaviors of
program
participants
• Can be expensive
Adapted from: Maberry Consulting Group

29. Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
5. Focus groups To explore a topic in
depth through group
discussion, such as
reactions to an
experience or
suggested change,
understanding
common complaints
(useful in evaluation
and marketing)
• Quickly and
reliably get
common
impressions
• Can be an efficient
way to get a range
and depth of
information in a
short time
• Can convey key
information about
programs
• Hard to analyze
responses
• Need a good
facilitator for
safety and closure
• Difficult to
schedule 6-8
people together
Adapted from: Maberry Consulting Group

30. Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
6. Case studies To fully understand or
depict participant
experiences in a
program, and conduct
comprehensive
examination through
cross comparison of
cases
• Fully depicts
participant’s
experience in
program input,
process and results
• Powerful means to
portray program to
outsiders
• Usually quite time
consuming to
collect, organize
and describe
• Represents depth
of information,
rather than breadth
Adapted from: Maberry Consulting Group

31. Basic analysis of quantitative (numeric) dataBasic analysis of quantitative (numeric) data
1.1. Make copies of your data and store the original. UseMake copies of your data and store the original. Use
the copy for making edits, cutting and pasting, and sothe copy for making edits, cutting and pasting, and so
onon
2.2. Tabulate the information in one or more spreadsheets.Tabulate the information in one or more spreadsheets.
3.3. Examine the median, mean, and distribution (range) ofExamine the median, mean, and distribution (range) of
data sets; think about which is/are more meaningfuldata sets; think about which is/are more meaningful
4.4. Think about segmentation: Do you have sub-Think about segmentation: Do you have sub-
populations based on demographics or how muchpopulations based on demographics or how much
service was received?service was received?
5.5. Consider confounding variables: Is the effect due to aConsider confounding variables: Is the effect due to a
bias rather than your service or intervention?bias rather than your service or intervention?
Adapted from: Maberry Consulting Group

32. Basic analysis of qualitative (subjective) dataBasic analysis of qualitative (subjective) data
(interview or focus group responses, or written commentary on questionnaires)(interview or focus group responses, or written commentary on questionnaires)
1.1. Read through all the data.Read through all the data.
2.2. Organize comments into similar categories:Organize comments into similar categories:
concerns, suggestions, strengths, weaknesses,concerns, suggestions, strengths, weaknesses,
similar experiences, program inputs,similar experiences, program inputs,
recommendations, outputs, or outcome indicators.recommendations, outputs, or outcome indicators.
3.3. Look for patterns, associations and possible causalLook for patterns, associations and possible causal
relationships.relationships.
4.4. Keep all raw data as recommended by the funderKeep all raw data as recommended by the funder
after completion in case needed for futureafter completion in case needed for future
reference.reference.
Adapted from: Maberry Consulting Group

33. Evaluation is not only IndicatorsEvaluation is not only Indicators
 Evaluation: Indicators in the context of:Evaluation: Indicators in the context of:

Design & implementationDesign & implementation

Program relevanceProgram relevance

EfficiencyEfficiency

EffectivenessEffectiveness

ImpactImpact

SustainabilitySustainability

34. ExperiencesExperiences
 Write down the worst thing that anWrite down the worst thing that an
evaluator ever said to you about yourevaluator ever said to you about your
project…project…
 ……or the worst thing that ever went wrongor the worst thing that ever went wrong
on a grant-funded project at youron a grant-funded project at your
organization!organization!

35. WHAT?WHAT?
 WhatWhat do we do when something goesdo we do when something goes
wrong?wrong?

Partners don’t get alongPartners don’t get along

Leadership issuesLeadership issues

Disagreements over data interpretationDisagreements over data interpretation

Non-performance by partner(s)Non-performance by partner(s)

Change in conditionsChange in conditions

Conflicts of interestConflicts of interest

36. The Challenge of Being anThe Challenge of Being an
Evaluator (Sirotnik, 1999)Evaluator (Sirotnik, 1999)
If you…are looking for highly definitive,If you…are looking for highly definitive,
generalizable, cause/effect relationshipsgeneralizable, cause/effect relationships
between measurable, high-validity, independentbetween measurable, high-validity, independent
and dependent variables, then I suggest you goand dependent variables, then I suggest you go
to work for a pharmaceutical company…to work for a pharmaceutical company…
However, if you have a high tolerance forHowever, if you have a high tolerance for
ambiguity and like to mess around…withambiguity and like to mess around…with
contexts that are always colliding andcontexts that are always colliding and
changing…in these settings that are engaged inchanging…in these settings that are engaged in
long-term processes of renewal and change –long-term processes of renewal and change –
well then have I got a career for you!well then have I got a career for you!

37. If the evaluator ain’t happy, ain’t noIf the evaluator ain’t happy, ain’t no
one happy…one happy…
 If your project isn’t going well, it is theIf your project isn’t going well, it is the
evaluator’s job to tell you (and ultimately toevaluator’s job to tell you (and ultimately to
tell the funder)tell the funder)
 If you have an issue with your evaluator,If you have an issue with your evaluator,
first look at and fix issues with your projectfirst look at and fix issues with your project
 Most of the time, this will repair yourMost of the time, this will repair your
relationship with the evaluator, or at leastrelationship with the evaluator, or at least
make it tolerable!make it tolerable!

38. Conflict of PartnersConflict of Partners
 Partners have conflicting ideasPartners have conflicting ideas
 One or more partners are notOne or more partners are not
carrying through with activitiescarrying through with activities
or data collectionor data collection
 Personality clashesPersonality clashes
 Complaints about distribution ofComplaints about distribution of
resources or other complaintsresources or other complaints

39. Leadership IssuesLeadership Issues
 Poor communication – no contact, badPoor communication – no contact, bad
emails, late information and noticesemails, late information and notices
 Bills not paid on time, delays in processingBills not paid on time, delays in processing
registrations for activitiesregistrations for activities
 Bad record-keeping or data collectionBad record-keeping or data collection
 Failure to respond to formative evaluationFailure to respond to formative evaluation
 Mismanagement of project resourcesMismanagement of project resources
 Illegal activityIllegal activity

40. Different Interpretations of DataDifferent Interpretations of Data
 Evaluator sees the dataEvaluator sees the data
one way, project partnersone way, project partners
see it anothersee it another
 Lack of data leads toLack of data leads to
subjective assessmentsubjective assessment
 One party rejects dataOne party rejects data

41. Non Performance of PartnersNon Performance of Partners
 Failure to complete scope of servicesFailure to complete scope of services
 Communication issuesCommunication issues
 Significant, ongoing delaysSignificant, ongoing delays
 Personality clashes or personnel problemPersonality clashes or personnel problem
 Undermining professional relationshipsUndermining professional relationships
 Mismanagement of fundsMismanagement of funds

42. Change in ConditionsChange in Conditions
 New competitor (threat)New competitor (threat)
 Change in lawChange in law
 Sub-contractor or partner goes out ofSub-contractor or partner goes out of
business or quitsbusiness or quits
 Significant economic downturnSignificant economic downturn
 Change in leadership (project orChange in leadership (project or
organization)organization)

43. Conflict of InterestConflict of Interest
 Is the evaluator independent enough toIs the evaluator independent enough to
objectively evaluate the project?objectively evaluate the project?
 Would giving a positive or negativeWould giving a positive or negative
evaluation result in financial gain?evaluation result in financial gain?

Future contracts with one or more partnersFuture contracts with one or more partners
 Is there an appearance of conflict ofIs there an appearance of conflict of
interest?interest?

Employer/employee or family relationshipEmployer/employee or family relationship

44. Presentation Wrap UpPresentation Wrap Up
 Additional stories and comments on theAdditional stories and comments on the
evaluation processevaluation process
 Questions on evaluationQuestions on evaluation
 Reference materials for furtherReference materials for further
investigation and information on workinginvestigation and information on working
with an evaluator and the evaluationwith an evaluator and the evaluation
processprocess

45. Additional ReadingAdditional Reading
 Evaluating Professional DevelopmentEvaluating Professional Development, Thomas, Thomas
R. Guskey, ISBN 0-7619-7561-6, 2000.R. Guskey, ISBN 0-7619-7561-6, 2000.
 ““8 Smooth Steps: Steps to Your own8 Smooth Steps: Steps to Your own
Evaluation,” Joellen Killion,Evaluation,” Joellen Killion, Journal of theJournal of the
National Staff Development CouncilNational Staff Development Council, p 14, Fall, p 14, Fall
2003.2003.
 Practicing Evaluation: A Collaborative ApproachPracticing Evaluation: A Collaborative Approach,,
Rita O’Sullivan, ISBN 0-7619-2456-5, 2004.Rita O’Sullivan, ISBN 0-7619-2456-5, 2004.
 W.K. Kellogg Foundation Evaluation HandbookW.K. Kellogg Foundation Evaluation Handbook
& Development Guide,& Development Guide, www.wkkf.orgwww.wkkf.org, 2008., 2008.

46. Thank you for attending!Thank you for attending!
 Contact information:Contact information:

Lisa Glickstein:Lisa Glickstein: lglickstein@aps1.netlglickstein@aps1.net

Marjorie Dennis:Marjorie Dennis: marjorie_dennis@uml.edumarjorie_dennis@uml.edu