This document summarizes chapters 1 and 2 of "The 2010 User-Friendly Handbook for Project Evaluation" by Joy Frechtling. It discusses the reasons for conducting evaluations, including allowing for improvement and providing unanticipated insights. It outlines two types of evaluations: formative evaluations provide ongoing feedback, while summative evaluations assess the overall success of a completed project. Formative evaluations include implementation evaluations to check that a project is proceeding as planned and progress evaluations to check that goals are being met. The document contrasts program and project evaluations.

1. The 2010 User-Friendly
Handbook for Project
Evaluation
by Joy Frechtling Westat
Chapters 1 and 2
Summary by Farran Coker,Tracy Crosby, Savannah Baker, and
Manda Cawthon

2. Follow the link to viewThe 2010
User-Friendly Handbook for Project
Evaluation
https://www.purdue.edu/research/docs/pdf/2010NSFus
er-friendlyhandbookforprojectevaluation.pdf

3. Table of Contents
•Chapter 1: Reasons for
Conducting Evaluations
•Chapter 2: Evaluation
Prototypes

4. Chapter 1: Reasons for Conducting Evaluations

5. What is an evaluation?
• A comprehensive definition, as presented by the Joint Committee on
Standards for Educational Evaluation (1994), holds that evaluation is
“systematic investigation of the worth or merit of an object.”
• This idea of evaluations revolves around the idea of using evaluations for a
purpose meaning that once an evaluation is conducted, action must be taken
based on the results.

6. Why should evaluations be conducted?
• Evaluations allow for improvement. It is essential to understand what
objectives are being met and what objectives are not in order to make
decisions to facilitate improvement.
• Evaluations can always provide information on new insights that were
unanticipated.These insights can help drive future improvements.

7. What is the best way to conduct an evaluation?
• Evaluations should not be conducted in an adversarial mode, but instead
conducted in a manner that will provide insight for improvements.

8. Chapter 2: Evaluation Prototypes

9.  Evaluation: Includes different models and methods of data
collection within the lifespan of a project
Two basic types of evaluation:
1) Program evaluation
2) Project evaluation
What is the difference between a program and a project?

10. Programs and Projects
• Program: a coordinated approach to exploring a specific area related to the
NSF’s mission of strengthening science, mathematics, and technology.
• Project: a particular investigation or developmental activity funded by that
program. (NSF funds discrete projects to test the effectiveness of a program
based upon student need and performance.)

11. Program Evaluation
• The evaluation of a program determines the total value of a series of
projects.
• The process of evaluation includes the examination of the viability of
activities and strategies employed.
• The collection of data may be summative or done throughout the program
while it is underway.
• In any case, data is collected annually.

12. Project Evaluation
• Project evaluation focuses on an individual project funded under the
umbrella of a specific program.The collected information is used to enhance
the project as it develops.
• The data is used to guide the project while it is underway.This promotes the
obtainment of the defined program goals and objectives of the project.
• Examination of specific components could also be implemented.To what
extent have the goals been met?What is needed to make the project more
successful?

13. The Different Kinds of Evaluation:
FORMATIVE AND SUMMATIVE EVALUATIONS
Formative Assessment:
Purpose: provide information for project improvement.
Begins during project development and continues throughout the
duration of the project.
Two components: 1) Implementation Evaluation and 2) Progress
Evaluation
Summative Assessment:
Purpose: to assess the quality and impact of a fully implemented project.

14. Formative Evaluation

15. Formative Evaluation: Implementation Evaluation
Purpose: to assess whether the project is being conducted as
planned.
• This assessing of the project is conducted several times
throughout the span of the grant or contract provided by the
NSF.
• The project must be operating to the proposed plan. If not, then
modifications may be necessary.
• Also provides for documentation of the activities carried out.
Research based plans done in alignment with the program goals
are paramount.

16. Formative Evaluation: Progress Evaluation
Purpose: to assess progress in meeting the project’s ultimate goals.
• Internal and External (monitoring) is done to check for proper
procedures during the project.
• Progress Evaluation involves the collection of data to determine
if the established benchmarks have been met and to identify
unexpected deviations. The project is adjusted accordingly based
on this information.
• It is important early problems are corrected to promote individual
effectiveness.

17. Summative Evaluation

18. Purpose: to assess a mature project's success
in reaching its stated goals
• takes place after the project has been established and the time frame for
change has occurred
• collects information about outcomes and related processes, strategies, and
activities that have led to them
• needed for the decision making about the future of the intervention
• disseminate the intervention to other sites or agencies
• continue funding
• increase funding
• continue on probationary status
• modify and try again
• discontinue

19. Unanticipated Outcomes
• findings that emerge during data collection or data analyses that were never
anticipated when the study was first designed

20. Evaluation compliments but is
different from other kinds of data
collection activities

21. FastLane allows for basic data to be
collected across all programs in a
consistent and systematic fashion.

22. Types of Data Collection
Activities
Project Descriptors and Statistics

23. Formative Evaluation & Summative Evaluation
• gather information to answer
• descriptive information
• in-depth data collection activities
• support decision making
• range in cost (depending on questions asked and project complexity)

24. Performance Factors/Indicators
• performance indicator system: collection of statistics that can be used to monitor
the ongoing status of a program against a set of targets and metrics
• play critical role in the GPRA and PART activities
• provide information that can be measured against a set of goals and objectives
• examine accomplishments of the projects in a program
• unlikely to provide an explanation of why a project has succeeded or failed
• focus on tangible results used to focus policymakers, educators, and the public on
1. key aspects of how an educational program is operating
2. whether progress is being made
3. where there are problems

25. Basic Research
• descriptive information
• provide targeted in-depth exploration of issues
• broaden understanding
• explore conceptual models and alternative explanations for
• observed relationships

26. Using Evaluation
Information

27. Why assess worth or merit?
• to make changes or improvements in the status quo
• "evaluations should be judged by their utility and actual use" Patton (2008)
• create a context in which evaluation findings are actually used for decision
making and improvement
• provide an ongoing source of information that can aid decision making at
various steps along the way

28. Discussion Question
What types of evaluations have you used or plan to use in your
classroom? How have you used the results or how can the
results of these evaluations be used? Provide an example of
each type of evaluation, formative and summative.

29. References
Frechtling, Joy A., and Melvin M. Mark. The 2010 User-friendly Handbook of Project
Evaluation.Arlington,VA: National Science Foundation, Directorate for Education and Human
Resources, Division of Research and Learning in Formal and Informal Settings, 2010.Web.