This document discusses building evaluation capacity and establishing a culture of data within organizations. It emphasizes that collaboration between stakeholders is key to creating ownership over the evaluation process. Evaluation allows institutions to improve programs and policies by taking a systematic approach using data. Building evaluation capacity involves collaborating with stakeholders at all levels to incorporate evaluation as a core and continuous function. The goals of evaluation capacity building are to strengthen an organization's ability to design, implement, and use evaluations to foster organizational learning. Factors like leadership commitment, collaboration, developing analytic expertise, ensuring data quality, and building a data culture are important for establishing an effective evaluation system and using data for improvement.

1. BUILDING AGENCY CAPACITY
Creating an Evaluation Community and a
Culture of Data
Stephanie R. Miller, The Pell Institute

2. Collaboration is KEY!
• Allows for the diversity of opinions
• Creates a sense of “ownership”
• Engages important stakeholders
• Creates inherently interactive and
voluntary relationships
• Takes time and can be difficult to achieve
• Cannot be mandated, forced or faked
• Establishes clear, articulated and
common goals

3. Importance of Evaluation
• Allows institutions to better
select, oversee, improve and make sense of
programs and policies
• Definition: A collaborative and systematic
approach to making an assessment of a
program, policy or operation using data. The
goal of an evaluation is to improve the
effectiveness and/or efficiency of a
program, policy or operation. (Mark, Henry, &
Julnes, 2000)

4. Building Evaluation Capacity
• Evaluation becomes a core function of the organization
• Involves collaboration of all stakeholders (at all levels
of the organization)
• Builds an understanding of the importance of
evaluation and how it influences processes and
outcomes
• Evaluation should be a continuous, routine and
sustained effort
• Operates at multiple levels of the organization
• Evaluation has multiple purposes and goals
• Requires continuous adjustments and refinements

5. Goals of Evaluation Capacity Building
• The goal of ECB is to strengthen and
sustain an organization’s capacity to:
1) design, implement, and manage effective evaluation
projects
2) access, build, and use evaluative knowledge and
skills
3) cultivate a spirit of continuous organizational
learning, improvement, and accountability
4) create awareness and support for program evaluation
and self-evaluation as a performance improvement
strategy
(King & Volkov, 2005; King, 2007)

6. Elements of Evaluation Capacity Building
Evaluation
Culture
and
Capacity
COLLABORATION
Leadership
Commitment
Analytic Data
Expertise Quality
COLLABORATION

7. Leadership Commitment
• Commitment from the organization’s
leadership body is key to sustain
evaluation efforts
• Strategies
– Provide stakeholder team with continuous
resources, support, incentives and
opportunities

8. Collaboration ~ Cooperative
Partnerships
• Shared resources and expertise
• Diversity of opinions and experiences
• Defined roles and responsibilities
• Open, productive communication
• Strategies
– Join stakeholders in pursuit of common
goals
– Solicit support and involvement of
stakeholders

9. Establishing An Evaluation Culture
• Institutionalize evaluation
– Incorporated into “practice” and work responsibilities
– Formalized process to plan, execute and use
information from evaluations
– An established, natural part of the work environment
• Strategies
– Institute regular assessment
– Commit to self-examination and self-improvement

10. Analytic Expertise
• Engage with experts (internal and
external) who are knowledgeable of:
– Various research methodology
(quantitative, qualitative and mixed-
methods)
– Relevant subject matter
– Data analysis/interpretation and data
systems
• Strategies
– Build staff expertise

11. Establishing Data Quality
• Access to credible and reliable data is
essential
• Data should be consistent across all
levels of the organization
• Create a Culture of Data

12. Building a Data Culture
• Leadership is required at all levels of the
organization
• Belief that good data are an integral part
of evaluation and promoting positive
policy and program outcomes
• Willingness to invest time and resources
in achieving useful information

13. Components of a Data Culture
• Leadership and Collaboration: Data should be accessible to all
stakeholders
• Coordination and Articulation: There needs to be
communication between the institution and smaller programs to
insure standardized data collection
• Dynamic Data System: Data tool should be able to systematize
the data collection process, analyze data and report results
(standard reporting)
• Accuracy: Information must be correct and complete; data entry
should be standardize to ensure reliability
• Security: Protect student and staff confidentiality; data systems
must be secure
• Utility: Data have to provide the right information to answer
questions
• Timeliness: Data should be entered in a timely manner

14. Factors Affecting a Data Culture
• Polices and Regulations
• Standards and Guidelines
• Professional Development/Training
• Technology
• Data Entry Environment
• Timelines/Calendars
• Assessment and Reporting
• Integration into Practice/Culture ~ Community
of Practice
Source: www.data4ss.org

15. Using Data for Evaluation and
Program Improvement
• Why is Data Important?
– A blueprint for success, accountability and achievement
– Review/analysis of complete, unbiased and measurable
results
• Typical Use of Data
– Superficial use
– Required reporting (e.g. annual performance
reports, grants, program reviews, etc.)
• Misuses of Data
– Discarding of unfavorable data
– Heavy reliance on one source of data

16. Building A Culture of Data
• Data-Driven Decision Making
– Systematic process of data use
– Routine and collaborative use of data to
inform organizational, programmatic and
instructional improvement decisions
– Builds knowledge, uses a variety of data to
construct knowledge, to better understand
what is actually going on

17. Using Data for Evaluation and Program
Improvement
Data
Collection
Collaboration
&
Professional
Development
Report &
Communicate Data
Data Analysis

18. Using Data
• Collaboration and Professional Development
– Ongoing efforts
– Include key stakeholders
– Formulate key questions
– Establish desired outcomes
– Consider cross-program evaluation
• Data Collection
– Utilize multiple data sources and databases
– Use appropriate data
– Organize data

19. Using Data
• Data Analysis
– Conduct meaningful analyses
– Identify appropriate analysis techniques
– Incorporate multiple measures
• Reporting & Communicating Data
– Ensure data results are seen by the
appropriate stakeholders
– Create “easy-to-interpret” charts and graphs
– Establish periodic reporting

20. Creating Effective Data Tools
• Data warehouse
– Organized storage area for data elements
• Data analysis tool
– User-friendly analytic tool
– Customized and pre-formatted reports
– Expedited analysis
• Decision support tool
– Provides real-time recommendations or alerts
– Automatic actions for stakeholders

21. What is YOUR organization’s
experience?
• How has your organization incorporated evaluation into
its culture?
– Success/Pitfalls?
– Areas for improvement?
• Has your organization engaged in Evaluation Capacity
Building? What are the challenges?
• What is the culture of data at your organization?
– How is data used?
– Do you use a data system? What does it look like?
• Strengths/Weaknesses?
• Areas for improvement?

22. Resources
Evaluation Capacity Building
 Building Evaluation Capacity: Designing a Cross-Project Evaluation, Clewell &
Campbell, 2008
 Program Evaluation: an Evaluation Culture and Collaborative Partnerships Help
Build Agency Capacity, United States GAO, 2003
 Evaluation capacity building in complex organizations, Taylor-Powell & Boyd, 2008
http://www.isce.vt.edu/files/Evaluation_Capacity_Building.pdf
 Evaluation ToolKit: http://toolkit.pellinstitute.org/
 Evaluation: An Integrated Framework for Understanding, Guiding and Improving
Public and Nonprofit Policies and Programs, Mark, Henry & Julnes, 2000
 Developing evaluation capacity through process use, King, 2007
• A framework for building evaluation capacity based on the experiences of three
organizations, King & Volkov, 2005
Data
 Data-Driven Decision Making: A Powerful Took for School Improvement
https://www.erdc.k12.mn.us/promo/sage/images/Analytics_WhitePaper.pdf
 Using Data to Improve Schools: What’s Working
http://www.eric.ed.gov/PDFS/ED469227.pdf
 Data for Student Success: http://www.data4ss.org/