Building agency capacity
Upcoming SlideShare
Loading in...5
×
 

Building agency capacity

on

  • 396 views

Example PPT for viewing

Example PPT for viewing

Statistics

Views

Total Views
396
Views on SlideShare
394
Embed Views
2

Actions

Likes
0
Downloads
7
Comments
0

1 Embed 2

http://tsi2011.pbworks.com 2

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Building agency capacity Building agency capacity Presentation Transcript

  • BUILDING AGENCY CAPACITY Creating an Evaluation Community and a Culture of DataStephanie R. Miller, The Pell Institute
  • Collaboration is KEY! • Allows for the diversity of opinions • Creates a sense of “ownership” • Engages important stakeholders • Creates inherently interactive and voluntary relationships • Takes time and can be difficult to achieve • Cannot be mandated, forced or faked • Establishes clear, articulated and common goals
  • Importance of Evaluation • Allows institutions to better select, oversee, improve and make sense of programs and policies • Definition: A collaborative and systematic approach to making an assessment of a program, policy or operation using data. The goal of an evaluation is to improve the effectiveness and/or efficiency of a program, policy or operation. (Mark, Henry, & Julnes, 2000) View slide
  • Building Evaluation Capacity • Evaluation becomes a core function of the organization • Involves collaboration of all stakeholders (at all levels of the organization) • Builds an understanding of the importance of evaluation and how it influences processes and outcomes • Evaluation should be a continuous, routine and sustained effort • Operates at multiple levels of the organization • Evaluation has multiple purposes and goals • Requires continuous adjustments and refinements View slide
  • Goals of Evaluation Capacity Building • The goal of ECB is to strengthen and sustain an organization’s capacity to: 1) design, implement, and manage effective evaluation projects 2) access, build, and use evaluative knowledge and skills 3) cultivate a spirit of continuous organizational learning, improvement, and accountability 4) create awareness and support for program evaluation and self-evaluation as a performance improvement strategy (King & Volkov, 2005; King, 2007)
  • Elements of Evaluation Capacity Building Evaluation Culture and Capacity COLLABORATION Leadership Commitment Analytic Data Expertise Quality COLLABORATION
  • Leadership Commitment • Commitment from the organization’s leadership body is key to sustain evaluation efforts • Strategies – Provide stakeholder team with continuous resources, support, incentives and opportunities
  • Collaboration ~ CooperativePartnerships • Shared resources and expertise • Diversity of opinions and experiences • Defined roles and responsibilities • Open, productive communication • Strategies – Join stakeholders in pursuit of common goals – Solicit support and involvement of stakeholders
  • Establishing An Evaluation Culture • Institutionalize evaluation – Incorporated into “practice” and work responsibilities – Formalized process to plan, execute and use information from evaluations – An established, natural part of the work environment • Strategies – Institute regular assessment – Commit to self-examination and self-improvement
  • Analytic Expertise • Engage with experts (internal and external) who are knowledgeable of: – Various research methodology (quantitative, qualitative and mixed- methods) – Relevant subject matter – Data analysis/interpretation and data systems • Strategies – Build staff expertise
  • Establishing Data Quality • Access to credible and reliable data is essential • Data should be consistent across all levels of the organization • Create a Culture of Data
  • Building a Data Culture • Leadership is required at all levels of the organization • Belief that good data are an integral part of evaluation and promoting positive policy and program outcomes • Willingness to invest time and resources in achieving useful information
  • Components of a Data Culture • Leadership and Collaboration: Data should be accessible to all stakeholders • Coordination and Articulation: There needs to be communication between the institution and smaller programs to insure standardized data collection • Dynamic Data System: Data tool should be able to systematize the data collection process, analyze data and report results (standard reporting) • Accuracy: Information must be correct and complete; data entry should be standardize to ensure reliability • Security: Protect student and staff confidentiality; data systems must be secure • Utility: Data have to provide the right information to answer questions • Timeliness: Data should be entered in a timely manner
  • Factors Affecting a Data Culture • Polices and Regulations • Standards and Guidelines • Professional Development/Training • Technology • Data Entry Environment • Timelines/Calendars • Assessment and Reporting • Integration into Practice/Culture ~ Community of Practice Source: www.data4ss.org
  • Using Data for Evaluation andProgram Improvement • Why is Data Important? – A blueprint for success, accountability and achievement – Review/analysis of complete, unbiased and measurable results • Typical Use of Data – Superficial use – Required reporting (e.g. annual performance reports, grants, program reviews, etc.) • Misuses of Data – Discarding of unfavorable data – Heavy reliance on one source of data
  • Building A Culture of Data • Data-Driven Decision Making – Systematic process of data use – Routine and collaborative use of data to inform organizational, programmatic and instructional improvement decisions – Builds knowledge, uses a variety of data to construct knowledge, to better understand what is actually going on
  • Using Data for Evaluation and ProgramImprovement Data Collection Collaboration & Professional Development Report & Communicate Data Data Analysis
  • Using Data • Collaboration and Professional Development – Ongoing efforts – Include key stakeholders – Formulate key questions – Establish desired outcomes – Consider cross-program evaluation • Data Collection – Utilize multiple data sources and databases – Use appropriate data – Organize data
  • Using Data • Data Analysis – Conduct meaningful analyses – Identify appropriate analysis techniques – Incorporate multiple measures • Reporting & Communicating Data – Ensure data results are seen by the appropriate stakeholders – Create “easy-to-interpret” charts and graphs – Establish periodic reporting
  • Creating Effective Data Tools • Data warehouse – Organized storage area for data elements • Data analysis tool – User-friendly analytic tool – Customized and pre-formatted reports – Expedited analysis • Decision support tool – Provides real-time recommendations or alerts – Automatic actions for stakeholders
  • What is YOUR organization’sexperience? • How has your organization incorporated evaluation into its culture? – Success/Pitfalls? – Areas for improvement? • Has your organization engaged in Evaluation Capacity Building? What are the challenges? • What is the culture of data at your organization? – How is data used? – Do you use a data system? What does it look like? • Strengths/Weaknesses? • Areas for improvement?
  • Resources Evaluation Capacity Building  Building Evaluation Capacity: Designing a Cross-Project Evaluation, Clewell & Campbell, 2008  Program Evaluation: an Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity, United States GAO, 2003  Evaluation capacity building in complex organizations, Taylor-Powell & Boyd, 2008 http://www.isce.vt.edu/files/Evaluation_Capacity_Building.pdf  Evaluation ToolKit: http://toolkit.pellinstitute.org/  Evaluation: An Integrated Framework for Understanding, Guiding and Improving Public and Nonprofit Policies and Programs, Mark, Henry & Julnes, 2000  Developing evaluation capacity through process use, King, 2007 • A framework for building evaluation capacity based on the experiences of three organizations, King & Volkov, 2005 Data  Data-Driven Decision Making: A Powerful Took for School Improvement https://www.erdc.k12.mn.us/promo/sage/images/Analytics_WhitePaper.pdf  Using Data to Improve Schools: What’s Working http://www.eric.ed.gov/PDFS/ED469227.pdf  Data for Student Success: http://www.data4ss.org/