Building agency capacity


Published on

Example PPT for viewing

Published in: Technology, Business
  • Be the first to comment

Building agency capacity

  1. 1. BUILDING AGENCY CAPACITY Creating an Evaluation Community and a Culture of DataStephanie R. Miller, The Pell Institute
  2. 2. Collaboration is KEY! • Allows for the diversity of opinions • Creates a sense of “ownership” • Engages important stakeholders • Creates inherently interactive and voluntary relationships • Takes time and can be difficult to achieve • Cannot be mandated, forced or faked • Establishes clear, articulated and common goals
  3. 3. Importance of Evaluation • Allows institutions to better select, oversee, improve and make sense of programs and policies • Definition: A collaborative and systematic approach to making an assessment of a program, policy or operation using data. The goal of an evaluation is to improve the effectiveness and/or efficiency of a program, policy or operation. (Mark, Henry, & Julnes, 2000)
  4. 4. Building Evaluation Capacity • Evaluation becomes a core function of the organization • Involves collaboration of all stakeholders (at all levels of the organization) • Builds an understanding of the importance of evaluation and how it influences processes and outcomes • Evaluation should be a continuous, routine and sustained effort • Operates at multiple levels of the organization • Evaluation has multiple purposes and goals • Requires continuous adjustments and refinements
  5. 5. Goals of Evaluation Capacity Building • The goal of ECB is to strengthen and sustain an organization’s capacity to: 1) design, implement, and manage effective evaluation projects 2) access, build, and use evaluative knowledge and skills 3) cultivate a spirit of continuous organizational learning, improvement, and accountability 4) create awareness and support for program evaluation and self-evaluation as a performance improvement strategy (King & Volkov, 2005; King, 2007)
  6. 6. Elements of Evaluation Capacity Building Evaluation Culture and Capacity COLLABORATION Leadership Commitment Analytic Data Expertise Quality COLLABORATION
  7. 7. Leadership Commitment • Commitment from the organization’s leadership body is key to sustain evaluation efforts • Strategies – Provide stakeholder team with continuous resources, support, incentives and opportunities
  8. 8. Collaboration ~ CooperativePartnerships • Shared resources and expertise • Diversity of opinions and experiences • Defined roles and responsibilities • Open, productive communication • Strategies – Join stakeholders in pursuit of common goals – Solicit support and involvement of stakeholders
  9. 9. Establishing An Evaluation Culture • Institutionalize evaluation – Incorporated into “practice” and work responsibilities – Formalized process to plan, execute and use information from evaluations – An established, natural part of the work environment • Strategies – Institute regular assessment – Commit to self-examination and self-improvement
  10. 10. Analytic Expertise • Engage with experts (internal and external) who are knowledgeable of: – Various research methodology (quantitative, qualitative and mixed- methods) – Relevant subject matter – Data analysis/interpretation and data systems • Strategies – Build staff expertise
  11. 11. Establishing Data Quality • Access to credible and reliable data is essential • Data should be consistent across all levels of the organization • Create a Culture of Data
  12. 12. Building a Data Culture • Leadership is required at all levels of the organization • Belief that good data are an integral part of evaluation and promoting positive policy and program outcomes • Willingness to invest time and resources in achieving useful information
  13. 13. Components of a Data Culture • Leadership and Collaboration: Data should be accessible to all stakeholders • Coordination and Articulation: There needs to be communication between the institution and smaller programs to insure standardized data collection • Dynamic Data System: Data tool should be able to systematize the data collection process, analyze data and report results (standard reporting) • Accuracy: Information must be correct and complete; data entry should be standardize to ensure reliability • Security: Protect student and staff confidentiality; data systems must be secure • Utility: Data have to provide the right information to answer questions • Timeliness: Data should be entered in a timely manner
  14. 14. Factors Affecting a Data Culture • Polices and Regulations • Standards and Guidelines • Professional Development/Training • Technology • Data Entry Environment • Timelines/Calendars • Assessment and Reporting • Integration into Practice/Culture ~ Community of Practice Source:
  15. 15. Using Data for Evaluation andProgram Improvement • Why is Data Important? – A blueprint for success, accountability and achievement – Review/analysis of complete, unbiased and measurable results • Typical Use of Data – Superficial use – Required reporting (e.g. annual performance reports, grants, program reviews, etc.) • Misuses of Data – Discarding of unfavorable data – Heavy reliance on one source of data
  16. 16. Building A Culture of Data • Data-Driven Decision Making – Systematic process of data use – Routine and collaborative use of data to inform organizational, programmatic and instructional improvement decisions – Builds knowledge, uses a variety of data to construct knowledge, to better understand what is actually going on
  17. 17. Using Data for Evaluation and ProgramImprovement Data Collection Collaboration & Professional Development Report & Communicate Data Data Analysis
  18. 18. Using Data • Collaboration and Professional Development – Ongoing efforts – Include key stakeholders – Formulate key questions – Establish desired outcomes – Consider cross-program evaluation • Data Collection – Utilize multiple data sources and databases – Use appropriate data – Organize data
  19. 19. Using Data • Data Analysis – Conduct meaningful analyses – Identify appropriate analysis techniques – Incorporate multiple measures • Reporting & Communicating Data – Ensure data results are seen by the appropriate stakeholders – Create “easy-to-interpret” charts and graphs – Establish periodic reporting
  20. 20. Creating Effective Data Tools • Data warehouse – Organized storage area for data elements • Data analysis tool – User-friendly analytic tool – Customized and pre-formatted reports – Expedited analysis • Decision support tool – Provides real-time recommendations or alerts – Automatic actions for stakeholders
  21. 21. What is YOUR organization’sexperience? • How has your organization incorporated evaluation into its culture? – Success/Pitfalls? – Areas for improvement? • Has your organization engaged in Evaluation Capacity Building? What are the challenges? • What is the culture of data at your organization? – How is data used? – Do you use a data system? What does it look like? • Strengths/Weaknesses? • Areas for improvement?
  22. 22. Resources Evaluation Capacity Building  Building Evaluation Capacity: Designing a Cross-Project Evaluation, Clewell & Campbell, 2008  Program Evaluation: an Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity, United States GAO, 2003  Evaluation capacity building in complex organizations, Taylor-Powell & Boyd, 2008  Evaluation ToolKit:  Evaluation: An Integrated Framework for Understanding, Guiding and Improving Public and Nonprofit Policies and Programs, Mark, Henry & Julnes, 2000  Developing evaluation capacity through process use, King, 2007 • A framework for building evaluation capacity based on the experiences of three organizations, King & Volkov, 2005 Data  Data-Driven Decision Making: A Powerful Took for School Improvement  Using Data to Improve Schools: What’s Working  Data for Student Success: