Building Capacity to
Measure, Analyze and
Evaluate Government
Performance
Trachtenberg School Series on
Improving Governme...
My Key Points
Acceptance of a false dichotomy contrasting
program evaluation with performance
measurement has had deleteri...
What is Program Evaluation?
Program evaluation refers to both a mind set
and a set of skills
Evaluation involves:
◦ using ...
Framing Evaluation Questions
Design Evaluation

HOW

Resources/
Inputs

Activities

WHY

Outputs

Customers

Short term
ou...
A False Dichotomy
Program evaluation has been viewed an a
separate function and activity from
performance measurement
The ...
A Pernicious Dichotomy
Viewing performance measurement, or
monitoring, as something separate from
evaluation has resulted ...
Program Evaluation Skills Infuse Performance
Measurement and Enhance Organizational
Learning
Program Evaluation
•Design Pe...
Evaluation Should Also Infuse
Performance Management
Performance management: The use of
performance measurement informatio...
Evaluation for What?
Organizational learning
and
Accountability for
◦ ongoing organizational learning,
◦ program and organ...
Program
Program
Evaluation Across
Evaluation
Across the
the Programmatic
Programmatic
Life Cycle
Life Cycle

Improved
Perf...
Desired Outcome: Strategic Design to
Promote Evaluation
Evaluation should be treated as a mission support function, and
de...
Desired Outcome: Strategic Use of
Evaluation to Promote Organizational
Learning
Policy analysts and evaluation staff have ...
Desired Outcome: Evaluation
Competencies Held by Managers
Analyze the theory of change underlying programs
and policies
Fr...
Next Steps?
Think strategically and holistically about
evaluation to strategically:
connect and plan out the evaluation ef...
AEA: An Evaluation Roadmap for a
More Effective Government
Scope and Coverage
Management
Quality and Independence
Transpar...
Thank you!!
I can be reached at newcomer@gwu.edu

16
Upcoming SlideShare
Loading in …5
×

Building Capacity to Measure, Analyze and Evaluate Government Performance

518 views
341 views

Published on

Washington Evaluators Brown Bag
by Kathy Newcomer
June 29, 2012

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
518
On SlideShare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Building Capacity to Measure, Analyze and Evaluate Government Performance

  1. 1. Building Capacity to Measure, Analyze and Evaluate Government Performance Trachtenberg School Series on Improving Government Performance A Showcase for Sharing Promising Practices for Improving Government Performance Kathryn Newcomer, PhD June 29, 2012 1
  2. 2. My Key Points Acceptance of a false dichotomy contrasting program evaluation with performance measurement has had deleterious effects Identification of needed evaluation competencies should begin with a consideration of the desired outcomes, i.e., enhanced organizational learning and government performance The AEA “Evaluation Roadmap for More Effective Government” (2010) provides useful guidance on evaluation capacity building 2
  3. 3. What is Program Evaluation? Program evaluation refers to both a mind set and a set of skills Evaluation involves: ◦ using systematic social science tools to assess the strengths and weaknesses of programs, policies and organizations to improve their effectiveness, ◦ employing systematic data collection, analysis and judgment to address questions about programs and policies, and ◦ both ongoing measurement and one-shot studies across the life cycle of programs. Framing the most appropriate questions to address in any evaluation work is the key 3
  4. 4. Framing Evaluation Questions Design Evaluation HOW Resources/ Inputs Activities WHY Outputs Customers Short term outcome Intermediate outcome Longer term outcome (STRATEGIC AIM) 4 Process/Implementation Evaluation Outcome Evaluation Impact Evaluation Adapted from Evaluation Dialogue Between OMB and Federal Evaluation Leaders: Digging a Bit Deeper into Evaluation Science, July 2006. 4
  5. 5. A False Dichotomy Program evaluation has been viewed an a separate function and activity from performance measurement The separation has been perpetuated by both academics and practitioners who may view the functions as serving different clients and/or potential users, or whose professional training or experiences highlight differences rather than similarities 5
  6. 6. A Pernicious Dichotomy Viewing performance measurement, or monitoring, as something separate from evaluation has resulted in: ◦ reduced benefits from application of professional evaluation skills and standards in performance measurement practice, ◦ a proliferation of different offices with different and even non-communicative staffs in public agencies, ◦ reduced capacity for a strategic and holistic evaluation approach within organizations, and ◦ obstacles to the generation of organizational learning and improved programmatic performance. 6
  7. 7. Program Evaluation Skills Infuse Performance Measurement and Enhance Organizational Learning Program Evaluation •Design Performance Measures and Systems Ongoing Measurement and Reporting Ongoing Measurement and Reporting Program Evaluation Program Evaluation •Study / Share promising practices •Use data for outcome and/or impact studies •Study trends and address How / Why questions,, i.e., data analytics Ongoing Measurement and Reporting Ongoing Measurement and Reporting Program Evaluation •Compare / Contrast delivery variations. •Improve Measures 7
  8. 8. Evaluation Should Also Infuse Performance Management Performance management: The use of performance measurement information to: ◦ help set agreed-upon performance goals, ◦ allocate and prioritize resources, ◦ inform managers to either confirm or change current policy or program directions to meet those goals, and ◦ report on the success in meeting those goals. SOURCE: Serving the American Public: Best Practices in Performance Measurement, NPR, June 1997. 8
  9. 9. Evaluation for What? Organizational learning and Accountability for ◦ ongoing organizational learning, ◦ program and organizational performance improvement, and ◦ achievement of espoused policy and program goals. 9
  10. 10. Program Program Evaluation Across Evaluation Across the the Programmatic Programmatic Life Cycle Life Cycle Improved Performance ece Organizational Learning 10
  11. 11. Desired Outcome: Strategic Design to Promote Evaluation Evaluation should be treated as a mission support function, and designed in a strategic and holistic fashion Performance measurement processes should be designed and employed as one component of a strategic organization-wide evaluation approach, and better informed by program evaluation skills and standards, e.g., designs informed by theory of change, and measures informed by validity and reliability considerations The entire range of evaluation approaches, from qualitative and quantitative tools employed to assess to service quality, to cost effectiveness analyses to measure the results of regulations, should be embraced within the evaluation purview Evaluation should be situated within organizations to facilitate organizational learning, and staffed with experts in the specific field, as well as by representatives of multiple disciplines 11
  12. 12. Desired Outcome: Strategic Use of Evaluation to Promote Organizational Learning Policy analysts and evaluation staff have the competencies needed to support a comprehensive evaluation function, however--Competencies in evaluative thinking and analysis are needed by managers and executives, as well as budget and financial staff Strengthening needed evaluation competencies across the organization should be viewed as within the mandate for an evaluation office, e.g, the EPA model 12
  13. 13. Desired Outcome: Evaluation Competencies Held by Managers Analyze the theory of change underlying programs and policies Frame useful questions about program implementation and results Involve stakeholders effectively to enhance evaluation efforts Assess the competence, relevance and sufficiency of evidence (data) obtained through performance measurement processes and evaluation studies Analyze trends/and interpret performance data Interpret cost effectiveness and cost benefit models 13
  14. 14. Next Steps? Think strategically and holistically about evaluation to strategically: connect and plan out the evaluation efforts undertaken in an agency to benefit from the synergies among them, enhance organizational learning from all evaluation efforts, including performance measurement, co-locate evaluation & measurement efforts and staff in one office to better inform performance measurement and management processes, train managers and executives to enhance their evaluation competencies, and envision evaluation and learning as the responsiblity of all managers. 14
  15. 15. AEA: An Evaluation Roadmap for a More Effective Government Scope and Coverage Management Quality and Independence Transparency 15
  16. 16. Thank you!! I can be reached at newcomer@gwu.edu 16

×