The document summarizes findings from three recent GAO reports on the use of evidence in federal decision making. It discusses the results of GAO's 2017 survey of federal managers which found no significant increase in the use of performance measures or information in decision making. It also summarizes the GAO's assessment of GPRAMA implementation and key findings about quarterly performance reviews and program evaluation from the manager survey. The document concludes with a recommendation that OMB direct each agency to prepare an annual evaluation agenda.

1. PRESENTATION FOR
WASHINGTON EVALUATORS
N O VE M B E R 1, 2 0 17
G A O ST A F F :
Are Federal Managers Using
Evidence in Decision Making?
• P e t e r B e c k , S e n i o r A n a l y s t
• V a l e r i e C a r a c e l l i , S e n i o r S o c i a l
S c i e n c e A n a l y s t
• S h a n n o n F i n n e g a n , A s s i s t a n t D i r e c t o r
• B e n j a m i n L i c h t , A s s i s t a n t D i r e c t o r
• A d a m M i l e s , S e n i o r A n a l y s t
• S t e p h a n i e S h i p m a n , A s s i s t a n t D i r e c t o r
• S h a n e S p e n c e r , A n a l y s t
• B r i a n W a n l a s s , A n a l y s t

2. Three Recent GAO Products
on Federal Use of Evidence
• GAO-17-776SP: Results from a 2017 Survey of Federal Managers
• GAO-17-775: GPRA Modernization Act (GPRAMA) Implementation
• GAO-17-743: Survey Results Related to Program Evaluation
2

3. Results from
GAO’s 2017
Survey of
Federal
Managers
3

4. About the 2017 Survey
 Web-based survey of 4,395 federal managers conducted
from November 2016 to March 2017.
 Comparable to five previous surveys: 1997, 2000, 2003,
2007, and 2013.
 Asked for managers’ perspectives on a range of
management and performance topics, including:
 the availability and use of performance information in their agencies
 program evaluation efforts
 cross-agency and agency priority goals, and related quarterly
performance reviews
 Government-wide response rate about 67%.
4

5. Assessment of
GPRAMA
Implementation
5

6. GPRAMA Implementation
 Assessed implementation of the Act as it relates to
resolving four long-standing federal governance
challenges:
 Addressing cross-cutting issues
 Ensuring performance information is useful and used
 Aligning daily operations with results
 Building a more transparent and open government
6

7. Federal Managers’ Use of Performance
Information in Decision Making
 Survey results related to
 Federal managers use of performance information
 Practices that promote the use of performance information
 Quarterly performance reviews required by GPRAMA
Note: Unless described otherwise, survey results represent
responses to a “great” or “very great” extent on each survey item.
7

8. No Statistically Significant Increase in Presence of
Performance Measures from 2013 to 2017
8

9. No Statistically Significant Increase in Presence of
Performance Measures from 2013 to 2017
9

10. No Improvement on GAO’s Use of Performance
Information Index at the Government-wide Level
10

11. Use of Performance Information for
Selected Management Activities
11

12. Use of Performance Information to
Reward and Recognize Employees
12

13. Management Practices That Promote
the Use of Performance Information
In 2005, GAO identified 5 practices to enhance the use of performance
information in decision making and improve results:
• Demonstrating management commitment
• Communicating performance information frequently and efficiently
• Improving the usefulness of performance information
• Developing the capacity to use performance information
• Aligning agency-wide goals, objectives, and measures
13

14. Demonstrating Management Commitment
14

15. Demonstrating Management Commitment
Note: Senior Executive Service (SES) only
15

16. Communicating Performance Information
No statistically significant difference for two survey items on
federal managers communicating performance information
relative to 2013 or since those items were introduced in 2007.
In 2017, we estimate that:
• 44 percent of federal managers agreed that managers at
their level effectively communicate performance information
on a routine basis
• 34 percent agreed that managers at their level use
performance information to share effective program
approaches with others
16

17. Improving Usefulness
17

18. Developing Capacity
Agency training provided in the last three years
18

19. Aligning Goals and Measures
19
• In 2017, 50 percent of federal managers agreed that managers at their level took
steps to align program performance measures with agency-wide goals and
objectives.
• No statistically significant change from question introduction in 2007 to 2017.

20. • GPRAMA requires agencies to review
progress toward Agency Priority Goals at
least once a quarter.
• These data-driven reviews are also known as
quarterly performance reviews (QPRs).
• GAO has previously identified practices to
promote successful data-driven reviews.
Data-Driven Performance Reviews
20

21. Greater Involvement in QPRs Results in Greater Reports of
Them Driving Performance in line with Leading Practices
21

22. Greater Involvement in QPRs Results in Greater Reports of
Them Driving Performance in line with Leading Practices
22

23. The Director of OMB should work
with the Performance Improvement
Council to identify and share among
agencies practices for expanding
the use of data-driven reviews.
Related Recommendation
23

24. Program
Evaluation
Survey Results
24

25. Background
25
• GPRAMA changes aim to ensure agencies use performance
information to improve performance
• GAO and OMB encouraged agencies to build their capacity to
conduct and use evaluation
• Since GAO’s last survey of federal managers in 2013:
• How has agency access to and use of evaluation
changed?
• What factors facilitate or hinder their use of evaluation in
decision making?

26. Most Federal Managers Lack Current Program
Evaluations, but Find They Help Improve Programs
26
• 40% reported an evaluation had been conducted in last 5
years of their program, operation or project; 39% don’t know
(unchanged from 2013)
• About half the managers with evaluations report they
contributed greatly or very greatly to: assessing program
effectiveness or value, implementing changes to improve
program management or performance, and sharing what
works with others (unchanged)

27. Lack of Resources Hinders Evaluation Use
27
• Among evaluators with evaluations, lack of resources to
implement the study’s findings was the most frequently report
hindrance to use (29% rated great or very great extent, same
as in 2013)
• Managers reported modest concerns related to program
context and agency capacity or support for evaluation more
often than potential problems with study quality
• Few managers reported that lack of congressional support
was a great barrier to use (16%), but twice as many reported
they had not basis to judge (35%)

28. Leadership Support for Evaluation and Study’s
Relevance Facilitate Use
28
Facilitating use to a great or very great extent among managers
with evaluation:
• Top executive commitment or support for using evaluations to
make decisions (38%)
• Importance of an evaluation’s issues to agency decision
makers (36%)
• Staff involvement in planning or conducting evaluation studies
(35%)
However, 31% reported they had not basis to judge whether
congressional requests or mandates facilitated use

29. GAO Recommendation
29
Because
• Agencies have not expanded their evaluation capacity since
2013, but leadership support and staff involvement help
ensure study relevance and use, and
• GAO, OMB, AEA, and the Commission on Evidence-Based
Policymaking recommended developing an evaluation
agenda to target studies to the most important management
and policy issues,
OMB should direct each agency to prepare an annual agency-
wide evaluation agenda that describes key evaluation questions
and stakeholders consulted