Pilots vs. PARs: Some Preliminary Results Jerry Ellig & Henry Wray
Team of 3 experts with experience in govt. and/or performance management evaluates reports
12 criteria based on GPRA requirements
Evaluate reports from 24 CFO Act agencies
Each evaluation reviewed by a member of advisory panel
Entire report reviewed by entire advisory panel
How we score the reports 1-5 rating scale
4 criteria in each category
Criteria tightened each year to reflect previous year’s best practices
Total score can range from 12 to 60
Fails to meet expectations 1 Partially complete 2 Satisfactory 3 Shows innovation and creativity 4 Sets a standard for best practice 5
Pilots vs. PARs, fiscal 2007 24 % Difference 7.33 Difference 30.00 Pilot average score 37.33 PAR average score
Pilot vs. PAR scores, fiscal 2006-07 -12 -1 % Change -4.11 -0.4 Change 30.00 37.33 2007 34.11 37.73 2006 Pilot PAR
Whatever its impact on other users, the pilot was a step backward for public disclosure :
Access to performance information was delayed for several months.
Once released, the information was harder to find and use.
To the extent found, the information gave the public little if anything that was not available last November.
The one potential benefit for public disclosure: Some highlights documents (mainly those done voluntarily by non-pilot agencies) demonstrated the value they can add for lay readers, if done well .
The pilot provided limited performance information that the general public could reasonably access and use:
We found and used all three relevant documents (highlights, performance report, and financial report) for only 2 of the 9 pilot agencies.
We used the highlights for all 9 agencies (although 2 were not timely posted on line). However, most failed to provide an insightful overview of agency performance and lacked user-friendly links to relevant information in other source documents.
We used only 3 of 9 performance reports. We could not find the reports for 3 agencies by the due date. The remaining 3 reports were embedded in budget justifications and thus beyond the practical reach of the general public.
If the pilot continues, the highlights documents and their links to other source documents must improve substantially in order for the public to get any value from performance reporting.
Highlights documents can significantly enhance performance reporting for the general public. Therefore, they should be improved and expanded whether or not the pilot continues.
Our report will offer a number of specific suggestions for improving the highlights documents.