Auditor Roles in Government Performance Management


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Auditor Roles in Government Performance Management

  1. 1. National Performance Management Advisory Commission Issue Paper # 8 Topic: Auditor Roles in Government Performance Management Author: Paul D. Epstein Principal, Epstein & Fass Associates Manager, Auditor Roles in Government Performance Measurement Project 1 The Auditor Roles Project Team may update this paper as our various project principals have an opportunity to contribute, and as our new Advocacy Committee of expert state and local auditors begins to work. About the Issue There are many valuable roles and practices auditors can play, and have played, in advancing performance management in state and local government. Our research across North America has found numerous local, state, and provincial auditors playing one or more of these five major roles: ROLE 1. Audit Performance or Performance Management Systems: For example, performance audits give management valuable recommendations for improving program performance, one of the primary goals of performance management. And by auditing a government entity’s performance management system, auditors help management and elected officials understand how to improve the system and use it better to improve decisions, performance, and accountability. ROLE 2. Assess Performance Information: By auditing reliability and relevance of performance information and how it is communicated, auditors help build confidence in the information so managers, elected officials, and citizens are more likely to make good use of it. ROLE 3: Define or Measure Performance (or assist management in doing so): Auditors often help management find better measures of performance. And in a number of governments, auditors survey citizens, providing a valuable cross-cutting function by asking about services provided by multiple departments, and strengthening the perception of survey objectivity. ROLE 4: Encourage or Assist Management: Auditors have “encouraged” management through advocacy or other means (e.g., written guidance) to improve performance management. They also have served as key arm’s-length advisors or partners with management in developing or improving performance management systems that are stronger because of what the auditor brings to the effort. ROLE 5: Assist Elected Officials or Citizens: Auditors have related to elected officials and citizens concerning performance management in numerous ways. For example, they have often helped elected officials understand the importance of improving performance management, so legislators will provide important support for system improvements. These five roles and fourteen related practices are well documented in a 2004 book based on research of auditor practices. 2 The book and updated materials, including new examples from state and local auditors and articles describing how auditor practices add value to government performance, are available on the “Auditor Roles in Government Performance Measurement” website (, which has supporting material far beyond which this paper can provide. A case history of auditor practices in one city government is provided at the end of this paper. 1
  2. 2. Why should auditors be involved in improving performance management? Because they add value. Most of this paper describes how they add value, and the numerous examples documented on demonstrate that they indeed add value to performance management in many ways. A section near the end of the paper also provides several reasons why auditors should be involved, even though managing performance is a management function. In general, consider that in many governments, audit offices have one of the strongest pools of quantitative skills, performance analysis skills, and knowledge of good system controls and practices. And they can use the knowledge and skills independently, without being distracted by the demands of day-to-day management. Their independence also adds to the credibility of their recommended improvements, which can build broad support for performance management improvement among elected officials and citizens as well as managers. So, if an audit staff in a jurisdiction has such knowledge and skills to offer, it would be foolish not to involve them in how the government improves performance management. The “Best” Roles for Auditors to Play at Any Time is Situational Is any one auditor role or set of practices “better” or “more appropriate” than the others? From our research of audit organizations across North America, the answer is clearly, “No.” Or, more explicitly, the best role for a government auditor to play at any given time depends on the maturity of the government organization’s performance management system, and the willingness and ability of others in the government to improve that system. One of the central findings of our research, especially of audit offices that have at least ten years’ experience playing these roles, is that “auditors add more value by changing practices as entities’ systems evolve.” 3 This is explained, next, by describing each of the five roles and related practices, and explaining the circumstances in which each can be especially valuable in advancing performance management. To give those explanations the context of reality, this paper ends with a brief case history of the City of Austin, Texas, to show how auditors’ roles and practices in that city government have changed over the years to keep adding value in ways that are most effective, based on how performance management has evolved over time in Austin. ROLE 1. Audit Performance or Performance Management Systems Practice 1a. Audit performance: Performance auditing is a well-established practice in government, with at least a 40-year history of auditors taking an independent look at government programs and recommending improvements. It continues to be valuable at all levels of government today. Performance auditing has an interesting, mutually-enhancing relationship with performance management, in which they can add value to each other. For example: • When performance management of an agency or program is weak, a performance audit can identify those weaknesses and recommend, for example, improvements needed in how performance is measured, or in how performance data are used. Thus, performance auditing can help improve the value obtained from performance management systems. • A strong performance management system that produces relevant and reliable performance data can, in turn, make performance auditing more valuable. This is because auditors will not have to spend a lot of costly time and effort on primary data collection just to establish basic performance levels, as agencies will have data for auditors to use. This value can be captured in two ways 4 : Auditors can be more efficient and timely with performance audits. So, for the same overall auditing cost, they can produce more performance audits and complete them sooner, thus providing more recommendations agencies can use to improve performance each year. With basic agency performance levels reliably established, auditors can use the opportunity to conduct more thorough performance audits, such as deeper evaluations of impact on the 2
  3. 3. population served. These “deeper audits” can provide management or elected officials with valuable information for improving policies, resource allocation, or program designs. Practice 1b. Audit performance management systems: In addition to auditing a program or agency’s performance, auditors can assess the systems and processes agencies or entire governments use to manage performance. This can be especially valuable in the following circumstances: 5 • If an agency or entire government has no real performance management system, or a weak one, the auditor can establish the need for implementing or improving such a system, and articulate the benefits of doing so. This can help elected officials understand the need to invest in PM systems, and help management find ways to use that investment to best effect. • If a government is in the process of implementing a new PM system, or making major changes to an existing system, the auditor can play a valuable role reviewing agencies’ implementation progress, and recommending improvements to help agencies get up to speed faster with the new systems. Thus, new or improved PM systems will “mature” faster and will be more useful, at an earlier date, to both management and legislative decision makers. ROLE 2. Assess Performance Information Like Role 1, Role 2 is a traditional auditor role. Role 2 is “traditional” in that it involves auditors assessing information. This auditor role increases the usefulness of information in the performance management system, and helps build the confidence of managers, elected officials, and citizens in the information, so they are more likely to use it to improve performance and accountability. A closer look at the three “practices” of Role 2 suggests when each can be especially valuable. Practice 2a. Test relevance and reliability: As with auditing PM systems, testing relevance or reliability (or both) of performance information can be especially useful in the early stages of implementing a new or improved PM system. At such a time, this can best be done by auditors on a broad-scale, at a relatively fast pace (e.g., assessing selected performance measures from all agencies over one to three years), to build management and legislative confidence in performance data as a new or improved PM system is implemented. Once all or most agencies have made their measures relevant and established the controls to keep their data reliable, then the audit office can perform this practice less often. Instead of testing a large number of measures each year, auditors might “spot check” agency data from time to time, or simply make it a regular practice to test measures and data whenever they do program performance audits. From our entire framework, we have documented the most auditor examples for this practice. 6 In some cases, as in the City of Austin, auditors have trained managers in testing and improving data reliability. As management takes responsibility for data reliability, auditors can perform this practice less often. 7 Practice 2b. Assure performance reports: Beyond testing data reliability, auditors can play a valuable role making public performance reports more useful. In North America, Canadian provincial auditors have taken the lead on this practice. 8 But state and local auditors in the U.S. can also use criteria such as those suggested by the GASB to review reported performance information and recommend how it can be more effectively communicated and made more valuable to intended users. Practice 2c: Support external review (of performance information or reporting): Auditors have been prominent among the reviewers the AGA has used to evaluate state and local public performance reports. When state or local government organizations want to submit their performance reports for such a review, or have an external review of their PM system conducted, auditors can help their government entity prepare for such a review so they will get more from it. 3
  4. 4. Non-traditional, but Still Valuable, Auditor Roles in Performance Management We consider Roles 3, 4, and 5 to be “non-traditional” for auditors because in performing these roles, auditors step out of “traditional” audit processes. However, for several practices related to these roles, some auditors have followed auditing standards and considered initiatives to be audit projects. Alternatively, auditors can perform these practices them as “non-audit services.” Important “auditor independence” issues arise with non-audit services. The 2007 edition of Government Audit Standards (the “Yellow Book”) provides specific guidance to auditors on how to maintain independence while providing non-audit services. 9 The Yellow Book specifies some services (e.g., “third party surveys”) where no “safeguards” are needed to preserve independence. For other non-audit services, such as the auditor serving in an advisory capacity to management, Yellow Book safeguards represent good practice for any collaborative relationship. For example, one such safeguard involves spelling out, as in a memorandum of understanding, what the auditor will do and not do, and what management will do, making sure that the auditor does not perform any management functions. ROLE 3: Define or Measure Performance (or assist management in doing so) Practice 3a. Help choose measures or targets: The key word here is “help.” It is inappropriate for auditors to decide what performance measures management should select, or what performance expectations or targets should be set. But often, based on findings in audits or other studies, auditors have found either a lack of performance measurement, performance measures that are not sufficiently relevant, or a lack of a sound basis for setting targets. 10 Sometimes auditors have seen opportunities for more useful comparisons if measures are changed or added, and have recommended that measures used by other jurisdictions be adopted to enable external benchmarking, such as measures in ICMA’s comparative performance measurement program. In all cases, the auditor’s role is to provide recommendations or advice, and for management and elected officials to decide what the actual measures and targets should be. Some situations where this practice can be especially helpful are: • When either elected legislators or executive management feel that, for a particular program, service, or agency, current measures or targets do not give them a sufficient basis for decision making or assessing accountability. • When a program is new—particularly a relatively new kind of program without a long history or “industry standard” method of measurement (e.g., transit oriented development programs)—the auditor can help the agency running the program find a useful, practical way to measure and target the program faster than the agency might on its own; • When an auditor finds that an agency is not taking advantage of available methods (e.g., practices proven elsewhere) or available bases for comparison that could provide valuable information for improving performance and accountability; • When an auditor finds that the customers of a service, or members of the public with a particular interest in a program, do not identify with the measures used for the quality, timeliness, effectiveness, or results of that service or program. Practice 3b. Collect data: All of the cases of this practice we have documented involve auditors conducting or commissioning surveys or focus groups that cut across department lines, such as surveys of satisfaction and perceptions that ask about multiple services as well as cross-cutting issues such as “livability” or “business climate.” 11 Government Auditing Standards allow auditors to perform such surveys “as an independent third party” without impairing auditor independence. 12 Of course, there are many jurisdictions where management has conducted or commissioned surveys or focus groups. But auditors obtaining data from citizens, interest groups, or service customers is not intrusive on management as other kinds of data collection would be, such as collecting data that normally comes 4
  5. 5. from service employees recording information on daily operations. Situations in which it can be useful for auditors to use this practice to improve performance management include: • Where elected officials or management request the auditor to collect this kind of data. • Where management was not collecting this type of data, and the auditor determined that such information would be useful to elected officials and citizens for decision making or accountability. • Where there is a sense of a “disconnect” between how one or more agencies are reporting on their performance and how citizens or service customers perceive that performance; • Where there are signs that the public may not completely trust management to ask unbiased questions about service performance or to provide unbiased reports of results. ROLE 4: Encourage or Assist Management Practice 4a. Encourage management … to develop and implement performance management systems. (Examples at: Practice 4b. Assist management … in designing, improving, or maintaining performance management systems, or build the capacity of management to do so. (Examples at: practices/encourage-or-assist-management/practice-4b-description.html). The key words in these practices are “encourage” and “assist.” Developing, designing, implementing, and maintaining performance management systems are clearly management functions. But across the country, at the state and local levels, auditors have played valuable roles in encouraging or assisting management in developing and improving performance management systems and their use—for examples go to the web addresses linked above. Many of these examples have involved extremely effective, productive working relationships between auditors and managers. That is the case with a current example in King County, Washington, where the auditor convenes and facilitates a work group of representatives from executive departments, the judiciary, the County Council, and separately elected officials. The auditor’s role is spelled out as advisory, but has been valuable in helping the county develop a consistent approach to strategic planning and performance measurement. The effectiveness of the work group, and positive experiences across branches of government, are reflected in the Performance and Accountability Act passed by the County Council in July 2008, mandating performance management improvements recommended by the work group. 13 As in King County, such working relationships can readily be documented and managed in ways that do not impair auditor independence or have auditors intrude on management functions. While it is always useful to have good working relationships between management and auditors concerning performance management, perhaps the best times for these practices are: • When a jurisdiction is contemplating developing a new performance management system or making major improvements in an existing system, these practices can help bring together the best of the different knowledge, skills, and perspectives of auditors and management to produce a better system than either group would produce on its own. • When new performance management systems or major system improvements are being implemented, auditors can provide valuable training, technical assistance, or advice to help agency managers become productive users of the system sooner than otherwise possible. • When implementation of an existing PM system is uneven in a jurisdiction, auditors can help executive management assist those agencies that most need to improve how they use the system. • At any time, auditors can develop and provide guidance materials to encourage or assist managers in making better use of performance management systems. 5
  6. 6. ROLE 5: Assist Elected Officials or Citizens This is a potentially extensive role in which auditors assist or relate to citizens, non-executive elected officials, or other external stakeholders, such as interest groups and the media. Five different auditor practices are encompassed in this role, including: • Practice 5a. External advocacy [to] external stakeholders, concerning the development, implementation, or improvement of government performance management systems. • Practice 5b. Report performance: Produce and issue external periodic performance reports. • Practice 5c. Assist external decision making: Assist external stakeholders in using performance information to make decisions. • Practice 5d. Engage citizens … in determining performance goals, objectives, or measures. • Practice 5e. Assess citizen engagement … related to performance management. 14 Depending on the auditor reporting relationship in a jurisdiction, this role or some of these practices may be more or less appropriate for the auditor, for example: • Auditors who report to management may be least likely to play this role, but sometimes managers ask the auditor to do it (e.g., in Phoenix, the city manager asked the auditor to engage citizens to help make performance measures and reports more understandable and useful to citizens). • Elected auditors or those who report to a legislative body may be most likely to play this role, as they have a responsibility to assist the constituents or elected body they report to. • Auditors who report to independently-elected officials (e.g., Comptrollers) may not perform this role directly, but may help their “boss” play this role, and may encourage that official to do so. Auditors have made valuable contributions to help state and local legislators understand the need for improving performance management, and in formulating legislative guidance or mandates to do so. The King County example of the auditor “Assisting Management,” noted above, could also be listed under this role, as the auditor’s office helped draft parts of the Performance and Accountability Act. There are other examples of auditors drafting legislation or policy guidance used by legislators to advance performance management, as well as assisting legislators and citizens in other ways. 15 For four of these five practices, there are no apparent issues involving auditor independence or auditors playing a management role. However, Practice 5b. Report performance, involving issuing public performance reports of a government entity, is generally considered a management function. And independence questions persist, as auditors cannot audit their own report, especially in view of recently tightened government audit independence standards. (Some auditors check management’s data before reporting it. But they cannot audit their own analyses, interpretations, or communications approaches.) However, auditors have produced some of the best public performance reports, including several that have won certificates of achievement from the AGA. 16 Also, auditors have sometimes stepped in and provided this important public reporting service where management has not, providing important gains in public accountability. So, we are not quite prepared to say there are no circumstances in which auditors should prepare external performance reports. For now, we will continue to follow developments of this practice and audit independence standards. In Summary, Auditors Can Add Value to Performance Management in Many Ways State and local audit offices across the country have been helping improve performance management by performing the roles and practices described above. Some, such as auditors in the State of Florida, and the Cities of Austin (TX), Phoenix (AZ), and Portland (OR) have been doing it for over 15 years. 17 These and other auditors have changed their mix of practices over time as the performance management in their jurisdictions has changed, so they, as auditors, will add as much value to 6
  7. 7. performance management as they can at any given time. They have demonstrated that there is not one role that auditors should play in performance management, but a variety of roles and practices that auditors can play to add value. They have also demonstrated that it is valuable for auditors to change their mix of practices over time in order to best match what auditors do to the changing performance management environments of their governments. That way, auditors can optimize their roles, and add more value to improving performance management than if they always did the same things. But Why the Auditor? Couldn’t Management Do All These Things? If a few names of the above roles and practices are changed (e.g., change “audit” performance to “study” or “evaluate” performance), one could easily imagine management performing all of the above-described roles and practices. Particularly in larger jurisdictions that can afford to dedicate separate staff to oversee performance management, management does perform many of these practices. They even perform “encourage or assist management,” as, for example, a central performance management team under a chief executive that assists agency managers in improving their performance management practices. So, why involve auditors? There are a number of reasons. The reasons that most apply will differ from one jurisdiction to another. For example: • Auditors bring an independent perspective that adds credibility to performance management practices. They bring a perceived extra level of objectiveness to performance management systems, and their participation is likely to increase trust of elected officials and citizens. • Auditors do not face day-to-day concerns of running programs and services, which can distract from improving performance management. Managers must focus most on delivering services, but auditors can provide a helpful extra set of eyes and ears, and often have more time and freedom to research best practices from other sources, to find ways to improve performance management. • In many jurisdictions, the auditor’s office is a source of some of the strongest quantitative, analytic, and procedural skill sets, which are valuable for improving performance management. For example, when auditors cooperate in developing or improving a performance management system, they can help ensure that controls to assure data reliability are designed into the system. • Sometimes management just doesn’t do it. In some jurisdictions managers do not implement performance management systems, or they do not make the effort to improve existing systems or make them work well. In these cases, auditors can fill the void to improve accountability and sometimes stimulate performance gains. Should a jurisdiction and its citizens suffer from no performance management, or weak performance management, because improving performance management is a “management prerogative” that management chooses not to exercise? All in all, improving performance management is not something that either managers or auditors should do. In the best case, auditors and managers work cooperatively to bring their different perspectives and skills to bear on developing good performance management systems and on consistently improving those systems over time. Case History: City of Austin, Texas 18 From 1985, when the Austin City Auditor started conducting performance audits (Practice 1a. Audit performance), through 1992, very common audit findings across many city agencies involved weak performance measurement practices, with measures often not relevant or non-existent, data unreliable, and the use of data by management limited at best. So in 1992, the City Auditor encouraged the City Council (Practice 5a. External advocacy) to demonstrate its interest in performance measurement to city management. The City Auditor reports to the City Council. That year, the City Auditor’s Office drafted a resolution that the Council passed encouraging city agencies to measure and report performance. 7
  8. 8. Building on the Council resolution, the City Auditor began adding value in a new way, by assessing the systems used by the city to measure, report, and improve performance (Practice 1b. Audit performance management systems). The Auditor’s Office conducted entity-wide performance management systems audits or assessments in 1994, 1996, 1998, and 2002, using results both to continue advocating to Council (Practice 5a External advocacy) to require greater improvement, and to help management design and improve these systems (Practice 4b. Assist management). One approach taken to assisting management, in the late 1990s, involved the City Auditor’s Office serving on a committee with management to oversee agency business planning, including development of new, more relevant performance measures. In 2002, city management enhanced citizen access to city performance information by implementing a web-enabled searchable database called e-performance measures with access to performance information for all city departments. In 2002, city management also took more responsibility for the reliability of performance information by assigning Corporate Internal Audit staff (who report to management) to develop and implement a certification process to test performance measurement reliability (Practice 2a. Test relevance or reliability). In 2004, city management enhanced measures certification when the Budget Office piloted a self- assessment process in which each year, city departments self-assess the reliability of selected measures included in their annual business plans. These self-assessments are submitted by departments to the Budget Office for review. This self-assessment complements the formal certification process, still conducted selectively by Corporate Internal Audit, increasing the number of performance measures that are reviewed for accuracy, and increasing departmental ownership of measure reliability. The certification process is defined by the Budget Office in the city’s Managing for Results Resource Guide provided on-line for use by departments in conducting business planning and performing self- assessments of performance measures. Performance measurement in Austin has reached a level of maturity such that management expects and is required to provide performance information on city operations, be able to answer questions on the information, and be accountable for its reliability. Also, management has been upgrading systems that provide information to support effective performance measurement, and the City Auditor’s Office is planning projects to test the reliability of these systems. The trend in the city government to more integration and centralization of these types of information systems and sources will likely result in more centralized auditing. The City Auditor continues to audit relevant performance measures during performance audits of individual departments and issue areas, as well as assessing data reliability of selected systems that provide information used for performance measurement and other purposes. Just as management has been upgrading automation of performance information, the City Auditor’s Office has upgraded its reliability testing (Practice 2a. Test relevance or reliability) to include the use of software tools for data mining and analysis of entire data universes with less reliance on sampling, or geographic information software for testing the completeness and accuracy of geographic information that may be used in performance measurement. As the quality of department performance information has improved, it has become more useful not only to management and City Council, but also as a tool for the City Auditor. Now, the City Auditor’s Office reviews trends in department performance measures as a regular part of the office’s citywide risk assessment process for determining departments and programs to audit in the coming year, which 8
  9. 9. helps the City Auditor find higher-value targets for its regular program of performance auditing (Practice 1a. Audit performance). NOTES 1 “Auditor Roles” is a Project of The Institute of Internal Auditors (The IIA) in partnership with the Association of Local Government Auditors and the National Association of State Auditors, Comptrollers, and Treasurers, funded by the Alfred P. Sloan Foundation and The IIA. The analyses, interpretations, and views in this paper are those of the author(s), and do not necessarily represent the views of these organizations. 2 Epstein, Paul, Stuart Grifel, and Stephen Morgan Auditor Roles in Government Performance Measurement: A Guide to Exemplary Practices at the Local, State, and Provincial Levels (Altamont Springs, FL: The Institute of Internal Auditors Research Foundation, 2004) 3 Epstein, Grifel, and Morgan, pp. 19–23. 4 Epstein, Paul and Alina Simone, “Add Value to Government Performance and Performance Audits,” in GAP News, 3rd Quarter, 2007, and at [] Also see how Florida’s Office of Program Policy Analysis and Government Accountability (OPPAGA), has used both these approaches: using reliable agency data to audit performance and building on the data to conduct “deeper” audits, at []. 5 For example, see Epstein, Paul, “Performance Management Systems Offer High-value Audit Opportunities,” in GAP News, 1st Quarter, 2008, and at []. Also see examples from eight local, state, and provincial governments at []. 6 For example, see Epstein, Grifel, and Morgan, pp. 50–61, and []. 7 See Epstein, Grifel, and Morgan, p. 51, and []. 8 See Epstein, Grifel, and Morgan, pp. 61–69, and []. 9 “Organizational Independence When Performing Nonaudit Services” in Government Auditing Standards, (Washington, DC: U.S. Government Accountability Office, 2007), paragraphs 3.20–3.30, and Appendix I paragraphs A3.02–A3.03. For a discussion of auditor independence, with reference to these standards, as related to specific Auditor Roles and Practices described in this paper, see []. 10 For state and local examples of this practice, see []. 11 For examples, see []. 12 Government Auditing Standards (2007), Appendix I, (paragraph A3.03g). 13 For the King County (WA) example, see: [] 14 We have not identified auditors assessing citizen engagement that directly relates to performance management, but we have documented auditors assessing engagement for other purposes. We see this as a “potential future practice,” because if citizen engagement in performance management becomes more prevalent, it will be valuable for an independent auditor to assess and recommend improvements in the effectiveness and representativeness of that engagement. Our examples are at: [] 15 For examples, see: [] and []. 16 Among currently listed (2007) AGA Certificate of Achievement winners at [] the Palo Alto (CA), Portland (OR), and Prince William County (VA) reports were prepared by auditors. Other auditors’ reports have won this award in previous years. 17 For case histories of these audit offices performing these practices over time, see Epstein, Grifel, and Morgan, pp. 20–23, and []. 18 Source: []. 9