Strategic performance management_system_and_strategic_planning_cycle

910 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
910
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
49
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Strategic performance management_system_and_strategic_planning_cycle

  1. 1. April 2007 ­ UI Response to UNDP CoPERFORMANCE MANAGEMENT: A PRACTICAL GUIDE FOR MUNICIPALITIES Prepared byDr Malik Khalid Mehmood Assisted by  Katharine Mark  Ritu Nayyar­Stone  Sonia Ignatova UNDP, Bratislava RegionalCentreGrosslingova 35, 811 09BratislavaSlovak Republic March 2007 Project 08013-000Contract No. PS-2006/10
  2. 2. Table of ContentsINTRODUCTION AND SCOPE................................................................................................. 6 The Millennium Development Goals and their Role in this Guide .......................................................... 7 Focus of the Guide.................................................................................................................................... 8 Remainder of Guide.................................................................................................................................. 8STEP 1: BEGIN GOVERNING FOR RESULTS IN YOUR MUNICIPALITY(ORGANIZE THE EFFORT AND DETERMINE THE SCOPE) ........................................ 10 What is Performance Management? How Can It Help Local Governments? ........................................ 10 Focus on Service Outcomes ................................................................................................................... 11 Limitations.............................................................................................................................................. 11 Select the Scope and Coverage of the Performance Measurement Process ........................................... 13STEP 2: IDENTIFY OUTCOMES AND OUTCOME INDICATORS ................................. 18 Identify the service/program objectives, and customers......................................................................... 18 Select the important outcomes for your service/program....................................................................... 21 Categorize performance indicators ......................................................................................................... 26 Select outcome indicator breakouts (disaggregation) of each outcome indicator by key characteristics ................................................................................................................................................................ 28STEP 3: SELECT DATA COLLECTION SOURCES AND PROCEDURES..................... 33 Use agency records................................................................................................................................. 34 Survey Citizens (including Businesses).................................................................................................. 35 Data Quality Control .............................................................................................................................. 48 The Cost of Performance Measurement ................................................................................................. 51STEP 4: ANALYZE PERFORMANCE DATA....................................................................... 52 The Importance of Analyzing Performance Data ................................................................................... 52 How Performance Data Can Help .......................................................................................................... 52 Do Some Preliminary Work ................................................................................................................... 54 Examine the Aggregate Outcome Data .................................................................................................. 54 Examine "Breakout" Data ...................................................................................................................... 56 Examine Findings Across Indicators ...................................................................................................... 59 Make Sense of the numbers.................................................................................................................... 60STEP 5: REPORT PERFORMANCE RESULTS................................................................... 65 The Importance of Good Reporting........................................................................................................ 65 Internal Reporting................................................................................................................................... 65 External Reporting.................................................................................................................................. 69 Other Information that should be Included When Reporting Performance Data - In Both Internal and External Reports ..................................................................................................................................... 70 What If the Performance News Is Bad? ................................................................................................. 70 Dissemination of Performance Reports .................................................................................................. 71STEP 6: SET MUNICIPAL TARGETS ................................................................................... 77 The Importance of Setting Targets ......................................................................................................... 77 Set Targets .............................................................................................................................................. 79 Connect Local Targets to National Targets ............................................................................................ 83 2
  3. 3. Relation of Municipality Targets to Millennium Development Goals (MDGs) and Strategic Plans ..... 83 Concerns About Targets and Target Setting........................................................................................... 84 Final Comment ....................................................................................................................................... 84STEP 7: USE PERFORMANCE INFORMATION TO IMPROVE SERVICES ................ 86 The Importance of Using the Information .............................................................................................. 86 Undertake Service Improvement Action Plans....................................................................................... 86 Analyze Options/Establish Priorities...................................................................................................... 91 HOLD HOW ARE WE DOING? SESSIONS ....................................................................................... 93 Performance Budgeting .......................................................................................................................... 95 Capital Budgeting ................................................................................................................................... 99 Strategic Planning................................................................................................................................. 100 Motivate Your Employees.................................................................................................................... 100 Performance Contracting ...................................................................................................................... 101 Contribute to National and Regional Information Sources .................................................................. 102 Final Comment on Using Performance Information ............................................................................ 103STEP 8: BUILD MUNICIPAL CAPACITY.......................................................................... 105 Decide what training is required, to whom, and how much ................................................................. 106 FINAL WORDS ................................................................................................................................... 109APPENDIX A: Set of sample service outcome/quality indicators for a variety of municipal servicesAPPENDIX B: The millennium development goals and likely data collection sourcesAPPENDIX C: Sample customer questionnaire for municipal services, user survey for water service, patient feedback formAPPENDIX D: Sample procedures for rating certain quality elements of municipal services using trained observer ratings: street cleanlinessAPPENDIX E: Health status, mortality, country data for MDG 7, Target 10 3
  4. 4. PREFACEPreparation of this guide has been sponsored by the UNDP as part of an effort to support the achievementof the Millennium Development Goals.The Millennium Development Goals (MDGs) were derived from the United Nations MillenniumDeclaration, and adopted by 189 countries in 2000. The MDGs, as they are called, focus on the most critical aspectsof poverty, and the factors that contribute to both income and human poverty. They were re-confirmed in 2005 as theGlobal Development Agenda. The MDGs are a set of global goals whose achievement depends on theimplementation by countries of national MDG agendas aimed at achieving nationally adjusted global goals. TheMDGs have played a special role in developing and transition countries as they provide one set of goals that can helpthem and their cities as well set concrete targets and concentrate resources on meeting those. There are a large number ofother examples and resources around the world now available, with more and more national governments and citygovernmentschoosing to measure their performance and improve the services they provide their citizens.UNDP advocates the adoption by both national and local governments of an MDG agenda (see theMillennium Declaration, UN, 2000). While this Guide was first targeted at those countries and specifically themunicipalities of such countries which have decided to adopt an MDG agenda (for further discussion of the relationshipbetween national and local MDG agendas see Capacity Development for Localizing the MDGs, UNDP 2006), theperformance management methodology in this Guide can be used by all municipalities irrespective of whether theyhave adopted the achievement of the MDGs astheir goals.UNDP defines localizing the MDGs as the process of designing (or adjusting) and implementing localdevelopment strategies to achieve the MDGs (or more specifically, to achieve locally adapted MDGs targets). Thisimplies either adapting and sequencing the indicators and targets of existing localdevelopment strategies as needed, or elaborating a MDGs-based development strategy that reflects local priorities andrealities. For this approach to be successful, it should be locally owned and participatory/ UNDP and other UNagencies have produced Toolkits and Guides to help municipalities localize the MDGs and thus address regionaldisparities and marginalization at the sub-national level. (link to UNDPand UN-Habitat toolkits).There is a compelling logic to believe that unless the type of goals included in the MDGs are brought tothe local level (localized), national and global achievements will be skewed. National targets and indicatorsrepresent national averages. Achieving them would require targeted interventions in pockets of poverty which are oftenvery context specific. In order to impact the lives of people, goals, such as those in the MDGs need to be adapted to thecurrent level of development and translated into local realities, andembedded into local planning processes.A Special Note to Municipalities:Integrating the MDGs in the municipal performance framework can bring several advantages to your Cityand to your country, such as: - Make sure performance of services related to key development problems such as poverty, health, education and environment, is monitored and weak performance is identified and addressed; - Link your local development agenda to the national MDG agenda (if it exists) thus ensuring that 4
  5. 5. the work of your city contributes to nationally set goals (the setting of which you should have ideally participated); - Ensure that the work of your City contributes to the global development agenda and contributes to better and safer world which also benefits you city (poverty and the related crime and diseases, environment have no frontiers nowadays); - Increase the chances of benefiting from financial support to your City from central government or the donor community.This Guide is intended to support you in this effort. 5
  6. 6. INTRODUCTION AND SCOPEThis guide is aimed at helping all municipalities accomplish their goals by monitoring their ownperformance and using the information they get to improve the lives of all their citizens.Why establish a performance monitoring system to improve results? Many countries of Europe, the CISregion, and elsewhere in the world have initiated political and administrative decentralization processes. Decentralizationmeans transparency and accountability to taxpayers, as well as local governments thatstrive to continually improve the services they provide to their citizens.Local authorities play a vital role in improving the well-being of their citizens. They provide the mostbasic everyday services - such as solid waste collection, road maintenance, or access to water - as well as working in manyother ways to help their citizens out of poverty and improve their quality of life. Integrating a sound performancemeasurement process is only a first step. Often faced with very limited resources, poor quality infrastructure, andhistorically weak trust and communication between citizens and local government, local government officials and staffin Eastern Europe and elsewhere frequentlyfeel especially handicapped in trying to implement improved services that meet citizen needs.Setting up a system to monitor performance of municipality programs and policies will enable themunicipality and its agencies to: • Establish strategic plans, such as City Development Strategies • Regularly, monitor progress in meeting strategic plan and annual performance targets • Use performance budgeting as a means to links resources with results • Identify weak performing services and programs so that the necessary actions can be taken to improve performance • Allocate their own resources (not just city budget funds, but also city staff and equipment) in the most effective way • Help motivate public employees to continually focus on improving the quality of their services to citizens • Identify best practices in order to learn from high performing entities • Compare performance across localities, regions, and countries to help identify achievable targets for the municipalitys own performance and identify areas that need additional strengthening or resourcesAs suggested above, and as will be made clearer in this Guide, establishing a performance monitoring canhave multiple benefits for the work of a municipality, ranging from strengthening strategic planning,learning what service approaches work well and which do not improve budgeting and justifying theallocation of funds for initiatives to improve service delivery, to encouraging the reporting of results to citizens. TheGuide suggests steps a municipality can take both to improve collection of information on performance - "performancemeasurement" - and to use that information to help get better results -"performance management."The use of performance indicators and targets to improve conditions for citizens has increased over thelast decade as local and national governments around the world have become increasingly aware of thevalue of results-based decision-making. 6
  7. 7. THE MILLENNIUM DEVELOPMENT GOALS AND THEIR ROLE IN THIS GUIDEThe MDGs provide a clear framework for national and local development efforts taking a holistic multi-dimensional approach to poverty reduction and human development. They link the global, the national and local levelsthrough the same set of goals and provide a target-based, measurable framework for accounting for national and localdevelopment results. As already indicated, the MDGs, focus on criticalaspects of poverty, and the factors that contribute to poverty. These include essential areas such as health,education, access to drinking water, or adequate shelter. The eight MDGs listed below: Goal 1: Eradicate extreme poverty and hunger Goal 2: Achieve universal primary education Goal 3: Promote gender equality and empower women Goal 4: Reduce child mortality Goal 5: Improve maternal health Goal 6: Combat HIV/AIDS, malaria and other diseases Goal 7: Ensure environmental sustainability Goal 8: Develop a global partnership for developmentThe Goals are accompanied by "indicators" and "targets" which are more specific, to allow countries tofocus on particular areas that are important to monitor. Governments and municipalities that have chosento adopt an MDG agenda can adapt the global targets and indicators to national and local circumstances. They are shouldalso include additional outcomes and indicators important to the specific localconditions.All the MDGs, with associated targets and indicators, are listed in Appendix B. We have marked theindicators that might be of particular interest to local governments as part of the indicators they choose tomonitor. Some examples of MDGs and the indicators used to monitor their progress are:Goal 2: Achieve universal primary education Indicator 7. Proportion of pupils starting grade 1 who reach grade 5Goal 5: Improve maternal health Indicator 17. Proportion of births attended by skilled health personnelGoal 7: Ensure environmental sustainability Indicator 29. Proportion of population with sustainable access to an improved water source Indicator 30. Proportion of people with access to improved sanitationBecause of the international interest in MDGs, they have helped focus national attention on theimportance of identifying priority outcomes and using performance measurement tools to reach those outcomes. Mostinternational donors, including the UN system as a whole naturally support the MDGsand are aware of the importance - more broadly - of monitoring performance in key areas of public service. Theadoption of an MDG agenda and initiatives related to improving public service delivery can act as effective advocacytools and help the mobilization of resources for local development. As cities begin to establish their ownperformance measurement systems, it will be useful to have allies at the national and international level, who may beable to provide resources and data to strengthen the localefforts.This Guide will refer to the MDGs throughout, identifying associated resources for local governments,and suggesting opportunities for using some aspect of the MDGs. 7
  8. 8. This guide shows cities, first, how municipalities can use performance measurement and performance management to improve their services and their responsiveness to citizens; how they might use outcome indicators, including MDG indicators, as part of their performance monitoring; how they can set local targets for each of their indicators and reach the desired outcomes; and how they can also contribute to the efforts of the country itself to set and meet country-wide targets to mitigate poverty and improve the quality of life at the national level.FOCUS OF THE GUIDEThe material in this guide focuses on "Governing for Results." Most suggestions contained here areintended to encourage municipalities and their agencies to seek and use information on the results (the benefits) thattheir services and programs are achieving for their citizens. This means explicitlyconsidering the specific likely effects on their citizens when making policy and program decisions.Achieving results at as low a financial cost (that is, being efficient) is another important area of municipalperformance. However, this is a subject for another guide.This guide focuses on the process of regularly monitoring the outcomes of a municipalitys services. Byregularly, we mean at least annually for purposes such as budgeting, and more frequent monitoring of performance byindividual agency managers, such as quarterly or monthly. The focus is on developing a practical process thatmunicipalities can adapt to their own situation for regularly, and reliably, tracking their own progress on outcomes ofimportance to their citizens. The Guide does not discuss more in- depth evaluation studies, in which, if resources areavailable, the government can sponsor more in-depth look into how well a particular program or service is performingand why. Such in-depth studies can on occasion be very useful to municipalities. However, the regular monitoring ofoutcomes discussed herereport can often provide considerable information for such studies.This guide is complementary to two Guides that have been produced by UN Agencies: 1) Toolkit forLocalizing the MDGs (UNDP); 2) Localizing the MDGs: a Guide for Municipalities and Local Partners (UN-Habitat). The two publications deal with strategic planning which integrates and localizes the MDGs. As willbecome clear from this guide, performance management can be an element of an MDG- based strategic planning andimplementation processes, or any strategic planning process, and a tool for monitoring the implementation of strategicplans. The readers of this guide are therefore encouraged toalso consult at least one of the two documents mentioned above.REMAINDER OF GUIDEThis Guide will help you develop high quality performance management systems or improve the onesalready in place. The guide is organized in accordance with the basic steps for instituting performancemanagement in your local government, as listed below. Step 1. Organize the Effort and Determine the Scope Step 2. Identify Outcomes and Outcome Indicators Step 3. Select the Data Collection Sources and Procedures Step 4. Analyze the Performance Data Step 5. Report Performance Results 8
  9. 9. Step 6. Set Municipality Targets for Each Performance Indicator Step 7. Use the Performance Information to Improve Services and Establish Accountability Step 8. Build Municipal Capacity for Governing for Results This guide takes the reader through each of these steps, with each chapter corresponding to one step. The flow chart below depicts the usual order of these steps. It is important to note, however, that in many cases these steps are iterative. For example, after setting targets (Step 6) usually data will again be collected (Step 3) and analyzed (Step 4) in order to monitor progress, and then reported (Step 5), perhaps leading to the establishment of new targets (Step 6). STEP STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 1 ChooseSelect Outcomes Data Analyze Report Set UseServic & Identify Collection Targetses Indicators STEP 8 - Build Municipal Capacity A number of Appendices are provided. These include: Appendix A: A candidate set of outcome indicators for a number of municipal services. Use these to give you ideas for the kinds of indicators you might want to select in your municipality. Appendix B: Millennium Development Goals with associated targets and indicators and likely data collection sources for indicators applicable to municipalities. Appendix C: Examples of a sample customer questionnaire for municipal services, an illustrative user survey for a water service, and a patient feedback form from a hospital in India. Appendix D: Procedures for rating certain quality elements of municipal services using "trained observer ratings" Appendix E: Data for a number of performance indicators to illustrate data that could help a municipality set its own targets for those indicators. 9
  10. 10. Step 1. BEGIN GOVERNING FOR RESULTS IN YOURMUNICIPALITY (ORGANIZE THE EFFORT AND DETERMINE THE SCOPE) WHAT IS PERFORMANCE MANAGEMENT ? HOW CAN IT HELP LOCAL GOVERNMENTS?Performance management - sometimes referred to as "governing for results" - is a system of regularlymeasuring the results (outcomes) of public sector programs, organizations, or individuals, and using this information toincrease efficiency in service delivery. Public officials need regular feedback on theeffectiveness of their services in order to make improvements, while at the same time the public wants toknow that the government is spending their tax money in the best way to meet citizens needs.What is the difference between Performance Measurement and Performance Management? - PerformanceManagement goes one step further, using the measurements of performance to manageservices better.Traditionally, this kind of information has been hard to get - emerging only piecemeal throughcomplaints or occasional anecdotes. Over the last three decades, performance management has become increasinglypopular way for governments around the world - both local and national governments - tomanage their programs and services to achieve the results their citizens expect. BENEFITS OF A PERFORMANCE MANAGEMENT SYSTEM TO LOCAL GOVERNANCE • Improving service quality and outcomes; • Improving resource allocation and justifying agency budgets or service cuts; • Making public agencies accountable for results to elected officials and the public; • Increasing the citizens trust in the local government; and • Making work more interesting and satisfying for public employees because of its citizen focus.Governing for Results has encouraged governments both to make the results they are seeking explicit andto design programs and policies to actively and directly seek those results. Monitoring progress towards those resultsprovides a constant feedback into the policy and implementation process to improve effortsto achieve their objectives.
  11. 11. 10
  12. 12. FOCUS ON SERVICE OUTCOMESPerformance management is based on a simple concept: a focus on service outcomes, or actual results,rather than only on the quantity of service that an agency provides. This implies assessing the performance ofthe government based on the implications of services to customers, not on physical outputs. The work of the localgovernment is measured by what the citizen or user of the service actually experiences - are the roads in good enoughcondition so that children can get to school in the rainy season? Do the children stay in school until they graduate? Dopregnant women visit the primary health clinic during pregnancy, and does that result in healthier babies? Is garbagecollected regularly, and doesthat have an effect on health?This simple idea, however, means that many people in local government need to think in a different way.It will not be enough to measure how many kilometers are paved, or whether the clinic has the right staff. It will beimportant to see what the results of those efforts are in order to know whether they are workingwell.Through tracking performance indicators, and clearly linking those indicators to the results that the localgovernments wants to see, the system provides decision makers with better information. With this informationthey can make better decisions—and show why they made those decisions. Usingperformance management, local governments can demonstrate their commitment to providing qualityservice.This way of thinking can proceed at several levels. For example, a city may believe it is very important toprovide a "safe, health, and clean environment for all the citizens of the municipality." That may lead to the identification ofa number of outcomes that will be sought in several different services: good quality service at the primary health clinic,public awareness about health hazards, better solid waste collection, clean water sources. Each of those in turn can requirea number of different outcomes. For example, the head of solid waste management may want to ensure that streetsare clean, citizens satisfied with collection service, landfills are well managed, and that the service has full costrecovery so that goodservice is sustainable. LIMITATIONS Municipal officials need to recognize important limitations of the performance information that would come from the steps discussed here. These include: • The regularly collected annual performance information discussed in this report will not tell you WHY the recorded performance levels were good or bad. (However, a well- designed performance measurement process can provide useful clues. For performance indicators that indicate unexpectedly low, or high, results, more in- depth evaluations will be needed to get at the causes). • Similarly, the performance information does not tell municipality officials what has to be done to correct problems identified by the performance measurement data. • Performance measurement information provides information about past performance. It does not by itself tell what future result will be. However, information on past performance provides a major source of information for estimating future results, such as needed for making budget, policy, and program decisions. 11
  13. 13. GET STARTEDDepending on their size, governance system, and capacity constraints, governments around the world areusing different tools to govern-for-results. One or more of the following approaches might be used to startdeveloping a performance management system in your city: Develop and track selected performance indicators in each service sector. Make policy decisions based on the information and disclose this information in city performance reports and its budget. Develop service improvement action plans in priority sectors. Apply performance management to the internal processes of the local government, for example to increase municipal revenue, or reduce the time it takes for citizens to register births or marriages. Implement a comprehensive performance management system in your city that combines strategic planning, setting goals and objectives for each service sector, citizen participation and using performance information. A municipality can choose to adopt an MDG agenda and integrate MDG Goals in its strategic plan. For each of the MDG Goals this municipality should set targets and indicators which reflect local circumstances.Whoshouldbeinvolved?Many stakeholders play a role in the process of implementing and using a performance managementsystem. Some of the key actors are described here.Mayor The Mayor should be a principal user of performance information especially in establishing major policies and in reviewing city programs and its budget. In addition, the Mayor will play a major role in setting the climate for the shift to a results-orientation. The Mayors support is important, for making sure that adequate resources are allocated to implementing and, later, sustaining the process..City Council The support of the council will be essential to the success of the enterprise, not only through the provision of funding when necessary, but also through underlying the importance of performance information by requesting it and using it. Elected council members will find outcome information to be very useful in carrying out their responsibilities, enabling them to more easily understand the impact of city services on their constituents, and to make decisions in their appropriation and oversight roles.DepartmentHeads The heads of different departments or institutions play a crucial role in facilitating and using performance data.National and Regional Government These entities should have their own performance management processes. Many of their agencies are likely to need performance data (such as on health, education and welfare) to make their own policy and program decisions - and to provide information that will enable them to set national targets (such as for MDG indicators). 12
  14. 14. National and regional agencies may also use the performance indicators as a basis for identifying local areas that require special assistance, training, and help in achieving equity in the country, or in identifying best practices that can be shared with other localities to improve service everywhere.International Donors Donor loans or grants sometimes stipulate the condition that infrastructure projects or grants-in-aid be subject to detailed performance monitoring, or tied to achievable results. For example, the World Banks output- based-aid involves delegating service delivery to a third party (private firm or NGO) under contracts that tie payment to particular outputs or results delivered. Other agencies like USAID, CIDA, and DFID also use performance monitoring for specific projects, own inter-agency performance, and in some cases link them to country plans. A number of international donors, including the UN system, are focusing assistance efforts on helping countries attain the MDG targets and may want to see the connections between municipal programs and those targets. These donors can provide valuable assistance in capacity building within municipalities, by providing training on performance management systems, and creating incentives for performance monitoring in service delivery.Non-Governmental Organizations Different types of NGOs can play two valuable roles in improving service delivery: (1) in providing important quality public services themselves -- in which case they should themselves have their own performance management process, and (2) playing a watchdog or advocacy role in increasing citizen awareness of their rights to better quality public services.Business Community Businesses are a major consumer of government services (such as water, transportation, and, economic development services). In addition, many services are delivered through contracts. To ensure quality services, the municipality can use performance contracts with these businesses. s Data for performance monitoring will, thus, also have to be obtained from, or at least with the cooperation of, these businesses.Citizens Citizens are the major consumers of public services. And they pay (via their tax payments or fees), for many services. Citizens are also a major source of information needed to evaluate services and are an important source for identifying the outcomes that should be tracked, both annually and for strategic plans.SELECT THE SCOPE AND COVERAGE OF THE PERFORMANCE MEASUREMENT PROCESSWhich services should you include?You might choose to start with one service (or program), several, or cover all municipal services andprograms. It is recommended here that you attempt to cover all your services so that all municipality staff are encouraged tofocus on results. Realistically, however, you may need to start with a few services at a time, so that the successes in onearea can serve as motivation to introduce performance managementmore widely. 13
  15. 15. There are several different ways to decide where to start. One method is to identify the departments thatmight be "easiest" - for example, areas where data is easily available, or some inexpensive improvements are most likely toyield rapid results. Another might be to start with departments in which the leadershipis already very interested in adopting the new approach, which is also likely to make the pilot efforteasier.Another approach to choosing a starting place can be to look at citizen priorities. To do so might slowdown the process, but does identify an area that is likely to yield improvements in citizen satisfaction in the short term.International donors are often extremely supportive of such consultations with citizens, and can help fund such efforts.Some more detail about such an approach is provided in Step 7 on Using Performance Management, under the descriptionof Strategic Planning. If there is strategic planning, thestrategic goals identified will determine the services that are to be monitored.The Millennium Development Goals can also provide guidance on what outcomes to select. A step bystep approach on how a municipality should select services that contribute to the MDGs might be asfollows:Step 1 - Review the MDG Goals and select those global Goals which are relevant to local realities (in a transition or developing country context, all should normally be relevant)Step 2 - Identify the services which are related to those GoalsStep 3 - Adapt the indicators to local circumstancesStep 4 - Review the national (if they exist) and global indicators related to the selected (and adapted) targets and select those which are relevant to local circumstancesStep 5 - Identify additional outcomes and indicators which are relevant to local circumstances and contribute to the MDG goalsStep 6 - Collect baseline data for all indicatorsStep 7 - Set for each indicator targets appropriate for the locality, given the priorities, citizen preferences, needs, and available resources, and also bearing in mind as possible benchmarks, such sources as national MDG targets, performance in other localities, and adapting those targets to local circumstancesStep 8 - Identify non-MDG related city goals, identify indicators and set targetsLocal governments in different countries are responsible for different types of functions. In addition, insome countries those functions may be subject to change, especially where a process of decentralization may beunderway. It makes sense to start with services that are fully under the control of the locality, because that is whereimproved decisions will have the greatest impact, but there have also been instances when performance measurement canalso be applied to functions that are mixed - i.e., shared between central and local government -- or even largely centralfunctions. It can be especially difficult in sharedfunctions to ascertain the effectiveness of the service, and measuring performance can provide useful input on thevarious aspects of the service. Thus, the central government might gain information about how well local governmentsare carrying out a task, or performance information might show that centralfunding or regulations arent yielding the results that were expected.Exhibit 1-1 provides a list of exclusive and shared functions in Albania as of 2006. It can be noted thatwhile most cities that have used performance management in Albania have focused first on exclusive functions, suchas solid waste collection, parks, street cleaning, or water provision, there have also beenseveral efforts to measure performance in areas of shared functions, such as in education and socialassistance. Some of those examples will be provided in this Guide. Exhibit 1-1. Functions of Communes and Municipalities in Albania 14
  16. 16. Exclusive Functions of Communes and Municipalities I. Infrastructure and Public Services a. Water supply b. Sewage and drainage system and [flood] protection canals in the residential areas; c. Construction, rehabilitation and maintenance of local roads, sidewalks and squares; ç. Public lighting; d. Public transport; dh. Cemeteries and funeral services; e. City/village decoration; ё. Parks and public spaces; f. Waste management; g. Urban planning, land management and housing according to the manner described in the law. II. Social Cultural and Recreational Functions a. Saving and promoting the local culture and historic values, organization of activities and management of relevant institutions; b. Organization of recreational activities and management of relevant institutions; c. Social services including orphanages, day care, elderly homes, etc. III. Local Economic Development a. The preparation of programs for local economic development; b. The setting [regulation] and functioning of public market places and trade networks; c. Small business development as well as the carrying out of promotional activities, as fairs and advertisement in public places; ç. Performance of services in support of the local economic development, as information, necessary structures and infrastructure; d. Veterinary service; dh. The protection and development of local forests, pastures and natural resources of local character. IV. Civil Security a. The protection of public order to prevent administrative violations and enforce the implementation of commune or municipality acts; b. Civil security. Shared Functions of Communes and Municipalities a. Pre school and pre university education; b. Priority health service and protection of public health; c. Social assistance and poverty alleviation and ensuring of the functioning of relevant institutions; ç. Public order and civil protection; d. Environmental protection; dh. Other shared functions as described by law.Source: Law on Organization and Functioning of Local Governments, No. 8652, dated 31.07.2000A Good Approach: Establish a Municipal Steering Committee and Working GroupsOnce you have determined the scope of your performance measurement process, a good way to beginimplementing it is to establish a high-level, across-government, Steering Committee to oversee the process. TheSteering Committee can then establish a Working Group to lead work on the details ofimplementation. The Steering Committee should include such persons as: - A representative of the Mayor - A high-level official of the finance/budget office - A high-level official of the human resources (personnel) office - Several department heads - Information technology high-level official 15
  17. 17. The Working Group should have representatives from the departments carrying out or overseeing thework in question, from the financial department, and from a number of related areas. Encourage eachparticipating municipal department to have its own working groupExhibit 1-2 provides examples of the types of people that might be included in these department workinggroups. Such groups should consider including a representative from outside the government to obtain abroader, consumer, perspective. Exhibit 1-2. Example of Working Group CompositionSolid Waste Working Group Head of Technical and Land AdministrationCity Manager Municipal Services ManagerSocial economic department Head of Planning and InformationDirector of financial department Construction and Design TeamHead of the solid waste collection company Municipality administratorsSanitation team Representatives from business firmsHealth offices administration Financial Management teamMunicipality level social teamMunicipality level economic team Construction and Maintenance of Asphalt andEnvironmental protection authority Gravel Roads Working GroupRepresentative from an NGO or citizen group Technical and Land Administration Departmentinterested in city cleanliness Technical team Administrative Support servicesEducation Working Group Urban Development and Construction Bureau -Deputy Mayor local branchSchool principalRepresentative from the parent-teacher Municipality-Wide Working Groupassociation Deputy MayorRepresentative from the Education Committee Head of public works departmentof the City Council Director of finance department Other department headsLand Management Working Group Representatives of selected NGOsWhat should be the functions of these working groups? The government-wide working group should have such task as: - Developing a government time table for implementation - Identifying and defining the types of indicators to be included - Identifying staff training needs and making arrangements for the initial training efforts - for both management and line personnel. - Develop a communication strategy - Communicate with local government bodies, City Council, civil society and ordinary citizens - Arranging for the development of guidelines for major data collection procedures - In general, guiding such steps as those described in this guide in Steps 2-8 16
  18. 18. The department working groups should have similar tasks but focused on their specific needs. 17
  19. 19. Step 2. IDENTIFY OUTCOMES AND OUTCOME INDICATORS IDENTIFY THE SERVICE/PROGRAM OBJECTIVES, AND CUSTOMERSFor each service or program included in the municipalitys performance measurement process, themunicipality should start by identifying the services objectives. What is the service intended to do forthe city and its citizens? What are the primary benefits desired?A good statement of objectives should identify the key intended benefits and who are the intendedbeneficiaries (such as all the municipalitys citizens or some particular segment). This process should also identify thepossible unintended effects, both beneficial and negative effects. Each of these will helpformulate the outcomes that will be tracked. Ask such questions as: • Who benefits from the program - in what ways? Which demographic groups are particularly affected by the program? • Who might be hurt by program activities - in what ways? • What persons who the program does not directly target might be significantly affected by the program? • Is the public-at-large likely to have a major interest in what the program accomplishes?Exhibit 2-1 provides examples of the objectives and affected citizen groups for a few services. Exhibit 2-1. Service, Objective, and CustomersProgram or Service Objective Customers or UsersSolid waste collection Clean city and neighborhoods City residentsSchools Better education Children and parents; employersFinancial Department Increase municipal revenue All municipal services and citizensRoad maintenance Safe and rideable roads City residents and city visitorsAll services Improved collection of fees All municipal services and citizensLand management Adequate housing City residents City residentsWater authority Healthy population Elderly people, their families,Social services Healthy and secure elderly their caretakersHousing peopleHealthNGOsExamples of key customer groups in different programs are shown in Exhibit 2-2. 18
  20. 20. Exhibit 2-2. Examples of Key Customer Groups in Different ProgramsA road construction program Citizens and transportation companiesA water treatment plant Citizens, businesses, and visitors to the communityA vocational school program: Students, parents, and local businesses who recruit the schools graduatesA sports facility Athletes and the general publicA municipal park Adults, children, and senior citizens in the community, and visitors SELECT THE OUTCOMES TO TRACK FOR EACH SERVICEIdentifying the specific outcomes that you will try to achieve, given the services objectives, is one of themost important parts of this process. What isanoutcome? An outcome is the result of a service, from the point of view of the citizens, especially the customers for the service. We can start by thinking about the various steps that go into delivering a service: First, there are inputs - these are the resources we use, for example, money or employees of the municipality. Second, there are outputs - these are the products that the city department, contractor, or agency produces, such as kilometers of road repaired, tons of garbage collected. Third are the outcomes - these are the results of the service: the roads are in good condition, the city streets are clean, etc. It is useful to identify two primary levels of outcomes; "intermediate" outcomes and "end" outcomes. We can think of the higher outcomes - the "end outcomes" or "ultimate" outcomes • as the real purposes of what we are doing. For instance, the improved health of citizens that comes from a clean city, or the ability to go to work or school quickly and safely that is made possible by good roads. We call these "end outcomes." An intermediate outcome is also a result, not just an output, but the accomplishment of something that is likely to lead to an end outcome.Exhibit 2-3 diagrams the "causal" relationship among these categories. Funding and people are needed toimplement activities. Those activities are expected to produce outputs that are expected to lead tointermediate outcomes and then to end outcomes.Exhibit 2-3. Building Towards Results 19
  21. 21. End Outcome Intermediate Outcomes Outputs InputsBelow are some sources of information that can help you identify what outcomes your municipalityshould track. Each source is likely to have its own perspective on what is, or should be, important to citizens and thecommunity a whole. Most, probably all, services and programs will each need to consider multiple outcomes inorder to be comprehensive as to what is important to citizens and thecommunity. • Discussions or meetings with customers and service providers • Customer complaint information • Legislation and regulations • Community policy statements contained in budget documents • Strategic plans • Program descriptions and annual reports • Discussions with upper level officials and their staff, to identify future directions, new responsibilities, new standards at the national or regional level • Discussions with legislators and their staff • Input from program personnel • Goal statements by other governments for similar programs • Poverty Reduction Strategy Document (or other national strategy) • Sector Strategies • Regional Development StrategiesYou can obtain information on program results through meetings with customers (known as "focusgroups"); meetings with program staff; and meetings with other local government personnel.Exhibit 2-4 provides several different examples of outputs, intermediate outcomes, and end outcomes. Exhibit 2-4. Examples of Outputs, Intermediate Outcomes, and End OutcomesOutput Intermediate Outcome End OutcomeRoads are repaired Roads are in good condition Citizens can reach work, 20
  22. 22. school, markets and servicesClinics are built and staffed Pregnant women visit clinic Children are born healthyGarbage is collected Neighborhoods are clean Lower incidence of diseaseCustomers are billed Fees are paid Cost recovery enables adequate services to be providedWater is supplied Citizens have access to water Citizens are healthySchools have desks and Children attend school Children are educatedtextbooks How are outputs and outcomes different? An important element of performance measurement is that it differentiates between outputs and outcomes. In measuring what government does, the traditional focus has been on tracking expenditures, number of employees, and sometimes their physical outputs. An Output or an Outcome? Sometimes people are confused about the difference between an output and an outcome. A key question is how likely it is to be important to citizens and service customers. Outputs are usually the physically things that services and their employees did (e.g., paved 200 square meters of road), while outcomes are what those things are expected to accomplish from the viewpoint of the "recipient" of the service (e.g., road condition is good). The outcome focus of performance measurement, however, connects performance to the benefits of services for citizens and the community. For example, performance measurement is concerned not with the number of teachers employed, but with the reduction in student dropout rate. Of course, focusing on outcomes does not mean that you neglect outputs. Instead, a focus on outcomes provides a framework for you to analyze outputs in a meaningful way. In the above example, hiring more teachers or increasing the number of lessons taught does not necessarily reduce the number of students dropping out of school. It may mean that you also need special programs to improve the employment opportunities for parents for students who are dropping out of school. Or you might set up a preventive counseling program to help those students who are the most likely to drop out. Measuring the performance of programs targeted at decreasing the dropout rate would then tell you how successful or unsuccessful these programs are. Another example: Focusing on the percentage of your municipalitys roads that are in good, rideable condition, rather than on the number of square meters of road maintained, helps identify specific areas that most need maintenance attention.SELECT THE IMPORTANT OUTCOMES FOR YOUR SERVICE/PROGRAM 21
  23. 23. It is not always obvious what outcomes should be selected, but the best way to decide is to think aboutwhat is most important. Whoever is responsible for selecting the outcomes should brainstorm about outcomes beforemaking a final decision. Remember, there can be several "layers" of outcomes - ranging from the end outcomes (e.g., the healthof citizens) to a number of intermediate outcomes (e.g., access to water, sufficient water pressure, and adequate costrecovery). Usually, you will want to track both intermediate outcomes and end outcomes to determine both whether theend outcome has been reached and which intermediate outcomes have been successful or might need to be adjusted. Endoutcomes are more important, but intermediate outcomes provide services/programs with earlier information onprogress and, thus, usually need to be tracked.One important source of outcomes that should not be neglected is consultation with end-users, that is, those who willbenefit from the service. This might include meetings with citizens in differentneighborhoods, or the use of information from a citizen survey. How do you "brainstorm" for outcomes? Brainstorming is a technique to help a group think creatively to come up with new ideas. The central "rule" is that everyone should say what he or she thinks openly and without inhibition. No one will be critical. A good way to start might be to have a large piece of paper (maybe on a flip chart) and ask everyone to suggest outcomes they would like to see. Go ahead and shout them out. Write everything down. After the "brainstorm", the group will discuss the choices to decide which outcomes they will focus on.Multiple outcomes are to be expected for any public service.Here are some examples of some outcomes that have been selected in cities in Eastern Europe for selectedservices:Solid waste collection:- Areas around collection points are rated "clean"- Citizens are satisfied with cleanliness in their neighborhoods- Full cost recovery via collection of garbage feesWater- More households are connected to the city water system- Citizens feel they have enough water when they need it- Increased water quality- Full cost recovery through water tariffsOutcomes contributing to the Millennium Development GoalsSome local governments may want to consider in what ways their local functions contribute to reachingthe Millennium Development Goals. As a starting point, it is useful to note that most local government services areessential contributions to the Millennium Development Goals, although they are not specifically identified by theGoals, the Targets, or the Indicators. For instance, maintaining the adequacy of roads - a key local function in mostcountries - is essential to ensuring access to many primary services (clinics, schools, markets, water). AppendixB provides an annotated version of the MDGs,suggesting ways in which local services might be contributors to the Goals. 22
  24. 24. Local governments may choose to start their performance management efforts in areas related to one ormore Millennium Development Goal, choosing outcomes over which the local government has somecontrol and that are important to the community. An example might be: Goal 2: Achieve universal primary education Supporting outcomes: • Good condition roads to allow access to schools • School facilities are in good condition (in countries where local governments are responsible for school facilities)IDENTIFY AND DEFINE THE SPECIFIC PERFORMANCE INDICATORS THAT WILL BE MEASURED -FREQUENCY OF COLLECTION , UNITS, LEVEL OF DISAGGREGATIONFor each outcome you identify, you also need one or more specific outcome indicators, specific ways tomeasure progress toward that outcome. Outcomes Indicators are at the heart of performance management. They are theelements you will measure and track to see whether your local government is achieving the results it wants. For eachoutcome that is sought, measurable indicators need to be selected that permitthe government to assess the progress being made towards the outcome.An indicator must first of all be measurable. Not all outcomes of programs are measurable, or at leastdirectly measurable. You need to translate each outcome of the program into performance indicators thatspecify what you will measure. In some cases you may want several indicators for one outcome.Typically indicators start with the words "number of" or "percent of." In some cases, you will want tomeasure both the number and the percent; for instance you might want to measure the total number of children who havereceived vaccinations, and also the percent of children that represents, so that is clearhow many children are still at risk.Sometimes, when you cannot measure directly a particular outcome, you can use a substitute indicator, aproxy indicator. For example, for outcomes that seek to prevent something from occurring, measuring the number ofincidents prevented can be very difficult, if not feasible. Instead, governments track the number of incidents thatdo occur - a proxy indicator. These proxies are not ideal but they can be theonly practical approach.Each indicator needs to be fully and clearly defined so that data collection can be done properly andproduce valid data. For example, consider the important indicator: "Proportion of population with sustainable access to an improved water source." What do the words "sustainable," "access," and "improved water source," mean? Different people responsible for collecting the data for the indicator can easily define each of these terms differently. Next year different staff might interpret the terms differently than those collecting the data last year.An excellent source for definitions, especially for MDG indicators, is that provided by the United NationsDevelopment Group. The block below presents the definition provided by UNDG for the water-accessindicator used in the above example. The MDG indicators most likely to be directly relevant tomunicipalities are listed in Appendix B. 23
  25. 25. Exhibit 2-5Definition of the Indicator "Proportion of Population With Sustainable Access to an ImprovedWater Source, Urban and Rural"The percentage of the population who use any of the following types of water supply for drinking: pipedwater, public tap, borehole or pump, protected well, protected spring or rainwater. Improved water sources do notinclude vendor-provided water, bottled water, tanker trucks, or unprotected wells andsprings.Access to safe water refers to the percentage of the population with reasonable access to an adequatesupply of safe water in their dwelling or within a convenient distance of their dwelling. The Global Water Supplyand Sanitation Assessment 2000 Report defines reasonable access as "the availability of 20 litres per capita per day at adistance no longer than 1,000 metres." However access and volume of drinking water are difficult to measure, sosources of drinking water that are thought to provide safewater are used as a proxy.Source for this and the other MDG indicator definitions: Indicators for Monitoring the Millennium DevelopmentGoals. Definitions, Rational, Concepts, and Sources. 2003. New York: United Nations. This publication isavailable at undp.un.org -in six languages.Such available definitions should provide your municipality with a very good starting point. However, asthe definition in the block indicate, at least some tailoring to own local situation is likely to be necessaryto fully define each indicator.Appendix A provides a candidate set of outcome indicators for a number of typical municipality services.(The MDG indicators included in Attachment B are included in the set of candidate outcomeindicators presented in Appendix A.)You need to consider several factors when selecting performance indicators. Exhibit 2-5 suggests a set ofcriteria for selecting them. Rate each indicator according to these criteria. Exhibit 2-5. Criteria for Selecting Performance IndicatorsRelevance. Choose indicators that are relevant to the mission/objectives of the service and to what theyare supposed to measure.Importance/Usefulness. Select indicators that provide useful information on the program and that areimportant to help you determine progress in achieving the services objectives.Availability. Choose indicators for which data can likely be obtained and within your budget.Uniqueness. Use indicators that provide information not duplicated by other indicators.Timeliness. Choose indicators for which you can collect and analyze data in time to make decisions. 24
  26. 26. Ease of Understanding. Select indicators that the citizens and government officials can easilyunderstand.Costs of Data Collection. Choose indicators for which the costs of data collection are reasonable.Exhibit 2-6 is an example of indicators that you could use as a starting point for two different programs.This exhibit also contains a fourth major category of performance indicator: efficiency indicators. These are usuallydefined as the ratio of the cost of a particular service to the amount of product that wasproduced with that amount of expenditure.The unit of product traditionally has been one of the outputs. The efficiency indicator usually is of theform "cost per unit of output." However, a sole focus on output efficiency can tempt employees to speed up their work,sacrificing quality. A municipality that also collects outcome data can in many cases thenuse a much more true indicator of efficiency: "cost per unit of outcome." For example, the public works agency can then inaddition to tracking cost per meter of road repaired also track "cost per meter of roadrepaired that was improved from an unsatisfactory condition to a good condition." Exhibit 2-6. Illustrative Performance Indicators City of Bangalore, India Water Supply Environment Input Input Cost Cost Staff Staff Materials, equipment Materials, equipment Output Output Average number of hours of water supply per Number of persons per hospital bed, including both day government and private sector hospitals Ratio of number of stand-posts in slums to total Percentage distribution of waste water treated by slum household each method Daily consumption of water in litres per capita Percent of waste water treated and re-cycled for non- per day (LPCD) consumption purposes Outcome Outcome Percentage of water lost during distribution Noise pollution in decibels at selected locations total water supply Percentage of population suffering from pollution- Average citizen satisfaction rating with water resultant respiratory diseases quality Percentage of population suffering from pollution- Percentage of households having safe or resultant water-borne diseases potable water source located within 200 Pollution load per capita per day meters of the dwelling Efficiency Efficiency Cost of installing water harvesting equipment Average cost, per kilolitre, of waste water treatment (per kilo litre) Cost per person treated in hospitals by pollution- Cost per metered household resultant diseases 25
  27. 27. Adapted from "Bangalore City Indicators Programme." (December 2000). Government of Karnataka, Bangalore MetropolitanRegion Development Authority CATEGORIZE PERFORMANCE INDICATORSIt is good practice for a municipality and its agencies to categorize each of its indicators by suchcategories as those given above. This will help users of the performance information keep in mind therelative importance to the city and its citizens of the individual indicators.Input, output, and efficiency indicators are relatively familiar to program managers. Governmentsregularly use them to track program expenditures and service provided. Indicators of outcomes are much rarer even thoughthey are more helpful in determining the consequences or results of the program.Categories of performance indicators are described below, and examples are shown in Exhibit 2-7. It is important for you to recognize the differences between the following categories of information: Inputs Input data indicate the amount of resources (amount of expenditures and amount of personnel used in delivering a service. Outputs Output data show the quantity of work activity completed. A programs outputs are expected to lead to desired outcomes, but outputs do not by themselves tell you anything about the outcomes of the work done. To help identify outcomes that you should track, you should ask yourself what result you expect from a programs outputs. Outcomes (intermediate and end outcomes) Outcomes do not indicate the quantity of service provided, but the results and accomplishments of those services. Outcomes provide information on events, occurrences, conditions, or changes in attitudes and behavior (intermediate outcomes) that indicate progress toward achievement of the objectives of the program (end outcomes). Outcomes happen to groups of customers (e.g., students or elderly persons) or to other organizations (e.g., individual schools and/or businesses) who are affected by the program or whose satisfaction the government wishes to attain. Efficiency and Productivity These categories relate the amount of input to the amount of output (or outcome). Traditionally, the ratio of the amount of input to the amount of output (or outcome) is labeled "efficiency." The inverse, which is the ratio of the amount of output (or outcome) to the amount of input, is labeled "productivity." These are equivalent numbers. Exhibit 2-7. Examples of Performance Indicators Input Number of positions required for a program 26
  28. 28. Cost Supplies used Equipment needed Output Number of classes held Number of projects completed Number of people served Number of letters answered Number of applications processed Number of inspections made Outcome Crime rate Employment rate Incidence of disease Average student test scores Percent of youth graduating from high school Number of successful rehabilitations Number of traffic accidents Efficiency Cost per kilometer of road repaired (output based) Cost per million gallons of drinking water delivered to customers (output based) Cost per number of school buildings that were improved from "poor" to "good" condition (outcome based)Exhibit 2-8 contrasts output and outcome indicators for specific services or activities. Exhibit 2-8. Contrast Between Output and Outcome Indicators Output Indicators Outcome Indicators 1. Number of clients served. 1. Clients whose situation improved. 2. Lane kilometers of road repaired. 2. Percentage of lane kilometers in good condition. 3. Number of training programs held. 3. Number of trainees who were helped by the program. 4. Number of crimes investigated. 4. Conviction rates of serious crimes, and crime rate. 5. Number of calls answered. 5. Number of calls that led to an adequate response.As a summary of selecting performance indicators, Exhibit 2-9 provides an example of Objectives,Outcomes, and Indicators for a road maintenance program. The example also provides targets forimproving performance. Targets will be addressed later in this manual, in Step 5, Data Analysis. Exhibit 2-9. Example of an Objective, Outcomes, Indicators, and Targets Road Maintenance ProgramObjective:Provide safe, rideable roads to the citizens, by regular renovation and maintenance of existing roads andby upgrading of any unpaved roads in the municipality. 27
  29. 29. Outcomes:(1) Maintain municipalitys road surface in good, better, or excellent condition.(2) Reduce traffic injuries or deaths by improving the condition and clarity of road signs.Indicators for Outcome (1):Input: cost of paving the road, personnel, equipment; amount of equipment used.Output: kilometers of road paved; number of households having paved roads.Outcome: kilometers of road surface in good or excellent condition; percent of citizens satisfied with roadconditions.Efficiency: cost per kilometer of road paved; cost per kilometer of road in excellent condition.Indicators for Outcome (2):Input: cost of new road signs, personnel costs.Output: number of road signs improved; number of new road signs installed.Outcome: traffic injuries or deaths; road signs in good or excellent condition.Efficiency: cost per new or improved road signs.Target for Outcome (1):Ensure that 90 percent of the road surface is in good or excellent condition.Target for Outcome (2):Reduce traffic injuries or deaths during the year by 10% through improved road condition and clarity ofroad signs.An important element of selecting performance indicators is to define each indicator thoroughly so thatmeasurements will be made in a consistent way by different personnel and over time. For example in the abovemeasurement of roads in various conditions, The municipality agency needs to define how todetermine whether a meter of road is in excellent, good, fair, or poor condition.SELECT OUTCOME INDICATOR BREAKOUTS (DISAGGREGATION) OF EACH OUTCOME INDICATORBY KEY CHARACTERISTICSYour municipality and your agencies will find the outcome information considerably more useful formaking improvement if you breakout the outcome data by key customer and service characteristics. This will muchbetter enable users of the data to identify more precisely where problems, and successfulpractices, are present. Consider breaking our the outcome data into categories such as the following:• By geographical location;• By organizational unit/project;• By customer characteristics;• By degree of difficulty (in carrying out the task in question); and• By type of process or procedure you use to deliver the service.Each of these recommendations for indicator breakouts is discussed below.By geographical location Break out data by district, neighborhood, etc. The presentation of data by geographical area gives users information about where service outcomes are doing well and where they are not. 28
  30. 30. Exhibit 2-10 shows the percentage of respondents who rated the cleanliness of their neighborhood in Püspökladány, Hungary, as very clean and somewhat clean. Overall (for the entire city), 45 percent of respondents stated their neighborhood was very clean or somewhat clean. However, when you break up responses geographically (by districts) you begin to see interesting variation. While most of the districts got a similar rating on neighborhood cleanliness, only 26 percent of respondents in district 1 rated their neighborhood as very clean or somewhat clean. This shows that district 1 is a problem area, and the city needs to examine why residents in that district rated cleanliness so low. (Note: The seven districts in the city were categorized based on socioeconomic conditions. Respondents were asked, "How would you rate the cleanliness of the neighborhood you reside in from 1 to 5, where 1 is very dirty, and 5 is very clean?").By organizational unit/project Separate outcome information on individual supervisory units is much more useful than information on several projects lumped together. For example, it is useful to have separate performance information on each public works departmental unit, not only for all the units together. Another useful application of breakouts by organizational unit would be to have separate performance information on the different units of the police department. For example, response times could be examined for individual units that specialize in particular crimes or other emergencies.By customer characteristics Breakouts by categories of customers (e.g., age, gender, education) can be very useful in highlighting categories of customer services that are or are not achieving desired outcomes. For example, if the government finds that the daytime hours of operation for reporting a problem with city services are too limited, the government may consider opening a hotline in the evenings for citizens to contact them who otherwise are not able to call during the day. For another example, park staff may find that they have put too much effort into satisfying parents with children and that their parks are lacking facilities that the elderly can enjoy.By degree of difficulty All programs have tasks that vary in difficulty. A more difficult program will have a harder time achieving the results you desire, and therefore distinguishing the degree of difficulty of a program will drastically change your perception of its outcomes. To show good performance an organization is sometimes tempted to attract easier- to-help customers, while discouraging service to more difficult (and more expensive) customers. Reporting breakouts by difficulty will eliminate this temptation. Exhibit 2-11 gives an example of considering the difficulty factor in presenting performance information.By type of process or procedure you use to deliver the service Presenting performance information by the type and magnitude of activities or projects being supported by the program is very useful for you. For example, a street cleaning program can comprise sweepers, garbage cans and dumpsters, and garbage trucks. You should present data on each project in the program by (1) the type and amount of each activity; and (2) the indicators resulting from each projects efforts. 29
  31. 31. Exhibit 2-10.Geographic Location Breakout 30
  32. 32. Exhibit 2-11. Workload (Client) Difficulty Breakout Unit No. 1 Unit No. 2 Total Clients 500 500 Number Helped 300 235 Percent Helped 60% 47% Difficult Cases 100 300 Number Helped 0 75 Percent Helped 0% 25% Non-Difficult Cases 400 200 Number Helped 300 160 Percent Helped 75% 80%Note: If you only looked at aggregate outcomes for Units 1 and 2 together, you would unfairly evaluate Unit 2, which hada higher proportion of difficult cases.You can use breakouts for purposes such as the following: • To help pinpoint where problems exist as a first step toward identifying corrective action; • As a starting point for identifying "best practices" that might be disseminated to other program areas, by identifying where especially good outcomes have been occurring; and • As a way to assess the equity with which services have been serving specific population groupsA summary checklist of these breakout categories is given in Exhibit 2-12. Exhibit 2-12. Possible Client and Service Characteristic BreakoutsService CharacteristicsGender Examine outcomes for men and women separately.Age Examine outcomes for different age ranges. Depending on the program, the age groups might span a large range of ages (such as examining clients under 21, between 21 and 59, and 60 and older), or the program might focus on a much smaller age range (such as youth programs wanting to compare outcomes for youth under 12, 13-14, 15-16, and 17 or older)Race/Ethnicity Examine outcomes for clients based on race/ethnicity.Disability Examine outcomes based on client disability. For example, some programs might want to determine whether clients with disabilities rate services differently than those without disabilities, as well as the outcomes for clients with various types of disabilities.Educational level Examine outcomes for each client based on the educational level achieved before 31
  33. 33. starting service.Income Examine outcomes for clients grouped into specific income ranges based on the latest annual household income at the time clients began service.Household Examine outcomes for households of various sizes, generations, and number of children.Difficulty of Examine outcomes by incoming statute based on expected difficulty in being able toproblem at intake help the client. Inevitably, some clients are more difficult t help than others. For example, an employment program might want to consider the literacy level of its new clients. An adoption program might want to relate outcomes to the age and health of the children.Service CharacteristicsFacility/Office Examine outcomes for individual facilities or offices.Service provider Examine outcomes for clients of individual service providers, such as caseworkers.Type of Examine outcomes for clients who were served using each distinct procedure. Forprocedure example, a youth program might have used workshops, field trips, classes, etc.Amount of Examine outcomes for clients who received varying amounts of service. This mightservice be expressed as number of sessions a client attended, the number of hours of service provided each client, or whatever level of service measurement the program uses.Source: Analyzing Outcome Information: Getting the Most from Data, The Urban Institute, 2004. 32
  34. 34. Step 3. SELECT DATA COLLECTION SOURCES AND PROCEDURESIDENTIFY A COLLECTION METHODA performance indicator is not very useful until a feasible data collection method has been identified. ForMDG indicators, a United Nations publication (2003) provides general suggestions for data collection and sources. (SeeAppendix B for a list of the MDG indicators likely to be directly applicable to municipalities, and theirdata sources.) However, your municipality will need to work out the datacollection procedure details. For the MDG indicator used as an example in Step 2, "Proportion of the population with access to an improved water source," the UN report notes that the usual sources have been administrative records on facilities and surveys of households. (It states that the "evidence suggests that data from surveys are more reliable than administrative records and…provide information on facilities actually used by the population.) We note that another possible data collection procedure is the use of trained observer rating procedures to help determine what is available to households.Data for most of the MDG indicators are obtained from national census and survey (usually conductedevery two to five years) or agency records from national line ministries. In some cases data are alsocomputed directly by the countries National Statistical Offices, the World Bank or UNESCO Institute for Statistics. Inaddition to these national surveys or agency records you will need to track disaggregatedvalues for the indicators for your municipality.There are four primary sources of performance data: • Agency records • Surveys of citizens • Ratings by trained observers • Use of special measuring equipmentIn this guide we discuss the first three in some detail below.Several factors will affect your decisions of which sources to use for which indicators: • How applicable is it to the information you seek? (For instance, outcome information such as citizen satisfaction or ratings of service quality can only be obtained from surveys of citizens.) • What is the availability of sources from which you can obtain the information? • How much time and resources would it take to regularly collect the data? • What is the likelihood that reasonably accurate data can be obtained from the procedure? 33
  35. 35. USE AGENCY RECORDSExamples of performance data obtainable from agency records (sometimes call "administrative records")include the following (some of these records will be available locally, others from the nationalgovernment): • Incidence of illnesses and deaths in a hospital Tracking Citizen Calls and (end outcome indicator) Response Times in Indjija, Serbia • Results of test scores in schools (end outcome The key feature of Indjijas Sistem48 is a call center indicator) for citizens to make complaints, comments, or • Total percent of owed fees collected requests concerning any local government service. (intermediate outcome indicator) After the call is received, several things take place: • Number of complaints received (intermediate • Callers are guaranteed a response within 48 hours outcome indicator) • The complaint or request is forwarded to the • Percent of time equipment is operational (such as service in question for resolution equipment for street cleaning or public transit vehicles • Data about the call is logged and reported, (internal intermediate outcome including: indicator) — Time of call • Response time to respond to citizen requests for — Content Length tion a service, such as to determine eligibility for a Data on — calls areof time until resoluMayor in bi- the reviewed by the public welfare benefit, to obtain a business weekly meetings with the departments. Receiving permit, to receive emergency medical attention, and recordfing citizen calls provides anasimponrtangt measure o citizen satisfaction, as well poi tin etc. (intermediate outcome indicator) to specific areas of particular concern. • Cost per kilomet er of road maint ained (efficiency indicator) • Size of workload, for example number of buildings needing inspection or number of kilometers of street that need to be repaired (used for calculating outcome and efficiency indicator values) Why is it useful to use agency records?The advantages of using agency records as data sources are their availability, low cost, and programpersonnels familiarity with the procedures. Since agency record data are already collected and available, this has been themajor source of performance data used by local governments. This information can,thus, form the starting point for your performance measurement system.For some performance indicators, an agency might need to obtain information from another municipalagency or even from another level of government. For example, one of your public welfare agenciesmight need record data from the health department to process an application for disability benefits. Can you use existing processes?For performance measurement purposes, however, your agencies are likely to need to modify theirexisting processes. For example, you may have to modify the forms and procedures to enable you tocalculate service response times. This involves:
  36. 36. • Recording the time of receipt of a request for service; 34

×