• Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,181
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Measuring Performance July 2008
  • 2. EFFICIENCY UNIT i VISION AND MISSION Vision stateMent To be the preferred consulting partner for all government bureaux and departments and to advance the delivery of world-class public services to the people of Hong Kong. Mission stateMent To provide strategic and implementable solutions to all our clients as they seek to deliver people-based government services. We do this by combining our extensive understanding of policies, our specialised knowledge and our broad contacts and linkages throughout the Government and the private sector. In doing this, we join our clients in contributing to the advancement of the community while also providing a fulfilling career for all members of our team. This brief was researched and authored by the Research Division, Institute of Public Administration, Ireland (www.ipa.ie/research). The Research Division provides applied research services for policy makers in a wide range of public service organisations, drawing on an extensive network of contacts and experience gained over more than thirty years. other efficiency Unit DocUMents The Efficiency Unit has produced a number of guides on good practice on a wide range of areas, including outsourcing and contract management. These may be found on the Efficiency Unit website at www.eu.gov.hk. Measuring Performance
  • 3. Foreword We have all heard of the consultant jargon which states that ‘what gets measured gets done’. But quite often, establishing meaningful performance targets and measures ii is not a straightforward task. As public managers we believe we are diligently measuring the performance of our organisations every day. Oversight agencies also keep us on our toes. But how good are we at this task? Performance measurement is an integral part of the process of delivering services to the community. We have been practising the target-based performance measurement system for over a decade and we hold ourselves accountable over the use of financial resources through the Controlling Officers’ Report. But overseas’ literature and experience show that performance measurement is an ongoing pursuit not only by the private sector but also by governments. To ensure that we are in tune with the public at all times, it is of paramount importance that we continue to adapt and perfect our performance measures so as to reflect the desired social outcomes that we pledge to achieve. We know that performance measurement in the public sector is a much more complex task than that in the private sector. This report resonates with many of our beliefs. It also recognises the fact that it takes time and resources to compile the necessary indicators to construe whether a social outcome is achieved or not. Nevertheless, governments in different parts of the world are fine-tuning their systems to make them more outcome/output-based than input-based. Setting appropriate performance measures, establishing challenging and yet achievable targets, and defining results – these are effective means of focusing our combined efforts to adhere to social objectives when delivering public services. We should always remind ourselves to measure the right things, and not just the easy ones. Increasingly, governments worldwide are using performance measures and targets, following consultation with departmental stakeholders. Experience overseas has yielded mixed results. I hope that this report will assist departments in identifying the most appropriate opportunities for improving their own regimes. Comments, feedback and sharing are most welcome. Head, Efficiency Unit July 2008 Measuring Performance
  • 4. contents 1 execUtiVe sUMMary 2 1. PerforMance MeasUreMent: Key challenges 6 2. iMProVing PerforMance MeasUreMent in BUsiness Planning 7 3. enhancing the MeasUreMent of Policy oUtcoMes 19 4. MeasUring PUBlic sector ProDUctiVity 27 5. ensUring that PerforMance MeasUres are UseD anD UsefUl 31 enDnotes 42 references 44
  • 5. Executive Summary A key challenge nowadays is to focus and interests of clients, customers and 2 measurement systems on results. citizens. Business plans should not be Government policies and programmes internal-looking documents that focus deliver two main types of results: solely on what managers and staff see as outputs (the direct products and services important. Measures should be developed produced) and outcomes (the that address the critical issues surfaced by consequences of those outputs for citizens as part of the planning process. society). In terms of this overall focus on results, four main measurement Having determined a set of measures for challenges are addressed in this report. inclusion in the business planning process, These are: improving performance it is useful to check the quality of the measurement in business planning; proposed performance measures. Good enhancing the measurement of quality measures are needed if they are policy outcomes; measuring public to be useful. Various sets of criteria have sector productivity; and ensuring that been developed for assessing the quality performance measures are used and of performance measures. useful. Business plans and associated performance iMProVing PerforMance measures are an important part of the MeasUreMent in BUsiness public accountability process for the Planning use of public funds. In this accountability context, some measures may be used Nowadays, business planning is a properly for individual or programme vital element in most public service accountability purposes, while other organisations. An important first step measures may be misleading if used in developing performance measures for such accountability purposes but for business plans is to get a clear nevertheless be important for the understanding of the range of overall management of the programme performance issues associated with the or activity. Both intermediate and final area under scrutiny. Public organisations outcome measures, for example, are in many countries are now using beyond the direct control of staff working the logic model approach to help in the area. Nevertheless, these outcomes structure thinking about developing should be reported as part of the business a range of performance measures planning process. But they should be seen covering inputs, outputs and outcomes. as contributing to giving an account on performance rather than being used to By focusing particular attention on hold staff to account. outcomes, measures are more likely to be developed that address the needs Measuring Performance
  • 6. EXECUTIVE SUMMARY enhancing the MeasUreMent addressing time-lag and attribution of Policy oUtcoMes issues 3 Whole-of-government and cross- Time-lag and attribution problems are cutting policy outcome measures significant when developing outcome- oriented performance measures. Time-lag Some governments, at both national and issues refer to the fact that it can often be local levels, have developed measures many years before the full effects of a policy that have a high-level focus on the final are seen in practice. Attribution issues refer outcomes of policies concerning social, to the difficulty in attributing causation economic and environmental issues. with regard to policy outcomes to particular These measures do not enable tracking of interventions. Changes in infant mortality individual agency performance, but they rates, to take one example, are influenced do provide greater clarity on desired high- by a myriad of factors, some associated level outcomes, and set a context within with health-care practices, some with social which individual agency performance and economic programmes, and some measures are developed. with educational programmes. Identifying the precise contribution of a particular At both the whole-of-government and programme or agency to such final sectoral levels, there is increasing attention outcomes is often, in practice, impossible. on the need to develop measures that cut There are no easy answers to these across organisational boundaries and focus challenges. But the separation of outcomes on the social and economic outcomes into intermediate and final outcomes, and desired by citizens. The interest here is the measurement of each, offers one way on measures that track and encourage forward for managers. Another approach joined-up thinking and practice on the part to dealing with attribution is to use impact of government organisations. For example, evaluation. policy units in government departments often have responsibility for developing and outlining the intermediate and final outcome objectives for the areas where they have policy responsibility. While it may be the task of other delivery agencies to develop measures and collect information on performance against the objectives, the department should report on performance against these measures as part of its overall policy responsibility. Measuring Performance
  • 7. EXECUTIVE SUMMARY MeasUring PUBlic sector the performance of employees and are 4 ProDUctiVity associated, in many instances, with higher levels of performance than either no, or The evidence from studies of public general ‘do your best’, targets. But how do sector productivity measurement is that we know when a ‘good’ target has been productivity measurement is still in its set? What might a challenging yet realistic early stages. Despite efforts going back target look like? The identification and use to the 1980s, the productivity measures of baseline data can be very helpful in this being produced need to be interpreted context when setting targets. cautiously. There is also the danger that over-simplistic use of the measures can engaging citizens in the process of lead to perverse consequences. It is clear performance measurement that no single figure of productivity can be used for public sector activities, unless It is vitally important to develop measures there is clear and widespread agreement of issues that citizens want measured in that it is an appropriate measure. A range performance measurement systems. The of supporting information is needed to question is not whether the customer measure productivity change. should be engaged in performance measurement, but how best to engage the ensUring that PerforMance public. Two issues of central importance MeasUres are UseD anD are: means of engaging the public and the UsefUl reporting of performance measures to the public. Performance measures need to be seen in a wider management context if they are Methods of engaging the public include: to be used and useful. Among the main focus groups; neighbourhood meetings; issues that need to be addressed when citizen satisfaction surveys; report cards for placing performance measures in this programme users; web-based discussion wider context are: linking measures and forums; and web-based surveys. Each targets; engaging citizens in the process method of engagement has advantages of performance measurement; and using and disadvantages, and may vary in terms incentives and sanctions to encourage the of cost. The use of a diversity of approaches use of performance measures. offers a good way forward. linking measures and targets Also, the way in which performance measures are reported back to citizens Targets are an important element in making affects the degree of engagement of performance measures useful. Good the public and ultimately the usefulness targets can lead to enhanced performance. of many performance measures. Issues Challenging and specific targets improve such as the provision of measures on Measuring Performance
  • 8. EXECUTIVE SUMMARY a neighbourhood basis and the use of comparative performance benchmarks 5 can be helpful here. Using incentives and sanctions to encourage the use of performance measures Performance measures are rarely used simply on the basis that they have been produced. Performance measurement is not just a technical exercise; it has strong cultural and political components. In such an environment, incentives and sanctions (carrots and sticks) can play an important role in encouraging the use of performance measures. The commonly used incentives and sanctions are: legislation; review; award/recognition; assurance; and involvement/feedback. Measuring Performance
  • 9. 1. Performance Measurement: Key Challenges A key challenge nowadays for managers is To overcome such difficulties, a results 6 to focus measurement systems on results. and measurement-oriented culture is The public and politicians are increasingly needed, involving managers and staff concerned with what is being delivered throughout government. In a supportive for the public money being used to fund culture, performance measures1 can help government programmes. They want tell the performance story of a programme to know that the money is being spent or organisation. to achieve the purposes for which it was allocated, but also and equally importantly In terms of this overall focus on results, that the money is being spent wisely. Hence and to address the issues raised above, the concern with results. Government four main measurement challenges are policies and programmes deliver two addressed in this report. These are: main types of results: outputs (the direct � Improving performance products and services produced) and measurement in business planning outcomes (the consequences of those • Enhancing the measurement of outputs for society). policy outcomes � Measuring public sector productivity But a focus on outcomes, while very � Ensuring that performance measures important, can also be very difficult. It is are used and useful often hard to connect the performance of programmes and managers with A particular emphasis is placed on distant and complex outcomes. There addressing the needs of the citizens. is a risk of measurement becoming a Ultimately, we need to be sure that we paper-based exercise that does not really are measuring the results that the citizens change practice. People may fall back on want to have measured. measuring what is easiest to measure, which may not be the most important issue from the viewpoint of the citizens. Goal displacement may occur, where measures themselves become the objectives, taking attention away from what the programme should be delivering. There is also often a tendency to fall back on input measures, as these are familiar, even though they tell us only about efficiency, not effectiveness. Measuring Performance
  • 10. 2. Improving Performance Measurement in Business Planning Public sector organisations are increasingly as the results chain. At the start of the chain using performance measures as part of are the outputs, results that managers can 7 their business planning process. Divisional largely control, e.g. drafting of legislation or and work unit business plans are now a giving of grants. At the end of the chain are common feature in many government the final outcomes, the end results sought, agencies. In this section, issues that are of e.g. improvements to health, the economy particular relevance to the development or the environment. And in-between is a of performance measures for business sequence of intermediate outcomes that planning are outlined. First, steps to are intended to lead to the final outcomes, be taken and issues to consider in the e.g. changes in people’s attitudes and development of performance measures behaviours. for business plans are presented. Second, methods of assuring the quality of The United Way of America (1996) notes: performance measures are reviewed. Third, “A programme logic model is a description the use of measures for management of how the programme theoretically and accountability purposes is discussed. works to achieve benefits for participants. And finally, a specific current issue for the It is the ‘if-then’ sequence of changes that day-to-day business of many organisations the programme intends to set in motion – assessment of the performance of through its inputs, activities and outputs. government websites – is examined. Logic models are useful frameworks for examining outcomes. They help you DeVeloPing PerforMance think through the steps of participants’ MeasUres for BUsiness Plans progress and develop a realistic picture of what your programme can expect to Using the logic model approach to accomplish for participants. They also help identify performance measures you in identifying the key programme components that must be tracked to assess An important first step in developing the programme’s effectiveness." performance measures is to get a structured understanding of the range of performance issues associated with the programme2 under scrutiny. Public organisations in many countries are now using the logic model approach to help structure thinking about developing a range of performance measures covering inputs, outputs and outcomes. Figure 1 shows the results spectrum produced from a logic model. This is sometimes referred to Measuring Performance
  • 11. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING figUre 1 logic MoDel resUlts chain 8 Outputs (goods and services produced by programme) Intermediate outcomes (immediate effects of the outputs) Intermediate outcomes (medium-term effects of the outputs) Final outcomes (the final or long-term consequences) An illustrative example of measures derived and management of the organisation and from using a logic model approach is given service users all have their own information in Figure 2 – evidential breath testing of needs. The incentive to develop a balanced drivers to improve road safety. set of performance measures, incorporating both financial and non-financial measures, choosing measures that meet stems from trying to meet the needs of stakeholder needs, particularly these different stakeholders. Initiatives citizens’ needs such as the logic model and the balanced scorecard (Kaplan and Norton, 1993) have An important issue when developing spread through many private and public performance measures for business organisations in an effort to ensure that planning is to consider the needs of a performance measures meet the needs of range of stakeholders when selecting various stakeholders. measures for business plans. For example, politicians, central finance officials, staff Measuring Performance
  • 12. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING figUre 2 Using the logic MoDel to DeVeloP PerforMance MeasUres: eViDential Breath testing 9 input objective To ensure that programme and administrative expenditure levels are contained as agreed input measures � Programme expenditure • Recurrent • Capital � Administrative expenditure • Pay related • Non-pay related output objective To test drivers for evidence of alcohol levels output measures � Number of drivers tested intermediate outcome objectives: To change the attitudes and behaviour of drivers towards alcohol consumption To reduce numbers tested who are over the legal limit To reduce the number of convictions for drink driving offences intermediate outcome measures � Survey findings of attitude and behaviour change � Numbers tested over the legal limit � Number of convictions for drink driving offences final outcome objectives: To reduce alcohol-related road accidents To improve road safety final outcome measures � Number of alcohol-related road accidents � Number of deaths and injuries on the roads linked to alcohol Measuring Performance
  • 13. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING A key stakeholder challenge for managers What is needed is some means of involving is to select performance measures for their citizens in the selection of the measures 10 business plans that measure the issues used in business plans. For example, the and results that service users and citizens Oregon Progress Board established an want to see measured. Ho (2007, http:// eight-member advisory committee to give www.businessofgovernment.org/pdfs/ state agencies a better sense of how citizens HoReport.pdf ) suggests that this citizen view the state’s current performance focus is often missing in the traditional measurement system. To do so, four approach to performance measurement state agencies offered their most recent development. In the traditional approach, annual performance measure reports for managers start by asking themselves and members of the advisory committee to their staff what their vision and objectives read and respond to. This led to initiatives are. They then develop performance such as more use of comparative data to measures on the basis of this work. Ho benchmark performance measures. identifies four main problems with this (See http://www.oregon.gov/DAS/OPB/ approach: docs/kpm/NCCI/NCCIfinal.doc for more � It assumes managers are capable of details.) establishing programme vision and objectives that reflect public concerns A more detailed case of good practice in and priorities. citizen involvement is given in the case � Managers may be trapped by their study of the Des Moines (USA) citizen- own blind spots and fail to see beyond initiated performance assessment project. what they routinely do in programme delivery and planning. And the public The Des Moines project was considered may view the measures as a tool to a success as it led to the development showcase the achievements of the of performance measures that were administration. actively used by officials but also were of � The traditional approach assumes that direct interest to citizens. The measures performance measures by their own addressed issues that were of concern to weight can influence how elected citizens and were identified by them as officials think about effectiveness and very important. But there are limitations resource allocation. But in practice to this partnership approach. In the Des politicians may not pay much Moines case, the president of Des Moines attention to measures if they do not Neighbors (an umbrella organisation for believe that the measures reflect their neighbourhood groups) and the city constituencies’ concerns and priorities. manager both stepped down within � Managers may focus on their own a short period of each other. With the priorities rather than on how they can loss of two key supporters the project best collaborate with other agencies lost momentum in 2005. An annual and the public to deliver results. This performance report continues to be issued, can emphasise a ‘silo’ mentality rather but the civic engagement process has than a ‘joined up’ perspective. not been maintained. This illustrates that maintaining such a partnership approach requires continuity of commitment, and that the loss of key individuals who drive the process may break the continuity. Measuring Performance
  • 14. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING case stUDy 11 Des Moines citizen-initiateD PerforMance assessMent Project Des Moines, Iowa, has a population of about 200,000 people. Between 2001 and 2004 the city implemented a citizen-initiated performance assessment (CIPA) project. Public officials and neighbourhood representatives jointly developed performance measures. The process went through several stages: Topic selection – Participants wanted to measure the quality of life in the city and what the city had done to improve the well-being of residents. Under the broad heading of nuisance control, a small number of key issues emerged from the process, e.g. odour control, noise pollution, traffic law enforcement and clearance of abandoned housing. Measure selection – For each topic, critical issues were identified and measures developed to track these critical issues, as illustrated below: toPic critical issUe PerforMance MeasUres odour Where are the problems? Number of complaints received control - by neighbourhood Are citizens satisfied with the air Satisfaction level of surveyed quality and the current level of citizens - by neighbourhood odour control? Satisfaction of city response Level of satisfaction with the after a complaint is filed service received after a complaint was made a month previously Data collection – Public officials tracked nuisance complaints over time and by location. They also conducted sample surveys of residents’ satisfaction with city services and citizens’ perception of quality of life in their residential area. Citizens also directly participated in digital surveys in selected neighbourhoods, using handheld electronic devices to document problems such as potholes and graffiti. Engagement of citizens and other stakeholders in reporting – Performance measurement results were actively disseminated to the city council and directly to the public. Source: Ho, 2007, p.18-23. Performance reports can be accessed at http://www.ci.des-moines.ia.us performancereport.htm Measuring Performance
  • 15. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING More generally, while having significant assUring the QUality of 12 benefits, involving citizens in the PerforMance MeasUres UseD development of performance measures in BUsiness Planning can be a challenging activity, and a number of issues need to be borne in mind if taking Having determined a set of measures for this approach: inclusion in the business planning process, � Managers must be committed to the it is useful to check the quality of the process and provide leadership to work proposed performance measures. Good with citizen groups. If this leadership is quality measures are needed if they are to not there, the process will not be taken be useful. Various sets of criteria have been seriously. developed for assessing the quality of � There are risks involved in engaging performance measures. Perhaps the most citizens in performance measurement. common is to check measures against Measures may reveal problems that the SMART criteria (specific, measurable, present challenges to managers and achievable, relevant and time-bound). elected officials. An illustrative example of this process is � There is a risk that citizen groups given in Table 1. In the Irish Department consulted may not be representative of of Agriculture and Food, performance the wider citizenry. Steps must be taken measures proposed by divisions for their to ensure that citizen engagement is business plans have been subject to a comprehensive and inclusive. quality analysis using SMART criteria from � Citizen engagement is resource- within the economic and planning unit. intensive, particularly in terms of The benefits of this process in terms of the time commitment required of improving the specificity of the measures management and staff. It is also and ensuring there is a link between demanding of citizens themselves. measures and targets can clearly be seen � There is no single citizen view of from this example. performance. More likely there will be a variety of views expressed as to the importance of measures selected and what they mean. Balancing and interpreting these divergent views presents challenges to managers. None of this is to suggest that citizen engagement in performance measurement is not important. But an awareness of the challenges and limitations can facilitate a more effective approach to such engagement. Measuring Performance
  • 16. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING taBle 1 exaMPles of PerforMance MeasUres analyseD Using the sMart criteria 13 Measure Specific Measurable Achievable Relevant Time- bound Level of Relates to In terms What is Relevant Not animal a specific of number identified to several identified, diseases item (animal of disease here is an objectives but disease levels indicator. in the perhaps levels) Target not statement not identified, of strategy applicable therefore achievability cannot be commented on Percentage Relates to a In terms of As above As above Time of specific item percentage criteria payments (payment within identified within deadline deadline (protocol protocol protocols) deadlines) deadlines Using MeasUres in BUsiness business plans, are an important part of Planning: DistingUishing the public accountability process for the BetWeen ManageMent anD use of public funds. In this accountability accoUntaBility PUrPoses context, it is important to recognise that measures may play different roles, Business planning is intended as part of a depending on the aspect of performance process of improved management within being measured. Some measures may be public service organisations. Business used properly for individual or programme plans are meant to ‘step down’ to the accountability purposes. Other measures divisional level the high-level goals and may be misleading if used for such objectives contained in strategic plans. As accountability purposes but nevertheless such, business plans, and consequently be important for the overall management the performance measures contained in of the programme or activity. This issue is Measuring Performance
  • 17. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING best illustrated by way of example, as set changes in these measures directly to the 14 out in Figure 3. programme. Figure 3 sets out performance objectives Both intermediate and final outcome and measures used to assess an indicators are beyond the direct control of anti-smoking television advertisement staff working in the area. But as Schacter campaign. Input, activity and output (2002) notes: ‘They are nevertheless things measures in this example can be used that you must monitor, because they have to help assess individual and team profound relevance to the design and the performance of staff working in this area. implementation of your program. These It is the responsibility of the staff working are the results that you are managing in the area to ensure that the campaign for, even if you can’t control them.’ These material is designed, tested and runs on outcomes should be reported on as part the television, within agreed resource of the business planning process. But they allocations. Staff can be held to account for should be seen as contributing to giving an this work. account on performance rather than being used to hold staff to account. Including The intermediate outcome objectives and outcome indicators such as these is measures – assessing whether people see important for the successful management the advertisements and if their attitudes of the programme. and behaviour are affected – are clearly beyond the direct responsibility of people working in the area. But these measures provide information about the results of the programme that can readily be directly attributed to the programme. It is possible to judge if the advertising campaign has been used and if it has had an effect on people’s attitudes and behaviour. As such, these measures can be used to assess programme performance and hence programme accountability. The final outcome measures – less smoking and lower incidence of smoking-related diseases – are affected by many other factors apart from the campaign itself. These measures are clearly relevant to judgements about programme performance, but it is not possible to directly attribute Measuring Performance
  • 18. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING figUre 3 Using MeasUres for Different accoUntaBility PUrPoses 15 inputs $$$ Full Time Equivalent Staff activities � Design and test campaign material Fair reflection performance of individual � Produce campaign material (Measure: material produced by dd/mm/yyyy) outputs � Advertisements run on television (Measure: number of schools receiving campaign material) intermediate � People see the advertisements outcomes (Measure: number of viewers) of programme Fair reflection performance � Peoples’ attitudes affected (Measure: degree of attitude change) � Behaviour affected (Measure: degree of behaviour change) final � Less smoking performance outcomes programme Relevant to (Measure: level of smoking) � Lower incidence of smoking-related diseases (Measure: incidence of smoking-related diseases) Source: adapted from Schacter (2002) Measuring Performance
  • 19. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING assessing the PerforMance What then is to be done to allow a more 16 of goVernMent WeBsites sophisticated measurement of website performance? A first step is to be clear A specific measurement issue, yet about the purposes and objectives of the one of growing prominence for many website. Is it there to enable people to fill organisations, is the measurement of the out a survey, use interactive applications performance of government websites. built into the site, download important This issue is often dealt with in the context information and so on? In the language of business planning for an organisation, of the logic model, what are the main when looking ahead to challenges to be inputs, outputs and outcomes required addressed in the coming year. Websites have of the website? Measures can then become a key channel of communication be developed to address these issues between governments and citizens. As (see Table 2, taken from Stowers, 2004, such they must address the needs of very http://www.businessofgovernment. wide-ranging and differing audiences. org/pdfs/8493_Stowers_Repor t.pdf Usability is crucial. But simply tracking measures such as the number of hits can The case study, on page 18, highlights be misleading. A site may get a lot of hits two initiatives to measure website yet visitors may stay on the site for only a performance, one from Australia and few seconds, suggesting they did not find one from the USA. Each develops a wide it useful once there. A site with a smaller variety of measures against which website number of hits may have people staying performance is judged. Comparative on for longer and returning frequently, assessment of the websites of different suggesting that it is meeting a need. government agencies is encouraged. Measuring Performance
  • 20. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING TABLE 2 POTENTIAL WEBSITE PERFORMANCE MEASURES 17 Input Measures Output Measures Outcome Measures Application development Number of hits Accessibility of services and hardware set-up or user contact Number of site pages Staff costs sessions meeting accessibility Other development Number of criteria costs downloads of Accuracy of the assistance Other vendor costs or information as documents measured by percentage Staff time for Time users spend accuracy rates in random application development on a site fact checking Other development time Number of times Adequacy of information Vendor time for transactions as measured by staff and development purposes completed, or citizen surveys the times online Ease of use as measured by forms have been pop-up or other surveys accessed and Citizen satisfaction with completed site Monetary amounts Service quality processed through Percentage of time when website is down and not each site available Webpage errors Efficiency Cost per transaction Total cost per user session Final outcomes Cost savings from e-government Staff time savings from e-government Measuring Performance
  • 21. IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING case stUDy 18 MethoDs for MeasUring WeBsite PerforMance assessing federal government websites – australia The Australian federal government commissioned a private company, UsabilityOne, to conduct an assessment of ten federal government websites. The company developed a usability compliance audit for evaluating websites. The audit involves 151 criteria, based on acknowledged usability principles and extensive user-testing experience. The main areas covered in the audit include navigation, content and content writing, design and graphics, search, error prevention and recovery, trust, internationalisation, window titles, news and press releases plus branding and company information. Two analysts independently reviewed each website against the criteria, and arrived at an overall compliance score for each website. Full details of the project can be accessed at http://www.usabilityone.com/. the american customer satisfaction index (acsi) methodology The ACSI was developed by researchers at the University of Michigan to evaluate customer satisfaction generally, and in 2001 ForeSee Results began to use the methodology to assess customer satisfaction with federal agency websites. ACSI, which uses a set of well- researched and benchmarked questions and research processes, uses a causal statistical model to predict customer behaviour, such as the probability of return visits to the website. Satisfaction with websites is examined using pop-up surveys. These results are then analysed using the model, which includes measures both of customer satisfaction and of customer loyalty. Results for individual websites can be compared to a nationally developed index based on more than 200 organisations. For more details see http://www.foreseeresults.com/ and Stowers, 2004, http://www.businessofgovernment.org/pdfs/8493_Stowers_Report.pdf, p.26. Measuring Performance
  • 22. 3. Enhancing the Measurement of Policy Outcomes The use of performance measures to These examples are given, not to suggest facilitate judgements about the efficiency that measures have no role in policy 19 and effectiveness of policy work presents work, but rather to caution about raising significant challenges. As Perrin (1998) expectations too highly in terms of the notes, a requirement to produce more role that measures may play. As part of policy-focused measures can lead to broader efforts to improve evidence on situations where there is the widespread performance, measures can be helpful development of easy-to-count measures for managers and staff engaged in policy which have little or no relationship to what work. the policy is supposed to be achieving. Peripheral activities that are easy to quantify Whole-of-goVernMent become the focus of reports that nobody anD cross-cUtting Policy uses. oUtcoMe MeasUres Such actions as described above arise Some governments, at both national and in part because of cultural and ‘political’ local levels, have developed measures reasons, but there are also inherent that have a high-level focus on the final technical difficulties in ‘measuring’ policy outcomes of policies concerning social, work. There are a number of particular economic and environmental issues. issues: These measures do not enable tracking of � Whole-of-government challenges, individual agency performance, but they such as multiple high-level outcomes, do provide greater clarity on desired high- which must be pursued simultaneously level outcomes, and set a context within and that sometimes are in conflict with which individual agency performance each other; or where policy is initiated measures are developed. Such high- in one organisation but responsibility level measurement systems have been for delivering on outcomes rests with developed because of a recognition of other agencies. the limitation of relying on economic � Time-lag problems, associated with measures: the fact that many years may elapse between the initiation of a policy and “Economic indicators have traditionally its implementation. been used to assess the economic ‘state � Attribution problems, where it of the state’. Strong economic growth, is impossible to disentangle the low inflation and unemployment were impact of a particular policy on final regarded as indicative of a healthy outcomes because these outcomes economic climate and believed to result are also affected by other policies and in prosperity for citizens. However, citizens influences. have become increasingly concerned about their relative quality of life, expressed in terms such as quality of education and Measuring Performance
  • 23. ENHANCING THE MEASUREMENT OF POLICY OUTCOMES health care, availability of recreational/ Two longstanding and relatively successful 20 cultural opportunities, clean environment, whole-of-government measurement and safety from crime. Accounting and systems are Oregon Shines and Alberta’s economic based measurement systems Measuring Up, examined in the case study were not designed to address these issues; below. thus, governments have introduced new systems for measuring progress, including policy outcome based performance measurement (Ogata and Goodkey, 1998).” case stUDy Whole-of-goVernMent oUtcoMe MeasUreMent oregon shines The state of Oregon, USA, adopted a development strategy in 1989 called Oregon Shines. The strategy addressed quality-of-life issues as well as economic diversification. Oregon Shines II was adopted in 1997 as a successor strategy, and Oregon Shines III is planned for 2009. Progress against the strategy is tracked through Oregon Benchmarks, an annual report which tracks over ninety measures against benchmark targets in seven categories: economy, education, civic engagement, social support, public safety, community development and environment. State agencies link their key performance measures to these benchmarks where it makes sense to do so. For details go to http://www.oregon. gov/DAS/OPB/docs/obm/New_Benchmark_Numbers.doc. Measuring Up, alberta In 1993 the province of Alberta, Canada, started a three-year strategic planning and performance measurement system. In 1995 the first annual Measuring Up report was published. Measuring Up contains sixty-five core performance measures related to fourteen government goals. These measures focus on issues such as infrastructure capacity, literacy and numeracy rates, crime rate and water quality. The intention is to develop outcome-based measures to provide information on progress towards long- term targets. The targets are selected and driven by political leadership, and through public consultation. State agencies link their business plan objectives to relevant priorities. For details go to http://www.treasuryboard.gov.ab.ca/1089.cfm Measuring Performance
  • 24. ENHANCING THE MEASUREMENT OF POLICY OUTCOMES At both the whole-of-government and Additionally, in a critique of earlier PSAs, sectoral levels, there is increasing attention Neely and Micheli (2004) note that there 21 on the need to develop measures that was little use in the public sector of what has cut across organisational boundaries and become best practice in the private sector that focus on the social and economic – developing success or strategy maps outcomes desired by citizens. The interest when designing measurement systems. here is on measures that track and Success/strategy maps are intended to encourage joined-up thinking and practice help people identify which are the critical on the part of government organisations. measures, and can be helpful when rationalising the number of measures in use. The UK government is acknowledged as a leader in the development of cross-cutting Another common cross-cutting situation targets and measures, in part through the is that of policy-oriented units operating development of cross-cutting Performance in government departments who have Service Agreements (PSAs) (see case study overall responsibility for policy but where below). It is too early as yet to judge the implementation of that policy is through success of this initiative. But it has been agencies operating under their control. In welcomed by the Select Committee on this instance, policy units may sometimes Treasury, though the Committee cautions not develop measures themselves, saying that the cross-departmental nature of the that it is the responsibility of the agencies new PSAs poses a challenge for a system to develop performance measures. The of accountability based on departmental argument used here, following on from the reporting and the work of departmental logic model approach, is that policy units select committees. They also stress that should be responsible for developing and the new agreements, to be effective, must outlining the programme theory including genuinely reduce the number of measures the intermediate and final outcome in use, and not simply bring together objectives. While it may then be the task of diverse topics within a PSA, and that there the relevant agencies to develop measures must be corresponding reductions in and collect information on performance measures down the delivery chain (see against the objectives, the department http://www.parliament.the-stationery- should report on performance against office.co.uk/pa/cm200607/cmselect/ these measures as part of its overall policy cmtreasy/279/27909.htm and http://www. responsibility. parliament.the-stationery-office.co.uk/pa/ cm200708/cmselect/cmtreasy/55/5507. htm). Measuring Performance
  • 25. ENHANCING THE MEASUREMENT OF POLICY OUTCOMES case stUDy 22 UK cross-DePartMental PUBlic serVice agreeMents In 1998, in the first Comprehensive Spending Review, the UK government introduced a new performance measurement framework based on Public Service Agreements (PSAs). While generally welcomed, these PSAs suffered from a number of drawbacks. The system was seen as top-heavy and unwieldy, with too many measures created at local level to feed into the system. Particular difficulties arose from targets held jointly between departments, and some issues that involved more than one department were not adequately captured. In the 2007 Comprehensive Spending review, the government announced a smaller suite of thirty new PSAs that outline the government’s highest priorities and span departmental boundaries. The agreements are cross-departmental, although there is a lead department for each agreement. Departments are required to produce a cross- departmental Delivery Agreement for each PSA, informed by consultation with the delivery chain. These Delivery Agreements set out how each PSA is to be delivered and the measures used to track performance against PSAs and their associated delivery. Agreements can be accessed via http://www.hm-treasury.gov.uk/pbr_csr/psa/pbr_ csr07_psaindex.cfm An example of this approach is set out number of plans developed, the number in Table 3. Here, in an Irish example, the of actions implemented from the plans focus is on the Local Development Social and the quality of the plans (for example Inclusion Programme (LDSIP), overseen the percentage of plans inspected that fall by the Department of Community, Rural below a determined quality standard). In and Gaeltacht Affairs. Implementation this case, the programme logic is that as a and administration of the programme result of developing high-quality plans, and is overseen by an agency ADM Ltd. on the subsequent implementation of these behalf of the Department. An intermediate plans, local groups contribute to sustained outcome objective for the LDSIP is that social and economic development in their local groups develop innovative plans communities, which is one of the final and programmes to improve the local outcome objectives of the community and social and economic infrastructure. local development programme. Readily applied measures here are the Measuring Performance
  • 26. ENHANCING THE MEASUREMENT OF POLICY OUTCOMES With regard to this final outcome objective overall responsibility for the programme, it of sustained social and economic is important that the Department use these 23 development in communities, Table 3 measures in its own reporting procedures. gives examples of measures gathered In this way, an overview is provided of by ADM Ltd. that can be used to track the outcomes of the community and progress. While ADM Ltd. has responsibility local development programme for public for developing and gathering the data on accountability purposes. these measures, as the policy body with taBle 3 coMMUnity anD local DeVeloPMent MeasUres interMeDiate oUtcoMes objective Development by local groups of innovative plans and programmes to improve the local social and economic infrastructure Measures � Number of plans developed � Quality of plans produced � Number of actions implemented final oUtcoMes objective Sustained social and economic development in communities, with a special focus on areas of disadvantage Measures � Number of people placed into jobs � Number of people supported into self-employment Measuring Performance
  • 27. ENHANCING THE MEASUREMENT OF POLICY OUTCOMES aDDressing tiMe-lag anD success of the programme solely on the 24 attriBUtion issUes basis of final outcome measures, though clearly they provide vital information to Time-lag and attribution problems are inform an overall judgement. Similarly, the significant when developing outcome- outputs of the programme – the number oriented performance measures. Time-lag and value of grants issued – do not on issues refer to the fact that it can often their own give a satisfactory picture of be many years before the full effects of performance. a policy are seen in practice. Attribution issues refer to the difficulty in attributing This is where intermediate outcome causation with regard to policy outcomes measures can help. The programme logic to particular interventions. Changes in is that as a result of receiving supports, infant mortality rates, to take one example, firms develop their research capability are influenced by a myriad of factors, some and improve their links with third-level associated with health-care practices, some research institutes, thus contributing to with social and economic programmes, the final outcome objectives. Measures and some with educational programmes. such as level of research skills before and To identify the precise contribution of after receiving the grant (assessed by a particular programme or agency to survey) and type and quality of links with such final outcomes is often, in practice, third-level research institutes can track impossible. these intermediate outcome objectives. Measuring the intermediate outcomes There are no easy answers to these can give a better sense of programme challenges. But the separation of outcomes performance. And if the programme logic into intermediate and final outcomes, is demonstrated to be faulty, a new theory and the measurement of each, offers one can be developed to address the issue. way forward for managers. Take as an The intermediate outcome measures help illustrative example the case of a research give a picture of the performance of the and development (R&D) grant scheme R&D grant scheme. Neither the output nor aimed at encouraging innovation and the final outcome measures alone could the development of new products. Final give such a picture. Intermediate outcome outcome measures, such as number and measures help flesh out the performance type of new products on the market, and story. turnover attributed to new products and services, may be influenced by factors other than grants given to firms to develop their R&D capabilities. The national and international economic situation, local educational initiatives and so on also have an impact. It is not possible to examine the Measuring Performance
  • 28. ENHANCING THE MEASUREMENT OF POLICY OUTCOMES Using impact evaluation (see case study below on increasing tax revenue in Pennsylvania). By knowing the 25 Another systematic means of developing actual impact of intervention towards the performance measures that address final outcomes, appropriate performance attribution and time-lag issues and that may measures can be designed. be useful in certain circumstances is impact evaluation. Impact evaluation is defined by the World Bank (2006) as an assessment of the impact of an intervention on final outcomes. It assesses the changes arising from an intervention that can be attributed to a particular project, programme or policy case stUDy increasing tax reVenUe in PennsylVania The Pennsylvania state government commissioned a large, randomised control trial to evaluate the effectiveness of various approaches to improving tax compliance by businesses that were late in paying their sales taxes. The trial randomly assigned 7,000 such businesses to receive one of seven letters, ranging from threatening to pleading, and made use of outcome data that the state had already collected for other purposes – namely, whether the businesses paid their taxes. The trial found that a letter containing a short (one-third page) statement that tax is due and that the business is liable produced significantly more tax revenue than the state’s existing letter (full-page, detailed letter with boxes that the businesses check to indicate why they have not paid the tax). The trial results indicated that the state’s use of the short letter for all late-paying businesses could generate US$6 million annually in increased revenue. Source: http://www.whitehouse.gov/omb/part/2004_program_eval.pdf, p.12 Measuring Performance
  • 29. ENHANCING THE MEASUREMENT OF POLICY OUTCOMES A key element in impact evaluation is Quasi-Experiments – Like RCTs, these 26 identifying the ‘counterfactual’: What would evaluations assess the differences that have happened had the intervention not result from an intervention and the result taken place? This involves the identification that would have occurred without the of a comparison group who are as alike intervention. However, the control group as possible to the group who receive the is not randomly assigned. Instead, it is intervention but who were not subject to designed on the basis that the evaluator the intervention. The most frequently used judges how to minimise any differences methods of identifying the counterfactual between the two groups, or it may be a are the following: pre-existing group. Use of comparison group studies does increase the risk of Randomised control trials (RCTs) misleading results because of the difficulty – An RCT is a study that measures an in eliminating bias in the selection of the intervention’s effect by randomly assigning control group. individuals (or other units, such as schools or hospitals) into an intervention group, The benefits and challenges of impact which receives the intervention, and into a evaluation are well described by the World control group, which does not. Following Bank Independent Evaluation Group the intervention, measurements are taken (http://www.worldbank.org/ieg/ecd/). to establish the difference between the intervention group and the control group. Because the control group simulates what would have happened if there were no intervention, the difference in outcomes between the groups demonstrates the final outcome or impact one would expect for the intervention. There are, however, many programmes for which it would not be possible to conduct an RCT. To carry out an RCT, there must be a possibility of creating a control group who will not receive the intervention. For practical, legal, and ethical reasons, this may not always be possible. Measuring Performance
  • 30. 4. Measuring Public Sector Productivity A number of initiatives have taken place papers on public sector productivity in several countries in recent years aimed measurement, e.g. education and health 27 specifically at improving the measurement sector productivity measurement (see of public sector productivity. The http://www.statistics.gov.uk/about/data/ experience of a small number of countries methodology/specific/PublicSector/ at the forefront of attempts to improve output/default.asp for details). the measurement of productivity is outlined here: the UK, Finland, Sweden and MeasUring PUBlic sector Australia. ProDUctiVity in finlanD MeasUring PUBlic sector Finland, along with the UK, is widely ProDUctiVity in the UK regarded as a world leader in public sector productivity measurement. In 1995 a Since 1988, the Office for National project was established to measure public Statistics has been progressively moving sector productivity in Finland. It was located away from the output=input approach in Statistics Finland, the national statistics to productivity, and incorporating direct office. The aim of the project was: ‘to measures of the volume of government develop a measurement and monitoring output in the national accounts. By 2005, system for government sector production these direct output estimates accounted and productivity by using an output for two-thirds of general government indicator method to measure the volume final consumption. In the context of of output’ (Niemi, 1998). In 1997 the scope this focus on output measurement, the of the project was expanded to include UK government commissioned Sir Tony the measurement of the productivity of Atkinson to undertake a review of the local government services. measurement of government output in the national accounts. This review (Atkinson, Under the terms of the project, for central 2005) provides a comprehensive overview government services the final output of developments and recommendations and the output measures are specified for future progress. by the agencies themselves. Examples of output measures are given in Table 4. The The UK government accepted the findings agencies for which input and output data and recommendations of the Atkinson are gathered cover about 80 per cent of review, and the Office for National Statistics the compensation of employees in central (ONS) has the lead role in taking forward government. Initial results show growth the recommendations. To this end, the rates of output and productivity varying ONS has set up the UK Centre for the extensively. Measurement of Government Activity (UKCeMGA). The UKCeMGA has issued a number of interesting reports and research Measuring Performance
  • 31. MEASURING PUBLIC SECTOR PRODUCTIVITY TABLE 4 EXAMPLES OF OUTPUT INDICATORS IN THE CENTRAL GOVERNMENT IN FINLAND 28 Consumer Ombudsman’s Office number of petitions to market court marketing instructions (number of ) contractual terms negotiated statements on legislative initiatives cases solved individually replies to written enquiries Courts number of cases settled National Food Administration number of letters guiding supervision number of administrative decisions and memos number of publications number of statements number of training events new instruction materials State Audit Office number of annual audits supplementary audits international audits expertise activities, statements Universities number of degrees completed (generally separated into graduate and postgraduate degrees) adult education and continuing education measured, for example, in days or number of courses (depending on the university) number of publications (research) Source: Niemi, 1998 Measuring Performance
  • 32. MEASURING PUBLIC SECTOR PRODUCTIVITY MeasUring PUBlic sector a 20 per cent increase in output. ProDUctiVity in sWeDen � To measure the productivity of an 29 agency as a whole means overheads Sweden has been measuring public sector must be taken into account. productivity since the mid-1980s. An Expert � Results differ depending on the kind of Group on Public Finances (a subcommittee price index chosen (Ministry of Finance, under the Ministry of Finance) established 1997). a steering group to conduct the work. The steering group was supported by Statistics MeasUring PUBlic sector Sweden, the national statistics office. ProDUctiVity in aUstralia Particular focus is given to the In 1993 the Council of Australian development of output measures for Governments established the Review of services. These include items such as the Government Service Provision to provide number of admitted patients for in-patient information on the effectiveness and medical care, the number of learning hours efficiency of government services for education, traffic volume as measured (Australian Productivity Commission, by vehicle kilometres for public roads, 2006). The review is conducted annually, number of flying hours for the air force. overseen by a steering committee of Attempts are made to adjust the quantity senior representatives from the central of outputs for quality variations where agencies of all the state governments, with data are available. A number of lessons the assistance of a secretariat provided by are drawn based on the experience of the Productivity Commission. Performance producing annual productivity measures information is provided on fourteen service over a period of time: areas covering six main government � Productivity varies greatly from year to functions: education; justice; emergency year. To assess any given year, a time management; health; community services; series of several years’ information is and housing. needed. � Most agencies have several categories The report includes performance of output. Attributing the same weight comparisons across jurisdictions for the to all categories may yield deceptive services, using a common method. Both results. In the enforcement service, for outputs and outcomes are measured, as example, counting cases dealt with well as efficiency, effectiveness and equity. regardless of category gives a 2 per Government funding per unit of output cent decrease in output from 1981 to delivered is typically used as a measure of 1992. If, however, different weights technical efficiency, e.g. recurrent funding are assigned to different categories of per annual curriculum hour for vocational output, to take into account differences education and training. Where there are in composition of the cases, the result is shortcomings in the data, other measures Measuring Performance
  • 33. MEASURING PUBLIC SECTOR PRODUCTIVITY of efficiency are used (including partial There is some evidence of output measures 30 productivity ratios such as staff level per produced under the various initiatives student in government schools, staff being used in a limited manner in resource per prisoner in corrective services and allocation decisions. The OECD (2007) has administrative costs as a proportion of reviewed the use of output measures in total expenditure in services for people the budgetary process. It finds that in most with a disability) (Australian Productivity sectors and cases, performance measures Commission, 2006). are loosely connected to decisions in the budgetary process. Output measures learning the lessons froM are often used by ministries of finance PUBlic sector ProDUctiVity in the budgetary process along with MeasUreMent initiatiVes other information on performance and on political priorities to inform budget The evidence from studies of public sector allocations. But the OECD warns that a productivity measurement is that the latter direct linkage, where results determine is still in its early stages. Despite efforts going funding, creates incentives for gaming back to the 1980s, the productivity measures the system, such as manipulating the being produced need to be interpreted data. Ministries of finance have taken a cautiously. There is also the danger that an cautious approach to using performance over-simplistic use of the measures could information to financially punish or reward lead to perverse consequences. It is clear agencies. When programmes show poor that no single figure of productivity can performance against outputs, the most be used for public sector activities, unless common course of action is that resources there is clear and widespread agreement are held constant and the programme that it is an appropriate measure. The reviewed during the course of the year. Atkinson (2005) recommendation that a range of supporting information is needed to measure productivity change, and not to rely solely on a single productivity measure, is one that should be applied generally. Measuring Performance
  • 34. 5. Ensuring that Performance Measures are Used and Useful Performance measures are not always “A jurisdiction or agency is employing a used, or used in ways that their designers PerformanceStat performance strategy if it 31 intended. Van Thiel and Leeuw (2002) coined holds an ongoing series of regular, frequent, the phrase the performance paradox. The periodic, integrated meetings during which performance paradox refers to the situation the chief executive and/or the principal where there is a weak correlation between members of the chief executive’s leadership performance measures and performance team plus the individual director (and the itself. Measures lose their value over time top managers) of different sub-units use and can no longer discriminate between data to analyse the unit’s past performance, good and bad performance. As a result, the to follow-up on previous decisions and relationship between actual and reported commitments to improve performance, to performance declines. establish its next performance objectives, and to examine the effectiveness of its Behn (2008) too makes the point that overall performance strategies (Behn, 2008, performance measures do not always of p.2).” themselves guarantee good performance. He examines the use of performance Behn identifies seven big mistakes that measures in structured settings, organisations often make in applying which he refers to as PerformanceStat. PerformanceStat approaches. These are PerformanceStat is Behn’s label for the outlined in Table 5. approach initiated in 1994 by the New York Police Department when they created The key message emerging from both Van CompStat, a strategy designed to reduce Thiel and Leeuw and Behn’s analyses is that the city’s crime rate. The approach was performance measures need to be seen in adopted by Baltimore and applied to a wider management context if they are the whole of city government, where it to be used and useful. Some of the main was labelled CitiStat.3 The approach has issues that need to be addressed when subsequently been applied in a number of placing performance measures in this other US cities and in Scotland. Behn uses wider context are: linking measures and the term PerformanceStat as a generic title targets; performance measurement and for such an approach, which he defines as citizen engagement; and using incentives follows: and sanctions. Measuring Performance
  • 35. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL taBle 5 32 seVen Big errors of the PerforMancestat aPProach to the aPPlication of PerforMance MeasUreMent Error No. 1: No clear purpose. In this case, the approach is applied as the latest ‘fad’ with little or no thought as to what results the organisation is trying to achieve or what better performance might look like. Error No. 2: No one has specific responsibilities. You need to be able to answer the question of who will be responsible for what. Error No. 3: The meetings are held irregularly, infrequently, or randomly. If meetings are not held regularly and frequently, they cannot provide good feedback on successes and failures. Error No. 4: No one person authorised to run the meetings. One person, with clear authority, needs to consistently run the meetings if they are to have an impact. Ideally, this should be the chief executive. Error No. 5: No dedicated analytic staff. PerformanceStat requires data for the measures used. For PerformanceStat to provide meaningful results, it needs a few analytical people working on it full-time. Error No. 6: No follow-up. Issues identified by the measures should be followed up from one meeting to the next, and not start over each time. Error No. 7: No balance between the brutal and the bland. Baltimore’s application of CitiStat is known for being tough and uncompromising with poor performers. Sometimes overly so. Others have tried to avoid this but in the process have achieved little more than presenting a picture of accomplishments. Balance is needed. Source: Behn, 2008, http://www.hks.harvard.edu/thebehnreport/Behn,%207PerformanceStatErrors.pdf Measuring Performance
  • 36. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL linking Measures and targets 2. Setting learning targets, as opposed to merely difficult-to-attain targets, may be most 33 Targets are an important element in making effective when tasks are complex. When tasks performance measures useful. Good are complex, challenging targets may be targets can lead to enhanced performance less helpful. In such situations, targets that (see local education authorities case encourage employees to explore strategies study below). In an extensive review of to tackle the task may improve performance. the literature on motivating employees, Perry, Mesch and Paarlberg (2006) 3. The target-performance relationship is identify three important propositions strongest when employees are committed to supported by evidence provided by their targets and receive incentives (monetary the literature with regard to targets: or otherwise), input and feedback related to the achievement of targets. A manager who 1. Challenging and specific targets improve wants to ensure that challenging targets the performance of employees. Target setting are met should try to improve the ability increases individual, group and work unit of employees to meet these targets and performance. In many instances, specific provide feedback on the results of their and challenging targets are associated efforts. with higher levels of performance than either no, or general ‘do your best’, targets. case stUDy target setting for local eDUcation aUthorities Theories suggest that organisational performance improves if targets for future achievement are used. Boyne and Chen (2006) explored this theory using panel data for 147 English local education authorities between 1998 and 2003. The dependent variables in the analysis are exam results for school pupils. The authors found that, controlling for other variables, the extent of performance improvement is influenced positively by the presence of a target. The results are consistent with the view that clear and quantified strategic priority targets lead to better organisational outcomes. Measuring Performance
  • 37. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL setting good targets – using baselines Sometimes a target range rather than 34 a specific figure may be appropriate, Knowing that linking performance especially where the measures are new measures to targets can enhance and untested. However, there is a need performance is useful, but not enough to guard against the games people on its own. How do we know when a sometimes play when setting targets. ‘good’ target has been set? What might a Organisations may set such modest challenging yet realistic target look like? targets that these are achieved quite easily. The World Bank suggests that the Conversely, unachievable targets may be identification and use of baseline data set that just end up demoralising staff. Use can be very helpful when setting targets. of the baseline measures can be important Kusek and Rist (2004) note that targets are here, especially if trend data are available, the quantifiable level of a performance to show what a realistic but challenging measure that a country or organisation target might be. Similarly, external scrutiny wants to achieve by a given time. They of targets, by key stakeholders or audit suggest that one method to establish institutions, can help in ensuring that over- targets is to start with a baseline indicator comfortable targets are not set. Some level, and include the desired level of examples of good and bad targets are improvement (taking into consideration outlined in Table 7. available resources over a specific time period) to arrive at the performance target (see Table 6 for a worked example). taBle 6 DeVeloPing targets Using Baseline Data Outcomes Measures Baselines Targets Nation’s children 1. Percentage of 1. In 2003, 75 per 1. By 2010, 85 per have better access eligible urban cent of children cent of children to preschool children enrolled in ages 3-5 ages 3-5 programmes preschool education 2. Percentage 2. In 2004, 40 per 2. By 2010, 60 per of eligible rural cent of children cent of children children enrolled in ages 3-5 ages 3-5 preschool education Primary school Percentage of Grade In 2005, 75 per cent By 2010, 80 per cent learning outcomes 6 students scoring scored 70 per cent scoring 70 per cent for children are 70 per cent or better or better in maths, or better in maths, improved on standardised and 61 per cent and 67 per cent maths and science scored 70 per cent scoring 70 per cent tests or better in science or better in science Source: adapted from Kusek and Rist, 2004, p.95 Measuring Performance
  • 38. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL taBle 7 35 exaMPles of gooD anD BaD targets Examples of good targets • We will reduce the number of missed household collections by 10 per cent by next year. • We will aim to collect 95 per cent of the council tax that is due next year. • We will increase the number of visits to local libraries by 20 per cent before the end of 2009. • We will cut the number of unfilled places in primary schools by 10 per cent by 31 December 2010. Examples of poor targets • We will improve the way we handle complaints. • We will buy as many books for the school as possible. • We aim to have the best bus service in the region. • We aim to increase co-operation between school and police authorities. • We will answer 75 per cent of all letters within 5 days. (a poor target if the remaining 25 per cent take 3 months to answer) Source: adapted from UK Audit Commission, 2000, p.24 PerforMance MeasUreMent � Performance measures developed with anD citizen engageMent citizen input are likely to have more support from politicians overseeing The importance of including measures that the work of public organisations. citizens want measured in performance � Citizens’ views may provide the measurement systems was discussed in opportunity and stimulus for Section 2. Generally, there are a number of innovation, suggesting new ways of reasons why the public should be engaged doing business. in performance measurement and the reporting of performance: The question is not whether the customer � As funders of the service, citizens’ views should be engaged in performance on how well services are provided are measurement, but how best to engage the an important direct consideration for public. Two issues of central importance the staff of public organisations. here are: means of engaging the public and � The direct involvement of the public the reporting of performance measures to may help legitimise the decisions the public. and priorities of public service organisations. Measuring Performance
  • 39. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL Means of engaging the public in provided by the World Bank). The use of 36 performance measurement citizen report cards began in Bangalore,4 India, and the practice has since spread Ho (2007) suggests a portfolio approach to many countries including Canada, to engaging the public in performance Denmark, the Philippines, Sweden, Ukraine measurement. Among the methods of and the UK. The case study below gives engaging the public are the following: details of the Filipino report card. � Focus groups � Neighbourhood meetings Three main types of institutional � Citizen satisfaction surveys arrangements for report cards are � Report cards for programme users possible: � Web-based discussion forums � Report card by civil society organisation. � Web-based surveys This was the model adopted in Bangalore, where the Public Affairs Centre developed Each method of engagement has the report card in response to anecdotal advantages and disadvantages, and may evidence of customer dissatisfaction with vary in terms of cost. Ho argues for a municipal services. The report card gives diversified portfolio approach that contains citizen feedback on both quantitative multiple strategies to receive diverse and and qualitative dimensions of the balanced input from citizens from all walks selected public services. The strength of of life. this approach is the independence and Two of the strategies noted above have credibility attached to the performance received particular attention in recent measures. The limitations of the approach years – report cards and citizen surveys. relate to the fact that not many civil society Experience in using each is discussed briefly organisations have the technical capacity below, to illustrate some of the benefits and willingness to take on such a job. and challenges of involving the public in � Report card by government service performance measurement. provider agency. This approach has been used in Canada and the UK. Here, a Citizen report cards government agency takes responsibility for the report card production, although the Citizen report cards are participatory actual survey and draft report preparation services that provide quantitative feedback is often contracted out to a commercial on user perception of the quality, adequacy organisation. A strength of this approach is and efficiency of public services (see http:// the ownership of the exercise by the public web.worldbank.org/WBSITE/EXTERNAL/ agency. This is also a potential limitation, TOPICS/EXTSOCIALDEVELOPMENT/EXTPC as the public and politicians may question ENG/0,,contentMDK:20507680~pagePK:1 the independence and objectivity of the 48956~piPK:216618~theSitePK:410306,00. findings. html for details of the report card approach � Measuring Performance
  • 40. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL case stUDy 37 institUtionalising the filiPino rePort carD The Department of Budget and Management (DBM) in the Philippines initiated a programme on the development of performance measures for public agencies so as to assess their effectiveness and guide future budget allocations. As part of this process, they decided to pilot the use of the report card as a means of getting regular user feedback on key public services. The incentive for service providers to respond to customer feedback is enhanced, knowing that they will be regularly monitored by DBM. Analysis and report preparation is contracted out to a credible and independent civil society organisation. An advisory panel, comprising representatives of service providers, other government agencies, the private sector, sectoral groups and prominent experts has been established to provide advice and guidance on the report card process and its integration with the budgetary process. Some specific lessons emerged from piloting the report card: � Focus on key performance measures. The first report card spread the net wide and tried to cover many aspects of service delivery. Subsequent rounds are to be limited to a few key performance measures. The intention is that the measures selected will have a significant overlap with those used by DBM in monitoring outputs and processes, facilitating the comparison of results to provide a comprehensive picture of performance. � Revising the questionnaire. Overlaps in survey questions were identified. Greater clarity of wording was identified as needed in some questions. Focusing of the questionnaire on the key performance measures common with those used for monitoring outputs and processes by DBM was identified as a significant issue. � Improving cost-effectiveness. Limiting questions and refining the questionnaire should bring about considerable savings. Also, limiting the scope to a few principal and common performance measures is intended to focus the attention of public service providers and policy makers and result in concrete actions and follow-up. Source: http://siteresources.worldbank.org/INTPCENG/1143333-1116505690049/ 20509283/Filipino+Report+Card+on+Pro-chapter7-report-cards.pdf Measuring Performance
  • 41. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL � Report card by government performance can correspond with 38 oversight agency. This approach has been more ‘objectively’ measured outcomes, used by the US and the Philippines. It supporting the use of citizen surveys as a involves a government co-ordinating measure of government performance. agency engaging an independent civil society organisation to undertake the However, the researchers urge caution design and preparation of the report card in interpreting their findings. Street in consultation with the public service cleanliness may be an issue that lends itself provider agencies. This approach is the to measurement through a citizen survey, most comprehensive both in terms of compared to other more complex or less process and product. Similar limitations visible services. But the findings suggest exist to the civil society organisation that, in the right circumstances, citizen approach, in terms of the availability of the surveys can be used to judge, and report technical capacity to undertake the work. on, the quality of government services, and may be particularly useful where it is too Citizen surveys complex or costly to gather data in other Report cards can be seen as a specific ways. type of citizen survey. Citizen surveys are generally and widely used to develop measures to assess customer satisfaction with services. But there are concerns about the representativeness and validity of citizen surveys. How do we know if the right questions are being asked? Are they an accurate indication of performance? Is the cost of obtaining a sample large enough to be representative really justified? In terms of the benefits of citizen surveys, work at the City University of New York provides some interesting pointers (Van Ryzin and Immerwahr, 2007 and Van Ryzin, Immerwahr and Altman, 2008). Using data from New York City’s street cleanliness scorecard, combined with responses to a citizen survey, the researchers found a clear and consistent correlation between the scorecard results and citizen rating of street cleanliness. The results suggest that citizen judgements about government Measuring Performance
  • 42. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL The researchers also note that in analysing derived importance because, although citizen surveys, it can sometimes be difficult more analytically complex, it has a better 39 to make sense of the ratings citizens give predictive ability with regard to consumer to various services and to use these ratings behaviour. to meaningfully compare performance. They advocate the use of importance- reporting of performance measures performance analysis (see Table 8). to the public This charts the relationship between service performance ratings and service The way in which performance measures importance as seen by the customer. Efforts are reported back to citizens affects the should be focused on improving the high degree of engagement of the public importance-low performance sector. and ultimately the usefulness of many performance measures. Ho (2007), after Two alternative means of measuring the examining a wide variety of performance importance of a service are possible: reporting practice, identifies nine lessons stated importance, in response to a survey for reporting performance measurement question that explicitly asks respondents results to citizens: to rate importance, and statistically derived importance, based on a regression model 1. Focus on outcomes and intermediate of overall customer satisfaction in which outcome measures in public reporting. the services are the independent variables. Citizens are primarily interested in the On the basis of comparative analysis, results delivered by programmes. the researchers recommend the use of taBle 8 the iMPortance-PerforMance griD Low importance High importance High Potential overkill in terms of Doing a good job, providing performance effort and possibility of slack services that citizens value resources Low From a citizen satisfaction Services that citizens value performance perspective, services that could but which suffer from poor be given a lower priority performance ratings Source: adapted from Ryzin and Immerwahr, 2007, p.217 Measuring Performance
  • 43. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL 2. Make the design of public Citizens’ views of services and 40 performance reports citizen friendly. programmes cut across organisational The use of figures, graphs and charts and other boundaries. They want to to replace/complement/supplement know about the issues that concern tables of numbers and text can make them, not the part of an issue that a big difference. falls under one particular level of jurisdiction. Reporting of performance 3. Provide geographic segregation measures should be organised by of data. Many citizens want topics that citizens are concerned performance measures to be broken about. down by neighbourhood rather than be presented for the whole 7. Use web-based reporting to keep citizens geographical area covered by more informed. Web-based reporting the programme. For example, the can be updated regularly and cost- Boston Indicators project website effectively. It also facilitates user has a feature known as ‘Geography engagement. For example, visitors at a Glance’ which highlights to the online version of Oregon’s some of the key socio-economic 2007 Benchmark Report (http:// measures by neighbourhoods benchmarks.oregon.gov/) can select (see http://www.bostonindicators. the benchmarks that interest them org/IndicatorsProject/). and generate a customised report. 4. Provide comparable performance 8. Blend e-reporting and paper copies. benchmarks. Citizens like to compare Paper copies of reports are still their neighbourhood with that important. It is not an either/or of other equivalent communities. situation with regard to e-reporting. This helps citizens contextualise Paper reports, for example, can be performance measures. structured to lead people to the website later. 5. Provide stories to explain and elaborate the data. Qualitative descriptions 9. Guarantee data accuracy and reliability. of specific events can complement Public trust and credibility with quantitative measures. Stories of regard to the performance measures successes and failures illustrate reported is vital. Actions such as measures and make them more conducting performance audits to meaningful. verify the reliability and validity of the data behind the measures can help 6. Organise performance information secure public trust. So can balanced by community concerns, not by reporting of both successes and departmental or agency structure. failures. Measuring Performance
  • 44. ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL Using incentiVes anD Annual Reports Award. Such initiatives sanctions to encoUrage recognise excellence in the reporting of 41 the Use of PerforMance performance measures. They can be used MeasUres to highlight and spread good practice. Performance measures are rarely used Assurance. Four levels of assurance can be simply on the basis that they have been identified (CCAF-FCVI, 2007, p.38): produced. Performance measurement is • Assurance is inferred if the user must not just a technical exercise; it has strong assess the report content to determine cultural and political components. In such how much confidence to place in the an environment, incentives and sanctions performance measures. (carrots and sticks) can play an important • Assurance is affirmed if management role in encouraging the use of performance affirms its legal responsibility for measures. Amongst frequently used reporting. incentives and sanctions are the following: • Assurance is described if management describes the basis for its judgements, Legislation. Making performance the steps it has taken to validate measurement and reporting a legislative measures, and the limitations of the requirement can help create a performance data. culture in an organisation. Measuring • Assurance is corroborated if a third performance is more likely to be seen as party (such as an auditor) examines the an important activity worthy of senior report and adds assurance (or delivers management attention. Canada’s Revenue cautions). Agency, for example, has a legislative requirement to produce a performance Involvement/feedback. Many studies report and has stated that this has had a show that if measures are to take root positive impact. in organisations, staff must have an opportunity at some stage to influence the Review. Internal and external reviews choice of measures used. Staff, as well as of performance measures and reports customers, should also be given feedback can help ensure that measurement is on performance rather than simply feeding taken seriously. The Office of the Auditor information up the line. Examples of ways General in Canada has regularly reviewed of encouraging participation include departmental reporting of performance developing mixed task teams to develop measures. measures and regular meetings between managers and staff to discuss results. Award/Recognition. In 2006 the New South Wales Public Bodies Review Committee, in conjunction with the Public Accounts Committee, hosted the first Premier’s Measuring Performance
  • 45. Endnotes 1. The term performance measure as used here includes indicators. In the literature, 42 measures and indicators are sometimes differentiated, with measures referring to quantitative assessment of normative performance and indicators giving a quantitative or qualitative assessment of relative performance. 2. The term programme as used here is shorthand for a variety of objects of performance measurement including programmes, projects, operations, administrative systems, etc. 3. CompStat originated in the NewYork City Police Department in 1994, under leadership of Police Commissioner William Bratton and his Deputy Commissioner Jack Maple. On a weekly basis, personnel from each of the Department’s 76 Precincts, 9 Police Service Areas and 12 Transit Districts compile a statistical summary of the week’s crime complaint, arrest and summons activity, as well as a written recapitulation of significant cases, crime patterns and police activities. These data are forwarded to the Chief of Department’s CompStat Unit where they are collated and loaded into a citywide database. The data are analysed by computer and a weekly CompStat Report is generated. The CompStat Report captures crime complaint and arrest activity at the precinct, patrol borough, and citywide levels, and presents a concise summary of these and other important performance measures. Comparisons to previous years’ activity are shown. Precinct commanders and members of the agency’s top management can see emerging and established crime trends as well as deviations and anomalies, and can make comparisons between commands. Also, several American cities, inspired by its success in Baltimore, are engaging with a technology-enabled management approach called CitiStat. The mayor of the city has a database of indicators established that tracks performance against his or her political priorities, such as reducing lead levels in children’s blood from traffic fumes, and also internal management issues such as overtime and absenteeism. The mayor then holds regular meetings with city managers to discuss these data and their trends. The mayor persistently questions managers about problems and strategies to deal with these problems. Specific targets, assignments and deadlines are given to address issues raised. The data are actively used to try to improve performance (see Henderson, 2003, http://www.businessofgovernment.org/pdfs/ HendersonReport.pdf ). Measuring Performance
  • 46. ENDNOTES 4. Inspired by a private sector practice of conducting client satisfaction surveys, a group of people in Bangalore, India, concerned about the city’s worsening standards of public 43 services, started an exercise in 1993 to collect feedback from public service users. User views of the quality, efficiency and adequacy of the various services were aggregated to produce a ‘report card’ that rated the performance of public service providers in the city. The findings received media attention, which helped mobilise citizen and government support for reform, and encouraged the rated agencies to improve their standards. The exercise was repeated in 1999. Report cards provide a useful means by which citizens can signal to public service agencies their level of satisfaction or dissatisfaction with services and provide pressure for change. For further details see http://siteresources.worldbank.org/INTPCENG/1143333-1116505690049/2050927 5/making.pdf Measuring Performance
  • 47. References Atkinson A.B. (2005), 44 Atkinson Review: Final Report – Measurement of Government Output and Productivity for the National Accounts, Final report, Basingstoke: Palgrave Macmillan UK Audit Commission (2000), On Target: The Practice of Performance Indicators, London: Audit Commission Australian Productivity Commission (2006), Report on Government Services 2006, Report of the Steering Committee for the Review of Government Service Provision, Canberra: Australian Productivity Commission (http://www.pc.gov.au/) Behn, B. (2008), The Seven Big Errors of PerformanceStat, Policy Briefs, John F. Kennedy School of Government, Harvard University (http://www.hks.harvard.edu/thebehnreport/Behn,%207PerformanceStatErrors.pdf ) Boyne, G.A. and A.A. Chen (2006), ‘Performance targets and public service improvement’, Journal of Public Administration Research and Theory, Vol.17, No.3, pp.455-477 Boyle, R. (2005), Civil Service Performance Indicators, Committee for Public Management Research Discussion Paper No.29, Dublin: Institute of Public Administration (http://www.cpmr.gov.ie/publications/discussion-papers/) CCAF-FCVI Inc. (2007), What Can We Learn from Effective Public Performance Reporting? Ottawa: CCAF-FCVI (http://www.performancereporting.ca/documents/PPR-Good_Practices.pdf ) Measuring Performance
  • 48. REFERENCES Efficiency Unit (2000), Step-by-Step Guide to Performance Measurement, 45 Efficiency Unit, The Government of the Hong Kong Special Administrative Region (http://www.eu.gov.hk/english/publication/files/step_by_step_guide_to_pm.pdf ) Henderson, L.J. (2003), The Baltimore CitiStat Program, Washington D.C.: IBM Center for the Business of Government (http://www.businessofgovernment.org/pdfs/HendersonReport.pdf ) Ho, A.T. (2007), Engaging Citizens in Measuring and Reporting Community Conditions: A Manager’s Guide, Washington D.C.: IBM Center for the Business of Government (http://www.businessofgovernment.org/pdfs/HoReport.pdf ) Kaplan, R.S. and D.P. Norton (1993), ‘Putting the balanced scorecard to work’, Harvard Business Review, September-October, pp.134-147 Kusek, J.Z. and R.C. Rist (2004), Ten Steps to a Results-Based Monitoring and Evaluation System, Washington, D.C.: The World Bank Ministry of Finance (1997), Public Sector Productivity in Sweden, Stockholm: Ministry of Finance Neely, A. and P. Micheli (2004), Performance measurement in the UK’s public sector: linking the national to the local agenda, CIMA New Public Sector Seminar, Edinburgh, 28-29 October 2004 (http://www.som.cranfield.ac.uk/som/research/centres/cbp/downloads/2004BAM_ PMpaper.pdf ) Measuring Performance
  • 49. REFERENCES Niemi, M. (1998), 46 ‘Measuring government sector output and productivity in Finland – application of the output indicator method’, Agenda item 1, OECD meeting of national accounts experts, Statistics Directorate, STD/ NA(98)4, 22-25 September, Paris: OECD OECD (2007), ‘Towards Better Measurement of Government’, OECD Working Papers on Public Governance, 2007/1, Paris: OECD Publishing. doi:10.1787/301575636734 (http://www.oecd.org/dataoecd/11/61/3813 4037.pdf ) Ogata, K. and R. Goodkey (1998), ‘Redefining government performance’, Alberta Finance, Canada (http://www.finance.gov.ab.ca/publications/measuring/cambridge_paper.html) Perrin, B. (1998), ‘Effective use and misuse of performance measurement’, American Journal of Evaluation, Vol.19, No.3, pp. 367-379 Perry, J.L., D. Mesch and L. Paarlberg (2006), ‘Motivating employees in a new governance era: the performance paradigm revisited’, Public Administration Review, Vol. 66, No. 4, pp.505-513 Pollitt, C., S. Harrison, R. Bal, G. Doswell and S. Jerak (2007), ‘Conceptualising the development of performance measurement systems’, paper presented at European Study Group on Performance and Quality conference, Madrid, 19-22 September, (http://soc.kuleuven.be/io/egpa/qual/madrid/papers/paper%20Pollitt%20et%20al.doc) Measuring Performance
  • 50. REFERENCES Schacter, M. (2002), Not a Tool Kit: Practitioners Guide to Measuring the Performance of Public Programs, 47 Ottawa: Institute on Governance Stowers, G. (2004), Measuring the Performance of E-Government, Washington D.C.: IBM Center for the Business of Government, (http://www.businessofgovernment.org/pdfs/8493_Stowers_Report.pdf ) United Way of America (1996), Measuring Program Outcomes: A Practical Approach, Alexandra, Virginia: United Way of America Van Ryzin, G.G. and S. Immerwahr (2007), ‘Importance-performance analysis of citizen satisfaction surveys’, Public Administration, Vol. 85, No. 1, pp.215-226 Van Ryzin, G.G., S. Immerwahr and S. Altman (2008), ‘Measuring street cleanliness: a comparison of New York City’s scorecard and results from a citizen survey’, Public Administration Review, Vol. 68, No. 2, pp.295-303 Van Thiel, S. and F.L. Leeuw (2002), ‘The performance paradox in the public sector’, Public Performance and Management Review, Vol. 25, No. 3, pp.267-281 World Bank (2006), Impact Evaluation – The Experience of the Independent Evaluation Group of the World Bank, Washington D.C.: The World Bank Independent Evaluation Group (http://lnweb18.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/35BC4 20995BF58F8852571E00068C6BD/$file/impact_evaluation.pdf ) Measuring Performance
  • 51. EfficiEncy unit 13/f., West Wing central Government Offices 11 ice House Street central Hong Kong Email: euwm@eu.gov.hk tel: 2165 7255 fax: 2524 7267 Website: www.eu.gov.hk
  • 52. Measuring Performance