Arief Wiroyudo

Since 2000, Indonesia has...
progress? How can we tell success from failure? The public services production can be illustrated on the
following figure....
The OECD (2002) defines monitoring and evaluation as follows:
Monitoring is a continuous function that uses the systematic...
Logical Framework of Development Process

Source: Framework for Managing Programme Performance Information, National Treas...
increases the working units workload due to the multiplicity of evaluation. Moreover this condition also
made the ambiguit...
specific table formats for the line-ministries, institutions and regional government work units to report
on program imple...
the decision makers reluctant to use current MONEV information because monitoring and evaluation
results and reports are o...
Recently, Bappenas has new working unit which have specific responsibilities on MONEV activities at
Deputy Level (same wit...
matter. All activities must be documented and posted to the integrated MONEV database system.
Improving the role of Indone...
Upcoming SlideShare
Loading in …5

Integrated Development Performance Monitoring and Evaluation System in Indonesia to Support National Development Planning Process


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Integrated Development Performance Monitoring and Evaluation System in Indonesia to Support National Development Planning Process

  1. 1. INTEGRATED DEVELOPMENT PERFORMANCE MONITORING AND EVALUATION SYSTEM Arief Wiroyudo INTRODUCTION Since 2000, Indonesia has been undergoing a massive transformation from centralized to decentralized government and regional autonomy. This new governmental system is fundamentally reshaping the political, economic and social structure of the country. The big-bang decentralization leads to the less role of central government in the regional development. Therefore, there are new tasks and responsibilities in terms of function assignment and financing to local government as consequences of decentralization. Underlying the national policy on performance monitoring and evaluation (MONEV) is accomplishment of government efforts in implementing development programs that improve the capability of departments and agencies to allocate limited resources appropriately and to manage them efficiently. The successful implementation of performance evaluation is seen as an institutional development that is needed to provide better information as an input for the next planning process. The importance of the MONEV quality awareness implies that those suppliers (line-ministries and/or government officials in local levels) of evaluation reports are responsible for the provision of good MONEV information demanded by decision-makers. However, the activities of MONEV in Indonesia still face some problems, such as many rules that are required by sector and local government to make reports to central government, there was less impact of MONEV activities to the next plan, lack of synchronization and overlapping issue, and many evaluation activities are very costly in terms of money and time. Integrated MONEV of wholly national development performance becomes a big challenge for Indonesia. It is essential to link and synchronize both information on monitoring and evaluation activities between central and regional government in order to formulate the next national development plan. The purpose of this paper is to emphasis the need of Integrated Development Performance MONEV System in Indonesia to support national and regional development planning process efficiently. This paper has some points as main discussions to introduce that can be shown as follows: 1. To give a theoretical framework of MONEV development system 2. To Analyze the national development MONEV system in Indonesia case 3. To recommend some key points in order to develop an integrated development MONEV system in Indonesia MONITORING AND EVALUATION DEVELOPMENT SYSTEM OVERVIEW: A THEORETICAL FRAMEWORK From time to time, there has been a global change in public sector management as consequences of public forces to make government to be more accountable to their stakeholders. These are some concerns and questions being raised by internal and external stakeholders, and many governments are struggling in addressing and answering them. The public citizens are really caring about the services delivered by the government. For examples questions are, Have policies, programs, and projects led to the desired results and outcomes? How do we know we are on the right track? How do we know if there are problems along the way? How can we correct them at any given point in time? How do we measure Page 1
  2. 2. progress? How can we tell success from failure? The public services production can be illustrated on the following figure. Production of Public Services Source: Hideaki Tanaka, 2009 Monitoring and Evaluation: What Is It All About? Evaluation is very important phase in policy analysis. As one of the procedures of the policy analysis integrated framework, it cannot be separated with other procedures. Monitoring as the previous procedure provides the fact and information that will be assessed by evaluation action. The policy performance information produced through evaluation will be used as a feedback for next procedure in policy analysis framework. According to Dunn (1994), the relationships of monitoring and evaluation activities on a cycle of an integrated framework of policy analysis can be described as following figure. An Integrated Framework of Policy Analysis Source: William Dunn, 1994 Source: William Dunn, 1994 Page 2
  3. 3. The OECD (2002) defines monitoring and evaluation as follows: Monitoring is a continuous function that uses the systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. Evaluation is the systematic and objective assessment of an ongoing or completed project, program, or policy, including its design, implementation, and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision making process of both recipients and donors. According to these two definitions, it is confirmed there are some different activities but monitoring and evaluation are complementary each other on a planning and budgeting cycle process. Monitoring gives information on where a policy, program, or project is at any given time (and over time) relative to respective targets and outcomes, while evaluation gives evidence of why targets and outcomes are or are not being achieved. Why We Do Evaluation? Evaluation is a complement to monitoring in a condition when a monitoring system sends information that the efforts are not in the track, and then good evaluative information can help clarifying the realities and trends noted with that monitoring system.Patton&Sawicki (1986) explained that the policy analysis process does not stop with the implementation of the apparently superior policy. Even after a policy has been implemented, doubt may remain as to whether the proper problem was identified, whether an important aspect of the problem was ignored, or whether the policy conclusion or recommendation might been wrong. These concerns require us to monitor and evaluate policies and programs to see that the correct alternative is implemented, to assure that it does not irregularly change form, and to determine whether it is having the preferred impact, whether it should be redesigned or modified, or whether it should be terminated. Mandated by certain regulations, the government (central and local governments) is responsible for programs and activities which contribute as a basis for budget establishment. Each program and activity produces outputs (immediate results) and outcomes (medium term results). Over the longer terms outcomes lead to impact (long term results). Logical framework of development process can be described as following figure. Page 3
  4. 4. Logical Framework of Development Process Source: Framework for Managing Programme Performance Information, National Treasury, Republic of South Africa, May 2007 Furthermore, focusing on evaluation system, it contributes to three basic functions (Mackay, 1998): • Accountability: making sure that public institutions, and their staff, are held accountable for their performance. • Allocation: making sure that resources are allocated to those activities which contribute most effectively to achieving the basic objectives of the institution. • Learning: making sure we learn from our successes and failures, to do things better in future. ANALYSIS OF THE NATIONAL DEVELOPMENT EVALUATION SYSTEMIN INDONESIA CASE The economic crisis in the late 1990’s became to catalyst that prompted the Government of Indonesian to give high efforts to reform the public sector starting in the early 2000. Furthermore, in this era, Indonesia has been undergoing a massive transformation from centralized to decentralized government and regional autonomy. One of the main focuses of the reforms was in the area of public financial management, with the goal of promoting good governance, greater transparency and accountability in the conduct of government affairs. Despite having traditionally a strong statistical system, monitoring and evaluation is a major weakness of the Indonesian government planning system. Feedback of progress and results achieved is rarely provided, despite the numerous requirements for filling out forms and lodging them. Most of those forms are for recording purposes only, not for monitoring progress and evaluating results. At the present time, there are so many regulations related with MONEV activities, therefore this condition Page 4
  5. 5. increases the working units workload due to the multiplicity of evaluation. Moreover this condition also made the ambiguity of responsible on evaluation result. MONEV plays a key role in the existing integrated planning, budgeting and implementation development activities. This circumstance has reflected that state ministries must evaluate performance of programs according to the regulation standards. However, it is generally recognized that MONEV mechanism and reporting systems in Indonesia are not very efficient and effective as can be seen from following problems: Legal Aspect Problem Legal aspects for MONEV of development activities in Indonesia mainly are derived from 3 laws. Firstly, Law No. 25/2004 on National Development Planning System which related with national development plan from all sectors as a whole. Secondly, Law No. 17/2003 on National Finance which related with budget disbursement, and thirdly, Law No. 32/2004 on Local Government which related with regional or sub national development plan. These laws provide a new framework for financial management in the public sector, and establish the basic principles of transparency and accountability, as well as regulate the planning, budgeting and the accounting systems. The relationships of those development activities can be seen as follows. Development Budget and Planning Process in Central and Regional Level 20 years 5 years Annual National Longterm Plan Ministries ’Workplan Ministries’ Budget Work Plan Detailed State Budget National Midterm Plan Gov’t WorkPlan Draft State Budget State Budget Central Government Ministries Strategic Plan Synchronized in the National Forum on Dev’t Plan Regional Mid Term Plan Regional WorkPlan Draft Reg Budget Regional Regional Work Unit Strategic Plan Regional WorkUnit WorkPlan Reg Budget Work Plan Detailed Regional Budget Law of Dev’t Plan Budget Local Government Regional Longterm Plan Law of Finance Source: Directorate Allocation of Development Funding, Bappenas According to the Law No. 25/2004 on National Development Planning System, it states that ministries, institutions and regional government work units are responsible for conducting MONEV activities of their development plans of the previous period. This law also stipulated that the more detailed procedures for MONEV of the implementation development plans will be issued through Government Regulations. The prominent government regulation of those regarding MONEV activities of national development plan implementation was issued in 2006. This regulation No. 39 Year 2006 provides Page 5
  6. 6. specific table formats for the line-ministries, institutions and regional government work units to report on program implementation, and formats for the consolidation of such information. The second government regulation, relating to MONEV of regional development plan implementation is Government Regulation No.8 Year 2008. Interestingly, this regulation does not refer to any article of Law 25/2004 as the legal basis for issuing the regulation, but it takes Article No. 154 of another Law (Law No.32/2004) as its legal basis. The linkage between national and regional development planning is acknowledged in article 150 of Law 32/2004, which states that regional planning is part of the national development planning. This government regulation refers to the need for issuance of several Ministerial Regulations (Minister of Home Affairs) to regulate the National Development Plan Forum/Consultation process, integrated planning and budgeting, and MONEV of regional development plans. In sort, according to those examples above, we can see that there are many regulations on MONEV activities. Each regulation asked for the specific forms as mandates, however many of them seem not correlated each other on an integrated MONEV development system. Lack of synchronization and Overlapping issue Initially, there were many definitions of MONEV activities of national and regional development in Indonesia. Many government institutions had evaluation specified in their mandates, but often meaning different things. This circumstance also made the ambiguity condition about who would be responsible for MONEV supervision or coordinator. Furthermore, this MONEV activity often occur overlaps condition in implementation. In fact, MONEV reporting is an additional burden to already busy of responsiblestaffs. In some experiences regarding MONEV activities, there are so many reports that should be prepared by the MONEV providers (planning bureau), so they cannot do anything but just fill-up the MONEV tables rather than make strategic plans. Lack of feedback from MONEV reports According to the theories and concepts explained earlier, MONEV results feedback actually used as a valuable input for the next planning process. However, in Indonesia case this MONEV report is not fully used as input for the next planning. The poor quality of the MONEV documents, un-measurable information, and there were no institutions that capable to develop MONEV reports with integrating cross-sector issues are some critical problems for creating a good MONEV report as an input for decision makers at the top level for the next strategic development plan activities. In many chances most of the reports are only focus on aid disbursement and physical progress, instead of program or activity performance indicators. As a consequence, the impact of development, especially for social aspects, was often lacking in monitoring and evaluation. So, it is difficult condition for Bappenas as a planning institution to come up with such good development planning with those limited information provided by line ministries. Lack of resources to carry out MONEV In many cases, monitoring activities of development performance in Indonesia are carried out by junior staffs of government officials. Typically, the activities of data input, table formatting, and recording are some main activities within this monitoring phase. However, for the evaluation, this advanced activity still carried out by the lower staffs rather than more advanced and skilled staffs. This circumstance made Page 6
  7. 7. the decision makers reluctant to use current MONEV information because monitoring and evaluation results and reports are often not well compiled and poorly information. For example, project implementers still have to report progress using a wide range of formats, which indicates a lack of streamlining and rationalization in reporting. Up to now, many top level decision makers have realized the importance of MONEV. They want to have staffs whose capability to carry out the MONEV activity and can provide them useful information to develop policies. Unfortunately, the capacity development for MONEV training is still limited. There are no certain trainings and courses to carry on development MONEV activity in local level or even central level. The activity mostly is derived from senior to junior staff without knowing the essential meaning and importance of MONEV activity itself. RECOMMENDATION According to some existing problems, there are some future expectations for improvement of MONEV activities. Hence, there are some recommendations as follows: Establishment of National Coordination Group. In Indonesia, practically departments and agencies are responsible for conducting MONEV activities. Departments were not prepared to hand over their control of departmental evaluation to Bappenas. According to the respected laws, the concept/framework of having Bappenas as functionally responsible to the MONEV system and the departments to responsible for doing the actual evaluation work was the key to the acceptance of an overall approach for performance evaluation in Indonesia. As the current situation, Bappenas with its limited staff would never have been able to do the monitoring and evaluation work as needed. Retaining functional responsibility, prescribing how evaluation work was to be carried out and reported, and having departments do the evaluation work, was the only viable solution. Departments are closest to the development work and they have access to the information needed for the MONEV activities. Unlike developed countries where there may be many alternative sources of information possible for evaluation, in developing countries there is only one real source of information which is the departments. It confirmed that evaluation work can only be performed well if the departments are involved. However, without central guidance from Bappenas, departments would do poorly evaluation or would do evaluation work using a wide range of different approaches that make the evaluation result cannot use to fulfill the requirement as a basic input need for the next planning process. Bappenas would then be unable to fulfill its national task regarding performance evaluation. Considering that condition above and also many regulations related to the MONEV activities, the existence of so called “National Coordinator Group” will be very essential part. This board would essentially more focus on coordination and collaboration activities in the development and implementation of MONEV processes that are used by ministries at the present time. By Giving Bappenas a role as “policy director’ of sector ministries is a best solution to have comprehensive consideration of National program priorities. The main task of this national coordination group is to provide documents or reports as a mandate to direct the development of performance management and indicators of national development as a whole, and to report to the Cabinet through a designated Minister on direction, progress, issues and problems. Establishment of Indicator Resource Group. Bappenas now has the functional responsibility for performance evaluation in overall while departments actually carry out the evaluation work. Bappenas works closely with departments to develop and use performance indicators for GOI developmental projects (donor-assisted or not), as part of the budgeting system. The proposed members of this group are the staffs in Bappenas who have responsibilities in MONEV activities (MONEV working unit). Page 7
  8. 8. Recently, Bappenas has new working unit which have specific responsibilities on MONEV activities at Deputy Level (same with Directorate General Level). Together with the staffs from ministries/agencies who have similar responsibilities in MONEV of their each ministry/agency and internal organization which perform as ODA providers, this group is expected to develop a good quality of KPIs (Key Performance Indicators). Rational selection of indicators will help to address the classic problem of too many indicators. On the other hand, the use of hierarchy of indicators at different levels will help to focus on the national strategic outcomes through bottom-up filtration until monitoring and evaluation information needs are met at the district, sectoral, provincial and national levels. Quantitative indicators need to be supplemented by informed judgment and common sense.Furthermore, it is important to build a review mechanism into the system. The various sets of indicators for all levels should be reviewed and challenged in terms of their effectiveness, cost implications, data quality and source of data collection, and linkages among performance indicators. The use of these performance indicators for development projects brings Bappenas and the departments closer together through both parties using performance indicators as a common language. With this arrangement, ODA providers now can support the national policy on performance evaluation by cooperating with related national organizations in the development and using performance indicators for development project inputs, outputs, results and impacts. The use of the same set of structured performance indicators for a given development project by both ODA providers and Indonesia as recipient country should strengthen MONEV capacity development. Parallel reporting systems for recording the achievements of ODA provider projects should be discouraged. Instead, as much as possible, ODA providers should adapt to the evaluation systems which developed by recipient country or in this case Indonesia. Establishment of Integrated MONEV System Data Base. At this time, the availability of data are scattered in each ministry/institution, meanwhile there is no coordination among data providers. It is difficult to have cross-sector and cross-region analysis and prepare a consolidated report. Not all indicators data are available regularly, especially for data that should be obtained from sector side (nonsurvey data). There are many data have been collected by sector institutions but some of them are not processed properly even there should be guidance forms based on some mandated regulations. Furthermore, there is no synchronized perception in filling the form and data processing of sector data. Therefore, the integrated MONEV system will serve as a multi-stakeholder platform. It will foster and enhance partnerships, collaboration, and programmatic synergies among stakeholders. It is an opportunity to have a common agreement on indicator metadata, data collection and data processing issues, data flow management, system support, and management of database development and maintenance. Under Bappenas supervision, Central Statistical Board (BPS) can handle those activities as a source of data provision, data collection, data sharing, and data management of documented all MONEV activities. This BPS will produce a kind of MONEV system database that can be accessed by all stakeholders. Development of implementation schedules for new MONEV processes. The existing condition of MONEV activities has given the increasing number and complexity of the MONEV functions that required from the working unit staffs, along with increasingly complex reporting requirements, it may not adequately power capacity to provide the necessary quality of the reports. Furthermore, as the degree of preparation needed increases with complexity, it becomes necessary to ensure that the extra time, costs, and resources are efficiently used. So, the integrated schedule for all MONEV activities should be developed. Starting from monitoring until ex-post evaluation should be established with clearly agenda, who, what, where, when, and how the MONEV activities be conducted are critical Page 8
  9. 9. matter. All activities must be documented and posted to the integrated MONEV database system. Improving the role of Indonesia Development Evaluation Community (InDEC). Knowledge from the same organization abroad that brings new ideas and contributes in improving the MONEV policies can be expanded from the presence of establishing community evaluator in Indonesia. It should be admitted that the MONEV policies have not started in its optimal at this point in time. Generally, this concern is caused by the lack of skills of the people responsible in the evaluation of the tasks towards the implementation, the limited information about the procedure in conducting work scheme evaluation, segregation of system, and evaluation reports as well as the minimum coordination among the facilitators. As many countries practices according to the involvement of evaluation community beside the government side, it is also believed that intensive interaction and exchange of information as well as the thought among MONEV actors spread to professional, academics, government officers, and NGOs both locally and internationally are urgent to create added value and high advantage in creating better climate in the MONEV system in Indonesia. --o0o-- Arief Wiroyudo adalah perencana pertama pada Direktorat Perkotaan dan Perdesaan, Bappenas REFERENCES Barberie, A. (1998). Indonesia’s National Evaluation System. Evaluation Capacity Development Working Paper Series No. 3. World Bank. Washington DC Dunn, W. (1994). Public Policy Analysis: An Introduction. Pearson Education, Inc. New Jersey. Kusek, J. Z. & Rist, R. C. (2004).Ten Steps to a Results-Based Monitoring and Evaluation System.World Bank.Washington, DC. Mackay, K. (1998). Public Sector Performance-The Critical Role of Evaluation. World Bank. Washington DC. McNamara, C.(2007). Field Guide to Nonprofit Program Design, Marketing and Evaluation. Authenticity Consulting, LLC Patton, C. and Sawicki D.S.,(1986). Basic Methods of Policy Analysis and Planning. Prentice Hall, Englewood Cliffs Solihin, D. (2008). Setting Up the Development Performance Evaluation System: Overview on Current Practices and Future. Directorate for Regional Development Performance Evaluation, Bappenas. Tanaka, H. (2009). Performance Measurement and Evaluation. Presentation of Public Expenditure Management: Theory and Practice. November, 2009 OECD (Organisation for Economic Co-operation and Development). (2002). Glossary of Key Terms in Evaluation and Results-Based Management. OECD/DAC, Paris Bappenas. (2004). Mid Term Development Plan Document. Bappenas, Jakarta Bappenas and Direktorat Perekonomian Daerah. (2006). Pengolahan dan Analisis Laporan Monitoring Pelaksanaan Dana Alokasi Khusus (DAK), Jakarta, Bappenas and Direktorat Perekonomian Daerah DepartemenKeuangan.(2007). Sistem Informasi Keuangan Daerah Direktorat Jenderal Perimbangan Keuangan Pusat dan Daerah [Regional Fiscal Information System of the Directorate General of Central and Regional Fiscal Balance] ( Page 9