• Save
Measuring Institional Effectiveness by Performance Audit
Upcoming SlideShare
Loading in...5
×
 

Measuring Institional Effectiveness by Performance Audit

on

  • 1,143 views

Primer on Performance Audit to measure Institional effectiveness

Primer on Performance Audit to measure Institional effectiveness

Statistics

Views

Total Views
1,143
Views on SlideShare
1,143
Embed Views
0

Actions

Likes
2
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Measuring Institional Effectiveness by Performance Audit Measuring Institional Effectiveness by Performance Audit Presentation Transcript

  • Performance Audit
  • What is Performance Audit - I? INTOSAI Auditing Standard 1.0.40Performance audit concerned with the audit of economy,efficiency and effectiveness and embraces: Audit of the economy of administrative activities inaccordance with sound administrative principles andpractices and management policies; Audit of the efficiency of utilization of human, financialand other resources, including examination ofinformation systems, performance measures andmonitoring arrangements and procedures followed byaudited entities for remedying identified deficiencies; and Audit of the effectiveness of performance in relation tothe achievement of the objectives of the audited entityand audit of the actual impact of activities compared withthe intended impact
  • What is Performance Audit - II?“Performance auditing is anassessment of the activities of anorganization to see if theresources are being managedwith due regard foreconomy, efficiency andeffectiveness and accountabilityrequirements are being metreasonably.”Muhammad Akram Khan, D-
  • Our Legal Mandate for Performance Audit Comptroller & Auditor General’s (Duties, Powers & Conditions of Service) Act, 1971 16 17 15 19 14 20 13 23
  • Objectives of a Performance Audit Quality of information and advice available to governmentfor the formulation of policy Existence and effectiveness of administrative machinery toinform the government whether program objectives andtargets have been determined with a view to fulfilling policyobjectives Whether and to what extent, stated program objectiveshave been met Economy, efficiency, effectiveness, equity and ethics of themeans used to implement a program/activity Intended and unintended, direct and indirect other impactsof programmes/activities, e.g. environmental impact ofgovernment activity Compliance with applicable laws and regulations in thecontext of the performance audit objectives
  • Performance vs. Regularity Audit - I Features Regularity Performance Generally covers a Covers the program over a financial period period of time Coverage is for the Focused only on a part of the Scope whole of the entity for entity’s the period programs Audit to be completed by a Coverage is selective stipulated time
  • Performance vs. Regularity Audit - II Features Regularity Performance Attestation of Covers the subject/program over a financial accountability period of time; Audit of financial systems Focused only on a part of the entity’s activities/program Audit of internal control for Coverage is selective safeguarding assets and completeness and accuracy of Accounts Evaluation of financial records Audit of economy, efficiency and effectiveness Objectives Audit of propriety of Audit of internal controls – 3Es, administrative decisions adherence to management policies, timely and reliable financial and management information Limited to financial matters Extends to non-financial /governance issues too Test for assuring compliance to Assessment of compliance to laws, regulations and rules applicable laws and regulations required in the context of audit objectives Opinions generally Conclusions related to audit with reference to standardized objectives set by auditors requirements
  • Performance vs. Regularity Audit - III Features Regularity Performance Financial statements Variety of forms of evidence per se, accounting documents, etc.; Budgetary assumptions and Quite often qualitative in appropriation nature authorizations, etc.; Evidence Transaction documents Conclusive nature of Persuasive rather than evidence conclusive - evidence related to pre-determined audit objectives Materiality by amount Materiality guided more by the nature or by context rather than amount alone
  • Performance vs. Regularity Audit - IVFeatures Regularity PerformanceAcademic Always knowledge-based: ofbase Generally laws, social sciences, accounting economics, development knowledge studies, public affairs, science and technology, etc More or less Varies widely from subject –Approach & standardized: manual generally serves asmethodology manual generally basic framework forms the entire basis Widely varying and subjective with ample scope forAssessment Standardized interpretations & assessmentCriteria with little scope criteria support the audit for subjectivity objectives, which are unique to the subject of performance audit
  • Performance vs. Regularity Audit - VFeatures Regularity Performance Opinion on the Conclusions on 3Es financial statements; Opinion on Report presented in compliance to variety of forms laws, regulations and rulesReport Generally opinion on Wide-ranging in nature, financial statements open to interpretations in standardized and subjective judgment format Related to specific Generally separate financial periods- publications on each there is a periodicity subject of performance of reporting audits as and when conducted
  • Typology of Program Evaluation Type ObjectiveProcess Assesses program activities’ conformance to statutory and regulatory requirements, program design and professional standards or expectationsOutcome Focuses on processes, outputs and outcomes (including unintended effects) to judge program effectivenessImpact Compares program outcomes with an estimate of what would have happened in its absence - employed when external factors are known to influence the program’s outcomesCost Assesses the least costly alternative to meet toEffectiveness achieve a single goal or objective
  • Changing Jargon & Avatars of Performance Audit
  • New Dimensions of PA’s Operating Environ - I New models of governance Greater role for NGOs in program/service delivery Public Private Partnerships Direct transfer of development funds to provinces Enhanced delegation of spending powers to relatively junior state officials Commercial exploitation of nation’s natural resources by private sector & their regulation Mass privatization of public utilities & their regulation by the state
  • New Dimensions of PA’s Operating Environ - II Mass privatization of public utilities & their regulation by the state Rising education and standard of living begets greater demand for accountability Better education also provides better skill sets to SAI personnel Declining standards of probity in public life Young generation no longer used to government inefficiency, waste and corruption Right to Information Act, Lok Pal Bill, etc. Increasing judicial intervention for accountability Corruption affects foreign investment and retards domestic saving and investment
  • Why is Performance Audit good for SAIs? Concentrates the audit effort in a publicprogram/project Utilizes slack in diverse skill sets andsynchronizes multi-disciplinary evaluation by SAIs Administrative-cum-financial external appraisalsynched with program is greater help to policyplanners, makers and implementers Synergize policy delivery mechanisms withconcurrent accountability Well-suited for developing countries wheregovernment still delivers most public programs Greater public visibility, debate andaccountability of public resources
  • The Performance Audit Schema Economy Efficiency Effectiveness Performance Audit
  • The Performance Audit Schema
  • Key Terms of Performance Audit Accountability Efficiency Activities Management Assessment Organization Economy Reasonableness Effectiveness Resources
  • Accountability & ReportingLegislative control over public financesestablished by:
  • Assessment & Reasonableness • Auditor formulates a judgment on the basis of relevant and reliable evidence, discusses with management and validates I procedures • Considers constraints of the management and operating environment II • Does not base its judgment on hindsight wisdomIII
  • Organization, Management & Resources Performance auditing takes an overview of the activities and functions of an organization as a whole. In exceptional cases, it may review an isolated project or a programme as well Resources of an organization consist of money, men, materials, and machines. Performance auditing reviews all these resources Management covers such functions as planning, organization, resourcing, directing and controlling. Performance auditing reviews all these phases of management cycle. It is not concerned merely with the use of resources.
  • The 3E Schema
  • What are the 3 Es? EconomyAcquiring resources at the lowest cost keeping in view the objectives of the organization Efficiency Relationship of inputs and outputs - utilization of resources Effectiveness Extent to which an organization achieves its objectives
  • How is PA different from InternalAppraisal? Audit Finding Criteria Cause Conditions Effect
  • Criteria Characteristics Reliable Comparable AcceptableComprehensible Complete Objective
  • What is an Audit Criteria?
  • What is Audit Criteria in Real Life?
  • A Program Logic Model Socio-economic ProblemsNeeds Objectives ActionInputs Processes OutcomeOutput Results Impact on Socio-economic Problems
  • Where do the 3Es fit in? Allocation of Inputs Use of Inputs Processes Outputs Results > Objectives Impact > Objectives
  • What is Economy? Spending only that much which is barely essential to achieve organizational goals For example, if lowest bidder leads to interruption in operations, it may not be the most economical manner of buying these supplies Therefore, while reporting on economy the, auditor should keep in view effectiveness of the organization in achieving its goals
  • How do I judge economy?Right Right RightTime Place Cost Capital Demand Idle HR OperatingProcurement Maintenance Automated Down Time assemblyAvailability Salvage Value
  • Yardsticks for Judging Costs • Costs estimated in the form of streams over the life-time of the asset I • Cost streams then discounted to the present using an appropriate discount rateII • Not a simple comparison of one-time capital cost, usually done in traditional auditingIII • Calculation of minimum cost presumes estimates of cost would be prepared for allIV available alternatives
  • What is Efficiency?An economic state is one inwhich every resource isoptimally allocated to serve eachperson in the best way whileminimizing waste andinefficiency
  • Can Government Always be Efficient? • Unlike profit-making entities, government is a I developmental entity in which there are few choices • Government is slave to people’s will represented by II the Legislature • Government focuses on entire nation and itsIII problems • Availability & Flexibility of use of resources limitedIV • Public policy is therefore a political animalV • Yet Governments remain accountable to the peopleVI
  • How do I measure efficiency? • Are all levels of the organization aware of theirI goals? • Is there a reliable system in place to monitor theII output of each level? • Does management receive regular reports onIII current levels of efficiency or services rendered? • What are the measures taken by management toIV improve its efficiency? • What efficiency measures are feasible in theV given situation?
  • How does efficiency relate to productivity? Productivity = Amount of goods or services produced / amount of resources used i.e. ratio of input and output or capital output ratio Efficiency is the relationship of actual input/output (productivity) to a prescribed performance standard Time taken for producing 80 bags of cement is one machine-hour is productivity of the plant This level of productivity may however, be 80% of the standard which is 100 bags per machine- hour Efficiency is, therefore, measured in terms of rate of return of production, work done in a time span or unit cost
  • Prerequisites for Measuring Efficiency Outputs should be uniform and repetitive, e.g. filling in prescription by the pharmacist in a hospital Non-uniform output like police programme, writing press briefs by Information Department etc., cannot be measured There must be an agreed standard of performance against which actual output has to be measured, e.g. in case of a school the number of class-contact hours for a teacher in a week should be laid down as a standard If standards do not exist then output may be measured with previous years Sometimes measurements are compared between two geographical locations, or, comparison with private sector - data of past years should be available
  • Measuring Efficiency in Government - I Assess whether the objectives of and the means provided (legal, financial, etc.,) for a new – or ongoing – public sector program are proper, consistent, suitable or relevant to the policy Assess the effectiveness of public sector programs and/or individual components, i.e. assess whether objectives are met Assess whether the observed direct or indirect social and economic impacts of a policy are due to the policy or to other causes, but only if it can be established with evidence
  • Measuring Efficiency in Government - II Identify factors inhibiting satisfactory performance or goal-fulfilment Assess whether the programme complements, duplicates, overlaps or counteracts other related programmes Assess the adequacy of the management control system for measuring, monitoring and reporting a program’s effectiveness Identify ways of making programmes work more effectively
  • Advantages of Efficiency Measures I • Control quality and quantityII • Demonstrate achievementsIII • Prepare Budgets and PlansIV • Rationalize pricing of productsV • HR performance appraisal
  • Efficiency Measurement Tools– IInter-authority ComparisonComparison with similarauthorities, delivering similar programs andoperating in comparable environments withfollowing illustrative parameters: Operating expenses Usage of service or the client population Unit cost of service provision Manpower levels Performance indicators
  • Efficiency Measurement Tools – IIInternal Comparison Comparison of costs between different servicesor facilities provided by the same authority atdifferent locations or in different departments For example, repair costs by class of vehicle, orenergy consumption by schools or residentialunits Large range of performance indicators can bedeveloped by the auditor from the internal datasuch as: ratio of administrative cost to operational costs ratio of over time payments to total payroll number of issues per hour in a library
  • Efficiency Measurement Tools – IIIPast Performance & Targets Comparison with past years may show potential areas for saving Auditor should analyze cost and output trends Targets set and achieved by the management for itself should be used as comparison measures
  • Judging Effectiveness - I • Are objectives precisely defined ?I • Are objectives based on reliable facts andII forecasts? • Are policy instruments to operationalize objectives based on detailed analysis ofIII alternatives? • Can progress be reasonably attributed to theIV activities and policies of t he auditee? • Does the auditee adequately monitor theV effectiveness of a policy instrument?
  • Judging Effectiveness - II • In measuring effectiveness a distinction needs to be madeI between output and outcome • Output relates to results of certain inputs within theII organization • Outcome relates to the results external to the organizationIII
  • Performance Measures for EffectivenessThree main problems: Jointness: Where a number of different policies maycontribute to satisfying unmet needs Ex: Educational standards may be affected by the sizeof classes, the quality of teachers and the supply ofequipment External factors: May affect the outcome of a project orprogramme Ex: Income and social status of consumers Cost: Sometimes programmes cannot be carried out inthe most effective manner due to prohibitive cost. Ex: It may be more effective to have more teachers thanto provide more books. But the cost of the former may beprohibitively high.
  • What should the Auditor do? Decide indicators in consultation withmanagement Also keep in view any national orinternational standards already available Objectives laid down in the plan maybe taken as a bench mark for some ofthe outcomes In no case should the auditorrestrict his audit criteria to theoriginal plan only
  • Some PA Tools We Use I • SurveyII • Statistical samplingIII • BenchmarkingIV • Focus groupsV • InterviewsVI • Case studiesVII • Qualitative data analysisVIII • Program Logic ModelIX • Flow charts
  • Potential Impact Areas of PAEconomy Reduction in costs through better contracting, bulk buying, etc Reduction in costs through economies on usage of personnel or other Resources Greater outputs from same inputsEfficiency Remedying duplication of effort or lack of coordination Better sub-objectives and targetsEffectiveness Better achievement of objectives by changing the nature of outputs or improved targeting Shorter waiting listsService Improved equity in access to programmesPlanning & Better control and management of humanControl resources, assets, projects and resources Improved forms of account, including commercial formatsAccountability Better and/or more accurate performance indicators
  • PA of I-T Systems - I Strategic and operational management of IT withinthe entity, including assurance that IT is included inthe overall corporate governance of the entity IT project management within the entity, includingthe entity’s record in meeting legislative and otherdeadlines; The risk management practices of the entity inrelation to IT IT system design, development and maintenancecontrols Compliance with standards, including externalstandards Application controls Processing controls, including audit trails
  • PA of I-T Systems - II Business continuity arrangements Data integrity, including sampling of data(possibly using CAATs) Access controls and the physical and logicalsecurity of networks and computers, includingInternet firewalls Controls to safeguard against illegal software Performance management and measurement Other issues that arise during the audit
  • Software used in I-T PA Audit Teamrisk – risk assessment tool for preparing risk based audit plan TeamMate Teamschedule – tool for scheduling staff (PWC) resources to audit Teammate (EWP) – electronic work paper system Teamcentral – web-based issue tracking system Crystal Ball Uses Monte Carlo simulation in (US GAO) conjunction with a cost estimation spreadsheet on the costs to give a range of values and the probability or the certainty of the values - useful for performing cost and uncertainty analysisEarned Value Analysis Measure progress of project Provide schedule and budget variances on a particular date Forecast completion date and cost of completion of project
  • Sources of Performance Measures Citizen surveys: Especially in areas of socialservices, police, garbage collection etc. - extremecare be exercised in determining sample size andpreparing questionnaires Trained observer ratings: Used to rate conditionsin a particular area, using a pre-designated ratingcriteria such as street cleaning Industry standards: Comparison of the resultswith prevalent standards in that area or with similarprograms Internal records: Occupancy rate for residentialhomes, user complaints, etc.
  • Some Words of Caution - I Auditor not expected to carry out theabove exercise himself Primary duty is to see if the auditee hasa reliable system to evaluate its owneffectiveness In that case, he would focus his attentionon the adequacy of these procedures In case the auditee does not have areliable mechanism to measure its owneffectiveness the auditor may have to do ithimself
  • Some Words of Caution - II Simple explanation of possible reasons forshortfall in achievement of goals not enough Ex: Not enough to say that the agency could notachieve its goals because of cost over run or timeover-run Instead auditor should analyze the reasons infollowing two categories: Factors that could have been influenced by theagency Factors that could not be influenced by theagency such as Government policy, applicablerules, socio- political environment, appropriationof funds and availability of trained manpower, etc.
  • Format of PA Report - IIf the management has a system of effectiveness measurementthe PA auditor: Documents the managements system of effectivenessmeasurements Assesses the appropriateness of performance indicators Verifies the results and compares them with the appropriate standard Formulates an opinion on the effectiveness of managements systemEffectiveness measures differ from efficiency measures Ex: A government department has the mandate to develop andeconomically backward region Efficiency measures would be the number of industries whichmoved to the backward region Effectiveness measures would relate to the figures of unemploymentor income level
  • Format of PA Report - IIIf there does not exist (or is weak) a system ofeffectiveness measurement then the auditorhas to generate his or her own dataAuditor should be careful about: Externalities of the programme Cost of data collection Method should be agreeable withmanagement
  • Some Sample Report Layouts - IReport No. 9 of 2009-10, National Rural Health Mission Executive Summary Introduction Planning & Monitoring Convergence Fund Flow Management Infrastructure Development & Capacity Building Procurement & Supply of Medicine & Equipment Information, Education & Communication Achievements in healthcare Conclusions & Summary of Recommendations
  • Some Sample Report Layouts - II Report No. 35 of 2010-11 Management of Defense EstatesPrefaceSummary of RecommendationsIntroductionLand norms, records & ownershipLand useManagement of LeasesManagement of old grant bungalowsConclusion
  • Some Sample Report Layouts - III Report No. 18 of 2010-11 Taxation of payments to non-residents Preface Executive Summary Introduction Oversight on forex flows Controls on Remittances Sectorwise Analysis Mistakes in Assessment
  • Concluding Concerns • PA reporting an administrative appraisalI than financial/certification audit • Fine line of distinction between audit ofII policy and audit of policy implementation • Clash of perspectives and interpretationsIII between audit and management • Charges of data manipulation andIV interpretation by auditors • Charges of auditors turning blind eye toV management replies
  • The Performance Auditor’s ProfileWho you ARE: √ Independent evaluator Economist/Statistician Public policy analyst Accountant/Auditor Aid to administration Who you are NOT: Legislative policymaker X Implementing executive Judicial officer RTI activist/Journalist Anti-corruption crusader