A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

1. Audit | Tax | Advisory
MEASUREMENT OF RESULTS (CONDITIONAL
TRANSFER MODELS)
The Role of the External Concurrent Audit
March2014

2. Audit | Tax | Advisory
Agenda
• Audits, External Audits, Internal Audits,
Concurrency
• Costs and Benefits expected from the ECA
• Experiences
• Lessons learned
• Conclusion

3. Audit | Tax | Advisory
The 4 Qs
Sooner or later, during the design stage of any
given program (the sooner, the better), the
team in charge of strategic definitions will be
faced with the 4 questions related to the
Audit function:

4. Audit | Tax | Advisory
The 4 Qs
Should we include the
Audit function in our
Project?
Should we define an
Internal Audit or an
External Audit?
Should our project
consider a Concurrent
Audit?
What should we do
with the results of
the Audit?

5. Audit | Tax | Advisory
Building (possible) answers
• The Audit function is not equivalent to the Audit structure.
Some organizations may create an Internal Audit structure, while
others may outsource such task. Some may design a Monitoring
and Evaluation Area. Smaller customer-driven organizations may
develop this function in the Customer Care Area.
• However, every organization must evaluate the need for the
Audit function, which is in charge of assessing what was done,
what should have been done and of suggesting corrections in
the event of discrepancies. Without such feedback,
organizations operate blind-folded throuought their lifecycle,
without being aware of the detours until the end of their
operation.

6. Audit | Tax | Advisory
External Audits vs. Internal Audits
• In the case of standard External Audits, the achievement of
the audited organization’s goals is not seen as their mission.
The goal is to ensure the veracity of the reported financial
information.
• Internal Audits focus on the process and the organization’s
compliance with the defined procedures. The result is
relevant in terms of the measurable result of the process.
• The is no such dilemma: they are two models, with different
goals and methods, which may be independent from each
other in the worst case scenario, and complementary in the
best.

7. Audit | Tax | Advisory
The ECA Model
• The External Concurrent Audit (ECA) appears between these
two models with the obvious tensions and overlaps in terms
of conceptual boundaries.
• It maintains its mandate of external independence and
opinion source , but it incorporates most of the tools of the
Internal Audit for the evaluation of processes and
measurement of results, as well as the improvement
opportunity approach.

8. Audit | Tax | Advisory
What kind of Projects are eligible for
an ECA?
• Long-term operation
• Distributed operation model
• Multiple parties involved
• High transactional component among players
• Introduction of New Management Models
• Results- Based Management Models
• Changing Operation Frameworks

9. Audit | Tax | Advisory
What kind of Projects are eligible for
an ECA?
The ideal projects are those in which the complexity
of the operation in terms of change over time, the
number of participants and geographic distribution
requires the implementation of a frequent and
close evaluation and report scheme to ensure that
the information is provided in a timely manner to
enable the adjustment or verification of the
operation.

10. Audit | Tax | Advisory
The Problem of Audit Results
• Regardless of the answer to the previous questions,
we must always define what to do with the Audit
results, taking into account the following:
– There is always the possibility of detours (even in the
design).
– Detected variations can be taken as flaws or as
opportunities for improvement.
– Hiding (or regulating) detours does not improve a project’s
performance.
– The more detailed the expected scope, the more specific
the Audit results will be.

11. Audit | Tax | Advisory
Should we include
the Audit function
in our Project?
Should we define an
Internal Audit or an
External Audit?
Should our project
consider a Concurrent
Audit?
What should we do
with the results of
the Audit?
The Audit function is inevitable.
Therefore, it is better to plan it
appropriately.
Both standard models complement
each other, which means they can
co-existwithin a Project.
Not every project is eligible to
benefit from an ECA model. It is
essential to identify those that are.
Audits create results which are
commonly expressed as detours; we
must take this into account from the
design stage to plan any actions that
should be taken.

12. Audit | Tax | Advisory
Benefits of the ECA Model
• Early detectionof detours and improvement
opportunities
• Incorporationof the perception of the different
players involved
• Fast adjustmentof work programs to regulatory
changes
• Cooperation in terms of the complianceof goals
by means of uniform approval criteria
• Creation of multidisciplinary teams, which
enables the fluent interactionof players

13. Audit | Tax | Advisory
Costs of the ECA Model
• Intensive time application
• Steep learning curves
• Counterpart availability
• Careful strategic definition of scopes
• Strategic decision about what to do with the
results
• Effort to document administrative decisions

14. Audit | Tax | Advisory
Our Experience with ECA
PROAPSREMEDIAR
BID
2003
PISMIPPLANNACER
APL1
BIRF
2005
PISMIPPLANNACER
APL2
BIRF
2009
FESP
BIRF
2011

15. Audit | Tax | Advisory
What did we learn?
• What to do
• When to do it
• How to do it
• What to do it with
• Who to do it with

16. Audit | Tax | Advisory
ECA Roadmap
Identification of Central
Process
Development of Work
Plan
Implementation of Work
Plan
Planning of work cycle
Fieldwork scheduling
Field Survey
Consolidation and
Analysis
Report Completion

17. Audit | Tax | Advisory
Risks associated with Project
Management
• Scope Risks
– The Work Plan might not cover the expected scope.
– The Work Plan might cover the expected scope but it
might not meet the strategic needs.
• Control Risks
– Excessive decentralization
– Too inflexible
• Detection Risks
– The Work Plan might not evolve fast enough to detect any
emerging variations.
– Variations across independent evaluators
• Relevancy Risks
– Observations in one site may not be relevant in another.

18. Audit | Tax | Advisory
Challenges
• To include local realities into Program guidelines
• To coordinate the agendas of resources and
counterparts in order to meet defined deadlines
• To match the consultant’s expectations with the
auditor’s responsibilities
• To adjust work programs to regulatory changes
without affecting deadlines

19. Audit | Tax | Advisory
ECA Guidelines
• Multidisciplinary local team
• Ongoing revision of Work Programs
• Ongoing training and updating
• Important supervising and monitoring component
• Constant contact with counterparts

20. Audit | Tax | Advisory
ECA Guidelines
• Multidisciplinary local professional team
• Ongoing revision of Work Programs
• Ongoing training and updating
• Important supervising and monitoring component
• Constant contact with counterparts

21. Audit | Tax | Advisory
AUDITORESDE
CAMPO
PROJECT LEADER
IT COORDINATOR AUDIT COORDINATOR
TECHNICAL AUDIT
COORDINATOR
AUDIT SUPERVISOR
TECHNICAL AUDIT
SUPERVISOR
IT AUDITOR FIELD AUDITOR
TECHNICAL FIELD
AUDITOR
AUDIT
COORDINATION
TEAM
TECHNICAL AUDIT
COORDINATION
TEAM
EXPERT IN
STATISTICSCENTRAL BASE
LOCALBASE
MIXEDBASE

22. Audit | Tax | Advisory
ECA Guidelines
• Multidisciplinary local team
• Ongoing revision of Work Programs
• Ongoing training and updating
• Important supervising and monitoring component
• Constant contact with counterparts

23. Audit | Tax | Advisory

24. Audit | Tax | Advisory
ECA Guidelines
• Multidisciplinary local team
• Ongoing revision of Work Programs
• Ongoing training and updating
• Important supervising and monitoring component
• Constant contact with counterparts

25. Audit | Tax | Advisory

26. Audit | Tax | Advisory
ECA Guidelines
• Multidisciplinary local team
• Ongoing revision of Work Programs
• Ongoing training and updating
• Important supervising and monitoring component
• Constant contact with counterparts

27. Audit | Tax | Advisory
Work Cycle
External Concurrent Audit
Month 1 Month 2 Month 3 Month 4
Two-month period
under analysis
Report
submission
Field auditor survey Supervising visits Report
review
Field auditor
survey
Set up of
sampling
Report
completion

28. Audit | Tax | Advisory
ECA Guidelines
• Multidisciplinary local team
• Ongoing revision of Work Programs
• Ongoing training and updating
• Important supervising and monitoring component
• Constant contact with counterparts

29. Audit | Tax | Advisory
Key Success Factors
• Highly-trained work team
• Efficient communication
• Adaptability to changes in the operation
model
• Constant counterpart feedback
• Careful planning of times and resources

30. Audit | Tax | Advisory
Lessons Learned
• Steep learning curves call for the careful
planning of changes in the ECA throughout
the project’s lifecycle.
• The strengthening of the auditor’s role
through the adoption of their observations as
binding for the program requires a negotiation
effort on the part of those running the
Program in the short term, but it leads to a
boost of results in the long term.

31. Audit | Tax | Advisory
Conclusion
• The ECA scheme provides the independence
of an External Audit with a methodological
framework based on process evaluation
• Not every Program will benefit from an ECA
model.
• Those Programs that benefit from an ECA
model will clearly obtain greater benefits from
this model than with a standard Audit model.