The document summarizes insights from recent audit reports regarding evaluation practice. It discusses four reports that examined pest management in Western Australia, earthquake preparedness in British Columbia, bushfire preparedness in Queensland, and the effectiveness of catchment management authorities in Victoria. The reports found weaknesses in evaluation and monitoring activities. They recommended improvements such as developing strategic plans, prioritizing activities, tracking preparedness, and reporting on the effectiveness of programs. Auditors see evaluation as an important accountability tool, but often find that more evaluation is needed.

1. What auditors are saying about
evaluation…
AES 15, International Evaluation Conference
5-9 September 2015, Melbourne
SEPTEM,BER 2015

2. I will present some lessons and insights for
evaluation from a range of recent audit reports
by regulators and assurance providers, from
across a range of Australian and international
jurisdictions.
Effectively, is a snapshot of some of the things
auditors have said about evaluation during 2014-
2015.
WHAT YOU WILL HEAR
SEPTEMBER 2015

3. I will talk about four reports.
All provide insights into evaluation practice and use.
All make recommendations that evaluators might find
interesting, and that reach across the boundary between
assurance and evaluation.
WHAT ELSE?
SEPTEMBER 2015

4. I have been summarising reports into a simple five-
stage format for a number of years now, for my own
use and reference.
The simple flow I use is:
• What was the report looking at?
• What does it say? and
• What does that mean?
You can find them on Slideshare.net
IN 5 SLIDES
SEPTEMBER 20153

5. WA Auditor General Performance Audit
Report on Managing Plant and Animal
Pests (Dec 2013)…
OAGWA looked at how well government pest management
programs were being run, focussing on the Departments of
Food and Agriculture (DAFWA), Conservation and Wildlife
(DPaW) and Fisheries (DoF).
OAGWA surveyed 692 private land managers as part of
the audit.
JANUARY 2014

6. While the legislative framework seems to be in place,
it just hasn’t been implemented yet.
The lead agency, DAFWA, is following an ‘invasion
management model’ used by others, but is struggling
to get it running well.
There are information gaps, and like other cross-
agency programs, there seems to be a bit of agency
‘siloing’ and competition going on.
WHAT DOES IT SAY.
JANUARY 2014

7. The AG found that they still haven’t managed to pull
together :
• a strategic plan for pest management,
• a current picture of pest impact, or
• a risk-based approach to prioritising activity.
In the last 10 years, compliance notices issued has
fallen by 99 percent. Over the same time period,
DAFWA funding has been cut, with a 32 percent
reduction in staff numbers.
WHAT ELSE DOES IT SAY.
JANUARY 2014

8. The auditor found limited evaluation of the
effectiveness of the mechanisms used to control pests.
DAFWA just didn’t have the monitoring program to
support it.
The auditor also through that the reporting of
effectiveness of programs inadequate.
This is a trifecta result.
WHAT ELSE DOES IT SAY.
JANUARY 2014

9. • It looks pretty bad.
• Sounds like the Auditor General found that WA
has starved on-ground pest management
activities, while failing to set up an effective
governance framework.
• Evaluation was seen as a key part of the
governance framework, to demonstrate that
operational outcomes are being achieved.
• The DAFWA response dodges evaluation.
The comments on these slides are the views of Tim Kirby, Sydney.
You should always read the report itself before putting any money on the line.
WHAT DOES IT MEAN.
Tim Kirby, Sydney CA, CIA, LA-EMS
au.linkedin.com/in/timkirbysydney
JANUARY 2014

10. British Columbia Auditor General audit
of Catastrophic Earthquake
Preparedness (Mar 2014)…
Recent studies have shown that British Columbia,
Canada, is prone to a catastrophic earthquake, like other
bits of the Pacific Rim.
The Auditor General looked at how well prepared
Emergency Management BC was.
MARCH 2014

11. The auditors found that EMBC was under-resourced, had
under-planned and under-reported.
Recent studies indicate that British Columbia is vulnerable
to a big earthquake, and the recent Christchurch 2010-2011
quakes are cited as a wake-up call.
The auditors found that British Columbia is not adequately
prepared for a big earthquake. This is the same finding
reached by an audit in 1997.
WHAT DOES IT SAY.
MARCH 2014

12. Disaster preparedness relies on planning and
training, and the auditor found that neither were up
to scratch.
The plans were based on analysis that wasn’t
current best practice. The emergency training and
exercises were insufficient.
Funding budgets for EMBC hadn’t changed since
2006.
WHAT ELSE DOES IT SAY.
MARCH 2014

13. The auditor concluded that EMBC couldn’t
demonstrate that it was adequately prepared to
manage a disaster.
They found that EMBC wasn’t tracking
preparedness, which would make it difficult to
respond effectively to a disaster, and wasn’t
reporting on preparedness to the public.
WHAT ELSE DOES IT SAY.
MARCH 2014

14. • BC is prone to a big earthquake, but isn’t really
prepared. And the EMBC isn’t saying so.
• This worried the auditor so much, that he called on
individuals to make their own plans.
• Australian emergency management agencies should
read this report cover to cover, so they spend the
new funding package wisely.
• Visit BC now, because it will take some time to
recover from the big one.
The comments on these slides are the views of Tim Kirby, Sydney.
You should always read the report itself before putting your reputation, or any money on the line.
WHAT DOES IT MEAN.
Tim Kirby, Sydney CA, CIA, LA-EMS
au.linkedin.com/in/timkirbysydney
MARCH 2014

15. QAO Bushfire Preparedness
performance audit (Dec 2014)…
Three recent reviews of bushfire risk made 168
recommendations that related to bushfire safety and the
Queensland Fire and Emergency Services.
As the fire season approached, the Queensland Audit
Office looked at how well the QFES implemented the
recommendations.
JANUARY 2015

16. The auditor didn’t like weaknesses in coordination
and leadership, recommending that QFES
centrally coordinate bushfire risk data and more
actively chair Fire Management Groups.
The auditor also spotted some basic planning
problems, including an inappropriate focus toward
response, rather than preparedness and prevention;
and no links between different planning scales.
WHAT DOES IT SAY.
JANUARY 2015

17. The TOM bushfire management tool used by QFES is
well-tested, but could be more useful if can get live or
recent data and be more broadly applied.
The auditor indicates TOM implementation has a
‘reliance on local knowledge and fluid interpersonal
communication channels’.
This tongue-twisting statement could indicate some key
people are not playing like a team.
WHAT ELSE DOES IT SAY.
JANUARY 2015

18. “Queensland can respond to, prevent and prepare for
bushfires better now than five years ago, due to those
actions it has taken up from the three reviews”.
But program evaluation was still weak, with.
• no reporting of hazard reduction burn effectiveness,
• no monitoring of the effectiveness of educational
material designed to prepare communities for the
threat of bushfire.
WHAT ELSE DOES IT SAY.
JANUARY 2015

19. • Evaluation works. Queensland bushfire
management has got better.
• But it looks like links between local,
regional and state-wide agencies are weak.
• This may not matter for each bushfire
response, but the lack of teamwork may
make it hard for governments to allocate
ongoing funding.
The comments on these slides are the views of Tim Kirby, Sydney.
You should always read the report itself before putting your reputation or any money on the line.
WHAT DOES IT MEAN.
Tim Kirby, Sydney CA, CIA, LA-EMS
au.linkedin.com/in/timkirbysydney
JANUARY 2015

20. VAGO Report on CMA Effectiveness
(Sep 2014)…
Over the last five years, Victorian catchment management
authorities have received $486M from the Victorian
Government and another $233M from the Commonwealth.
The Auditor General of Victoria looked at how well four
CMAs were going, and how well DEPI was supporting and
monitoring CMAs.
NOVEMBER 2014

21. The auditor found that state-wide catchment
management planning had a ‘fragmented and
short-term’ focus, contrasting nicely with the
‘integrated and long-term’ requirement of the
legislation.
Regional level planning seemed better, but was
hampered by short-term resourcing and poor
‘strategy ownership’ by partner agencies
WHAT DOES IT SAY.
NOVEMBER 2014

22. Monitoring, evaluation and reporting was weak. This
seems to be an expected finding for auditors looking at
NRM programs though.
Goulburn Broken CMA came out best for MER though,
because they had the longest running MER framework and
routine performance reporting to the board.
Victorian Catchment Management Council also comes out
of the audit looking good.
WHAT ELSE DOES IT SAY.
NOVEMBER 2014

23. CMA governance was being adequately supported by
DEPI, although it sounds like the independence of the
boards is more limited than other jurisdictions (like
NSW).
The auditor found ‘no statewide goals, priorities and
measures for integrated catchment management’.
Wimmera CMA cops a few hits, eg. for its 7 year old
community engagement plan
WHAT ELSE DOES IT SAY.
NOVEMBER 2014

24. • While regional level natural resource
planning is going OK, the lack of integration
at the state-wide level may be contributing to
declining natural resource condition.
• Victoria was not leading the way in MER,
or integrated catchment management.
• Agencies like the VCMC, and the NSW NRC,
are needed to highlight gaps for larger
departments.
The comments on these slides are the views of Tim Kirby, Sydney.
You should always read the report itself before putting your reputation, any money, on the line.
WHAT DOES IT MEAN.
Tim Kirby, Sydney CA, CIA, LA-EMS
au.linkedin.com/in/timkirbysydney
NOVEMBER 2014

25. • Auditors like evaluation- it is an important
accountability tool.
• Auditors want more evaluation.
• Performance audit reports might be useful
to evaluators.
The comments on these slides are the views of Tim Kirby, Sydney.
You should always read the report itself before putting any money on the line.
WHAT DOES IT MEAN.
Tim Kirby, Sydney CA, CIA, LA-EMS
au.linkedin.com/in/timkirbysydney
SEPTEMBER 2015