This document outlines Meat & Livestock Australia's (MLA) development of an evaluation framework to systematically assess the value of its programs and meet increasing demands for accountability from government and industry stakeholders. Some key points: 1. MLA developed criteria for an effective evaluation framework, including the ability to evaluate programs' triple bottom line impacts, compare ex-ante and ex-post evaluations, and clearly communicate outcomes. 2. MLA worked with the Centre for International Economics to design a framework addressing these criteria across MLA's diverse marketing, market access and R&D programs. 3. The framework involves evaluating programs at different stages of their lifecycles to identify opportunities for improvement and guide future investment and

1. 2009AUSTRALASIAN EVALUATION SOCIETY INC.
AWARDS FOR EXCELLENCE IN EVALUATION 2009
“There is nothing more difficult to take in hand,
more perilous to conduct, or more uncertain in
its success, than to take the lead in the
introduction of a new order of things!”
Machiavelli
CATEGORY: BEST EVALUATION POLICY AND SYSTEMS AWARD
Meat & Livestock Australia (MLA) & The Centre for International Economics (CIE)
MEASURING THE VALUE
OF MLA PROGRAMS
“Learning by doing”

2. 1
BACKGROUND
As Australia’s premier research and marketing enterprise for the Australian red meat
industry, Meat & Livestock Australia Ltd’s (MLA’s) core activities are: working to improve
market access, building demand for Australian red meat, conducting research and
development (R&D) to provide competitive advantages from paddock to plate, and
partnering with industry to build capability.
MLA is majority funded by compulsory livestock transaction levies, processor levies and
matching government funds for R&D work. Other income is generated through activities
such as commercialisation of R&D and partnering with industry participants. Currently the
red meat and livestock industry is valued at $16 billion per annum.
MLA has constantly sought ways to more accurately and consistently measure and
report the benefits and costs of its activities. Periodically, since its inception in 1998, MLA
conducted ex-ante and ex-post evaluations using a distinct evaluation approach for each
program, however this led to infrequent communication that was not meaningful for key
stakeholders.
The largest single contributor to MLA R&D programs is the Australian Government,
through matching R&D contributions. The Australian Government is increasingly calling on all
Research & Development Corporations (RDCs) to account for the public investment in their
programs, requesting verifiable triple bottom line information on R&D program benefits in
relation to National Research Priorities.
Although interest in measuring the value of MLA programs had stemmed mainly from
the Australian Government, there is also evidence of a growing focus on accountability
among industry stakeholders.
By 2005, when MLA had been in existence for six years, the MLA Board deemed it
appropriate to systematically assess the value of much of its work.
An evaluation strategy was required for analysis of the net benefits of each program
within a coordinated structure capable of identifying new monitoring and measurement
parameters going forward.
MLA subsequently developed a comprehensive framework for program evaluation
across its diverse marketing, market access and R&D programs. (See appendix, Why Does
MLA Need A Framework For Independent Evaluation, for a detailed description.)
“Rigorous evaluation across MLA’s RD and
marketing programs ensures the net impact of
MLA programs can be determined in terms of
industry and wider community benefits.”
“Peer review at various levels has informed
the development of an evaluation framework
that fosters greater understanding by program
managers and increased engagement with key
stakeholders.”
LiveCorp program managers aligning programs
with strategies.
Sheepmeat councillors participate in peer review of
the counterfactual scenario.

3. 2
The MLA Executive Committee developed the
following criteria for judging the successful
development and implementation of an evaluation
framework that would meet this challenge. The
Executive Committee pronounced that:
1. A SATISFACTORY EVALUATION
FRAMEWORK WOULD:
 be appropriate for the consistent evaluation of
marketing, market access and RD programs
 be appropriate for triple bottom line analysis to
evaluate program outcomes against traditional cost-
benefit benchmarks and wider societal/environmental
considerations
 be sensitive to the influence of the program lifecycle
on evaluation processes and criteria
 be comparable between ex-ante and ex-post
evaluations
 permit easy identification of key parameters and
methodologies for ongoing monitoring and
measurement of program Key Performance Indicators
(KPIs)
 be rigorous, with all evaluations conducted using this
framework standing up to scrutiny from both
Government and industry
 represent a transparent approach, with any
measurement techniques applied also being
transparent
 be simple and comprehensible to MLA stakeholders
 be an efficient evaluation process that does not drain
resources.
2. A SATISFACTORY CORPORATE
LEARNING OUTCOME WOULD BE
EVIDENT WHEN:
 program managers demonstrate awareness of the
benefits of using the framework to identify and monitor
more appropriate program KPIs
 program managers demonstrate an understanding of
the rules that dictate where equivalent evaluation
approaches should be used in marketing, market
access and RD programs, and where there should be
departures from this equivalent approach
 program managers demonstrate an understanding of
how information on program evaluation can best be
incorporated into program proposals
 program managers demonstrate an understanding of
how to establish appropriate measurement and
monitoring procedures to allow effective and consistent
evaluation of new programs
 program managers demonstrate a sense of framework
ownership and a willingness to use it in all ongoing
work.
3. A SATISFACTORY STAKEHOLDER
COMMUNICATION OUTCOME
WOULD BE DEMONSTRATED BY:
 a sample set of communication brochures that
illustrates how to communicate the outcomes of a
program evaluation in simple and relevant terms to
various external key stakeholder groups as
nominated by the program manager
 presentation of program evaluation outputs in terms
that are relevant and understandable to internal
stakeholders at various levels including: program
investors, MLA executives, other program
managers and personnel at individual project
levels.
Meat Livestock austraLia
evaluationseries
Eating quality
The industry impact
2.1 improving eating
quality
MLA_Eat_Quality.indd 1 22/8/07 5:16:40 PM
Meat Livestock austraLia
evaluationseries
Food safety:
predictive microbiology
The industry impact
1.1 enhancing product
integrity
MLA_Food_Safety.indd 1 22/8/07 5:18:06 PM
Meat Livestock austraLia
evaluationseries
Market access
The industry impact
1.2 Maintaining and
liberalising access to
world meat markets
MLA_Market _Access.indd 1 25/1/08 2:51:01 PM
evaluationseries
Red meat nutrition
marketing
The industry impact
2.2 Enhancing the
nutritional reputation
of red meat
MLA_RedMeatNutri.2.2.indd 1 9/02/09 1:59 PM

4. 3
SITUATION
ANALYSIS
“Sustainability can be enhanced by
increasing …transparency and by
implementing sound monitoring and
evaluating performance systems. This
assures not only continued funding but
also probably attracts additional sources
of funding.” (International Service
for National Agricultural Research)
In an increasingly tight funding environment
intensified by international economic decline,
RDCs are challenged by growing competition for
a share of the public funding pool.
The possibility that Australian Government
policymakers and industry investors could follow
other countries, who have adopted competitive
funding mechanisms to redirect priorities,
increase accountability and reduce costs, has
placed greater pressure on RDCs to demonstrate
a favourable return on stakeholder investment in
light of their responsibility “to ensure funding is
allocated to achieve the highest return possible”1
.
The imperative for RDCs to justify their
worth escalates the need for transparency and
heightened communication with key sponsors
including the Australian Government and peak
industry bodies.
While MLA recognises that such
communication is dependent on a consistent
body of knowledge that can only be collected
through thorough evaluation, a system to
appraise its performance did not exist prior to
2005, hampering efforts to demonstrate
accountability to key investors and alignment
with the Australian Government’s National
Research Priorities2
and Rural Research and
Development Priorities3
.
The organisation perceived the situation as a
key chance to address this weakness and
assume a competitive posture.
MLA drew on its strengths: a professional
internal resource base committed to program
improvement (effectiveness, efficiency and triple
bottom line benefits4
); and external expertise in
evaluation management, engaging the CIE to
assist its development of an evaluation
framework that could withstand – and would
welcome – intense scrutiny of its programs.
1
MLA Program Evaluation Strategy 2007-10, MLA
2
Department of Education, Employment and Workplace Relations 2003, Australian
Government
3
Department of Agriculture, Fisheries and Forestry 2007, Australian Government
4
Atkinson, Dr L 2008, ‘The evaluator as change agent in history: challenges and
strategies for success’, Australasian Evaluation Society Inc. paper delivered on
September 12, 2008
LiveCorp program manager presenting KPIs. Kate Roberts presenting to program managers on Social Impact Assessment.
“We have common need for reliable
information in directing our investment and
need to continue discussion on this issue.”
– Professor Beth Woods,
Assistant Director-General Innovation,
Primary Industries and Fisheries,
Queensland Department of Employment,
Economic Development and Innovation
“I have been telling them (other
stakeholders) about what we and MLA did
(ex-post evaluation), in the lamb industry,
and how interesting and useful it was”
– Geoff Kroker,
Manager Strategy Development
Victorian Department of
Primary Industries

5. 4
Broad Goal
Purpose
Next Users
End Users
Partners/
Collaborators
NARRATIVE SUMMARY
Build maintain stakeholder
confidence in MLA
1. Reporting to Government
2. Engagement with industry
stakeholders
3. Guiding future investment
4. Building a culture managing for
outcomes
MLA Staff:
• MLA Board
• Managing Director
• Executive Committee
• Program Managers
Target Stakeholder Audiences:
• Federal Minister
• Parliamentary Secretary
• Australian Government
Department of Agriculture,
Fisheries and Forestry (including
agencies)
• Politicians
• Policy makers
• CRRDCC Secretariat
• Other RDCs
• Peak Industry Councils
• Industry Associations
• Producer Groups
• Industry Leaders
• Industry cynics
Evaluation Collaborators:
• Service Providers
• Other RDCs
• CRRDCC Secretariat
• PIC EDs staff
• Other RDCs
• Federal State Government Staff
MEANS OF VERIFICATION
• MLA Membership register
• Membership survey
• Feedback at AGM
• Annual review of Annual
Operating Plan (AOP) by PICs
• CRRDCC reporting feedback
• Levels of co-investment,
collaboration, and alignment with
NRPs as reported in Annual
Report
• Regular independent meta-
analysis of the results arising from
the framework.
• Results reported using a
scorecard for tracking progress
towards stated purpose and
objectives.
• Improvement in the alignment of
AOP KPIs and industry outcome
measures.
• Extent of alignment is peer
reviewed by MLA and PIC staff on
an annual basis.
• Regular survey of target audience
members in Governments, PICs,
and CRRDCC to assess value of
information
• Collection of evidence that
information has been used in
press releases, speeches, reports,
and policy documents.
• Evidence of shared and endorsed
ex-ante reviews
• Evidence of shared and endorsed
ex-post reviews
• Evidence of commitment to and
co-investment in collaborative
evaluation frameworks
MEASURABLE INDICATORS
• Membership numbers
• Membership satisfaction
• PIC support for MLA
• On-going support for levy
• On-going co-investment in RD
by Australian Government
• Collaboration between RDCs
1. Reporting to Government
a. Simple Relevant
b. Meeting compliance requirements
c. Independent objective
d. Understandable Useful for
Policy
2. Engagement with industry
stakeholders
a. Meeting member expectations
b. Simple Relevant
c. Understandable Useful for
Policy
3. Guiding future investment
a. Consistent across programs
b. Triple bottom line reporting
c. Used to better understand the
value add arising from leveraging
d. Comparable ex-ante to ex-post
4. Building a culture managing for
outcomes
a. Staff have easy access training
b. Outcome KPIs developed
reported
c. Measurement is transparent
objective
d. MERI is resource efficient
e. Use of program logic in program
planning
f. Understandable Useful for
Program improvement
g. Staff are rewarded
h. Expert verification
• Recorded evidence of program
improvement
• Value of evaluation results as an
input into decision making and
policy development by both
industry and Government
• All partners participate in the
development of the program logic
for all collaborative programs.
• All partners endorse and support
the inclusion of measurement
evaluation activities in to program
design in order to meet the data
needs of the evaluation
framework
ASSUMPTIONS
• Australian Government continues
to demand evaluation reporting
for continued support of the RDC
model
• PICs and Producers continue to
demand evaluation reporting to
support the RDC model
• There is no catastrophic disruption
to supply or market access
• Other RDCs continue to remain
open to collaboration and
committed to harmonisation of
evaluation reporting
• On-going support from the MLA
Executive Committee for the MLA
Program evaluation Strategy 2007
– 10
• PIC staff and resources will be
available to undertake these
reviews.
• That the current hard copy and
web based versions of the
evaluation results continue to
reach target audiences.
• The target stakeholder audiences
have a capability to interpret and
use results from evaluations.
• Evaluation capacity building
remains a high priority for all
partners and collaborators.
• Evaluation framework is efficient
and does not become a drain on
program resources.
‘People-Focused’ Logic for Capacity building projects: Dr Jess Dart, Workshop # 8, AES Conference 2005
LOGICAL FRAMEWORK MATRIX FOR ADOPTION
OF EVALUATION AT MLA (DART, 2005)

6. 5
The MLA Program Evaluation Strategy 2007-10
and the supporting program evaluation schedule
drive the evaluation process at MLA (see table
right). These are reviewed and agreed with the
MLA Executive team, on an annual basis.
The core lines of inquiry for the evaluation
framework are best summed up in MLA’s five
key evaluation questions. It is important that
these questions are sensitive to the influence of
the program lifecycle:
 Are MLA programs adding value?
 Are MLA’s collaborations efficient and effective?
 Does MLA facilitate innovation from concept
through to commercialisation?
 Does MLA facilitate increased industry
capability and investment in innovation?
 Is MLA identifying – and correctly managing
– industry risk?1
1
MLA Program Evaluation Strategy 2007-10, MLA
IMPLEMENTATION
IDENTIFYING OPPORTUNITIES FOR PROGRAM IMPROVEMENT
ACROSS THE PROGRAM EVALUATION LIFECYCLE
The diagram illustrates how the MLA framework and specific business/corporate
functions support programs throughout their various program evaluation lifecycle stages:
ENGAGEMENT:
Ex-post
analysis
INPUT:
Ex-ante
review
OUTPUT:
AOP* KPI
Review
OUTCOME:
Monitor
Report
BUSINESS SERVICES
• Commercialise/Adopt
• Corporate Communications
CORPORATE SERVICES
• Evaluation
• SAP/Finance/Contracts
ALL MLA PROGRAMS HAVE LIFECYCLES
KPI
ENGAGEMENT
IMPACT
TIME
Early indicators
e.g Investment +
engagement levels
Progress
- Ongoing engagement
- Satisfaction
- Project progress
Outputs
- Workshops
- Technology
- POS material
Potential benefit quantified
Awareness
Capability to adopt
Adoption
Measure impact
Adoption
benefit realised
KPI
OUTPUT
KPI
INPUT
KPI
OUTCOME
MLA EVALUATION SCHEDULE
*AOP: Annual Operating Plan
2007 2008 2009 2010
Enhancing product integrity 1.1 
Maintaining and liberalising access
to world meat markets 1.2 
Improving eating quality 2.1 
Enhancing the nutritional reputation
of red meat 2.2 
Aggressive promotion in the
market place (sheep) 2.5 
Increasing cost efficiency and
productivity – on farm 3.1 
Aggressive promotion in the market
place (beef) 2.5  
Improving industry and market
information 3.3  
Ensuring sustainability 3.4  
Developing new markets and
products (beef) 2.3 
Developing new markets and
products (sheep) 2.3 
Promoting industry integrity (beef) 2.4 

7. 6
ASSESSMENT
“Over time every MLA program will
be subject to thorough, independent
evaluation. This process imposes a
new discipline on the company. …It is
recognised that all staff need to share
and benefit from the evaluation process.
It is the responsibility of the Executive
and Managing Director to illustrate and
drive those benefits through the
company [and] this will be acted on.”
David Palmer,
MD MLA, 4 December 2008
Effective implementation of the framework is
facilitated by ensuring that the principles of the
MLA Evaluation Framework are aligned with the
company’s values, vision and strategy.
The table below summarises the challenges and
issues that have arisen during the implementation
phase. MLA management has responded with
strategies to increase the effectiveness of the
change management process, build evaluation
capacity, and create a more evaluation-focused
organisational culture.
These strategies have included: learning on the
job, using a ‘people-focused program logic’
approach to implement the framework, creating a
cross-functional team to champion
implementation, engaging key stakeholders,
linking outcomes to employee performance
measures, and leveraging off the Australasian
Evaluation Society (AES) network of expertise.
SUMMARY OF CHALLENGES TACTICS FOR CHANGE
CHALLENGES TO CHANGE SUCCESSFUL CHANGE TACTICS
1. Making the complex
simple
1. Alignment with MLA values, vision,
and strategy
2. Embedding a focus
on outcomes
2. Engaging key stakeholders in
setting better outcome KPIs
3. Maintaining independence and
consistency across programs
3. Engaging the Executive
Committee
4. Building evaluation
capability
4. Creating cross-functional teams,
learning by doing, reflecting
5. Developing a learning culture 5. Creating layers of independence
and transparency
SHARED GOALS
MEAT INDUSTRY
STRATEGIC
PLAN
GOVERNMENT
RESEARCH
PRIORITIES
INDUSTRY CONSULTATION PROCESS
MLA STRATEGIC PLAN
ANNUAL OPERATING PLAN
PROGRAMS
SUB PROGRAMS
WORK PLAN
STAFF PERFORMANCE PLANS
Linking evaluation with strategy
Meat Livestock austraLia
Why does MLA need
a framework for
independent
evaluation?

8. 7
QUALITY
EX-POST EVALUATION
APPROVAL PROCESS
MLA needs credible, rigorous and consistent
evaluation procedures across the entire range of
its RD and marketing programs to ensure
program assessments can effectively determine
the net impact of MLA programs in terms of both
industry and wider community benefits.
Each year the Executive Committee determines
the major ex-post evaluations that will make a
significant contribution towards meeting
objectives of the independent company
performance review scheduled for 2010. These
appraisals are subject to a scrupulous planning
and reporting process to ensure consistent, high
quality information is provided to program
managers to assist decision-making. The
following flow chart demonstrates the approval
process that supports this approach.
MLA EX-POST EVALUATION APPROVAL PROCESS
Approval of proposed MLA Program Evaluation Schedule
Manager Evaluation Program Improvement works with relevant
program managers to develop evaluation proposals
Evaluation team develops evaluation proposal
Independent expert verification team
Program Manager forms the evaluation team
Evaluation team develops the evaluation plan
Data collection analysis
Evaluation team develops draft evaluation report
Independent expert verification team
Evaluation brochure is published and communicated as appropriate
NO
NO
NO
NO
NO
YES
YES
YES
YES
YES
Manager Evaluation
Executive Committee
Executive Committee

9. 8
LEADING EDGE
The MLA Evaluation Framework embodies three
simple principles which guide program
managers in their development of individual
program KPIs.
These are consistent with the Council of Rural
Research and Development Corporations’ Chairs
(CRRDCC) evaluation guidelines and are based on
the Commonwealth Department of Finance and
Administration approach to program evaluation:
1Setting a baseline - establishing the change
in industry and public benefits that would
occur if MLA does nothing.
2Showing evidence of causality – providing
the evidence that proves MLA program
outcomes have led to more industry and public
benefits when compared to the baseline.
3Attribution – estimating the proportion of
additional benefits that can be reasonably
attributed to the contribution made by MLA
programs relative to other influencing factors.
These three simple principles are generic. They
are valid in every context within which MLA
programs operate. They can be consistently
applied using either a ‘bottoms up’ or ‘tops
down’ evaluation approach.
‘Bottoms up’ is always the preferred way to
conduct ex-post evaluations. This approach is
based on reviewing the theory of action for the
program and the observed changes in the
relevant KPI measures compared to the baseline.
This evidence is then used to substantiate the
benefits claimed by the program. It is a very
transparent process and any claims arising can
withstand close scrutiny:
 because the baselines are produced using real
data;
 because there is clear evidence of causal links
between program outcomes and observed
changes in industry performance; and
 because the theory of action means that
MLA’s role is well understood and therefore
attribution is more objective.
The decision to go ‘tops down’ for an ex-post
evaluation is normally due to a combination of
factors such as: a lack of evidence linking
program outcomes (e.g. adoption figures) to the
benefits being claimed by the program; and/or
situations in which a large number of parties has
contributed to the achievement of claimed
benefits, such as a multi-RDC project.
Unfortunately, although it often turns out to be
the only practical option, the ‘tops down’
approach to ex-post evaluation is perceived to be
more arbitrary and less transparent than the
‘bottoms up’ approach:
 because establishment of the baseline is
reliant on industry expert opinion;
 because causal linkages between program
outcomes and observed changes in industry
performance are based solely on justified
assumptions;
 because there is a need to test attribution
assumptions using a sensitivity analysis such
as breakeven modelling; and
 because multi-party collaboration means that
attribution has to be negotiated and therefore
is usually a very subjective process.
There have been numerous issues and lessons
arising from the application of the MLA
Evaluation Framework to the programs evaluated
so far:
 limited evidence for causal linkage between
program outcomes and observed industry impact
for both marketing and research;
 negotiating the attribution of observed
industry benefits across key stakeholder
contributions.
Framework implementation has been an
evolutionary ‘learning by doing’ process. Peer
review at various levels (detailed under ‘Quality’)
has informed the framework’s refinement and
improvement. This iterative approach has
culminated in the development of a framework
that can be adapted so as to be fit for purpose
(i.e. marketing vis-à-vis market access vis-à-vis
RD), while maintaining the rigour, transparency,
simplicity and resource efficiency necessary to
allow increased understanding by program
managers and engagement with relevant
stakeholders.

10. 9
CONTRIBUTION
TO KNOWLEDGE
The CRRDCC released a set of “Guidelines for
Evaluation” in May 2007 which were revised in
May 2009.
The guidelines assist evaluation of the impact of
RD programs funded by RDCs and were
established by the CRRDCC to report to
Government on the:
 collective returns from RDCs to industry
 collective public and spillover returns from
RDCs
 public and spillover returns that are conditional
on public contributions to RDCs.
It is intended that the processes and procedures
set out in the CRRDCC framework will represent
the minimum evaluation requirement of each
RDC. A systematic evaluation of a sample of
investments will constitute an ‘evaluation’ pool
and an analysis of the overall economic, social
and environmental returns is provided on an
annual basis.
EFFECTIVENESS
The discipline behind the process of negotiating
attribution of benefits to program stakeholders
has had the greatest impact on MLA staff and
MLA investment partners.
The MLA Evaluation Framework process for
apportioning benefits across program
stakeholders typically:
a. considers what other stakeholders were
involved;
b. considers the contribution of other
stakeholders; and
c. rules that the total estimate should incorporate
all stakeholders involved and add up to 100
per cent.
The process of determining attribution involves
verification with all program stakeholders.
EQUITY
MLA has adopted the Australasian Guidelines for
the Ethical Conduct of Evaluation (see http://
www.aes.asn.au/). These guidelines are
consistent with the current MLA Code of
Business Conduct Ethics (2006), but can be
specifically applied to the execution of the
evaluation work plan outlined in the MLA
evaluation strategy 2007 – 2010.
One of the key objectives of the MLA Evaluation
Framework is to enable an outcomes-focused
performance culture that rewards measurement
and achievement of KPIs.
An effective evaluation framework provides
findings that can be used for program decision-
making, program improvement and optimal
resource allocation. Moreover, it empowers MLA
staff to further sharpen the focus on achieving
real outcomes that align with the objectives of
the MLA Strategic Plan 2008 – 2012 (detailed
under Assessment).
The MLA Executive Senior Managers Forum is
a recent initiative that is partly the result of
implementation of the evaluation framework (see
diagram below). Its objectives are to:
1Ensure all staff gain a broader understanding
of, and have greater input into, the company’s
objectives and activities
2Ensure alignment of industry and government
priorities with MLA strategy and program
implementation
3Ensure program measures align with MLA
strategy measures to allow clear reporting to
stakeholders.
Implementation of the evaluation work schedule
promotes a collaborative culture of robust
program review and improvement within MLA. It
will allow successes to be rewarded and
replicated, and facilitate knowledge-share of
lessons learned. This enables changes that
underpin MLA’s continuous improvement in the
delivery of its programs and services.
The MLA Strategic Plan 2008 – 2012 works in
concert with this evaluation work schedule to
deliver a culture of collaborative inquiry and
participative reflection within MLA and with our
investment partners.
ProjObjectives
Activity KPIs
Contracted
Activities
MLA measures
AOP Objective KPIs
Strategy 1
Strategy 2
Strategy 3
Strategy n
AOP Strategy
Strategy KPIs
Projects
Industry Outcomes
MLA measure KPIs
Each KPIsfor AOP
Objectives
Accumulate
(Over 5 years)
KPIs Cascading Down –
from industry outcome to Strategic Plan
to AOP to project to contract activity
KPIs Cascading Up – from the Project
to AOP to Strategic Plan levels
Improving MLA governance through
strategic alignment of KPIs

11. 10
USE OF INTERNAL AND
EXTERNAL RESOURCES
The role of external evaluators (typically the CIE) is to
conduct evaluation in a way that builds the capacity of MLA
by supporting the evaluation team during any or all of the
evaluation phases. This role includes:
 facilitating the scope scale of the evaluation
 advising on appropriate data sources and methods of
collection
 designing and implementing various data collection
tools and instruments
 drafting the evaluation report.
Using internal staff in evaluations brings corporate
knowledge of both the program and MLA into the evaluation
process. Internal teams have the advantage of being able to
quickly modify the design of an evaluation when it becomes
apparent that an evaluation activity is not working.
Using external evaluators brings specific expertise in
evaluation design, data collection and analysis or reporting
to the evaluation. External evaluators also bring
independence to the process.
The ex-post evaluations of most MLA programs are
conducted by CIE. The program manager is required to
manage the process to ensure that the findings and lessons
learned are relevant and useful. This involves actively
negotiating the evaluation plan with the contractor/consultant,
monitoring data collection and analysis, and approving the final
evaluation report in consultation with the independent expert
verification team and the Executive Committee.
The activity diagram above illustrates the context and
interaction between each of the components of a successful
evaluation framework at MLA. It illustrates that reporting to
government and engagement with peak industry
councils/industry – the function of the MLA evaluation
brochure series – contribute to the overarching objective of
building and maintaining stakeholder confidence in MLA.
Program improvement is an equally important
outcome of the framework. Its embodiment will be observed
as guiding improvement in future investment strategy and
the creation of a more outcomes-focused management
culture at MLA.
The program evaluation schedule is now past the
half-way mark and there are very promising signs of sustained
program improvement at MLA (e.g. adoption of program logic
for planning etc.).
The chart above illustrates the results of a subjective
assessment against criteria for MLA’s progress towards
desired reporting and program improvement outcomes. The
comparison between 2005 and 2008 suggests good progress
has been made, particularly with respect to reporting to the
Australian Government.
CONCLUSION
We will know that we have been successful when:
 evaluations are meaningful to end-users
 MLA staff manage for outcomes
 evaluation is core to MLA values.
In essence, implementation of the MLA Evaluation
Framework has been all about establishing patterns of
behaviour that permit learning and reflection to occur:
 At the strategic level – the evaluation strategy sets the
timeframe and processes for learning and reflection.
Human knowledge is deeply contextual; it is triggered by
circumstance and need, and is revealed in action.
“We only need to know what we know when we need to
know it”.
 At an operational level – there is a range of ‘learning
by doing’ processes that work both in series and
parallel. These begin with development of an evaluation
proposal for approval by the Executive Committee and
finish with final production, stakeholder/peer review
workshop, and approval of the evaluation brochure by
the MLA Board.
The role of MLA’s Manager Evaluation Program
Improvement is to work with each of the relevant
stakeholders (guided by people-focused program logic,
detailed on p4) to facilitate the adoption of these patterns of
behaviour.
There is strong evidence that attitudes are gradually
changing and an evaluative culture is beginning to emerge.
This is not an ‘all of company at the same time’ approach,
but rather a ‘bit by bit – program by program’ evolution
unfolding over the course of the evaluation strategy’s
implementation.
SUSTAINABILITY
Culture
managing
foroutcomes
MLA Board
Evaluation reporting Program improvement
Reporting
to
Government
Guiding
future
investment
Stakeholder
confidence in
MLA
AOP KPI
Report
Engagement
with
PICs/industry
KPI
Reporting
Baselines
measurement
Context and interactions of evaluation at MLA
Levels of achievement for
framework objectives*
Ideal
Culture managing
for outcomes
Usefulness as a guide
for future investment
Satisfactory PIC
Response
Satisfactory Government
Response
0% 20% 40% 60% 80% 100%
MLA 2005
100%
25%
25%
25%
25%
32%
33%
42%
56%
MLA 2008
*2005 levels represents compliance with minimum statutory requirements.

12. i
APPENDIX
MEASURING PROGRAM VALUE THROUGH EX-POST EVALUATION
PAGE 1
Meat Livestock austraLia
Why does MLA need
a framework for
independent
evaluation?

13. ii
Meat Livestock austraLia
introduction
Performance evaluation is a critical success factor that allows
Meat Livestock australia (MLa) to demonstrate its delivery
of world-class services and solutions in partnership with the
red meat industry and government.
in 2005 MLa engaged the centre for international economics
(cie) to conduct an independent review of the way MLa
evaluated its performance and consequently developed
a more effective evaluation framework. implementing the
recommendations of this review ensures accountability to
stakeholders including red meat producers (MLa members),
the red meat industry, peak industry bodies, third party
investment partners, and the commonwealth Government.
the framework allows MLa to assess the industry impact of
its programs and their compliance with government priorities.
assessing actual program outcomes against the principal
goals – to build demand, increase market access, develop
competitive advantage and build industry capability – is
called ex-post evaluation. it enables MLa to report back to
management and to be answerable to stakeholders.
assessing anticipated program outcomes against goals is
called ex-ante evaluation, and this helps MLa to predict the
impact of its programs in advance.
independent evaluation of programs across MLa’s portfolio
of work is essential to accountability and supports internal
management of activities, helping to ensure that a learning
process is instilled within MLa. Backing this is an evaluation
work schedule developed by MLa to ensure that the
framework is consistently applied across all programs and
that it involves both ex-post and ex-ante evaluation.
the framework will also report on alignment of MLa program
outcomes with the new rural research and Development
Priorities, which complement and are informed by the
australian Government’s National research Priorities.
Why Does
MLa NeeD a
fraMeWork for
iNDePeNDeNt
evaLuatioN?
APPENDIX
MEASURING PROGRAM VALUE THROUGH EX-POST EVALUATION
PAGE 2

14. iii
Meat Livestock austraLia
how does the MLa framework for
independent evaluation work?
MLa programs are evaluated at various stages throughout their lifecycle
for their efficiency, economy and effectiveness in
achieving objectives based on established
criteria. each stage has specific implications
for the level and type of evaluation.
evaluation also depends on the program’s
reporting requirements.
Beyond the basic indicators – inputs and
outputs – the MLa evaluation framework traces
outcomes, impacts and benefits, providing
a comprehensive assessment of program
performance (chart 1).
a set of objectives is derived from the australian
red Meat industry strategic Plan 2004-2009 and
MLa’s own strategic planning processes.
these program or project objectives
determine the inputs required to achieve the
objectives. Monitoring inputs – investment
Objectives
Market
returns
Benefit-cost
ratio
Triple bottom line
Environmental Social
Inputs
Outputs
Outcomes
KPIs
Integrated
framework
Chart 1: Program evaluation overview
Benefits to stakeholders
Program evaluation is important for several reasons:
1. Responsibility for investing funds on behalf of the industry.
MLa makes investments today to produce future benefits. to ensure funding
is allocated to achieve the highest return possible, MLa must evaluate
programs ex-ante to determine the expected program benefits or pay-offs
and to draw comparisons.
2. Efficient operation.
efficient operation is dependent on effective project management, which can
only be achieved with program monitoring and evaluation.
3. Learning from the experience of different programs.
evaluation allows MLa to learn both from successful programs and from
those which need further improvement. the accumulated knowledge
provides insights that enable funding allocations to be channelled into higher
pay-off areas across MLa’s range of activities.
four key criteria ensure the evaluation
is robust:
1. Consistency across programs – this
ensures programs can be compared
and allows MLa to consistently apply
its evaluation framework across the entire
program lifecycle and across all programs.
2. Calculation of marginal return from investments – this involves
determining whether or not funds can be transferred from one program
to another to achieve a higher overall return from the total investment
in MLa’s activities.
3. Declaration of the key assumptions – this allows a ‘counterfactual’
case to be established which determines what would have happened in
the absence of outcomes generated by the program.
4. Estimation of the spill-over benefits arising from program
outcomes – this measures the indirect benefits enjoyed by consumers,
the community and other enterprises outside the industry value chain.
APPENDIX
MEASURING PROGRAM VALUE THROUGH EX-POST EVALUATION
PAGE 3

15. iv
Financial model
Assessment and
ranking of benefit
Chart 2: Integrated framework
Value chain model Global trade model
Other triple bottom line:
• environmental
• social
Integrated framework
Quantifiable
Not
quantifiable
costs or total dollars spent – at this level produces simple key performance
indicators (kPis) that are valuable for short-term project management and
monitoring.
inputs lead to outputs – tangible results of inputs, such as the number
of butchers trained in better meat display – and these provide another
opportunity to develop clear kPis that measure the quality of the output, not
just whether it was achieved.
outputs lead to outcomes – the changes in practice or behaviour resulting
from adoption of outputs, such as increased sales and ultimately better
returns to producers attributable to improved butcher training. outcome
kPis measure adoption as well as resulting changes, and measure the extent
to which expected results are achieved. Deriving outcomes often requires
detailed knowledge about both consumer and producer behaviour, and
involves translating outputs into the key differences they make to industry
supply and demand relationships. in the case of food safety or animal
welfare, the evaluation process translates the output into a risk reduction.
once a supply, demand or risk outcome is identified, it is possible, through
the integrated framework, to quantify the benefits and costs to the industry
that can be aggregated across programs (chart 2).
the environmental and social outcome components of the triple bottom line
can sometimes be quantified and included formally in program cost and
benefit assessments. however they are often not quantifiable, particularly in
dollars. such benefits are reported qualitatively in case studies.
for MLa to meet its evaluation requirements it must have a thorough
appreciation of how the red meat industry interacts. there are four key
elements to ensure the evaluation framework accounts for these interactions:
1. A value-chain model of the industry
2. MLA’s global meat industries model
3. A framework to handle triple bottom line
environmental and social issues
4. A financial model (to calculate net present
values of benefit-cost ratio streams) that
allows the effectiveness of the current MLA
portfolio to be evaluated against what
could be achieved if it was reallocated to
maximise MLA’s return on its total
investment budget.
further information
• For access to the MLA Program Evaluation
Framework manual, the MLA Strategic Plan
2007-2011 or information on MLA programs,
visit the MLA website: www.mla.com.au
• To download More From Less: the Strategic
Direction for the Australian Red Meat Industry
2004-2009, visit the Red Meat Advisory
Council website: www.rmac.com.au
• The Council of Rural Research and
Development Corporation Chair Guidelines
for Evaluation are available on the Rural RDC
website: www.ruralrdc.com.au
Level 1, 165 Walker St,
North Sydney NSW 2060
AUSTRALIA
Tel +61 2 9463 9333
Fax +61 2 9463 9393
www.mla.com.au
Published by Meat Livestock Australia Limited
ABN 39 081 678 364
August 2007
© Meat Livestock Australia (2007)
ISBN 9781741911626
Disclaimer: Where possible care is taken to ensure the
accuracy of the information contained in this publication.
However MLA cannot accept responsibility for the accuracy
or completeness of the information or opinions contained in
the publication. You should make your own enquiries before
making decisions concerning your interests.
Reproduction in whole or part of this publication is
prohibited without prior consent and acknowledgment of
Meat Livestock Australia.
Meat Livestock Australia acknowledges the matching funds provided by the Australian Government to support the research and development detailed in this publication.
APPENDIX
MEASURING PROGRAM VALUE THROUGH EX-POST EVALUATION
PAGE 4