So far, two UNDP-hosted workshops (June and October 2017) – attended by over 70 officials from departments under MoAC – have focused on prioritization for adaptation planning, using multi-criteria analysis (MCA); and developing a preliminary screening system for ranking and fine-tuning ongoing climate-sensitive projects and programmes. The workshop in June focused on providing an overview of MCA as a tool to priority actions. Participants gained a better appreciation of the process and key steps involved, as well as its strengths and limitations in the context of climate adaptation planning. Feedback and key insights were also gained by MOAC on how MCA could be used in the context of implementing Thailand’s new sectoral climate change strategy (ACCSP). In a follow-up workshop in October, participants learned the key steps to apply tools and methods in the context of their work. After identifying key areas from the revised five-year Agricultural Climate Change Strategic Plan 2017-2021, MoAC’s is enhancing its capacity with the support of the NAP-Ag programme to to prioritize these activities, which will be funded under the Ministry’s annual budgetary cycle and put forward to international climate funds.

1. Theory of Multi-Criteria Analysis
(MCA)
Workshop ‘Smart Decision-Making for Climate Change Adaptation Supported by Multi-Criteria Analysis’
Presented by Martin Rokitzki
(PlanAdapt, International consultant)

2. When to use MCA
Do cost-benefit analysis
One objective?
Benefits not in $
Impacts measurable?
More objective?
Benefits not in $
Impacts measurable?
Impacts difficult to
quantity?
MCA with expert panel
Do cost-effectiveness
analysis
Do multi-criteria
analysis
Yes to all No
Yes
Yes
One objective?
Benefits in $
Impacts measurable

3. What is MCA used for?
MCA techniques can be used to:
• Identify a single most preferred option,
• Rank options,
• Short list a limited number of options for subsequent detailed
appraisal,
• Distinguish acceptable from unacceptable possibilities.

4. What is multi-criteria analysis?
• MCA provides an entire decision-making framework from
problem definition to ranking/ comparing alternatives solutions.
• MCA in its simplest form consists of a combination of quantitative
and qualitative criteria which when assessed together yields a
final score which can facilitate decision-making.
• Many sub-forms exist such as stakeholder-oriented multi-criteria
analysis. There is no universal blueprint for its use.
• However, does not provide a prescriptive answer. It provides a
transparent and informative decision process which helps to
uncover people’s intuitive decision procedures by a structured
rational analytic process.

5. • Short introductory video: https://www.youtube.com/watch?v=-
JUaQcO4Dfo

6. Why has MCA emerged as an important
decision making tool?
• The use of MCA within adaptation has arisen due to the problems
associated with traditional decision-making tools such as cost-benefit
analysis (CBA) and cost–effectiveness analysis (CEA).
• These tools struggle to deal with uncertainty, concepts which cannot
be monetarised, and qualitative information.

7. Cost-Benefit Analysis
• CBA is a traditional decision-making tool, used when efficiency is the
only criteria.
• CBA calculates and compares all of costs and benefits, in monetary
terms.
• Benefit of CBA is that it uses a single metric.

8. Limitations of CBA
• CBA only assesses efficiency and not other issues e.g. equity
considerations related to the distribution of the costs and benefits
across stakeholder groups.
• Requires all benefits to be measured and expressed in monetary
terms.
• There is a danger that there are costs and benefits that cannot be
valued are simply ignored by decision-makers.
• Discounting costs and benefits incurred in the future can be very
controversial and subjective. It can be manipulated by certain
stakeholders to ensure they receive the outcome they desire.

9. MCA or CBA?
• CBA can be integrated into the MCA process, whereby CBA is used to
assess efficiency as a criteria within a MCA process.

10. Cost-effectiveness analysis
• CEA is used to determine which option is the more effective measure
in terms of cost-effectiveness.
• CEA is used to find the least costly adaptation options for meeting
selected targets.
• CEA arose in part as an alternative to CBA when benefits cannot be
monetarized.

11. CEA within MCA
• CEA is only appropriate when only one adaptation option will be
implemented (whereby only the option with the lowest cost-
effectiveness ratio is chosen).
• However, choosing a single option alone will rarely be the most cost-
effective policy, and the preferred option will be a combination of
options (although incremental CEA can overcome some difficulties).
• However, CEA is often not used as a standalone tool for decision
support and can be integrated into other decision-making processes
such as MCA.

12. MCA can overcome CBA & CEA issues
• MCA represents a decision-making tool which can address issues
associated with CBA and CEA.
• It can utilise both qualitative and quantitative information.
• It can incorporate wider disciplines (typically only economists are
involved with CBA, not adaptation or agricultural professionals).
• It can incorporate wider stakeholders than is possible within a CBA or
CEA process alone.

13. Additional strengths of MCA
• It is process of learning as well as decision-making by stimulating
discussion. Such discussions can allow decision-makers to better
understand the values and priorities of themselves and of other
stakeholders.
• It can facilitate conflict resolution by helping stakeholders to understand
different understandings of the problem and solutions.
• It is open to divergent values and opinions.
• It supports broad stakeholder participation and helps decision-makers to
make compromises and avoid dictating their judgement.
• This in turn helps to ensure legitimate processes as long as the process is
transparent and systematic.

14. Additional strengths of MCA
• It is open and explicit.
• The choice of objectives and criteria are open to analysis and to
change if they are felt to be inappropriate.
• Scores and weights are also explicit and are developed according to
established techniques and amended if necessary.
• Scores and weights also provide an audit trail.
• Performance measurement can be sub-contracted to experts, so need
not necessarily be left in the hands of the decision making body itself.
• It provides an important means of communication and collaboration
between stakeholders.

15. Limitations of MCA
• The inclusion of climate uncertainties remains relatively simplistic in
comparison with other technically advanced methods.
• The process can be time consuming. For instance, stakeholders must
be identified and allowed time to contribute.
• MCA provides no instructive way to resolve disputes or trade-offs.
• Stakeholders may also be reluctant to openly share their perspectives.
• Stakeholders may also be unable to contribute equally due to
resources.
• MCA aggregation methods can be too technically complex and
demanding with respect to information.

16. How often do Governments use MCA?
• MCA has been voluntarily used by governments across both
developed and developing countries despite not being mandated by
law).
• It has been used across all governance areas such as transport,
energy, forestry, health, housing, and waste management.
• Across these cases a range of stakeholders have been involved.
However, in some instances only experts were involved in the
decision making process.
• Such is the rise of MCA as a decision-making tool that the LDC expert
group commented one has to resort to MCA regarding the production
of NAPAs.

17. Determinants of MCA usage
Criteria for selecting a certain MCA techniques can be:
• Internal consistency and logical soundness.
• Transparency.
• Ease of use.
• Data requirements.
• Importance of the issue being considered.
• Time requirements for the analysis.
• Manpower resources available for the analysis.
• Ability and need to provide an audit trail.
• Software availability.

18. Institutional Analysis within MCA
• Institutional analysis is the first tool which should be employed when
adopting MCA
• Institutional analysis involves identifying relevant stakeholders who
should input in the multi-criteria analysis process
• Additionally, it identifies how they should be able to contribute.
• This process is vital to ensure both the process of decision-making as
well as the decisions made have democratic and technical legitimacy.

19. Institutional Analysis within MCA
• Involving stakeholders early on in the problem identification process
will help ensure greater acceptance of the final result as stakeholders
will feel that the willingness to cooperate indicates that decisions
have not already been made.
• A common criticism of MCA is that it can be too reliant upon the
priorities and preferences of select ‘experts’ or decision makers.
• Thus multi-criteria analysis decisions can lack democratic legitimacy
when subsequent outcomes affect groups who are insufficiently able
to influence decisions which affect them.

20. Institutional Analysis within MCA
• Institutional analysis can be conducted via the use of two participatory rapid
appraisal tools:
• (1) ‘Stakeholder Analysis’
• (2) ‘Stakeholder Mapping’
• Stakeholder analysis can identify and establish the relative importance and
influence of people, groups or institutions with an interest in a particular issue,
activity or project.
• Stakeholder mapping should be used in conjunction with stakeholder analysis. It
enables a decision-maker to better understand and manage the relationships
between stakeholders, including potential areas of agreement and conflict.
• These tools can be achieved through participatory focus group discussions with
various stakeholder groups.
• However, other tools such as anonymous questionnaires and key informant
interviews should also be used due to the limitations with participatory methods.

21. Stakeholder Analysis
1. Create table outlining each
stakeholder, along with their
associated interest, degree of
influence, and level of
importance (below).
2. Map out stakeholder analysis
using a matrix chart (right).

22. Stakeholder mapping
• Essentially uses a Venn diagram to understanding relationships
between stakeholders, in terms of:
• Conflicts.
• Alliances.
• Areas of mutual interest.
• Circles are used to depict relationships where:
• separate circles = no contact
• touching circles = information passes between institutions
• small overlap = some cooperation in decision-making
• large overlap = considerable cooperation in decision-making

23. Establishing Criteria
• Choosing which criteria to be adopted is a major source of conflict
between stakeholders. Different stakeholders have different priorities
and their own proposed criteria will reflect this.
• Criteria can be based around general principles of good development
or adaptation.
• Such criteria can also be used to screen out projects.
• Criteria can also represent more specific requirements resulting from
policy frameworks
• Such criteria can be used to score projects which pass quality control process.

24. Examples of Criteria
• Show list prepared

25. Criteria and Indicators
• Creating criteria is necessary but not sufficient when conducting an MCA.
• Associated indicators or groups of indicators are needed
• This enables stakeholders to identify that criteria have been objectively
been met or have been objectively scored.
• Criteria should reflect the full range of (policy) objectives (economic,
environmental, social)
• Criteria should not be too encompassing if they are to be aggregated.
• Aggregating too many indicators into few criteria can disguise serious
failings in some dimensions and increase the difficulty of identifying proper
remedial action.

26. Criteria and Indicators
• Grouping criteria together into sets that relate to components can be
helpful if the emerging decision structure contains a relatively large
number of criteria.
• Main reasons for grouping criteria are:
• (a) to help the process of checking whether the set of criteria selected is
appropriate to the problem
• (b) to ease the process of calculating criteria weights in large MCDA
applications, when it can sometimes be helpful to assess weights firstly within
groups of related criteria, and then between groups of criteria;
• (c) to facilitate the emergence of higher level views of the issues, particularly
how the options realise trade-offs between key objectives.

27. Major requirements to fulfil when
choosing criteria
• Completeness: Have all important criteria been included?
• Redundancy: Are some criteria not necessary or redundant?
• Operationality: Are the criteria measurable or defined?
• Mutually independent: Is the performance of one option against a criterion
independent of the performance of the same option against a second
criterion?
• Double counting: Are two criteria counting the same issue?
• Size: Are there too many criteria?
• Impacts occurring over time: Are time-differentiated impacts adequately
dealt with through the criteria?

28. Determining completeness
• A value tree can be used as a valuable aid to check whether all
relevant criteria have been overlooked.
• It is not necessarily bad design which misses all final considered
criteria, as MCA is learning process as well as a decision making
process.
• Major questions to ask are:
• Have we overlooked any major category of performance?
• Within each area of concern, have we included all the criteria necessary to
compare the options’ performance?
• Thirdly, do the criteria capture all the key aspects of the objectives that are
the point of the MCA?

29. Determining completeness
• Also referred to as a criteria tree it can visualise and make explicit
how stakeholder groups have participated as well as their priorities.
• This tool can be used to explore potential conflict between and within
stakeholder groups.
• A criteria tree can be structured in two ways.
• The multi-actor MCA can be adopted where by the criteria are clustered in
such a way that they contribute to each individual stakeholder’s priorities.
• Secondly, it is possible to structure the criteria tree along groups of effects
and study alternative scenarios created by using alternative sets of weights

30. Example of Value Tree

31. Example of Value Tree

32. Scoring and weighting within MCA
• Analysis is typically carried out on a performance matrix, where
criteria are represented on columns and options are represented by
rows.
• However, it is at this stage that MCA becomes complex.
• It needs to be decided what weighting is to be given to each criteria
which represents its relative importance to other criteria.
• It will need to be decided how to deal with trade-offs.
• It needs to be decided whether criteria compensate for each other.

33. Pictures of performance matrices

34. Scoring and weighting within MCA
• Within the area of sustainability, weightings are typically used as
importance coefficients.
• The process of choosing weights has been criticised as non-objective.
• However, all decision-making processes are imperfectly objective
including other tools such as social cost-benefit analysis.
• The process through which these are established can be a lengthy
process and a common source of conflict between stakeholder.

35. No Scoring and weighting within MCA?
• Influential MCA figures have argued against the elicitation of weights
and that criterion weights in the evaluation of public policies for
sustainability should be derived only from a plurality of ethical
principles (e.g., economic prosperity, ecological stability, or social
equity).
• Taking into account various dimensions simultaneously creates an
outcome in which it is impossible to optimise all the objectives at the
same time.
• This results in social and technical incommensurability.

36. To weight or not to weight
• While it is possible to not have weights between criteria, having explicit
weights provides some technical and democratic legitimacy to the process,
it also means results are auditable.
• While it is possible to see criteria as non-compensatory, MCA using this
approach is not very effective in distinguishing between options in real
applications.
• Normative criteria can be used to screen out instances where different
project elements are not substitutable
• E.g. within forestry ecological capital, such as a rainforest, cannot be offset
by the financial capital gained from its destruction. Therefore why not
screen out projects which involve cutting down rainforests.

37. Scoring options across criteria
• The first step is to set up consistent numerical scales for the
assessment of criteria.
• Need to ensure that the sense of direction is the same in all cases, so
that (usually) better levels of performance lead to higher value scores.
• It is conventional to allot a value score to each criterion between 0
and 100 on an interval scale.

38. Scoring options across criteria
• Three methods for scoring exist:
• Use a value function to translate a measure of achievement on the criterion
concerned into a value score on the 0 – 100 scale.
• Difficult to achieve with qualitative criteria.
• Elicit from the decision maker a series of verbal pairwise assessments
expressing a judgement of the performance.
• Such as the Analytic Hierarchy Process
• Use direct rating
• Used when a common agreed scale of measurement for the criterion in question does
not exist, or where there is not the time nor the resources to undertake the
measurement.
• Direct rating uses the judgement of experts to simply associate a number in the 0–100
range with the value of each option on that criterion.

39. Subjectivity in scoring
• All methods of scoring will involve subjectivity.
• This is especially likely to occur when qualitative criteria are scored
via a 0-100 quantitative scale.
• Even quantitative criteria can be influenced by subjectivity.
• For instance do jobs created for low-income and otherwise disadvantaged
groups have more value than jobs for other societal groups.
• Consequently, it is important to be transparent.

40. Reducing Subjectivity
• Reference to objectively measurable quantities.
• The use of individuals with expertise in both the concept under evaluation
(e.g. health impact) and the context of application (for example, in a
specific region).
• The specification or construction of an appropriate scale defined in terms
of performance against one or more objectively measurable criteria.
• A solid stakeholder engagement process.
• A multi-stage process in which initial scoring of options is carried out
independently by a number of experts, forming the basis for discussion.
• Use of an experienced facilitator who supports and challenges those
responsible for scoring the options.

41. Specific MCA techniques
• Lots of different MCA tools which facilitate aggregation of scoring,
reflecting the broad ways in which MCA is used in environmental policy.
• Many believe it is very important that ranking methods are simple to
guarantee consistency and transparency.
• Weighted sum approach has been the most common within climate change
adaptation.
• Some believe they should be non-compensatory to avoid that bad
environmental or social consequences are systematically outperformed by
good economic consequences or vice-versa.

42. Simplified example of weighted sum
scoring
• In this example 3 criteria exist, with 5 stakeholders
assessing their importance.
• An aggregation of the scores is used to determine weights
of criteria.
• Each stakeholder gives each criteria a score from 1
(lowest) to 10 (highest).
Criteria Scores Total average score weight
Criterion 1 (4+5+6+7+8) 30/5 = 6 6/21 = 0.285
Criterion 2 (5+6+7+8+9) 35/5 = 7 7/21 = 0.333
Criterion 3 (6+7+8+9+10) 40/5 = 8 8/21 = 0.380
Sum 21 1.00

43. Analytic Hierarchy Process
• AHP is the most commonly used aggregation method.
• AHP helps with converting subjective assessments of relative importance to
a set of overall scores or weights.
• Can be used for both criteria and option performance scores regarding
different criteria.
• Relies upon pairwise comparisons:
• Is criterion A more important than criterion B?
• If so, how much more important?
• However, subsequent weightings are derived from matrix algebra to derive
eigenvectors, the calculation of which requires computer software.
• These weightings can greatly contrast with stakeholder expectations.

44. Analytic Hierarchy Process
• Pairwise comparisons are generally found to be readily accepted in
practice as a means of establishing information about the relative
importance of criteria and the relative performance of options.
• However, the AHP technique has been often criticised, and
alternatives proposed which build upon the approach.
• E.g. REMBRANDT, MACBETH, SMART, DISCRIM
• Examples of MCA within adaptation planning have avoided this form
of aggregation.

45. Other important aggregation methods
• Multi-Attribute Utility Theory transforms diverse criteria into one common
dimensionless scale (0–1) of utility or value. Each criterion is ranked on a 0–
1 scale and combined based on the criteria weights to find a combined
score for each option. By picking the highest-ranking score, decision
makers maximize their utility functions for an option.
• Outranking Methods attempt to identify the dominance of one option over
others against the different criteria. Instead of using numerical values,
outranking methods use descriptive information through the combination
of information for each criterion for each option in an attempt to identify a
clear narrative that establishes dominance of one option over others.
Outranking methods are useful when criteria are not easily aggregated,
measurement scales vary widely, and units are incomparable (Kiker et al.,
2005).

46. Sensitivity Analysis
• Sensitivity and robustness analysis can be used to improve democratic and
technical legitimacy.
• It achieves this by demonstrating how robust results are if subtle changes
are made to the parameters of the study.
• For instance, weights between criteria or sub-criteria can be changed, or
changes to scores of criteria, or changes to indicators used.
• Participatory sensitivity analysis can also be achieved by enabling
stakeholders to comment on final results.
• Also helps to ensure democratic and technical legitimacy.
• Final results should intrinsically feel correct, if the outcome is surprising it
is likely the weightings need to be adjusted.