Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Theory of Multi-Criteria Analysis
(MCA)
Workshop ‘Smart Decision-Making for Climate Change Adaptation Supported by Multi-C...
When to use MCA
Do cost-benefit analysis
One objective?
Benefits not in $
Impacts measurable?
More objective?
Benefits not...
What is MCA used for?
MCA techniques can be used to:
• Identify a single most preferred option,
• Rank options,
• Short li...
What is multi-criteria analysis?
• MCA provides an entire decision-making framework from
problem definition to ranking/ co...
• Short introductory video: https://www.youtube.com/watch?v=-
JUaQcO4Dfo
Why has MCA emerged as an important
decision making tool?
• The use of MCA within adaptation has arisen due to the problem...
Cost-Benefit Analysis
• CBA is a traditional decision-making tool, used when efficiency is the
only criteria.
• CBA calcul...
Limitations of CBA
• CBA only assesses efficiency and not other issues e.g. equity
considerations related to the distribut...
MCA or CBA?
• CBA can be integrated into the MCA process, whereby CBA is used to
assess efficiency as a criteria within a ...
Cost-effectiveness analysis
• CEA is used to determine which option is the more effective measure
in terms of cost-effecti...
CEA within MCA
• CEA is only appropriate when only one adaptation option will be
implemented (whereby only the option with...
MCA can overcome CBA & CEA issues
• MCA represents a decision-making tool which can address issues
associated with CBA and...
Additional strengths of MCA
• It is process of learning as well as decision-making by stimulating
discussion. Such discuss...
Additional strengths of MCA
• It is open and explicit.
• The choice of objectives and criteria are open to analysis and to...
Limitations of MCA
• The inclusion of climate uncertainties remains relatively simplistic in
comparison with other technic...
How often do Governments use MCA?
• MCA has been voluntarily used by governments across both
developed and developing coun...
Determinants of MCA usage
Criteria for selecting a certain MCA techniques can be:
• Internal consistency and logical sound...
Institutional Analysis within MCA
• Institutional analysis is the first tool which should be employed when
adopting MCA
• ...
Institutional Analysis within MCA
• Involving stakeholders early on in the problem identification process
will help ensure...
Institutional Analysis within MCA
• Institutional analysis can be conducted via the use of two participatory rapid
apprais...
Stakeholder Analysis
1. Create table outlining each
stakeholder, along with their
associated interest, degree of
influence...
Stakeholder mapping
• Essentially uses a Venn diagram to understanding relationships
between stakeholders, in terms of:
• ...
Establishing Criteria
• Choosing which criteria to be adopted is a major source of conflict
between stakeholders. Differen...
Examples of Criteria
• Show list prepared
Criteria and Indicators
• Creating criteria is necessary but not sufficient when conducting an MCA.
• Associated indicator...
Criteria and Indicators
• Grouping criteria together into sets that relate to components can be
helpful if the emerging de...
Major requirements to fulfil when
choosing criteria
• Completeness: Have all important criteria been included?
• Redundanc...
Determining completeness
• A value tree can be used as a valuable aid to check whether all
relevant criteria have been ove...
Determining completeness
• Also referred to as a criteria tree it can visualise and make explicit
how stakeholder groups h...
Example of Value Tree
Example of Value Tree
Scoring and weighting within MCA
• Analysis is typically carried out on a performance matrix, where
criteria are represent...
Pictures of performance matrices
Scoring and weighting within MCA
• Within the area of sustainability, weightings are typically used as
importance coeffici...
No Scoring and weighting within MCA?
• Influential MCA figures have argued against the elicitation of weights
and that cri...
To weight or not to weight
• While it is possible to not have weights between criteria, having explicit
weights provides s...
Scoring options across criteria
• The first step is to set up consistent numerical scales for the
assessment of criteria.
...
Scoring options across criteria
• Three methods for scoring exist:
• Use a value function to translate a measure of achiev...
Subjectivity in scoring
• All methods of scoring will involve subjectivity.
• This is especially likely to occur when qual...
Reducing Subjectivity
• Reference to objectively measurable quantities.
• The use of individuals with expertise in both th...
Specific MCA techniques
• Lots of different MCA tools which facilitate aggregation of scoring,
reflecting the broad ways i...
Simplified example of weighted sum
scoring
• In this example 3 criteria exist, with 5 stakeholders
assessing their importa...
Analytic Hierarchy Process
• AHP is the most commonly used aggregation method.
• AHP helps with converting subjective asse...
Analytic Hierarchy Process
• Pairwise comparisons are generally found to be readily accepted in
practice as a means of est...
Other important aggregation methods
• Multi-Attribute Utility Theory transforms diverse criteria into one common
dimension...
Sensitivity Analysis
• Sensitivity and robustness analysis can be used to improve democratic and
technical legitimacy.
• I...
Upcoming SlideShare
Loading in …5
×

National Adaptation Plans Thailand - Theory of Multi-Criteria Analysis

637 views

Published on

So far, two UNDP-hosted workshops (June and October 2017) – attended by over 70 officials from departments under MoAC – have focused on prioritization for adaptation planning, using multi-criteria analysis (MCA); and developing a preliminary screening system for ranking and fine-tuning ongoing climate-sensitive projects and programmes.

The workshop in June focused on providing an overview of MCA as a tool to priority actions. Participants gained a better appreciation of the process and key steps involved, as well as its strengths and limitations in the context of climate adaptation planning. Feedback and key insights were also gained by MOAC on how MCA could be used in the context of implementing Thailand’s new sectoral climate change strategy (ACCSP).

In a follow-up workshop in October, participants learned the key steps to apply tools and methods in the context of their work.

After identifying key areas from the revised five-year Agricultural Climate Change Strategic Plan 2017-2021, MoAC’s is enhancing its capacity with the support of the NAP-Ag programme to to prioritize these activities, which will be funded under the Ministry’s annual budgetary cycle and put forward to international climate funds.

Published in: Government & Nonprofit
  • Be the first to comment

National Adaptation Plans Thailand - Theory of Multi-Criteria Analysis

  1. 1. Theory of Multi-Criteria Analysis (MCA) Workshop ‘Smart Decision-Making for Climate Change Adaptation Supported by Multi-Criteria Analysis’ Presented by Martin Rokitzki (PlanAdapt, International consultant)
  2. 2. When to use MCA Do cost-benefit analysis One objective? Benefits not in $ Impacts measurable? More objective? Benefits not in $ Impacts measurable? Impacts difficult to quantity? MCA with expert panel Do cost-effectiveness analysis Do multi-criteria analysis Yes to all No Yes Yes One objective? Benefits in $ Impacts measurable
  3. 3. What is MCA used for? MCA techniques can be used to: • Identify a single most preferred option, • Rank options, • Short list a limited number of options for subsequent detailed appraisal, • Distinguish acceptable from unacceptable possibilities.
  4. 4. What is multi-criteria analysis? • MCA provides an entire decision-making framework from problem definition to ranking/ comparing alternatives solutions. • MCA in its simplest form consists of a combination of quantitative and qualitative criteria which when assessed together yields a final score which can facilitate decision-making. • Many sub-forms exist such as stakeholder-oriented multi-criteria analysis. There is no universal blueprint for its use. • However, does not provide a prescriptive answer. It provides a transparent and informative decision process which helps to uncover people’s intuitive decision procedures by a structured rational analytic process.
  5. 5. • Short introductory video: https://www.youtube.com/watch?v=- JUaQcO4Dfo
  6. 6. Why has MCA emerged as an important decision making tool? • The use of MCA within adaptation has arisen due to the problems associated with traditional decision-making tools such as cost-benefit analysis (CBA) and cost–effectiveness analysis (CEA). • These tools struggle to deal with uncertainty, concepts which cannot be monetarised, and qualitative information.
  7. 7. Cost-Benefit Analysis • CBA is a traditional decision-making tool, used when efficiency is the only criteria. • CBA calculates and compares all of costs and benefits, in monetary terms. • Benefit of CBA is that it uses a single metric.
  8. 8. Limitations of CBA • CBA only assesses efficiency and not other issues e.g. equity considerations related to the distribution of the costs and benefits across stakeholder groups. • Requires all benefits to be measured and expressed in monetary terms. • There is a danger that there are costs and benefits that cannot be valued are simply ignored by decision-makers. • Discounting costs and benefits incurred in the future can be very controversial and subjective. It can be manipulated by certain stakeholders to ensure they receive the outcome they desire.
  9. 9. MCA or CBA? • CBA can be integrated into the MCA process, whereby CBA is used to assess efficiency as a criteria within a MCA process.
  10. 10. Cost-effectiveness analysis • CEA is used to determine which option is the more effective measure in terms of cost-effectiveness. • CEA is used to find the least costly adaptation options for meeting selected targets. • CEA arose in part as an alternative to CBA when benefits cannot be monetarized.
  11. 11. CEA within MCA • CEA is only appropriate when only one adaptation option will be implemented (whereby only the option with the lowest cost- effectiveness ratio is chosen). • However, choosing a single option alone will rarely be the most cost- effective policy, and the preferred option will be a combination of options (although incremental CEA can overcome some difficulties). • However, CEA is often not used as a standalone tool for decision support and can be integrated into other decision-making processes such as MCA.
  12. 12. MCA can overcome CBA & CEA issues • MCA represents a decision-making tool which can address issues associated with CBA and CEA. • It can utilise both qualitative and quantitative information. • It can incorporate wider disciplines (typically only economists are involved with CBA, not adaptation or agricultural professionals). • It can incorporate wider stakeholders than is possible within a CBA or CEA process alone.
  13. 13. Additional strengths of MCA • It is process of learning as well as decision-making by stimulating discussion. Such discussions can allow decision-makers to better understand the values and priorities of themselves and of other stakeholders. • It can facilitate conflict resolution by helping stakeholders to understand different understandings of the problem and solutions. • It is open to divergent values and opinions. • It supports broad stakeholder participation and helps decision-makers to make compromises and avoid dictating their judgement. • This in turn helps to ensure legitimate processes as long as the process is transparent and systematic.
  14. 14. Additional strengths of MCA • It is open and explicit. • The choice of objectives and criteria are open to analysis and to change if they are felt to be inappropriate. • Scores and weights are also explicit and are developed according to established techniques and amended if necessary. • Scores and weights also provide an audit trail. • Performance measurement can be sub-contracted to experts, so need not necessarily be left in the hands of the decision making body itself. • It provides an important means of communication and collaboration between stakeholders.
  15. 15. Limitations of MCA • The inclusion of climate uncertainties remains relatively simplistic in comparison with other technically advanced methods. • The process can be time consuming. For instance, stakeholders must be identified and allowed time to contribute. • MCA provides no instructive way to resolve disputes or trade-offs. • Stakeholders may also be reluctant to openly share their perspectives. • Stakeholders may also be unable to contribute equally due to resources. • MCA aggregation methods can be too technically complex and demanding with respect to information.
  16. 16. How often do Governments use MCA? • MCA has been voluntarily used by governments across both developed and developing countries despite not being mandated by law). • It has been used across all governance areas such as transport, energy, forestry, health, housing, and waste management. • Across these cases a range of stakeholders have been involved. However, in some instances only experts were involved in the decision making process. • Such is the rise of MCA as a decision-making tool that the LDC expert group commented one has to resort to MCA regarding the production of NAPAs.
  17. 17. Determinants of MCA usage Criteria for selecting a certain MCA techniques can be: • Internal consistency and logical soundness. • Transparency. • Ease of use. • Data requirements. • Importance of the issue being considered. • Time requirements for the analysis. • Manpower resources available for the analysis. • Ability and need to provide an audit trail. • Software availability.
  18. 18. Institutional Analysis within MCA • Institutional analysis is the first tool which should be employed when adopting MCA • Institutional analysis involves identifying relevant stakeholders who should input in the multi-criteria analysis process • Additionally, it identifies how they should be able to contribute. • This process is vital to ensure both the process of decision-making as well as the decisions made have democratic and technical legitimacy.
  19. 19. Institutional Analysis within MCA • Involving stakeholders early on in the problem identification process will help ensure greater acceptance of the final result as stakeholders will feel that the willingness to cooperate indicates that decisions have not already been made. • A common criticism of MCA is that it can be too reliant upon the priorities and preferences of select ‘experts’ or decision makers. • Thus multi-criteria analysis decisions can lack democratic legitimacy when subsequent outcomes affect groups who are insufficiently able to influence decisions which affect them.
  20. 20. Institutional Analysis within MCA • Institutional analysis can be conducted via the use of two participatory rapid appraisal tools: • (1) ‘Stakeholder Analysis’ • (2) ‘Stakeholder Mapping’ • Stakeholder analysis can identify and establish the relative importance and influence of people, groups or institutions with an interest in a particular issue, activity or project. • Stakeholder mapping should be used in conjunction with stakeholder analysis. It enables a decision-maker to better understand and manage the relationships between stakeholders, including potential areas of agreement and conflict. • These tools can be achieved through participatory focus group discussions with various stakeholder groups. • However, other tools such as anonymous questionnaires and key informant interviews should also be used due to the limitations with participatory methods.
  21. 21. Stakeholder Analysis 1. Create table outlining each stakeholder, along with their associated interest, degree of influence, and level of importance (below). 2. Map out stakeholder analysis using a matrix chart (right).
  22. 22. Stakeholder mapping • Essentially uses a Venn diagram to understanding relationships between stakeholders, in terms of: • Conflicts. • Alliances. • Areas of mutual interest. • Circles are used to depict relationships where: • separate circles = no contact • touching circles = information passes between institutions • small overlap = some cooperation in decision-making • large overlap = considerable cooperation in decision-making
  23. 23. Establishing Criteria • Choosing which criteria to be adopted is a major source of conflict between stakeholders. Different stakeholders have different priorities and their own proposed criteria will reflect this. • Criteria can be based around general principles of good development or adaptation. • Such criteria can also be used to screen out projects. • Criteria can also represent more specific requirements resulting from policy frameworks • Such criteria can be used to score projects which pass quality control process.
  24. 24. Examples of Criteria • Show list prepared
  25. 25. Criteria and Indicators • Creating criteria is necessary but not sufficient when conducting an MCA. • Associated indicators or groups of indicators are needed • This enables stakeholders to identify that criteria have been objectively been met or have been objectively scored. • Criteria should reflect the full range of (policy) objectives (economic, environmental, social) • Criteria should not be too encompassing if they are to be aggregated. • Aggregating too many indicators into few criteria can disguise serious failings in some dimensions and increase the difficulty of identifying proper remedial action.
  26. 26. Criteria and Indicators • Grouping criteria together into sets that relate to components can be helpful if the emerging decision structure contains a relatively large number of criteria. • Main reasons for grouping criteria are: • (a) to help the process of checking whether the set of criteria selected is appropriate to the problem • (b) to ease the process of calculating criteria weights in large MCDA applications, when it can sometimes be helpful to assess weights firstly within groups of related criteria, and then between groups of criteria; • (c) to facilitate the emergence of higher level views of the issues, particularly how the options realise trade-offs between key objectives.
  27. 27. Major requirements to fulfil when choosing criteria • Completeness: Have all important criteria been included? • Redundancy: Are some criteria not necessary or redundant? • Operationality: Are the criteria measurable or defined? • Mutually independent: Is the performance of one option against a criterion independent of the performance of the same option against a second criterion? • Double counting: Are two criteria counting the same issue? • Size: Are there too many criteria? • Impacts occurring over time: Are time-differentiated impacts adequately dealt with through the criteria?
  28. 28. Determining completeness • A value tree can be used as a valuable aid to check whether all relevant criteria have been overlooked. • It is not necessarily bad design which misses all final considered criteria, as MCA is learning process as well as a decision making process. • Major questions to ask are: • Have we overlooked any major category of performance? • Within each area of concern, have we included all the criteria necessary to compare the options’ performance? • Thirdly, do the criteria capture all the key aspects of the objectives that are the point of the MCA?
  29. 29. Determining completeness • Also referred to as a criteria tree it can visualise and make explicit how stakeholder groups have participated as well as their priorities. • This tool can be used to explore potential conflict between and within stakeholder groups. • A criteria tree can be structured in two ways. • The multi-actor MCA can be adopted where by the criteria are clustered in such a way that they contribute to each individual stakeholder’s priorities. • Secondly, it is possible to structure the criteria tree along groups of effects and study alternative scenarios created by using alternative sets of weights
  30. 30. Example of Value Tree
  31. 31. Example of Value Tree
  32. 32. Scoring and weighting within MCA • Analysis is typically carried out on a performance matrix, where criteria are represented on columns and options are represented by rows. • However, it is at this stage that MCA becomes complex. • It needs to be decided what weighting is to be given to each criteria which represents its relative importance to other criteria. • It will need to be decided how to deal with trade-offs. • It needs to be decided whether criteria compensate for each other.
  33. 33. Pictures of performance matrices
  34. 34. Scoring and weighting within MCA • Within the area of sustainability, weightings are typically used as importance coefficients. • The process of choosing weights has been criticised as non-objective. • However, all decision-making processes are imperfectly objective including other tools such as social cost-benefit analysis. • The process through which these are established can be a lengthy process and a common source of conflict between stakeholder.
  35. 35. No Scoring and weighting within MCA? • Influential MCA figures have argued against the elicitation of weights and that criterion weights in the evaluation of public policies for sustainability should be derived only from a plurality of ethical principles (e.g., economic prosperity, ecological stability, or social equity). • Taking into account various dimensions simultaneously creates an outcome in which it is impossible to optimise all the objectives at the same time. • This results in social and technical incommensurability.
  36. 36. To weight or not to weight • While it is possible to not have weights between criteria, having explicit weights provides some technical and democratic legitimacy to the process, it also means results are auditable. • While it is possible to see criteria as non-compensatory, MCA using this approach is not very effective in distinguishing between options in real applications. • Normative criteria can be used to screen out instances where different project elements are not substitutable • E.g. within forestry ecological capital, such as a rainforest, cannot be offset by the financial capital gained from its destruction. Therefore why not screen out projects which involve cutting down rainforests.
  37. 37. Scoring options across criteria • The first step is to set up consistent numerical scales for the assessment of criteria. • Need to ensure that the sense of direction is the same in all cases, so that (usually) better levels of performance lead to higher value scores. • It is conventional to allot a value score to each criterion between 0 and 100 on an interval scale.
  38. 38. Scoring options across criteria • Three methods for scoring exist: • Use a value function to translate a measure of achievement on the criterion concerned into a value score on the 0 – 100 scale. • Difficult to achieve with qualitative criteria. • Elicit from the decision maker a series of verbal pairwise assessments expressing a judgement of the performance. • Such as the Analytic Hierarchy Process • Use direct rating • Used when a common agreed scale of measurement for the criterion in question does not exist, or where there is not the time nor the resources to undertake the measurement. • Direct rating uses the judgement of experts to simply associate a number in the 0–100 range with the value of each option on that criterion.
  39. 39. Subjectivity in scoring • All methods of scoring will involve subjectivity. • This is especially likely to occur when qualitative criteria are scored via a 0-100 quantitative scale. • Even quantitative criteria can be influenced by subjectivity. • For instance do jobs created for low-income and otherwise disadvantaged groups have more value than jobs for other societal groups. • Consequently, it is important to be transparent.
  40. 40. Reducing Subjectivity • Reference to objectively measurable quantities. • The use of individuals with expertise in both the concept under evaluation (e.g. health impact) and the context of application (for example, in a specific region). • The specification or construction of an appropriate scale defined in terms of performance against one or more objectively measurable criteria. • A solid stakeholder engagement process. • A multi-stage process in which initial scoring of options is carried out independently by a number of experts, forming the basis for discussion. • Use of an experienced facilitator who supports and challenges those responsible for scoring the options.
  41. 41. Specific MCA techniques • Lots of different MCA tools which facilitate aggregation of scoring, reflecting the broad ways in which MCA is used in environmental policy. • Many believe it is very important that ranking methods are simple to guarantee consistency and transparency. • Weighted sum approach has been the most common within climate change adaptation. • Some believe they should be non-compensatory to avoid that bad environmental or social consequences are systematically outperformed by good economic consequences or vice-versa.
  42. 42. Simplified example of weighted sum scoring • In this example 3 criteria exist, with 5 stakeholders assessing their importance. • An aggregation of the scores is used to determine weights of criteria. • Each stakeholder gives each criteria a score from 1 (lowest) to 10 (highest). Criteria Scores Total average score weight Criterion 1 (4+5+6+7+8) 30/5 = 6 6/21 = 0.285 Criterion 2 (5+6+7+8+9) 35/5 = 7 7/21 = 0.333 Criterion 3 (6+7+8+9+10) 40/5 = 8 8/21 = 0.380 Sum 21 1.00
  43. 43. Analytic Hierarchy Process • AHP is the most commonly used aggregation method. • AHP helps with converting subjective assessments of relative importance to a set of overall scores or weights. • Can be used for both criteria and option performance scores regarding different criteria. • Relies upon pairwise comparisons: • Is criterion A more important than criterion B? • If so, how much more important? • However, subsequent weightings are derived from matrix algebra to derive eigenvectors, the calculation of which requires computer software. • These weightings can greatly contrast with stakeholder expectations.
  44. 44. Analytic Hierarchy Process • Pairwise comparisons are generally found to be readily accepted in practice as a means of establishing information about the relative importance of criteria and the relative performance of options. • However, the AHP technique has been often criticised, and alternatives proposed which build upon the approach. • E.g. REMBRANDT, MACBETH, SMART, DISCRIM • Examples of MCA within adaptation planning have avoided this form of aggregation.
  45. 45. Other important aggregation methods • Multi-Attribute Utility Theory transforms diverse criteria into one common dimensionless scale (0–1) of utility or value. Each criterion is ranked on a 0– 1 scale and combined based on the criteria weights to find a combined score for each option. By picking the highest-ranking score, decision makers maximize their utility functions for an option. • Outranking Methods attempt to identify the dominance of one option over others against the different criteria. Instead of using numerical values, outranking methods use descriptive information through the combination of information for each criterion for each option in an attempt to identify a clear narrative that establishes dominance of one option over others. Outranking methods are useful when criteria are not easily aggregated, measurement scales vary widely, and units are incomparable (Kiker et al., 2005).
  46. 46. Sensitivity Analysis • Sensitivity and robustness analysis can be used to improve democratic and technical legitimacy. • It achieves this by demonstrating how robust results are if subtle changes are made to the parameters of the study. • For instance, weights between criteria or sub-criteria can be changed, or changes to scores of criteria, or changes to indicators used. • Participatory sensitivity analysis can also be achieved by enabling stakeholders to comment on final results. • Also helps to ensure democratic and technical legitimacy. • Final results should intrinsically feel correct, if the outcome is surprising it is likely the weightings need to be adjusted.

×