Some practical guidance on how to reduce (but not eliminate) subjectivity when conducting qualitative risk assessments through the use pf practical and tailored descriptors and guidance.

1. https://www.linkedin.com/in/danielatkin/
@dan_m_atkin
Reducing subjectivity in qualitative risk assessments

2. Problem:
Qualitative risk assessments, particularly those based on
ISO 31000, generally make use of some form of likelihood
and consequence scale to determine a level of risk.
Often little or no guidance is given around what these
likelihood or consequence levels mean, leading to a large
amount of subjectivity
What is “Likely” or “Severe” to one person, is very
different to someone else.
Insignificant Minor Medium Major Severe
Almost
Certain
Moderate Moderate High Very High Very High
Likely Low Moderate High High Very High
Possible Low Moderate Moderate High Very High
Unlikely Low Low Moderate High High
Very
Unlikely
Low Low Low Moderate High
Risk Matrix – example only

3. Example:
A CIA study in the 90s asked 23 NATO military personal to
assign a level of probability against statements like Likely,
Probably, Improbable etc. (Huer 1999)
The figure to the right is a graphical representation of the
results of the survey.
One of the conclusions of the study was that additional
guidance was needed to reduce subjectivity across
audiences. E.g. a probability ratio or percentage
Huer 1999 - https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf
Image source: https://github.com/zonination/perceptions

4. Probability Insignificant Minor Medium Major Severe
Almost
Certain
> 90% Moderate Moderate High Very High Very High
Likely
< 90%
>= 70%
Low Moderate High High Very High
Possible
< 70%
>= 50%
Low Moderate Moderate High Very High
Unlikely
< 50%
>= 30%
Low Low Moderate High High
Very
Unlikely
< 30% Low Low Low Moderate High
Insignificant Minor Medium Major Severe
Almost
Certain
Moderate Moderate High Very High Very High
Likely Low Moderate High High Very High
Possible Low Moderate Moderate High Very High
Unlikely Low Low Moderate High High
Very
Unlikely
Low Low Low Moderate High
This Becomes This

5. PROBLEM SOLVED?

6. NO!
That recommendation is great for CIA Analysts and or NATO officials
who are well trained in probability and analysis, but what about the
rest of us?
For most people in any organisation, a percentage rating of probability,
e.g. 73% chance of occurring, is still likely to be interpreted very
differently by different people.

7. Tailored probability and consequence
descriptors that suit your organisation
There is no one answer or approach that will suit every organisation. However, subjectivity can be
reduced further by providing real world descriptors of what different scales mean to your
organisation.
A likelihood of “Likely” can be described as:
• Having happened several times in the organisations past, and
• Expected to happen once every two to three years
A consequence of “Major” can be described as:
• Financial loss over $10m
• Reputational damage, including media coverage lasting up to a week

8. Example:
Insignificant Minor Medium Major Severe
Almost
Certain
Moderate Moderate High Very High Very High
Likely Low Moderate High High Very High
Possible Low Moderate Moderate High Very High
Unlikely Low Low Moderate High High
Very
Unlikely
Low Low Low Moderate High
Risk Matrix – example onlyLikelihood table – example only
Descriptor Scale Description Indicative Return Period
Almost
Certain
5
Expected to occur in the next 12
months
Every year or more
frequently
Likely 4
Greater chance than not of
eventuating in the next couple of
years
Between one to three
years
Possible 3
Might occur once in the history of
the organisation
Between three to five
years
Unlikely 2 Has occurred from time to time Between five to ten years
Very Unlikely 1
Have heard of something like this
occurring somewhere else
Ten years or longer

9. Example:
Consequence table – example only
Descriptor Scale Reputation Financial Occupational Health & Safety Business interuption
Severe 5
Negative media coverage
lasting several weeks
Loss of $10m
Death or permanent disability
of employee
Unable to operate business
indefinitely
Major 4
Negative media coverage
lasting up to a week
Loss of $5m Long term injury to employee
Unable to operate business
for one month or longer
Medium 3
Negative media coverage
lasting up to three days
Loss of $1m
Significant injury requiring
hospital care
Unable to operate business
for one week or longer
Minor 2
Negative media coverage
lasting one day
Loss of $500k
Injury requiring lengthy time
off work
Unable to operate business
for a day or longer
Insignificant 1
Isolated single negative
media story
Loss of $100k Injury requiring first aid
Unable to operate business
for several hours

10. Benefits of this approach
You will not be able to completely eliminate subjectivity amongst different people
but you will:
• Create a common “anchor” or starting point for people to apply their judgements
• Customise those anchors to suit your business needs and context
• Get more engagement from the business to use these tools by using plain
language grounded in terms relevant to their work

11. GOLDEN RULE!
What is “High” or “Likely” for one organisation wont be the
same for another organisation
Descriptor Scale Description Indicative Return Period
Almost
Certain
5
Expected to occur in the next 12
months
Every year or more
frequently
Likely 4
Greater chance than not of
eventuating in the next couple of
years
Between one to three
years
Possible 3
Might occur once in the history of
the organisation
Between three to five
years
Unlikely 2 Has occurred from time to time Between five to ten years
Very Unlikely 1
Have heard of something like this
occurring somewhere else
Ten years or longer
Descriptor Scale Description Indicative Return Period
Almost
Certain
5
Expected to occur in the next 12
months
Every year or more
frequently
Likely 4
Greater chance than not of
eventuating in the next five years
Between one to five years
Possible 3
Might occur once in the history of
the organisation
Between five to ten years
Unlikely 2 Has occurred from time to time
Between ten to twenty
years
Very Unlikely 1
Have heard of something like this
occurring somewhere else
Twenty years or longer
Organisation One Organisation Two

12. Descriptor Scale Reputation Financial Occupational Health & Safety Business interuption
Severe 5
Negative media coverage
lasting several weeks
Loss of $10m
Death or permanent disability
of employee
Unable to operate business
indefinitely
Major 4
Negative media coverage
lasting up to a week
Loss of $5m Long term injury to employee
Unable to operate business
for one month or longer
Medium 3
Negative media coverage
lasting up to three days
Loss of $1m
Significant injury requiring
hospital care
Unable to operate business
for one week or longer
Minor 2
Negative media coverage
lasting one day
Loss of $500k
Injury requiring lengthy time
off work
Unable to operate business
for a day or longer
Insignificant 1
Isolated single negative
media story
Loss of $100k Injury requiring first aid
Unable to operate business
for several hours
Descriptor Scale Reputation Financial Occupational Health & Safety Business interuption
Severe 5
Negative media coverage
lasting several weeks
Loss of $1m
Death or permanent disability
of employee
Unable to operate business
indefinitely
Major 4
Negative media coverage
lasting up to a week
Loss of $800k Long term injury to employee
Unable to operate business
for one month or longer
Medium 3
Negative media coverage
lasting up to three days
Loss of $500k
Significant injury requiring
hospital care
Unable to operate business
for one week or longer
Minor 2
Negative media coverage
lasting one day
Loss of $100k
Injury requiring lengthy time
off work
Unable to operate business
for a day or longer
Insignificant 1
Isolated single negative
media story
Loss of $10k Injury requiring first aid
Unable to operate business
for several hours
Organisation One
Organisation Two

13. TOP TIPS FOR SUCCESS!
1. Use plain language
2. Give guidance on how to apply it
3. Review the levels and effectiveness regularly
4. Engage with stakeholders to develop scales – e.g. Finance
department for financial consequences
5. Get agreement on scales and rating levels with decision
makers – e.g. if the risk matrix is for the whole
organisation, agree it with the CEO and Board

14. Should we even use qualitative risk
assessments?
The short answer is……
Maybe

15. Should we even use qualitative risk
assessments?
There are many different methodologies out there that are far superior to
qualitative risk assessments at determining an accurate level of risk, including:
• Tornado diagrams
• Monte Carlo Simulations
• Decision Trees & Modelling
• Artificial Intelligence
Source: Alexei Sidorenko https://www.linkedin.com/in/alexsidorenko/

16. But the right mix of approaches and tools for your organisation depends
on the complexity, maturity and capability of your organisation.
One of the key principles of ISO 31000 is that your approach to risk
management is tailored to your organisation context.
For many organisations, a qualitative approach is the right fit due to the
type and frequency of risks they are managing, the type or organisation
they are and the capacity, capability and resources that are available to
engage with risk management.

17. Contact me for more
information
Have questions?
Like to add more?
Don’t agree?
Get in touch, I’d love to hear from you.
Kind regards,
Daniel
https://www.linkedin.com/in/danielatkin/
@dan_m_atkin
Daniel Atkin