More Related Content Similar to Assessing measuring oprisksama-khan011805 Similar to Assessing measuring oprisksama-khan011805 (20) More from Ujjwal 'Shanu' (20) 1. Assessing & Measuring Operational Risk
Why COSO is Inappropriate
ISDA PRMIA
London, United Kingdom
January 18, 2005
Ali Samad-Khan
President, OpRisk Advisory LLC
ali.samad-khan@opriskadvisory.com
www.opriskadvisory.com
4. When considering risk and control assessment, what are the priorities?
• Establish a disciplined process. It’s the process that matters, the
results are less important
• A good process lays the foundation for a good risk management
culture
• Establish a process that will produce the most reliable results. The
results are more important
• If it is clear to end users that the results are fictitious then the
entire risk management program will be discredited; the
operational risk program will be seen to be adding little value
• Demonstrate practical value and the program will be a success and
subsequently create the right culture
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
5. Consider introducing a disciplined process that produces accurate results
which facilitate educated decisions making.
RISKS
What type
of risks
do I face?
CONTROLS
Which are
the largest
risks?
How well are
these risks
being
managed?
Manage Controls through Cost Benefit Analysis
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
6. To ensure the goal is practical, one needs to express it in the context of a
business problem
• Consider two risks: Unauthorized Trading and Money Transfer
• Past Audits reveal that both risks are under-controlled
• To address Unauthorized Trading risk one must improve segregation
of duties and audit frequency. (Solution: hire four new staff; cost =
$600,000 per year)
• To address Money Transfer risk one must improve the system
(Solutions: buy new system; cost = $5 million)
• You have $4 million in your budget. Where do you invest your
money?
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
7. Effectively managing operational risk requires a foundation designed to
turn raw operational risk data into information that supports managerial
decision making
MANAGEMENT
Economic Profit
INFORMATION
DATA
FOUNDATION
• Risk strategy,
tolerance
• Roles and
responsibilities
• Policies and
procedures
• Risk definition and
categorization
• Loss data collection
• Risk indicator data
collection
• Control selfassessment
• Risk assessment and
analysis
• Automatic notification
• Expected Loss – how
much do I lose on
average
• Unexpected Loss –
how much I could
reasonably expect to
lose in a bad year
• Control Scores – how
good are the controls
I have in place
• Follow up action
reports
Management & Control Quality
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
• Awareness of real
exposures
• Knowledge of
controls quality
• Cost benefit analysis
• Improved risk
mitigation and
transfer strategy
9. What does operational risk include?
Transaction
Execution
Settlement
Technological
Inadequate
Supervision
Information
Key Man
Lack of
Resources
Reputation
Insufficient
Training
Theft
Relationship
Fraud
People
Fiduciary
Compliance
Legal/Regulatory
Criminal
Rogue Trader
Physical Assets
Customer
Poor
Management
Fixed Cost
Structures
Business
Business
Interruption
Strategic
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
10. The universe of operational risks spans causes, events and consequences
CAUSES
EVENTS
Inadequate
segregation of duties
Insufficient training
Lack of management
supervision
Inadequate
auditing procedures
Inadequate security
measures
•
•
•
Poor systems
design
CONSEQUENCES
Legal Liability
Internal
Fraud
Regulatory, Compliance
& Taxation Penalties
External
Fraud
Loss or Damage
to Assets
Employment Practices
& Workplace Safety
Restitution
Clients, Products
& Business Practices
Loss of Recourse
Damage to
Physical Assets
Write-down
Business Disruption
& System Failures
Reputation
Execution, Delivery &
Process Management
Business Interruption
Poor HR
policies
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
EFFECTS
Monetary
Losses
OTHER
IMPACTS
Forgone
Income
11. Placing loss data within a Business Line/Risk matrix helps reveal the risk
profile of each business
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
13. Risk can also be assessed using a likelihood-impact approach. This
approach has been well documented by the Committee of Sponsoring
Organizations of the Treadway Commission (COSO).
Source: COSO
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
14. The COSO view of risk assessment is based on the likelihood and impact
of a specific type of event; the output is probability weighted impact. The
high risk area is in the top right corner of the matrix.
COSO
3
6
9
Med (2)
2
4
6
Low (1)
1
COSO
2
3
Low (1)
Med (2)
High (3)
LIKELIHOOD
High (3)
IMPACT
Likelihood x Impact = Risk
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
15. Under the risk management industry approach, the high risk area is the
bottom right cell in the matrix.
Risk Management Industry
n/a
LIKELIHOOD
High (3)
n/a
n/a
Med (2)
COSO
Low (1)
Low (1)
Med (2)
High (3)
IMPACT
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
16. When compared, there are significant differences ….
Risk Management Industry
Likelihood
n/a
COSO
Low (1)
Low (1)
Med (2)
High (3)
3
6
9
Med (2)
2
4
1
COSO
2
3
Phantom
Risks
6
Low (1)
n/a
Med (2)
High (3)
Likelihood
n/a
High (3)
COSO
Low (1)
Impact
Med (2)
Impact
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
High (3)
Real
Risks
17. Under the COSO approach one calculates risk through likelihood-impact
analysis
Likelihood x Impact = Risk
Risk 1 :
10% x $10,000 = $1,000
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
18. Likelihood-impact analysis can yield more than one result
Likelihood x Impact = Risk
Risk 1 :
Risk 2 :
10% x $10,000 = $1,000
1% x $50,000 = $ 500
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
19. Using likelihood-impact analysis one can calculate multiple outcomes
Likelihood x Impact = Risk
Risk 1 :
Risk 2 :
10% x $10,000 = $1,000
1% x $50,000 = $ 500
.
.
.
.
Risk 999 : 5% x $25,000 = $1,250
Risk 1000 : 20% x $ 6,000 = $1,200
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
20. The many probability and impact combinations represent a continuum
20% x $ 6,000
P
10% x $10,000
5% x $25,000
1% x $50,000
0-10
1020
2030
3040
4050
Impact
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
22. Risk is measured using internal and external loss data. The two
measures of exposure are the aggregate mean and aggregate Value at
Risk (VaR).
RISK MATRIX FOR
LOSS DATA
INDIVIDUAL
LOSS EVENTS
LOSS
DISTRIBUTIONS
P
74,712,345
74,603,709
74,457,745
74,345,957
74,344,576
•
INTERNAL
FRAUD
Corporate Finance
Nu mb er
Mea n
Standard Deviatio n
Trading & Sales
•
Nu mb er
Mea n
Standard Deviatio n
Retail Banking
Nu mb er
CLIENTS,
PRODUCTS &
BUSINESS
PRACTICES
DAMAGE TO
PHYSICAL
ASSETS
EXECUTION,
DELIVERY &
PROCESS
MANAGEMENT
BUSINESS
DISRUPTION AND
SYSTEM
FAILURES
36
33
150
2
315
35,459
52,056
3,456
56,890
56,734
1,246
89,678
44,215
8,975
3,845
5,694
7,890
3,456
245
23,543
6,976
50
4
35
50
46
210
3
441
53,189
8,541
78,084
13,463
5,184
5,768
85,335
11,835
85,101
5,184
1,869
368
134,517
35,315
66,322
10,464
4
32
45
42
189
3
4,666
76,802
76,591
1,682
121,065
59,690
Standard Deviatio n
7,687
12,116
5,191
10,652
4,666
331
31,783
Commercial Bankin g
9,417
Nu mb er
Mea n
Standard Deviatio n
41
43,083
6,918
3
63,248
10,905
28
4,199
4,672
41
69,121
9,586
37
68,932
4,199
170
1,514
298
2
108,959
28,605
Nu mb er
Nu mb er
Mea n
Standard Deviatio n
Asset Manag ement
Nu mb er
Mea n
Standard Deviatio n
Retail Brokerage
Nu mb er
Mea n
Standard Deviatio n
Insuranc e
Nu mb er
37
3
26
37
34
153
2
2
3
4
321
38,774
56,923
3,779
62,209
62,039
1,363
98,063
48,349
9,814
4,205
6,226
8,628
3,779
268
25,744
7,628
44
4
31
44
40
184
2
386
46,529
7,472
68,308
11,777
4,535
5,045
74,651
10,353
74,446
4,535
1,635
321
117,675
30,893
58,018
9,154
40
3
28
40
36
165
2
347
41,876
61,477
4,081
67,186
67,002
1,472
105,908
52,217
6,725
10,599
4,541
9,318
4,081
289
27,804
8,238
48
50,252
4
73,773
33
4,898
48
80,623
44
80,402
198
1,766
3
127,090
417
62,660
8069
12719
5449
11182
4898
347
33365
9886
43
4
30
43
39
179
2
66,395
4,408
72,561
72,362
1,589
114,381
56,394
7,262
11,447
4,904
10,063
4,408
312
30,028
8,897
Nu mb er
Mea n
Standard Deviatio n
435
45,653
7,331
36
67,021
11,555
302
4,450
4,950
435
73,245
10,158
399
73,044
4,450
1,812
1,604
315
24
115,459
30,311
P
375
45,226
Standard Deviatio n
3,806
56,926
8,981
Mea n
Total
1
357
53,721
8,476
Standard Deviatio n
Agency Services
0
397
70,276
TOTAL LOSS
DISTRIBUTION
Frequency
of events
TOTAL
25
45
Mea n
167,245
142,456
123,345
113,342
94,458
EMPLOYMENT
PRACTICES &
WORKPLACE
SAFETY
3
47,870
Mea n
Payment & Settlements
•
EXTERNAL
FRAUD
36
VAR
CALCULATION
Severity
of loss
VaR
Calculator
e.g.,
Monte
Carlo
Simulation
Engine
Risk
Mean
99th Percentile
Annual Aggregate Loss ($)
0-10
1020
2030
3040
4050
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
23. Risk is measured using internal and external loss data. The two
measures of exposure are the aggregate mean and aggregate Value at
Risk (VaR).
RISK MATRIX FOR
LOSS DATA
INDIVIDUAL
LOSS EVENTS
LOSS
DISTRIBUTIONS
P
74,712,345
74,603,709
74,457,745
74,345,957
74,344,576
•
INTERNAL
FRAUD
Corporate Finance
Nu mb er
Mea n
Standard Deviatio n
Trading & Sales
•
Nu mb er
Mea n
Standard Deviatio n
Retail Banking
Nu mb er
CLIENTS,
PRODUCTS &
BUSINESS
PRACTICES
DAMAGE TO
PHYSICAL
ASSETS
EXECUTION,
DELIVERY &
PROCESS
MANAGEMENT
BUSINESS
DISRUPTION AND
SYSTEM
FAILURES
36
33
150
2
315
35,459
52,056
3,456
56,890
56,734
1,246
89,678
44,215
8,975
3,845
5,694
7,890
3,456
245
23,543
6,976
50
4
35
50
46
210
3
441
53,189
8,541
78,084
13,463
5,184
5,768
85,335
11,835
85,101
5,184
1,869
368
134,517
35,315
66,322
10,464
4
32
45
42
189
3
4,666
76,802
76,591
1,682
121,065
59,690
Standard Deviatio n
7,687
12,116
5,191
10,652
4,666
331
31,783
Commercial Bankin g
9,417
Nu mb er
Mea n
Standard Deviatio n
41
43,083
6,918
3
63,248
10,905
28
4,199
4,672
41
69,121
9,586
37
68,932
4,199
170
1,514
298
2
108,959
28,605
Nu mb er
Nu mb er
Mea n
Standard Deviatio n
Asset Manag ement
Nu mb er
Mea n
Standard Deviatio n
Retail Brokerage
Nu mb er
Mea n
Standard Deviatio n
Insuranc e
Nu mb er
37
3
26
37
34
153
2
2
3
4
321
38,774
56,923
3,779
62,209
62,039
1,363
98,063
48,349
9,814
4,205
6,226
8,628
3,779
268
25,744
7,628
44
4
31
44
40
184
2
386
46,529
7,472
68,308
11,777
4,535
5,045
74,651
10,353
74,446
4,535
1,635
321
117,675
30,893
58,018
9,154
40
3
28
40
36
165
2
347
41,876
61,477
4,081
67,186
67,002
1,472
105,908
52,217
6,725
10,599
4,541
9,318
4,081
289
27,804
8,238
48
50,252
4
73,773
33
4,898
48
80,623
44
80,402
198
1,766
3
127,090
417
62,660
8069
12719
5449
11182
4898
347
33365
9886
43
4
30
43
39
179
2
66,395
4,408
72,561
72,362
1,589
114,381
56,394
7,262
11,447
4,904
10,063
4,408
312
30,028
8,897
Nu mb er
Mea n
Standard Deviatio n
435
45,653
7,331
36
67,021
11,555
302
4,450
4,950
435
73,245
10,158
399
73,044
4,450
1,812
1,604
315
24
115,459
30,311
P
375
45,226
Standard Deviatio n
3,806
56,926
8,981
Mea n
Total
1
357
53,721
8,476
Standard Deviatio n
Agency Services
0
397
70,276
TOTAL LOSS
DISTRIBUTION
Frequency
of events
TOTAL
25
45
Mea n
167,245
142,456
123,345
113,342
94,458
EMPLOYMENT
PRACTICES &
WORKPLACE
SAFETY
3
47,870
Mea n
Payment & Settlements
•
EXTERNAL
FRAUD
36
VAR
CALCULATION
Severity
of loss
VaR
Calculator
e.g.,
Monte
Carlo
Simulation
Engine
Mean
99th Percentile
Annual Aggregate Loss ($)
0-10
1020
2030
3040
4050
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
What is the impact of
the tail on the mean?
24. By comparing changes in the control environment one can predict
changes in each business’ risk profile
VAR
CONTROL
ASSESSMENT/INDICATOR
SCORE
CAPITAL
Adjustment for
Quality of
Current Control
Environment
210
190
100
Current score
Previous score
50
0
Linking capital to changes in the quality of internal controls
provides an incentive for desired behavioral change
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
25. The risk management industry approach can be used to integrate
measures of risk and control, which can be used for allocating economic
capital
RISK MATRIX FOR CAPITAL
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
27. What other approaches can be considered for risk assessment?
• Directly estimate frequency and severity parameters through expert
judgment
• Estimate frequency and severity distributions using institutional
memory
• Estimate risk (VaR) directly using internal and external loss data and
disciplined scenario analysis
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
29. Even with significant amounts of historical loss data it is virtually
impossible to reliably estimate severity parameters.
Number of Events
Size of Loss
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
30. It is also very difficult to reliably estimate severity probabilities at different
quantiles. Multiple estimates often create internal inconsistency
1 in 1 years
P
= $1,000
1 in 10 years = $10,000
1 in 20 years = $25,000
1 in 100 years = $50,000
0-10
1020
2030
3040
4050
Impact
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
31. Disciplined scenario analysis has been found to be moderately reliable
and has produced valuable business benefits.
• The analysis is based on factual, historical (external) loss data
• Risk magnitude is clearly defined as potential loss at a specified
confidence level, such as 99%
• A 99% level event is defined to mean the second highest loss in one
hundred years
• This is further clarified – put into practical terms – based on loss
experiences of ten peer banks; (similar size, similar controls), the
second highest loss in the last ten years for the peer group
The whole purpose of this analysis is to allow the bank to compare
the magnitude of loss at the same probability level:
50 foot tidal wave vs. 100 tidal wave
$10 million money transfer loss vs. $100 million sales practices loss
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
33. We start with the risks, and using loss data identify control weaknesses
and their underlying causes
RISKS
LOSSES
Internal Fraud
167,245
142,456
123,345
113,342
94,458
RISKS
EDPM
CAUSES
CONTROL ISSUE
Segregation of duties
Vacation policy
Data manipulation
Data Integrity
LOSSES
CAUSES
74,712,345
32,603,709
457,745
5,345,957
44,576
Insufficient training
Lack of management
supervision
Risks are manifested in losses
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
INDICATOR
Staff Training Budget
Number of
Reconciliation Errors
Number of
Customer Complaints
34. The goal is to identify which specific controls ought to be assessed
RISKS
LOSSES
CAUSES
CONTROL ISSUE
Access to
information
Timeliness
of information
Internal Fraud –
Credit Card
Counterfeiting
2,000 events
Losses =$40M
Leakage of
Confidential
Information
Special clearance for
overseas travelers
Special procedure in
terms of Business
Intelligence Algorithms
Risks are manifested in losses
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
35. The next step is to examine the underlying business processes to better
understand how well the risks are currently being managed and controlled
Process Chart
Applications Received.
Data Entry
.
Completed entry applications forwarded to officers.
Credit Information Verification
Application processing officers. Officers’ decision:
- Approved / Declined / Further information
Allocation of Application
End
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
36. A control assessment scoring process must be relevant, consistent and
objective
Relevance
The control issues must be relevant to a business line and risk
Answer Choices
The answer choices should be consistent
Weighting
The control issues must be weighted according to relevance
Scale
All scores must be converted to a consistent scale,
e.g., 0 to 100
Normalization
The process for normalizing scores must be theoretically valid
Transparency
The process must be transparent to allow for buy in and to
identify opportunities for improvement
Validation
Responses must be validated to avoid “gaming” the system
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
38. Integrated risk measurement and management can produces value,
where the results are meaningful
• Risk assessment is feasible and practical, but must be implemented
only after one fully understand the meaning of the word risk and the
technical challenges.
• Control assessment is feasible and practical, but must only be
implemented after one understands the how to make a subjective
process more objective.
• Integrated risk and control assessment or measurement promotes
educated decision making, which in turn facilitates prudent risk
management an can contribute to the creation of a good risk culture.
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
39. COSO was conceived in the early 1990’s – the first attempt at
standardizing what was a subjective and inconsistently applied method for
identifying, assessing, controlling and managing operational risks
• However, in the context of modern operational risk management, we
have learned:
• The definition of risk magnitude under COSO is inconsistent with
that used in the risk management industry, including the BIS
• The approach is highly subjective, resource intensive and
generates a huge catalog of unmanageable ‘risks’
• COSO approach to risk assessment (likelihood - impact analysis)
focuses attention on what are likely to be phantom risks, not real
risks. It produces both false positives and false negatives.
• Any prioritization of controls based on this spurious and misleading
information may lead to enhancing controls in areas that are
already over-controlled, while ignoring areas of control weakness
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
40. When the answers are unclear…
… is it because we are asking the wrong
questions?
41. Biographical Information
.
Ali Samad-Khan is President of OpRisk Advisory LLC. He has eight years experience in operational risk measurement and
management and approximately twenty years of professional experience. His areas of expertise include: establishing an
integrated operational risk measurement and management framework, developing policies and procedures, internal loss event
database design and implementation; data quality assessment, data sufficiency, risk indicator identification, risk and control self
assessment, disciplined scenario analysis, causal/predictive modeling, advanced VaR measurement techniques and economic
capital allocation.
Mr. Samad-Khan has advised many of the world’s leading banks on operational risk measurement and management issues. His
significant practical experience in this field comes from managing the implementation of ten major operational risk consulting
engagements at leading institutions in North America, Europe and Australia. Key elements of the ORA framework and
methodology have been adopted by dozens of leading financial institutions worldwide and have also been incorporated into the
BIS guidelines.
Mr. Samad-Khan has frequently advised the major bank regulatory authorities, including: the Risk Management Group of Basel
Committee on Banking Supervision, the Board of Governors of the Federal Reserve System, the Federal Reserve Bank of New
York, the Financial Services Authority (UK) and the Australian Prudential Regulatory Authority. He also holds seminars and
workshops for the Bank of International Settlements (BIS) and the Institution of International Finance (IIF).
Prior to founding OpRisk Advisory, Mr. Samad-Khan was CEO of OpRisk Analytics LLC, which was acquired by SAS in 2003.
(From June 2003 to September 2004 Mr. Samad-Khan provided transitional support for the acquisition of OpRisk Analytics,
serving as SAS’ Head of Global Operational Risk Strategy.) He has also worked at PricewaterhouseCoopers (PwC) in New
York, where for three years he headed the Operational Risk Group within the Financial Risk Management Practice, in the
Operational Risk Management Department at Bankers Trust as well as the Federal Reserve Bank of New York and the World
Bank.
Mr. Samad-Khan holds a B.A. in Quantitative Economics from Stanford University and an M.B.A. in Finance from Yale
University.
Articles include: “Is the Size of an Operational Loss Related to Firm Size,” with Jimmy Shih and Pat Medapa, Operational Risk,
January 2000; “Measuring and Managing Operational Risk,” with David Gittleson, Global Trading, Fourth Quarter, 1998.
Working papers include: “How to Categorize Operational Losses – Applying Principals as Opposed to Rules” March 2002 and
“Categorization Analysis” January 2003.
Copyright © 2004, OpRisk Advisory LLC. All rights reserved.
Editor's Notes This slide shows how the actual adjustments to capital are factored in at the cell-level. In each instance, you see an initial calculation of required capital, absent of the control information, then the previous and current score that captures the magnitude and direction of change in internal controls, and the resulting final capital required. These values can be summed to arrive at a total figure for a given unit, in this example Corporate Finance.
In summary, the system measures the risk, net of controls, and it dynamically adjusts the risk measurement to reflect changes in internal controls. This puts an incentive process in place for the business managers in charge to do the right thing in order to reduce their capital charge!
