White paper model risk sept 2011

1,903 views

Published on

Model Risk guidance was provided by the OCC and FED in 2011. We comment on their proposal, review other relevant literature, and conclude with an user friendly checklist.

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,903
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
86
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

White paper model risk sept 2011

  1. 1. White Paper:Model RiskVersion: FinalSeptember 22, 2011
  2. 2. 2 Introduction Table of Contents New Supervisory Guidance on Model Risk Management ........................................................... 2 Introduction ............................................................................................................................ 3 Definitions .............................................................................................................................. 3 Model Usage .......................................................................................................................... 4 Better Practices Guidance ...................................................................................................... 5 Model development, implementation, and use ........................................................................ 5 Development ....................................................................................................................... 5 Implementation ................................................................................................................... 6 Use ..................................................................................................................................... 6 Model verification and validation ............................................................................................. 6 Model Risk Governance ......................................................................................................... 7 A Note on Third-Party Models ................................................................................................ 7 Self-Assessment of Your Model Risk Management Practices................................................. 8 Conclusion ............................................................................................................................. 9 Authors ................................................................................................................................... 9
  3. 3. 3 White Paper: Model RiskIntroductionIn early April 2011, the Office of the Comptroller of the Currency (OCC) and the Board ofGovernors of the Federal Reserve System (FED) released Supervisory Guidance on Model RiskManagement (the Guidance or OCC 2011-12). From our perspective, there was little new in theguidance. However, OCC 2011-12 codifies better practices that were previously familiar only tosector specialists, including the Capital Markets teams at the OCC and FED. So the news is notthat there is anything new, per se, rather the news is that better practices have been clearlycommunicated to OCC- and FED-supervised banks and thrifts via OCC 2011-12 i.The Guidance reinforces prior regulatory recommendations. From a regulatory perspective, thisguidance was issued to “provide comprehensive guidance for banks on effective model riskmanagement” ii. An unwritten cause of this issuance was that bank supervisors have observedsubstandard model risk practices across the industry at institutions both large and small.The Guidance notes that the use of models, especially complex models, varies across theindustry, so model risk management and mitigation should be commensurate with risk andmodel complexity. As is typical of regulatory issuances, it notes that internal controls, policies,and documented procedures are appropriate.The FDIC and NCUA were not included in this issuance, as they were with the InteragencyAdvisory On Interest Rate Risk Management (OCC 2010-1a) and the Interagency PolicyStatement on Funding and Liquidity Risk Management (OCC 2010-27a). As with OCC 2000-16(Model Validation) and SR 09-1 (Application of the Market Risk Rule in Bank HoldingCompanies and State Member Banks), it is likely that FDIC and NCUA supervisors will applyOCC 2011-12 to supervised institutions.DefinitionsA model is defined as “a quantitative method, system, or approach that applies statistical,economic, financial, or mathematical theories, techniques, and assumptions to process inputdata into quantitative estimates”. The definition of model is expanded to include “quantitativeapproaches whose inputs are partially or wholly quantitative or based on expert judgment,provided that the output is quantitative in nature”.The Guidance, congruent with OCC 2000-16, delineates three components of a model: 1. An information input component, which delivers data and assumptions to the model 2. A processing component, which transforms input into estimates 3. A reporting component, which transforms estimates into useful business information. The Guidance suggests there are two main sources of model risk:
  4. 4. 4 Model Usage 1. Use of a model with errors, including: a. an incorrect theoretical foundation b. improper implementation or maintenance c. inaccuracies in any of its three components (as above). 2. Incorrect or inappropriate use of a model a. Use a model in inappropriate circumstances b. Results that are not decision makers c. Lack of explanation or understanding of limitations The Guidance defines model risk as “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports”. This is a good summary definition as it also includes the potential for poor decisions based on the outputs of technically correct, fully validated models that were not totally understood by decision makers (e.g. Value at Risk models, 2. above). Eugene Derman’s 1996 article iii on Model Risk provides more information on the types of model risk: • Wrong model o Inapplicability of model o Incorrect model specification • Model implementation o Programming errors o Technical errors o Use of numerical approximations • Model usage o Implementation Risk o Data issues o Calibration errors Two of the primary tools for effectively managing model risk are model verification and model validation iv. Model verification refers to a process aimed at answering the question “did we build the model right”. Model validation refers to a process aimed at answering the question the question “are the results credible? Model Usage Industry observers and the Guidance recognize that the use of models in banks is pervasive in pricing, risk management, and reporting: • evaluate and underwrite credits • price loans and deposits • create and analyze forecasts and strategies
  5. 5. 5 White Paper: Model Risk • measure and monitor credit, liquidity, market, and operational risks • value exposures and financial instruments • report and disclose resultsThe Guidance states, “The use of models invariably presents model risk”. Derman noted, “Thisreliance on models to handle risk carries its own risks”. The authoritative Verification, Validation,and Accreditation of Army Models and Simulations v simply say, “There is no such thing as anabsolutely valid model”. Accordingly, we would suggest that best practices model riskmanagement practices recognize that model risk is unavoidable, but that effective model riskmanagement mitigates model risk within a cost/benefit context.Better Practices GuidanceThe Guidance highlights three areas for effective, best practices risk model management: • Model development, implementation, and use • Model validation • Governance, policies, and controlWhile the above framework is not new to bankers, the Guidance notes “A guiding principle formanaging model risk is "effective challenge" of models, that is, critical analysis by objective,informed parties who can identify model limitations and assumptions and produce appropriatechanges.”The notion of “effective challenge” is perhaps new to all but the largest of banks. In very largebanks, a specialized Model Risk or Model Validation unit reports to the Chief Risk Officer. Inmost banks, Internal Audit staff or the Chief Financial Officer’s team is tasked with modelreview. Usually, these units lack the expertise necessary to review the theoretical underpinningsand math of complex models. As a result, the review is ineffective, and frequently relies solelyon checklists emphasizing balancing to financials and policy compliance, which while necessaryare not sufficient.Model development, implementation, and useThis topic is broadly consistent with the definition of model verification, as defined above.Examples from the ALM model sector may illuminate the notion of “effective challenge” in thisarea.DevelopmentAs regards vendor model development and documentation, we have found that some models,even those with hundreds of client banks and that have been “certified” by a third party, have
  6. 6. 6 Model verification and validation fundamental valuation flaws. The flaws have been noted in both the valuation model math and in the documentation of how to implement the valuation tools. Perhaps this is due to the greater subject expertise of the model verification and validation team than the model development or certification teams. In any event, it is unlikely that community bank Internal Audit teams have sufficient expertise in valuation models to “effectively challenge” the use of a vendor valuation model or module, so the use of outside experts is important. Implementation We have also observed inadequate implementations of otherwise sound risk models. Recently, at a $1+ billion National Bank, we asked the ALM staff about the model implementation and subsequent model calibration. The staff readily noted that they had not updated its pricing, valuation, and prepayment configuration, including assumptions, over the past ten years since initial vendor implementation “because the model did it automatically”. Unfortunately, or fortunately, depending on your perspective, for the bank and readers of this publication, the state of the art over the past decade is such that no ALM model automatically updates every assumption appropriately and automatically. Use One of the key assumptions in the use of an ALM model is Core Deposit rate sensitivities. In many banks, unsubstantiated and undocumented rate sensitivity estimates are used. This has the impact of suggesting that their bank is either “asset and/or liability sensitive”. The use of alternative assumptions, via sensitivity analysis, market or institution statistical studies, or other quantitative approaches may affect the magnitude and/or directionality of bank risk exposures. Again, the typical Internal Auditor is unfamiliar with the risk reporting consequences of key assumptions, let alone the impact such changes may have on product pricing, hedging strategies, and balance sheet allocation decisions. Model verification and validation The Guidance outlines three core elements of an effective validation: 1. Evaluation of conceptual soundness and assessment of the quality of the model design and construction 2. Ongoing monitoring, including process verification and benchmarking 3. Outcomes analysis, including back-testing
  7. 7. 7 White Paper: Model RiskThe topic of Model Validation has been covered in previous BALM articles, including ours, andwill likely be the topic of futures articles, including ours. However, we would note that modelsare used in other parts of the world in addition to banking and present an example vi ofverification and validation activities. Accordingly, we suggest that “model validation” as used inBanking typically refers to “model verification and validation” in other sectors.Model Risk GovernanceTo control model risk, banks need a strong, thorough model governance process. This includesoutlining the important roles for the Board, senior management, the model operators/ownersand the model validators. The key is defining a workable structure that ensures the integrationof high quality model forecasts with management decision making and regulatory requirements.The challenge is to do this in a way that does not create a system crippled bymicromanagement processes.The following summarizes key requirements for effective governance: • Board of Directors and Senior Management - establish a strong model risk management framework that fits into the broader risk management of the organization • Policies and Procedures – require model risk management activities be formalized through policies and documented procedures to implement them • Roles and Responsibilities – address ownership, controls and compliance • Internal Audit - assess the overall effectiveness of the model risk management framework • External resources – specify in writing third-party resources engaged to assist with model validation and review, compliance functions, or other activities in support of internal audit • Model inventory – require the maintenance of a comprehensive set of information for models implemented for use, under development for implementation, or recently retired where applicable • Documentation – maintain detailed documentation of model development and validation so that parties unfamiliar with a model can understand how the model operates, its limitations, and its key assumptions.A Note on Third-Party ModelsSome banks rely on vendors to run financial models and prepare risk reports for them. Theguidance addresses the validation of vendor and third-party products by stressing that althoughthe process may be somewhat modified, the validation principles should be the same as appliedto in-house models. This helps bank management better understand the vendor product and itscapabilities, applicability, and limitations.
  8. 8. 8 Self-Assessment of Your Model Risk Management Practices The Board and bank management should ensure that the model works effectively if they use reports to make decisions. Management should be diligent in tracking changes the vendor makes to the model including for each reporting period a list of any model changes and an explanation of the expected effects on the resulting reports. Self-Assessment of Your Model Risk Management Practices Developing, implementing, and maintaining an effective model requires ongoing diligence in oversight, validation and controls management. Ask yourself, or have your Internal Audit ask these questions for each model your bank uses: 1. Are the model results significant to managing your bank’s financial position and/or defining future strategy? 2. Does the model have a sound theoretical foundation? 3. Are the assumptions reasonable and supported? 4. Do senior management and the Board understand the model’s primary purpose, methods, and calculations? 5. Are model limitation regularly communicated and understood? 6. Are the data inputs accurate? 7. Are model results translated into understandable analytical reports that support decision making? 8. Has your model been validated within the last twelve month period? 9. If your model has been validated, have significant findings and recommendations been addressed? 10. If the model is a third-party model, then answer these additional questions: a. Do you have an appropriate process in place for selecting vendor models? b. Do you require that the vendor provide developmental evidence explaining the product components, design, and intended use, to determine whether the model is appropriate for your bank’s products, exposures, and risks? c. Does your vendor regularly provide appropriate testing results, showing their product works as expected? d. Is your bank clearly aware of the model’s limitations, detailed assumptions and where the model’s use may be problematic with respect to your particular products? e. Does your vendor regularly report on and conduct ongoing performance monitoring and outcomes analysis, and make appropriate modifications and updates over time? f. Are your bank’s model customization choices documented, justified and understood? g. If your model vendor provides input data or assumptions, are you provided with details and is the relevance to your bank’s situation understood and assessed? h. Are you conducting ongoing monitoring and outcomes analysis of vendor model performance using the bank’s own outcomes?
  9. 9. 9 White Paper: Model Risk i. Do you have an in-house model expert who understands the systematic modeling procedures that can help the bank understand the vendor product and its capabilities, applicability, and limitations?ConclusionThe recent Guidance alerts banks to the emphasis that the OCC and FED are placing oneffective model risk management. Over the past two years, we have seen an increase in formaland informal regulatory orders related to effective model risk management for market, credit,and liquidity risk models. These orders have also noted the effectiveness and frequency ofBoard, ALCO, and Credit Committee reporting. We suggest that you use the “Self-Assessment”above and review the Guidance as time allows.AuthorsFred Poorman Jr., CFA, Managing Principal, Bank Risk AdvisorsHoward Stern, PhD, Senior Consultant, Bank Risk AdvisorsTerry Treadwell, CPA, Senior Consultant, Bank Risk Advisorsi Existing regulatory guidance includes– the FDIC’s Supervisory Insights (Winter 2005) article on ModelGovernance; OCC Bulletin 2000-16, the current guiding directive on ALM model verification for nationalbanks, as well as mandates outlined in various other regulatory directives -- the most recent IRR advisory,2010-1A, and discussion in FDICs Supervisory Insights (Winter 2009).ii Unless otherwise noted quotations are from the Guidanceiii http://www.ederman.com/new/docs/gs-model risk.pdfiv http://ppc.documents/model-verif-validation-full.pdfv http://www.apd.army.mil/pdffiles/p5 11.pdfvi http://ppc.documents/model-verif-validation-full.pdf

×