MRM
- 1. Model Risk Management Disclaimer:The views and opinions expressed in this article are those of the authors own and do not necessarily reflect the view / views of the organizations the author has worked / working with. KV N K Prasad
- 2. Model Risk Managementcan be understood as effective and robust mechanism of identification and resolution of risk of model failure due to incorrect inputs, flawed assumptions, and incorrect model design or model misuse. The Model Risk Management has been gaining significant importance across the financial institutions and regulators. The extract from OCC SR-11-7 also clearly highlights the scope of Model Risk Management function “…while model validation remains at the core of the new guidance, but the broader scope of model risk management encompasses model development, implementation, and use, as well as governance and controls related to models”. Model risk can set-in anywhere in the lifecycle of the model
- 3. 1. First Lineof Defense 2. Second Line of Defense 3. Third Line of Defense Model Implementation Model Methodology Model Development Model Documentation Model Change management Model Change Implementation Model Owners ( Credit Risk, Market Risk, Operational Risk)User Model Validation & On-going Monitoring Internal Audit 1 2 3 Model Risk Committee Board Risk Committee Board The value of sophisticated MRM extends well beyond the satisfaction of regulatory regimes, effective MRM can improve an institution’s earnings through cost reduction, loss avoidance, and capital improvement.
- 4. A three-tiered approachto addressing model risk has emerged, with defined roles and responsibilities between the first, second and third lines of defence. Third Line A full-scope validation is expected to be conducted by a person or team qualified to assess the model, and the validators must be independent of the model developer/builder. Validation covers certain specific procedures, including a review of the governance surrounding the model, an assessment of the data inputs, the conceptual design and framework and tests of the implementation accuracy and calculation processes, including back-tests of the outputs or other outcome analysis to assess their reasonableness. Finally, the third line of defence, generally internal audit, reviews the work of model risk management to affirm that the model risk management framework provides effective challenge, that appropriate internal and external resources were used, that processes and procedures were properly followed and that data integrity was maintained throughout First Line Second Line The first line of defense is the model developer/owner, who is responsible for building the model to acceptable standards, consistent with academic literature and industry standard practices, and then documenting it adequately. Documentation standards have become rigorous. Expectations are that a knowledgeable person can pick up the documentation and not only understand it, but be able to run it, use it and possibly make changes to it. Seamless integration between 1st, 2nd and 3rd lines of defense is key for MRM success
- 5. S.No Firm LossYear Cause 1 JPMC $5-7 BN 2012 Model Control & usage 2 GFC $14 TN 2007-10 Correlation Assumptions 3 NAB $1-2 BN 2001-03 Rate Methodology 4 LTMC $4-5 BN 1997 Model Assumptions 5 American Consulting Firm $145 MN 2005 Model not effectively calibrated for stress events 6 UK Based Investment Bank $200 MN 2008 Model misused by model users 7 Financial Firm in Japan $83 MN 1997 "Vanilla" derivatives pricing applied for exotic derivatives 8 American Multinational Insurance Company $85 MN 2008 Failure of model validation and governance. 9 Dutch Multinational Bank $43 MN 2005 Modeling Errors & modeling assumptions Model risk can lead to financial loss, poor business and strategic decision making, or damage to a bank’s reputation. - FED/OCC Guidance on Model Risk Management (April 2011) There are numerous cases where banks have failed in regulatory stress testing due to various MRM issues resulting into hefty fines
- 6. Phase – I Phase- II Phase - III Basic MRM Infrastructure • Model Inventory • MRM policy - Standards & Governance • Work flow Tools • Basic governance & communication forums • Separate model validation and governance team Enhanced MRM Infrastructure • Model Inventory - tiers criteria • MRM policy - Standards & Governance for various models in different tiers • Automated / Semi-automated work-flow tools • Enhanced governance & communication forums • Enhanced controls & process management • Separate model validation and governance Sophisticated MRM Infrastructure • Model Inventory - tiers criteria • MRM policy - Standards & Governance for various models in different tiers • Fully automated work-flow tools • Sophisticated governance & communication • Sophisticated controls & process management • Integrated MRM dashboards • MRM center of excellence • Resource optimization infrastructure & process From To Large financial institutions are currently at Phase – II and are in transition to Phase – III A fully matured MRM provides strategic benefits which are way beyond compliance exercise
- 7. Model Usage 1.Source ofdata being used for mode development. 2.Through checks of codes & data logs along with logic 3. Assess data relevance & reconciliation with standard reports 4. Assessment of assumption and data treatments This includes a thorough review of the assumptions and methodologies employed, as well as the data set used; it also involves identifying where the key assumptions are useable and where they are not 1. Alternative theories & approaches 2. Model Metrics & limitations 3. Model Accuracy 4. Robustness & Stability Do the outputs capture the behaviour seen in the real world? In principle this is simpler; in practice it may be harder (than the conceptual analysis). However, it is clearly necessary, and has the ability to reject a number of poor models with a clear rationale 1. Model Backtesting - Discrimination - Calibration - Stability 2. AvE Analysis 3. Model Benchmarking Input Data Conceptual Soundness Outcome Analysis Mode implementation 1. Production codes and change controls 2. System & platform implementation reports 1. Ensuring that model is used for the purpose it is intended 2. Review & document scope & limitations Model Validation – Process Flow Final Deliverable : Final Model Validation report – with “Fit for Purpose” & key findings Model validation is defined within regulatory guidance as “the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives, and business uses. Note: At each stage process controls are established to highlight key issues with stakeholders and signal major “showstoppers “
- 8. 1. Every yearMRM (model validation team) should review dozens of models by a date that is set in stone. 2. Every year there are numerous models that are recalibrated or fully redeveloped. 3. The model validation inventory keeps increasing with surge of more number of models for different purposes ( CCAR / DFAST, IFRS 9, CECL, ALLL , PD, LGD, EAD, E-CAP, Fraud, Collections, origination and Behavioral etc…,) 4. There are new guidelines that govern different class of models (Stress Testing Vs. CELC) and enhancements to model validation landscape are constantly being added. 5. Model development team keeps going back to drawing board and want to work on models as long as possible 6. Limited number of model validation resources. 7. Delay in providing feedback by regulators ( usually for CCAR its late summer when the feedback is received - if there are too many rebuild – there is too much rush) 8. Lack of established channels & transparency to highlight and resolve the key validation issues. 9. In consistent scoping theWork, enabled by in-effective processes 10. Lack of shared accountability. Model risk is a very young field and is still in search of its commonly accepted definitions and well established standards and practices, resulting into too many constraints
- 9. 1. Having anagile model validation process is key to achieve success and avoid last minute rush. 2. In the agile model validation approach – validation team will not wait for model to completed. 1. The development & validation go hand in hand. 3. Key mile stones are set –and as and when the development team completes the miles stone – the validation will follow thru. 4. The critical success factor for agile model validation success is ongoing feedback from validators to developers, while maintaining independence. 5. Second critical success factor for agile model validation is to hard-stop on agreed-upon dates when developer will put “pencils down” and stop iterating on the model build so validation can be finalized. 6. Agile model validation requires robust controls and process enhancements to maintain high standards. 7. Last of all, but not least – train … train and re-train the validation resources on various new developments & methodologies of validation. But I should caution that if you seek to plot out all your moves before you make them—if you put your faith in slow, deliberative planning in the hopes it will spare you failure down the line—well, you're deluding yourself
- 10. As the thirdline of defense to manage risk, the audit function is increasingly expected to perform self-testing that should challenge the model’s conceptual design, data reliability, and risk management controls. In this capacity, the role of auditing and self-testing is not to replicate model validation activities, but rather to assess how effectively a model risk management framework meets business and regulatory requirements and how models perform under controlled environment. Model Audit – Process Flow Regulatory Response Scoping 1.The key deliverable of the audit function is to look thru the regulatory findings and coordinate with 1st and 2nd lines of defense in addressing the findings in an appropriate way. Process & Control Reviews The audit function closely monitors the documents, policies, procedures and standards. Collect relevant model documents, validation reports, testing files Assessment of completeness of submission of model development documents and validation documents. Regulatory Submission Support Review submitted documents against regulatory expectations and internal policies and industry leading practices. Assessment model development documents and validation documents inline with regulatory expectations Create best documentation and reporting practices GAP Assessment Identify gaps in model development methodology Identify gaps in model validation . Create resolution plan to bridge identified gaps . Development of audit assessment reports Key Success: Supervisors observe that issues they find in first and second line have been already identified by audit
- 11. 1. Planning andscopes are clearly and communicated ex ante 2. Audit focuses on evaluating processes and controls, not individual models 3. Audit has sufficient standing in the organization to feel comfortable pointing out shortcomings and to have them addressed in model development & model validation environments. 4. Audit is able to identify thematic issues and root causes of non-adherence to policies and procedures 5. Model audit outcomes are consistent with regulatory expectations 6. Model audit findings are regularly updated to senior management and respective risk committees in the firm. 7. Model audit should result into measurable findings – that are appropriately critical, and has clear conclusions The regulatory environment has become more complex due to unacceptable conduct remediation of model risk management issues. Consequently, financial firms continue to be compelled to demonstrate compliance with legal mandates through documented assurance assessments.
- 12. 1. As arisk discipline, model risk is a very young field and is still in search of its commonly accepted definitions and well established standards and practices across spectrum of different models and regulatory regimes. 2. Regulatory attention on MRM continues to grow in coming days. 3. Sustainability of MRM practice is key for delivering long term goals 4. Constantly strive to demonstrate that MRM is not a mere supervisory process, but is a loss prevention center of excellence.