Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Model Validation

901 views

Published on

Protiviti presentation at the Trepp Banking Symposium, 10-9-2015.

Published in: Business
  • Be the first to comment

Model Validation

  1. 1. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. TREPP BANKING SYMPOSIUM MODEL RISK MANAGEMENT OCTOBER 9, 2015
  2. 2. TABLE OF CONTENT • MODEL DEFINED • EFFECTIVE CHALLENGE • TOP ISSUES • THREE LINES OF DEFENSE • VALIDATION TASK & TECHNIQUES • ROLE OF INTERNAL AUDIT • QUESTIONS
  3. 3. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. A Model Defined A model is a computer-based mathematical construct that processes data inputs and produces outputs used in business decision making 3 A model is not a decision-maker A model is only as good as the assumptions and inputs that go into it Even a great model will “fail” if it is used inappropriately Even a great model that has been used appropriately will need to be updated for environmental changes
  4. 4. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Financial Models Differ From Other User Developed Tools 4 • Provided by vendors or developed in house using a wide range of platforms. • An approximate representation of the real world that can be used to calculate prices, risks, reserves, and strategies for financial products. • An estimate based on a set of assumptions about the behavior of the people, organizations, risks, and other models participating in the marketplace • A model may reside within a user developed tool. A Quantitative Financial Model is: • Also known as an End User Tool, Spreadsheet or just tool. • End user tools tabulate known or agreed values. • If a calculation is objectively true with no possibility of an incorrect assumption, it is not a model. A User Developed Tool is: Model Validation and Audit should provide effective challenge around controlling risks associated with all business tools.
  5. 5. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Effective Challenge In Model Risk Management Effective challenge for model risk involves three guiding principles: independence from model developers, competent resources, and an influence within the organization that values and supports the function’s role. 5 Senior Management & Board Oversight Risk Identification & Assessment Risk Measurement Risk Reporting, Monitoring & Management Effective Challenge Resources Organizational Structure Risk Culture Policies & Practices Independence/Incentives • The effective challenge function (e.g. Model Validation, Governance Committees, Internal Audit) or model risk should be distinctly separate from the model development process • Challenge should also be appropriately supported with well- designed compensation practices Competence • Individuals in the challenge function require technical expertise and modeling skills to effectively validate model assumptions, limitations, usage, and outcomes • Understanding of business processes and financial knowledge enhance the challenge function’s value in risk management Influence • Effective challenge has explicit authority and responsibilities supported by policies and procedures as well as Senior Management’s commitment • The challenge function has stature and visibility within the organization and is a respected partner to the business
  6. 6. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Guidance For Effective Challenge of Model Risk Management The OCC/FED supervisory guidance on model risk management outlines key principles and control mechanisms for model risk, including effective challenge. 6 • Developers should consider models’ impact on business performance from user feedback and should factor in suggestions and criticism from sources independent of the business • Existence of effective challenge is required to identify model limitations and assumptions Model Development, Implementation, and Use • Validation of models should be performed by a function independent of the developers • Each model should be reviewed at least annually to assess material changes and whether validation activities are sufficient. • Models should be validated for conceptual soundness including the validity of assumptions, limitations, and outcomes • Models from 3rd party vendors should be evaluated for appropriateness to the institution’s situation and require ongoing monitoring by the institution and the vendor Model Validation • Board and Senior Management should support effective challenge through establishment of a robust model risk framework that includes policies and procedures outlining the roles and responsibilities for model validation and quality assurance functions • Risk Management functions are responsible for managing model validation and review processes to ensure effective challenge occurs • Internal Audit evaluates the design and effectiveness of controls for model risk management and communicates audit findings to the business for remediation • Use of external resources for model validation, compliance, or audit requires clear lines of communication to internal parties for identified issues Model Governance, Policies, and Controls
  7. 7. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Top 10 Issues In Model Validation 7 Data constraints1 Lack of explanation for business judgment2 Inadequate model documentation3 Invalid or unrealistic assumptions4 Too many performance testing exceptions5 Difficulty in choosing the appropriate level of conservatism6 Lack of alternative model analysis7 Inadequate validation of vendor models8 Inadequate understanding of modeling software functionalities9 Model risk has inadequate stature in the organization10
  8. 8. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Model Governance & Validation 2nd Line of Defense Line of Business 1st Line of Defense Industry Leading Framework for Model Risk Management 8 Model Governance and Model Validation Model Governance Committee(s) Board of Directors Model Owners and Model Users Internal Audit 3rd Line of DefenseInternal Audit
  9. 9. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Industry Leading Practices 9 First Line of Defense Model Developer Second Line of Defense Risk Management / Model Validation Group Third Line of Defense Internal Audit The Issuance of OCC bulletin 2011-12 / FRB SR 7-11 has prompted a shift in all banks to place heightened emphasis on Model Risk Management, including model validation The largest banks have begun to appoint a Chief Model Risk Officer Medium-sized banks have adopted model risk policies designed to drive a structured model risk governance process with a focus on effective challenge Some banks with assets over $10 billion now have a dedicated model validation function Model Risk Management Infrastructures focuses on three levels of control Executive Management and The Board of Directors are expected to review and approve a Bank’s model risk governance process
  10. 10. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Roles in Model Risk Management Model validation and internal audit ensure controls are designed adequately and implemented effectively to reduce the likelihood of erroneous model output or incorrect interpretation of model results. 10 Activity Players Development Implementation Production Reporting Validation Model Developers • Code • Document assumptions, theory, and testing • Conduct user acceptance tests (UAT) • Monitor performance • Support user interpretation of model results • Provide code and documentation Independent Validators • None (maintain independence) • Review implementation • Periodic reviews • Ensure model reports are used appropriately • Evaluate • conceptual soundness, outcomes analysis, • ongoing monitoring Internal Audit • Evaluate adequacy of development guidelines • Test completeness of development documentation • Test systems development life cycle (SDLC) • Evaluate model risk monitoring controls • Evaluate completeness of reports • Evaluate design adequacy of validation guidelines • Test effectiveness of validations
  11. 11. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Validation Tasks 11 • Interview model developers and other key stakeholders to obtain an understanding of the models • Review and assess whether the documentation is sufficient to allow a third party to reconstruct the model if necessary Interviews & Model Documentation Review • Review developmental evidence for the model • Review assumptions • Conduct impact analysis for underlying assumptions • Assess theoretical limitations • Assess whether theoretical and conceptual framework aligns with the purpose Model framework, Theory & Design • Review calculations within ranges of industry practices and regulatory requirements • Independently replicate calculation components • Independently replicate the model calculation process and benchmark outputs Process, Code & Math • Review the model control environment by testing a sample of model inputs if applicable • Review the reasonableness of the calculation of input variables • Review and evaluate missing data handling process • Review and evaluate the performance of input model if applicable • Assess the reasonableness of key parameter inputs and assumptions • Assess the alignment of qualitative and quantitative assumptions, supporting business rationale, and statistical analysis, if applicable Model Input Quality • In-sample, out-of-sample and out-of-time sample validation tests • Multiple testing approaches may be applied for different types of models Performance Testing
  12. 12. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Common Validation Techniques There are a variety of techniques that can be applied in the performance of model validations. Some of the various techniques are described below. Many of these techniques are applied in combination with other testing approaches. 12 Techniques Description Back Testing Compare actual outcomes with model predicted outcomes using statistical measures. Sensitivity Analysis Evaluate the sensitivity of changes in model output in relation to changes in input (macro variables, interest rates, home prices, etc). Benchmarking Compare internal model results to other like models (e.g. industry models). Model Replication Replicate design, attributes, and mathematical form of models. Compare outputs between parallel replica and the actual model. Cash Flow Replication Estimate cash flows outside of the model validating that costs, expenses, instrument attributes, and assumptions have been properly considered. Calibration Testing Where model inputs are not available. Calibrating the score or log-odds to observed default rates over various time periods. Calibrations techniques such as determining calibration of FICO scores to default rates. Assumption & Theory Review Justify the reasonableness of assumptions and modeling simplifications.
  13. 13. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Industry Model Validation Approaches Below is an illustration of where certain techniques maybe useful given model type or subject matter. The appropriateness of specific procedures must be determined by qualified personnel and in the context of the model’s design and intended use. 13 Model Types Back Testing Sensitivity Analysis Bench- marking Model (or Modular) Replication Cash Flow Replication Calibration Testing Assumption Theory & Review Scorecards a a a Interest Rate a a a a a Forecasting Models a a a a Valuations a a a a Marketing a a a Operational Risk Models a a a a Economic Capital Models a a a Roll Rate Models a a a Pricing Models a a a a a a Liquidity Models a a a a AML Thresholds a a a
  14. 14. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. Model Audit Approach—3rd Line Review 14 Process Test of Design Test of Effectiveness Sample Size (#) Organization structure/roles & responsibilities N/A N/A Risk reports submitted to the Board and/or Senior Management Development standards & documentation Identification Risk rating Usage Performance metrics & thresholds Issues closure / extension Change management Adjustments / overrides Annual review Validation
  15. 15. © 2015 Protiviti Inc. CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. 15 Questions?
  16. 16. 16

×