Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reducing Regulatory Drag on Analytics Teams


Published on

Insights No. 82—Automated workflows standardize and speed model management processes

Banking regulators are increasing scrutiny of analytic models, peeling back layers of the onion with probing questions. They want to know not only how models affect credit policies and customer decisions, but about the processes used for developing, validating, deploying and updating them.

Banking executives, increasingly aware of the full dimensions of model risk, are also asking pointed questions.

Finding answers can add drag to the performance of analytics teams—even pulling them away from high-value work that leads to competitive advantage.

Published in: Technology
  • Some of these essay writing guides can be very helpful, but I don't think you should support and promote services that sell essays (written by others) to students, who then turn them in as their own work. That's just wrong, on many levels. Check out for loads of helpful tips, advice and instruction to help students write their own standout essays!
    Are you sure you want to  Yes  No
    Your message goes here
  • Essay origin is one of the best essay writing service that helps to know about the purpose of the essay writing. It's called Good luck!
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Reducing Regulatory Drag on Analytics Teams

  1. 1. Make every decision countTM INSIGHTS WHITE PAPER Number 82 Reducing Regulatory Drag on Analytics Teams Automated workflows standardize and speed model management processes Banking regulators are increasing scrutiny of analytic models, peeling back layers of the onion with probing questions. They want to know not only how models affect credit policies and customer decisions, but about the processes used for developing, validating, deploying and updating them. Banking executives, increasingly aware of the full dimensions of model risk, are also asking pointed questions. Finding answers can add drag to the performance of analytics teams—even pulling them away from high-value work that leads to competitive advantage. To improve compliance and response time to detailed questions, leading banks are implementing formal model management processes throughout the analytic lifecycle. But while best practices may be understood, they can be challenging to deploy consistently across analytic teams. It’s also difficult to know if they’re being followed at the right level of granularity, such that no matter where regulators probe—and even with analytic staff turnover— all questions can be readily answered. This white paper examines how automated, configurable model workflow tools promote process consistency and accountability. We show how banks are using workflow at enterprise and departmental levels to improve model governance without creating extra work for analytic teams. In fact, by orchestrating model processes while automatically capturing key artifacts, decisions and sign-offs, workflow can lead to better model performance. Analytic teams are freed to spend more of their time creating and updating models. We’ll cover: • Turning work friction into work flow • Creating standard processes that fit the needs of diverse teams • Tracking model and characteristic lineages Improve compliance while spending 75% less time on it.
  2. 2. Reducing Regulatory Drag on Analytics Teams INSIGHTS WHITE PAPER November 2014 page 2 Banks, long at the vanguard of data analytics for business, have continued to expand their use of models. Predictive scorecards and other models for anticipating and responding to customer behavior now play a central role in every area of operational decision making. The sheer quantity of these models is increasing rapidly. What hasn’t advanced as quickly is model management. An August 2014 article by Butler Analytics pointed out that“some banks simply do not know how many models are actually deployed.”In other cases, model information is in so many places that preparing for audits or answering regulator inquiries becomes extremely labor-intensive. The Butler article reports that“the modeling staff in one major US bank now spend 80% of their time meeting regulatory requirements, detracting from much needed new model development.” While this is an extreme case, every bank is seeing an impact on the productivity and performance of its analytics teams. Both those building models and those compiling reports and answering inquiries have more work to do. And where one group is charged with both duties, compliance tasks can siphon away analytic resources from work that could be producing significant value for the bank. An executive in the mortgage division of one bank told FICO that, for a period of time, 25% of its analytics workforce had to be diverted to collecting, preparing and reporting on data required by regulators—costing the bank tens of millions of dollars. Yet some banks are moving ahead of the curve. They’re improving their ability to answer questions about analytics while lightening the burden on their analytics teams. In fact, a bank devoting 80% of modeler time to regulatory requirements could reduce that expenditure to 20% or less. Model Management— Getting Ahead of the Curve Over the past five years, analytic excellence has become a core requirement in today’s financial market… Modeling touches virtually every decision of the bank… IDC Financial Insights A Framework for Model Governance, June 2013 “ Organizations today face heavy regulatory pressures…To meet these challenges and mitigate risk, they need model management solutions that can reduce resources required to complete compliance audits, and encompass the full model lifecycle and risk-management continuum...” ”Peyman Mestchian Managing Partner at Chartis
  3. 3. page 3 Reducing Regulatory Drag on Analytics Teams INSIGHTS WHITE PAPER This more efficient, streamlined approach combines centralized model management with automated, configurable workflow tools. As depicted in Figure 1, workflows enforce approved processes at every stage of the model lifecycle. They capture granular information about what decisions and actions are taken and why. Populated by these workflows, the central repository maintains an inventory of all models in operations and under development—their purpose, data types/sources, key assumptions, exclusions, predictive characteristics, segmentation schemes, where model is being used, restrictions on usage, etc. The model management solution also automates scheduled validations and flags models for review when stability or performance metrics decline. This makes it much easier for banks to maintain models at peak performance, as well as spot and resolve any compliance issues. Analytics teams accessing this shared resource have detailed information at hand to explain, for example, how they analyzed population segments for a particular model—or across multiple models—and defend their choice of segmentation. They can readily justify why a new predictive characteristic was added during a model refresh and how it was developed. They can provide regulators with evidence that a rigorous analysis of all potential economic drivers was performed for Long Run and Downturn PD estimation. Turning work friction into work flow We’ve discussed how increasing regulatory scrutiny has affected the work processes of bank analytic teams. Now here’s the real rub: This burden is settling on analytics teams at exactly the moment when the importance of what they do for helping banks understand customers and create competitive differentiation has become strikingly clear. Banking executives want their analytic teams to bring more powerful, complex analytics to market as quickly as possible. But they also need them to capture all of the details and reasons along the way so that models and their usage are transparent and explainable to regulators, customers and executives. FIGURE 1: BEING PREPARED TO ANSWER ANY AND ALL REGULATOR QUESTIONS Banks can answer such questions with confidence when they have standardized, approved processes in place across the model lifecycle... ...and one place to access all process history details Regulators are asking more detailed“peel back the onion”questions about models, such as: When was this model redeveloped? Why was this new characteristic added? How was this characteristic calculated? New customer characteristic: Ratio velocity retail purchases to velocity ATM withdrawals (Predictive of rising credit risk) CENTRALIZED MODELMANAGEMENT BehaviorScoreRedevelopmentProcessHistory VALIDATION SUMMARY Plans Results Metadata VariablesRoles Configuration Processes ModelUsage DEFINITION Models AlertRulesets ValidationJobs Targets WorkspaceDashboard AdministrationProcessAuthoring AnalyticsTeam A Task Name Assignee Variable Value Task Activity Independentreview D.Berman IR_Approval_Status Approved Developmodel J.Kalabar Technical_Review BehaviorScoreRedevelopmentTechnicalReview.doc Dataqualityreview R.Santos DQ_Status Approved Assessdataquality J.Kalabar DQ_Report BehaviorScoreRedevelopmentDataQualityReport.doc Roleassignment L.Moreau Project_Team_Status Assigned DQ_Comments Keyfieldslookgood.NoproblemthatPromoCodeisscarcelypopulated. Model_Specs BehaviorScoreRedevelopmentSpecification.doc IR_Comments Greatwork.Let’sproceedwithimplementationtesting. 1 2 3 Model development documentation provides information on how the characteristic was developed as well as its predictive power. UPDATED BEHAVIOR SCORE Development Validations Redevelopment BEHAVIOR SCORE LIFECYCLE
  4. 4. page 4 Reducing Regulatory Drag on Analytics Teams INSIGHTS WHITE PAPER Executives also want to get answers that allay their concerns about model risk. It’s not just potential financial and reputational damage from regulatory noncompliance that’s worrisome, but the opportunity costs of delayed model deployments and competitive impacts from making decisions with underperforming models. Automated workflow reduces the friction of these additional requirements for the entire analytics team. For model development: Workflow guides users through standard processes for documenting actions and decisions at each step. (See Figure 2.) It eliminates manual data input by automatically capturing segmentation criteria, test results and other key information. It also orchestrates review cycles across departments, notifies stakeholders and ensures approvals are obtained for moving on to the next step. Dashboards provide at-a-glance status on the real-time progress of all models. For regulatory and internal reporting and inquiries: Workflow feeds data about model lifecycle processes into a centralized repository. Users compiling reports, preparing for audits or responding to queries work faster and more efficiently with this single source of all relevant information on models and modeling processes. They can very quickly, for instance, examine a model’s most recent validation test. But they can also just as easily peruse an entire process history encompassing every validation ever performed on that model, and the results and consequent actions in each instance. FIGURE 2: AUTOMATED WORKFLOW ORCHESTRATES MODEL LIFECYCLE PROCESSES WHILE CAPTURING DATA AND DOCUMENTATION FOR CENTRALIZED MANAGEMENT Models Processes MyTasks WorkspaceDashboard AdministrationProcessAuthoring AnalyticsTeam A Redevelop Behavior Model Task: Data Quality Review Your Actions Using the form below, review the model data quality assessment and supporting documentation, and render an approval decision Decision Action: Comments: Supporting Information Model Name: Change Status: Change Summary: Model Owner: Project Docments: Comment History: Behavior Score Pending Review Updating the model in response to validation review recommendations and to incorporate new transactional predictive characteristics J. Kalabar Behavior Score Redevelopment Techinical Review.doc Please note in the documentation which records were used for the out-of-time validation of the new parameters Approval Required: Model review triggered by out-of-range results during periodic automated validation; recommenda- tion to redevelop Process initiated by model manager Email to analytics team leader: Tasks required: Project approval role assignment Email to model developer: Task required: Data quality assessment Email to technical reviewer: Task required: Data quality approval Data documentation captured in shared repository Development Validations Redevelopment BEHAVIOR SCORE LIFECYCLE PROCESS MANAGEMENT 1 2 3 4 5
  5. 5. page 5 Reducing Regulatory Drag on Analytics Teams INSIGHTS WHITE PAPER Overall, this approach provides banks (and regulators) with complete visibility into how each model is developed, deployed and maintained. It ensures that the evidence banks need to explain and defend their analytic choices is always fully captured and readily accessible. Creating standard processes that fit the needs of diverse teams Banks vary greatly in how they use analytics across their enterprise and the scope of their efforts to standardize lifecycle model management processes. Here are some examples: • One FICO client, a top-five US bank, is moving to centralize management of every model across its vast enterprise. • A leading Asian bank is initially focusing on ensuring its models in development achieve Advanced Internal Rating Based status under the Basel III global standard. • A top-five Australian bank seeks to bridge current process inconsistencies around model tracking and validation of Basel rating models and decision models across different countries. For any initiative, flexibility to align workflows with the needs of analytic teams is essential. Managing best practices depends partly on the types of analytics that teams are developing. For instance, predictive models for forecasting customer behavior have their own specific requirements and methodological pitfalls to be avoided. So do descriptive models for improving population segmentation and prescriptive models for recommending best next actions. In addition, expert (judgment-based) models need to be documented in very different ways than empirically derived models. Requirements and methods also vary, of course, across geographies and markets. To accommodate this diversity, banks need workflow tools that incorporate automated business rules management technology. By authoring and changing business rules, analytics groups can easily adjust workflows—within standardized parameters and constraints—to their local needs. At the same time, centralized repositories enable analytic teams to share characteristic libraries and learn from each other’s documentation and validation results. They also support collaboration, where appropriate, across teams. FICO® Model Central™ provides consistent end-to-end model governance and a central repository for all of an organization’s analytic models. The solution manages models created in any vendor’s tool, showing their real-time status and monitoring their performance. New advanced workflow capabilities in version 5.0 orchestrate and manage model lifecycle processes. Automated, configurable workflows notify process participants when they have a new task and provide instant access to the information needed to complete it. As tasks are performed, workflows capture key steps, artifacts, decisions and sign-offs. Over time, Model Central compiles a comprehensive audit trail for model lifecycle processes— project initiation through data quality testing, model development, implementation, deployment, validation, and updates or replacement. Together, these advanced capabilities enable organizations to achieve peak model performance, while providing the transparency and accountability required by banking regulators and executives. Without diverting analytics teams from more valuable work, they help reduce model risk and raise compliance. CENTRALIZING MODEL MANAGEMENT Model Data Mart Tracking Monitoring Ongoing Validation Management Reporting Alerts Decision Simulation Decision Execution Scoring Services Decision Optimization Development Calibration Deployment Verification Model Data Mart ADVA NCED PROFES SIONAL DECIS IONING DEVELO PM ENT FOUNDATION
  6. 6. Reducing Regulatory Drag on Analytics Teams INSIGHTS WHITE PAPER For more information North America Latin America Caribbean Europe, Middle East Africa Asia Pacific +1 888 342 6336 +55 11 5189 8222 +44 (0) 207 940 8718 +65 6422 7700 FICO, Model Central and“Make every decision count”are trademarks or registered trademarks of Fair Isaac Corporation in the United States and in other countries. Other product and company names herein may be trademarks of their respective owners. © 2014 Fair Isaac Corporation. All rights reserved. 4072WP 11/14 PDF The Insights white paper series provides briefings on research findings and product development directions from FICO. To subscribe, go to Tracking model and characteristic lineages The concept of model lifecycle management is broader than the life of any particular model. As analytics proliferate across organizations and the pace of change in financial services markets accelerates, banks need to start thinking in terms of lineages. Regulators may, for instance, ask pointed questions about why a retired model was replaced with the current one, and which customer characteristics were given greater predictive weight in the process. When a customer characteristic is changed by its author/owner, banks need to know which models incorporate that characteristic so they can manage all downstream effects. State-of-the-art model lifecycle management takes this broader view. Automated workflows help banks capture a complete lifecycle history of all models and their components. For each model, users can quickly track the lineage of any predictive customer characteristic—generated during development, harvested from a previous model, taken from a shared library, etc. For each characteristic, they can see everywhere it is currently used or was previously used—predictive models, segmentation strategies, decision strategies, etc. Another advantage of this approach is that banks have the opportunity to evaluate the value of individual customer characteristics over time. Increasingly far-reaching and detailed regulatory scrutiny is making it more important than ever for banks to put standard, approved model management processes in place. At the same time, banking executives want more visibility into and control over the full dimensions of model risk, including both compliance exposure and performance issues. Automated workflows that feed model lifecycle management solutions help banks lower model risk by improving process consistency and accountability. And they do it without clipping the wings of analytic teams—in fact, they offer efficiencies that can help them soar. To learn more about best practices for model management, visit the FICO Blog and read these Insights papers: • Customer Centricity: Four Bank Success Stories (No. 78) • Satisfying Customers and Regulators: Five Imperatives (No. 75) • Comply and Compete—Model Management Best Practices (No. 55) Conclusion