Closing loop in analytic development and deployment


Published on

With FICO’s Analytic Closed- Loop Process, FICO estimates that a lending organization utilizing 10 scorecards measuring various types of behavior can see $500,000 or more in monetary benefits per year. Learn more at

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Closing loop in analytic development and deployment

  1. 1. Analytics has long been the core of the decision-making process in customer management. The technology, and its level of predictive precision, has advanced significantly over the past 30 years as organizations have accumulated and taken advantage of vast amounts of customer data, and moved from manually derived scorecards to developing sophisticated custom models. Yet, there’s always a need for improvement—new conditions or objectives call for new ways to boost the predictive power of analytics. That’s certainly the case in today’s customer management environment. In our volatile, still-uncertain economy, and with today’s new brand of consumer demanding personalized, highly responsive treatment, organizations need to raise the bar, yet again, in making even smarter, faster customer decisions. One way to do that is by expediting the development and deployment of analytic models. Most often, the purpose of developing a new model is to supplant an existing model with one that yields greater predictive power—generated at least in part by more recent and richer data. But if a new model takes months or up to a year to develop and deploy, two problems arise: One, for all its potential, while the model sits on the sidelines the organization compromises its ability to make better, more profitable decisions, and as a consequence money is being lost; and two, the “more recent data” that the new model was built upon is no longer really recent, putting the model’s power in question when it is finally deployed. Make every decision countTM WHITE PAPER Closing the Loop and Eliminating Costly Delays in Analytic Development and Deployment With FICO’s Analytic Closed- Loop Process, FICO estimates that a lending organization utilizing 10 scorecards measuring various types of behavior can see $500,000 or more in monetary benefits per year.
  2. 2. ©2014 Fair Isaac Corporation. All rights reserved. page 2 Particularly today, it’s essential that organizations know unequivocally that they are applying the best of their analytic proficiency to every customer decision, and that rests largely on finding new opportunities to accelerate their analytic development and deployment. Organizations that can streamline development and deployment without sacrificing the quality of their output stand a far better chance of reaping the benefits of improved risk management and increased customer loyalty than those that don’t (see box below). Today, with developments in FICO® TRIAD® Customer Manager, organizations will find that opportunity. Historically, the major impediment to rapid development and deployment of analytics has been a disconnect between the data sources used for development of the models and the decisioning application(s) into which those models are deployed and strategies are executed. At deployment the data is normally the system of record whereas the development of the models most often uses an extract of the data customized and derived only when requested. The analytic process is therefore disjointed and wastes valuable time and resource funds. Connecting the two data sources to include regular updates limits the work to the initial implementation and allows a fresh source of the same data to be used in both model development and deployment. Whether relying on an in-house team or an external vendor, it’s not unusual for organizations to take several weeks or even months to pull the necessary data from their systems in a format that can be used for model development. Even after the data is gathered, more work may be necessary to “cleanse” the data, involving the time-consuming task of identifying and correcting any portions of the data that may be incomplete, inaccurate or incorrect. Meanwhile, as time passes, a deployed model in need of replacement is generating losses or missing out on revenue opportunities. As soon as the data is available, the model development process itself can be accomplished fairly quickly. However, once the models are built, it’s not uncommon for deployment and testing of the models to require much more time and effort. This can take several weeks, or months, depending upon the number of models being deployed. In all, the time required for data gathering, developing the models and deploying them can be as many as six to twelve months. Of course, beyond the time and costs of sub-par decisions caused by delayed rollout, resource drain also costs organizations. Closing the Loop and Eliminating Costly Delays in Analytic Development and Deployment Think of degrading models as cracks in your organization’s infrastructure that leak profitability. That’s particularly unsettling when an organization is aware of the problem, but can’t rectify it for months or up to a year due to lagging model redevelopment and deployment. Such was the case for probably every institution at the outset of the recession. For example, consider how much loss credit card issuers could have prevented with faster model development and deployment. Knowing that the plummeting economy was likely to result in job losses and delinquent payments, using technology to expedite the roll out of new models and strategies, issuers could have quickly tested tighter pre- collection treatments, or authorizations of over-limit accounts, based on the market’s rapidly changing customer data. They could have then implemented new, effective models and decision strategies in weeks rather than months, saving millions of dollars in profitability, and saving many account holders from delinquency and bankruptcy. How Much Does Faster Model Development and Deployment Matter? An Example: Improving Time to Value
  3. 3. ©2014 Fair Isaac Corporation. All rights reserved. page 3 Consider how much business users, IT staff and analysts spend on a lethargic rollout of a model—time that could be spent in more profitable pursuits. The data-gathering and extraction process often requires analysis of the data source(s) by business users to make sure the correct data is extracted and interpreted properly. Additionally, IT staff is typically required to develop the interface to extract the data. Once the models have been developed, IT (or outside vendors) help is required to implement the models into the decisioning application and, because of the possibility of implementation errors (typically, some amount of recoding of the model is required within the application), the analytics team and/or the business users need to thoroughly test the newly implemented analytics. All of this effort can add up to hundreds of hours or more—time that can be better spent by IT on other projects, by model developers in building and refining models, and business users in running their business. One way to reduce the overall time and costs to implement new analytics is by streamlining the process of getting data from the organization’s customer portfolio data sources into analytic tools and, from there, into their decisioning systems for testing and ultimate deployment into production. This is commonly referred to as an Analytic Closed-Loop Process. An example of this might look like what is depicted in Figure 1. Closing the Loop and Eliminating Costly Delays in Analytic Development and Deployment Streamlining the Analytic Development/Deployment Process Decisioning Application (System Configuration and Testing) Decisioning Application (Production System) Model Development and Optimization Tools Reports Analytic Data Mart Client Master File Strategy Deployment Scorecard Deployment Strategy Trees Scorecards (Models) Analytic Data Production Data FIGURE 1: STREAMLINED ANALYTIC DEVELOPMENT AND DEPLOYMENT PROCESS
  4. 4. ©2014 Fair Isaac Corporation. All rights reserved. page 4 The Analytic Closed-Loop Process depicted above suggests a “feedback” loop, or cyclical process involving the client’s customer master file(s) (for example, their bill-post system) and their decisioning application where the data from the master file is regularly fed into an analytic data mart. Once in the data mart, the data can be stored and extracted for use in analytic tools for scorecard and decision (strategy) tree development. Now, with the data extracted directly from the data mart, the analytic tools can be used to develop scorecards (using the client’s own data). Similarly, analytic tools can be used to develop and optimize decision trees and business rule flows. Once the analytics are developed, the next step in the process is to deploy them into the decisioning application where they can be included in simulations to determine if the analytics will perform as expected in a production environment. Finally, assuming that the results of the simulations indicate that the newly developed analytics will perform as desired, the new analytics can be introduced into the production environment where they can be monitored for actual results, and a new cycle of analytic development and deployment can begin. In order for this process to be efficient, and for organizations to gain the benefits of better analytics and faster deployment at low cost, it is critical that the data can move seamlessly between the client’s data sources, the analytic data mart and the analytic tools. Furthermore, data model alignment is required so that the analytics developed using the data can be easy to deploy back into the decisioning application. FICO has recently solved the challenge of rapid development and deployment as part of its most recent release of FICO® TRIAD® Customer Manager. In FICO’s version of the Analytic Closed-Loop Process, rather than pulling data in batches, the raw data from the master file is regularly fed into the TRIAD Analytic Datamart via the TRIAD standard interface. The data is stored in the data mart each time TRIAD is called upon to produce decisions. Once in the Analytic Datamart, the data can be stored over an indefinite period of time and can be extracted, on demand, in formats appropriate for use in FICO® Model Builder for scorecard development and in Model Builder’s optional Decision Trees Module for decision tree development. Using Model Builder, in conjunction with FICO’s newly developed FICO® Master File Characteristic Library, clients can quickly develop highly predictive behavioral scorecards using the extract from the data mart. Similarly, clients can rely on the Decision Trees Module to develop more profitable strategies. Clients can also develop decision trees based on specific needs and objectives—such as maximizing profits or minimizing cost given existing business constraints—by combining their use of the Decision Trees Module with the advanced modeling and simulation features of FICO® Decision Optimizer. In both cases, scorecards and decision trees can then be imported seamlessly back into the FICO TRIAD Customer Manager testing facility. TRIAD has long had the ability to import/export decision trees with Model Builder. But in TRIAD, a new feature makes it possible to import Model Builder-developed models directly into TRIAD. This feature imports all components of what is referred to as a score type. Closing the Loop and Eliminating Costly Delays in Analytic Development and Deployment FICO’s Advances in the Analytics Closed-Loop Process
  5. 5. ©2014 Fair Isaac Corporation. All rights reserved. page 5 A score type consists of one or more scorecards—such as behavior risk scorecards, attrition scorecards or collections scorecards—that have similar attributes. When importing a score type, the scorecards associated with that score type are imported, along with each scorecard’s description, good/bad scheme, log odds slope, log odds intercept, performance window, exclusion data, scorecard data (i.e., characteristics with their associated bins with multiple ranges, etc.) and reason codes. No configuration or reprogramming effort is required by the client; the import is accomplished through a couple of clicks on the FICO® TRIAD® Customer Manager score type import page. Since both FICO® Model Builder and TRIAD use the same FICO® Blaze Advisor® business rules management system repositories, what has been built in Model Builder is exactly what gets deployed into TRIAD; there is no need for additional unit testing of the models after they have been deployed. Now that the scorecards and decision trees have been deployed into TRIAD Customer Manager’s testing facility (within the TRIAD user interface) clients are free to run simulations against actual data from their production system to see how they might perform. Results may be reviewed in TRIAD Customer Manager’s estimator reports or within the TRIAD Graphical Estimator where actual historical data counts can be viewed at each segmentation break within the decision tree. If testing isn’t satisfactory, then refinements to the models can be made either in TRIAD directly (for the decision trees) or back in Model Builder. Once desirable results are achieved, then the new models can be deployed into the production system where adaptive control (also known as champion/challenger testing) can be applied. Adaptive control results will be regularly measured using TRIAD outcomes reports, portfolio performance reports and scorecard performance reports. Figure 2 shows a more detailed look at the FICO’s Analytic Closed-Loop Process in FICO TRIAD Customer Manager, focusing on scorecards. Closing the Loop and Eliminating Costly Delays in Analytic Development and Deployment Reporting FICO® TRIAD® User Interface Scorecard Performance Reports Web Export Web Estimator Report Records TRIAD® Analytic Datamart Config Data Client’s Production Environment Linkage Master File • Scoring • Decisions TRIAD® Execution Facility Control Tables Database Process Input/Output Analytic Data Extract File Score- Type File (Scorecard) FICO® Model Builder Linkage Data File Report Records FIGURE 2: FICO’S ANALYTICS CLOSED-LOOP PROCESS (SCORECARDS)
  6. 6. ©2014 Fair Isaac Corporation. All rights reserved. page 6 Clients can extend FICO’s Analytic Closed-Loop Process, not depicted in Figure 2, with a connection between the FICO® TRIAD Analytic Datamart and FICO® Model Central™ Solution. By interfacing the Analytic Datamart with Model Central, organizations can improve their vigilance and management of models and strategies with Model Central Solution’s advanced monitoring and governance capabilities. Model Central helps organizations uncover model degradation and support timely upgrades, streamline validation reporting to minimize compliance risk and quickly identify shifts that can impact portfolio profitability. Where data gathering could take weeks or even months, organizations that have installed FICO® TRIAD® Customer Manager 9.0 already have the required interface for export to FICO® Model Builder. Furthermore, TRIAD users can define data sets specifically to be used for extracting data for modeling purposes. These data sets may be saved for reuse. Scheduled exports using these data sets can be set to occur regularly and automatically. Organizations that have purchased Model Builder can build their own models, or they can deliver the data mart extract to FICO for custom model development. Where model implementation could take weeks or months, the deployment of models into TRIAD now takes literally just seconds and can be done directly by the business user. On average, FICO has seen clients reduce their development and deployment time of a new model by as much as two to six months. Consider the potential financial impact of being able to more rapidly develop and deploy scorecards. FICO estimates that a lending organization utilizing 10 scorecards measuring various types of behavior (including credit risk, attrition risk, collections, offer responsiveness, propensity to pay and cross-sell take up) can see $500,000 or more in monetary benefits per year. This would be the combined result of cost savings (reduced IT costs/operational risk, reduced development costs, maintained business relevance and predictive power and reduced opportunity costs) and additional revenue (improved competitive advantage through improved value of better predictive models). Beyond the monetary benefits, FICO’s Analytic Closed-Loop Process gives business users complete control over when to develop and implement new analytics, freeing them from any IT reliance or schedules. Since IT is not required to collect data, recode and test the model in the deployment application, model development and implementation can be performed independent of IT project cycles. No longer must business users schedule modeling projects months in advance; now they can work on them at any time without IT support. Of course, the extent of the financial and operational benefits of the FICO Analytic Closed- Loop Process depend upon the level of sophistication of an organization’s analytics team (the less sophisticated, the more lift can be gained from analytics tools), the number of models in place (the more models, the more overall gain in time and cost reduction), and the rate of model replacement (the more frequently models are replaced or introduced, the greater the gain from the closed-loop process). Custom model development and refinement subscription services are available for clients who don’t have the resources to maintain their own models. Again, because the data needed for development and refinement is now easily available via the Analytic Data Mart, clients can inexpensively gather analytic data and then deliver it to FICO, where FICO analytic scientists, using Model Builder, can develop models on the client’s behalf. Those custom models can be delivered to the client in a format that can then be quickly and inexpensively deployed in TRIAD or in Blaze Advisor. Closing the Loop and Eliminating Costly Delays in Analytic Development and Deployment Benefits of the FICO’s Analytic Closed-Loop Process
  7. 7. ©2014 Fair Isaac Corporation. All rights reserved. page 7 Similarly, pooled models, which are especially helpful to smaller organizations that don’t have large enough portfolios to benefit from custom model development, can be quickly developed and deployed. In particular, this type of process might be very well-suited through a cloud-based deployment of FICO® TRIAD Customer Manager, where a number of clients with similar portfolios can contribute to the data pool. Model implementation in a cloud environment would be transparent to the clients. Finally, having the ability to regularly collect data from multiple clients in a consistent format enables FICO to provide benchmark services to its subscribers, where they can see how they compare to other, similar organizations in terms of the effectiveness of their decisioning systems. There’s good reason for all the excitement about “big data” and how it can be used to extract new and valuable insights into customer behavior, especially in regard to creating more predictive analytics. Think about the potential for sharpening analytics and decisions with data sources such as text files, Twitter, Facebook, LinkedIn and many other sources of unstructured data. For example, call center logs may help detect a customer’s increased risk, or bankruptcy or attrition potential; or Facebook data could advance modeling for marketing. However, the Big Data challenge with decisioning applications, even those with extensible data models, is how to take advantage of these vast amounts of unstructured and undefined (or ambiguous) data. FICO has addressed this challenge with advances in its model development tools—FICO® Model Builder and FICO® Blaze Advisor® business rules management system— since both have the requisite capabilities to extract and analyze Big Data sources. One Big Data possibility is to develop new “Big Data Models” which can be used to augment the other types of risk behavior models. Such models, built predominately or exclusively by examining Big Data sources could then be executed outside of the decisioning application. Once calculated, the “Big Data Scores” can be added as external scores to the decisioning system’s input where they can be used as additional decision variables (or segmentation variables) within a decision tree or within business rules. In TRIAD, the scores derived externally and passed into the application system can be included in the analytic data mart and used as part of the testing process. Similar to the streamlined process of model deployment described in the FICO Analytic Closed-Loop Process, models developed using Model Builder can be seamlessly deployed to Blaze Advisor, thus making it easy to integrate the scores derived from the Big Data Models into the decisioning application. Closing the Loop and Eliminating Costly Delays in Analytic Development and Deployment New Frontiers in Analytic Data
  8. 8. Closing the Loop and Eliminating Costly Delays in Analytic Development and Deployment For more North America Latin America Europe, Middle East Asia Pacific Web information & Caribbean & Africa +1 888 342 6336 +55 11 5189 8222 +44 (0) 207 940 8718 +65 6422 7700 FICO, TRIAD, Blaze Advisor, Model Central and“Make every decision count”are trademarks or registered trademarks of Fair Isaac Corporation in the United States and in other countries. Other product and company names herein may be trademarks of their respective owners. © 2014 Fair Isaac Corporation. All rights reserved. 3077WP 03/14 PDF FICO (NYSE: FICO) is a leading analytics software company, helping businesses in 80+ countries make better decisions that drive higher levels of growth, profitability and customer satisfaction. The company’s groundbreaking use of Big Data and mathematical algorithms to predict consumer behavior has transformed entire industries. FICO provides analytics software and tools used across multiple industries to manage risk, fight fraud, build more profitable customer relationships, optimize operations and meet strict government regulations. Many of our products reach industry-wide adoption—such as the FICO® Score, the standard measure of consumer credit risk in the United States. FICO solutions leverage open-source standards and cloud computing to maximize flexibility, speed deployment and reduce costs. The company also helps millions of people manage their personal credit health. Learn more at Putting It All Together The combination of innovations involving new capabilities to more rapidly develop and deploy conventional analytics and the use of big data in model development will give those organizations that adopt them a significant advantage over their competitors. Early adopters will be able to deploy more predictive, fresher analytics, at a lower cost, and gain time to value and return on investment advantages.