Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mark Zangari, CEO, Quantellia at MLconf SEA - 5/01/15

1,498 views

Published on

Agency Theory: The existence of massive data sets in many arenas is creating new challenges. Most people know about the issue of spurious correlations that do not represent true cause-and-effect. However a second challenge is more insidious and costly: this is the expense – which can run into the billions of dollars – of managing data that does not lead to actionable and valuable outcomes for an organization. For this reason, organizations that can identify the 20% of data that represents 80% of value realize a substantial advantage.

In this talk, I introduce Agency Theory, which is a mathematical framework for analyzing decision models to solve this problem. Agency theory borrows key ideas from machine learning, to solve a different purpose: rather than finding a set of parameters that best fits a data set, the objective is to find a set of decisions that leads to the most favorable set of outcomes, along with the data that is most valuable in supporting those decisions. Just as many foundational aspects of machine learning can be understood using information theory, I’ll describe how entropy and related concepts underlie Agency, and how to use this approach to prioritize data management and improve decision making.

Published in: Technology
  • Be the first to comment

Mark Zangari, CEO, Quantellia at MLconf SEA - 5/01/15

  1. 1. Agency Mark Zangari CEO Quantellia
  2. 2. Agency Agency’s aims: 1. Extend using machine learning into a new domain of problems. 2. The beginnings of a formal technique for linking data science to business levers. 3. Instead of answering “Given input data A, what output data B does my model predict?”, Agency answers “If I do A to complex dynamical system S, with the intention of achieving objective B, what is the probability that S’s outputs are closer to B?”
  3. 3. Agency The four pillars for effective machine learning: 1. Availability of training data 2. System stability over characterstic times (a) training data period (past) (b) prediction period (future) 3. Availability of input data at the time the prediction is to be made. 4. Sufficient signal-to-noise in data for pattern recognition to be possible.
  4. 4. Agency The four pillars for effective machine learning: What can we do if one or more of the four pillars are not present?
  5. 5. Agency The four pillars for effective machine learning: What can we do if one or more of the four pillars are not present? Example from finance: Commercial lending vs. consumer lending.
  6. 6. Agency Consumer Credit: 1. Ample, accessible data 2. Large, consistent, classifiable population 3. Relevant variables easily measured 4. Strong correlations between input variables and credit risk. Consumer Credit Transactions Data Warehouse Rating System Credit Score
  7. 7. Bill Fair & Earl Isaac Contacted 58 of the nation’s top lending institutions in 1958 offering to show them how using data would help them make better credit decisions... Only one responded. Image composed from stock images and portraits of Fair and http://www.fico.com/en/about-us#our_history
  8. 8. Agency Consumer Credit: 1. Ample, accessible data 2. Large, consistent, classifiable population 3. Relevant variables easily measured 4. Strong correlations between input variables and credit risk. Commercial Credit for Small/Medium Businesses: 1. Little data available, hard to obtain 2. Each business different, many are very new with little history. 3. Complex inter-entity relationships affect credit risk. 4. Many correlations , hard to isolate those that are good predictors of credit risk.
  9. 9. Agency Fully automated credit rating is rarely used for scoring Small/Medium Business Loans. The process commonly used is a good example of Agency in action…
  10. 10. Agency Type of Business Loan Amount Probability of Default Typical Model Decomposition for Commercial Loan Decision Grant this Loan? Profitable Loan Book Desired ObjectiveBusiness Lever External Inputs
  11. 11. Agency Agency Principle One: Learning and other analytics are designed to discriminate actions that will increase the probability of the objectives being met, from actions that do not. So… What is an “objective?”
  12. 12. Agency Objective: Map every element in the set of outcomes to a measure of the favorability of that outcome occurring, relative to the other outcomes. Probability of default DesiredLoanFrequency
  13. 13. Agency Objective: Map every element in the set of outcomes to a measure of the favorability of that outcome occurring, relative to the other outcomes. Probability of default DesiredLoanFrequency
  14. 14. Agency Type of Business Loan Amount Probability of Default Typical Model Decomposition for Commercial Loan Decision Grant this Loan? Profitable Loan Book Desired ObjectiveBusiness Levers External Inputs Measure Measure Learn Interest Rate
  15. 15. Agency Agency Principle Two: If a system does not ideally support machine learning because it does not satisfy the four pillars, decompose it using a dynamic system model until each link between each node either: (a) Satisfies the four pillars, or (b) Can be described using known domain-specific relationships (formulas, rules, approximations, etc.)
  16. 16. Agency Financial Spreading Values Market & Industry Data Management Team Bios & Experience Relationships to other Entities Key Ratios Aggregate Statistics Scoring Probability of Default Known mathematical relationships Machine Learning (intangibles) Rating Machine Learning Transform Tables Typical Model Decomposition for Business Credit Rating Feature Engineering
  17. 17. Agency Management Team Experience Agency gives us insight into decisions via the model levers. E.g. What interest rate do we charge and how does this affect our success? Interest Rate 1 Management Team Experience Interest Rate 2 Interest Rate 1 Interest Rate 2
  18. 18. Agency Summary of characteristics where Agency becomes highly relevant and useful: • Sparse data • Complex emergent dynamics, including feedback loops, phase changes, and other non-linear effects. • Provides a way of including intangibles with few measureables in the model • Can utilize relationships that the training data does not make apparent.
  19. 19. Agency A (brief) mathematical representation of Agency: We draw analogy from the information measure: The “Agency” A of a lever L, which maps some part of the output space to favorable outcomes, f and some other part to unfavorable outcomes, with measure M(f) and M(u) is given by:
  20. 20. Agency Agency: • Helps business users connect real-world decisions with support from data. • Provides a architecture for using machine learning in situations that are not typically well suited to ML solutions. • Allows ML results to be extrapolated beyond information contained in the data, by integrating knowledge of system dynamics.

×