Presentation at SAS Analytics conference 2014
Predictive analytics has been applied to solve a wide range of real-world problems. Nevertheless, current state-of-the-art predictive analytics models are not well aligned with business needs since they don't include the real financial costs and benefits during the training and evaluation phases. Churn modeling does not yield the best results when it's measured by investment per subscriber on a loyalty campaign and the financial impact of failing to detect a churner versus wrongly predicting a non-churner. This presentation will show how using a cost-sensitive modeling approach leads to better results in terms of profitability and predictive power – and is applicable to many other business challenges.
This document summarizes a presentation on maximizing profit from customer churn prediction models using cost-sensitive machine learning techniques. It discusses how traditional evaluation measures like accuracy do not account for different costs of prediction errors. It then covers cost-sensitive approaches like cost-proportionate sampling, Bayes minimum risk, and cost-sensitive decision trees. The results show these cost-sensitive methods improve savings over traditional models and sampling approaches when the business costs are incorporated into the predictive modeling.
Slides of the paper http://arxiv.org/abs/1505.04637
source code is available at https://github.com/albahnsen/CostSensitiveClassification/blob/master/costcla/models/cost_tree.py#L15
Abstract:
Several real-world classification problems are example-dependent cost-sensitive in nature, where the costs due to misclassification vary between examples and not only within classes. However, standard classification methods do not take these costs into account, and assume a constant cost of misclassification errors. In previous works, some methods that take into account the financial costs into the training of different algorithms have been proposed, with the example-dependent cost-sensitive decision tree algorithm being the one that gives the highest savings. In this paper we propose a new framework of ensembles of example-dependent cost-sensitive decision-trees. The framework consists in creating different example-dependent cost-sensitive decision trees on random subsamples of the training set, and then combining them using three different combination approaches. Moreover, we propose two new cost-sensitive combination approaches; cost-sensitive weighted voting and cost-sensitive stacking, the latter being based on the cost-sensitive logistic regression method. Finally, using five different databases, from four real-world applications: credit card fraud detection, churn modeling, credit scoring and direct marketing, we evaluate the proposed method against state-of-the-art example-dependent cost-sensitive techniques, namely, cost-proportionate sampling, Bayes minimum risk and cost-sensitive decision trees. The results show that the proposed algorithms have better results for all databases, in the sense of higher savings.
Slides from my PhD defense
Example-Dependent Cost-Sensitive Classification
Applications in Financial Risk Modeling and Marketing Analytics
https://github.com/albahnsen/phd-thesis
This document discusses causal inference techniques for machine learning, including:
- Correlation does not imply causation, and observational data can be biased by confounding variables. Randomization and counterfactual modeling are introduced as alternatives.
- Inverse propensity scoring is presented as a method for estimating treatment effects from observational data by reweighting samples based on their propensity to receive treatment.
- Instrumental variable regression is discussed as another technique, using variables that influence the treatment but not the outcome except through treatment. Scalable methods for instrumental variable regression on large datasets are proposed.
- Challenges with weak instruments are noted, as instrumental variable estimates can become more biased than purely correlational models when instruments are weak
Beyond Churn Prediction : An Introduction to uplift modelingPierre Gutierrez
These slides are from a talk I at the papis conference in Boston in 2016. The main subject is uplift modelling. Starting from a churn model approach for an e-gaming company, we introduce when to apply uplift methods, how to mathematically model them, and finally, how to evaluate them.
I tried to bridge the gap between causal inference theory and uplift theory, especially concerning how to properly cross validate the results. The notation used is the one from uplift modelling.
These slides are from a talk I gave at Google Campus Madrid for the Machine Learning Meetup. The main subject is uplift modelling. Starting from a churn model approach for an e-gaming company, we introduce when to apply uplift methods, how to mathematically model them, and finally, how to evaluate them.
Naive Bayes is a classification algorithm that is suitable for binary and multiclass classification. It is suitable for binary and multiclass classification. Naïve Bayes performs well in cases of categorical input variables compared to numerical variables. It is useful for making predictions and forecasting data based on historical results.
Random Forest Classification is a machine learning technique utilizing aggregated outcome of many decision tree classifiers in order to improve precision of the outcome. It measures the relationship between the categorical target variable and one or more independent variables.
This document summarizes a presentation on maximizing profit from customer churn prediction models using cost-sensitive machine learning techniques. It discusses how traditional evaluation measures like accuracy do not account for different costs of prediction errors. It then covers cost-sensitive approaches like cost-proportionate sampling, Bayes minimum risk, and cost-sensitive decision trees. The results show these cost-sensitive methods improve savings over traditional models and sampling approaches when the business costs are incorporated into the predictive modeling.
Slides of the paper http://arxiv.org/abs/1505.04637
source code is available at https://github.com/albahnsen/CostSensitiveClassification/blob/master/costcla/models/cost_tree.py#L15
Abstract:
Several real-world classification problems are example-dependent cost-sensitive in nature, where the costs due to misclassification vary between examples and not only within classes. However, standard classification methods do not take these costs into account, and assume a constant cost of misclassification errors. In previous works, some methods that take into account the financial costs into the training of different algorithms have been proposed, with the example-dependent cost-sensitive decision tree algorithm being the one that gives the highest savings. In this paper we propose a new framework of ensembles of example-dependent cost-sensitive decision-trees. The framework consists in creating different example-dependent cost-sensitive decision trees on random subsamples of the training set, and then combining them using three different combination approaches. Moreover, we propose two new cost-sensitive combination approaches; cost-sensitive weighted voting and cost-sensitive stacking, the latter being based on the cost-sensitive logistic regression method. Finally, using five different databases, from four real-world applications: credit card fraud detection, churn modeling, credit scoring and direct marketing, we evaluate the proposed method against state-of-the-art example-dependent cost-sensitive techniques, namely, cost-proportionate sampling, Bayes minimum risk and cost-sensitive decision trees. The results show that the proposed algorithms have better results for all databases, in the sense of higher savings.
Slides from my PhD defense
Example-Dependent Cost-Sensitive Classification
Applications in Financial Risk Modeling and Marketing Analytics
https://github.com/albahnsen/phd-thesis
This document discusses causal inference techniques for machine learning, including:
- Correlation does not imply causation, and observational data can be biased by confounding variables. Randomization and counterfactual modeling are introduced as alternatives.
- Inverse propensity scoring is presented as a method for estimating treatment effects from observational data by reweighting samples based on their propensity to receive treatment.
- Instrumental variable regression is discussed as another technique, using variables that influence the treatment but not the outcome except through treatment. Scalable methods for instrumental variable regression on large datasets are proposed.
- Challenges with weak instruments are noted, as instrumental variable estimates can become more biased than purely correlational models when instruments are weak
Beyond Churn Prediction : An Introduction to uplift modelingPierre Gutierrez
These slides are from a talk I at the papis conference in Boston in 2016. The main subject is uplift modelling. Starting from a churn model approach for an e-gaming company, we introduce when to apply uplift methods, how to mathematically model them, and finally, how to evaluate them.
I tried to bridge the gap between causal inference theory and uplift theory, especially concerning how to properly cross validate the results. The notation used is the one from uplift modelling.
These slides are from a talk I gave at Google Campus Madrid for the Machine Learning Meetup. The main subject is uplift modelling. Starting from a churn model approach for an e-gaming company, we introduce when to apply uplift methods, how to mathematically model them, and finally, how to evaluate them.
Naive Bayes is a classification algorithm that is suitable for binary and multiclass classification. It is suitable for binary and multiclass classification. Naïve Bayes performs well in cases of categorical input variables compared to numerical variables. It is useful for making predictions and forecasting data based on historical results.
Random Forest Classification is a machine learning technique utilizing aggregated outcome of many decision tree classifiers in order to improve precision of the outcome. It measures the relationship between the categorical target variable and one or more independent variables.
This paper proposes using a "shrinkage" estimator as an alternative to the traditional sample covariance matrix for portfolio optimization. The shrinkage estimator combines the sample covariance matrix with a structured "shrinkage target" using a shrinkage constant to minimize distance from the true covariance matrix. The paper finds this shrinkage estimator significantly increases the realized information ratio of active portfolio managers compared to the sample covariance matrix. An empirical study on historical stock return data confirms the shrinkage method leads to higher ex post information ratios in portfolio optimization. However, the shrinkage target assumes identical pairwise correlations that may not fully reflect market characteristics.
Prediction of customer propensity to churn - Telecom IndustryPranov Mishra
- A logistic regression model was found to best predict customer churn with the highest AUC and accuracy.
- The top variables increasing churn risk were credit class, handset price, average monthly calls, billing adjustments, household subscribers, call waiting ranges, and dropped/blocked calls.
- Cost and billing variables like charges and usage were significant, validating an independent survey.
- A lift chart showed targeting the highest risk 30% of customers could identify 33% of potential churners. The model allows prioritizing retention efforts on the 20% riskiest customers.
The document provides an introduction to decision trees. It defines decision trees as visual representations of choices, consequences, probabilities, and opportunities that break down complicated situations into easier to understand scenarios. The document outlines the steps to create a decision tree, including drawing the diagram, using quantitative data like payoffs and probabilities, and calculating expected values. An example decision tree is provided about a factory owner deciding whether to expand. The key aspects of decision trees are summarized, including how they can help structure sequential decision problems and encourage clear thinking.
BigData Republic teamed up with VodafoneZiggo and hosted an meetup on churn prediction.
Telecom companies like VodafoneZiggo have long benefited from the fine art/science of predicting churn. Currently, in the booming age of subscription based business models (e.g. Netflix, Spotify, HelloFresh), the importance of predicting churn has become widespread. During this event, VodafoneZiggo shared some of its wisdom with the public, after which BDR Data Scientist Tom de Ruijter presented an overview of the modeling tools at hand, both classical, as well as novel approaches. Finally, the participants engaged in a hands-on session showcasing the implementation of different approaches.
PART 1 — Churn Prediction in Practice by Florian Maas
At VodafoneZiggo we are incredibly excited about Advanced Analytics and the enormous potential for progress and innovation. In our state of the art open source platform we store the tremendous amount of data that is generated every single second in our mobile and fixed networks. This means that we have a vast body of rich information, which if unlocked, can lead to something very special. As a company with a primarily subscription-based service model, churn plays a vital role in the daily business. Not only is the churn rate a good indicator of customer (dis)satisfaction, it is also one out of two factors that determines the steady-state level of active customers. During this talk, we will show how data science provides added value in the process of churn prevention at VodafoneZiggo. We will talk about the data and the modeling approach we use, and the pitfalls and shortcomings that we have encountered while building the model. We will also briefly discuss potential improvements to the current approach, which brings us to talk #2.
PART 2 — The Churn Prediction Toolbox by Tom de Ruijter
The second talk will show you the fine intricacies of predicting churn through different approaches. We’ll start off with an overview of different modeling strategies for describing the problem of churn, both in terms of a classification problem as well as a regression problem. Secondly, Tom will give you insights in how you evaluate a churn model in a way such that business stakeholders know how to act upon the model results. Finally, we’ll work towards the hands-on session demonstrating different model approaches for churn prediction, ranging from classical time series prediction to recurrent neural networks.
Python and the Holy Grail of Causal Inference - Dennis Ramondt, Huib KeeminkPyData
The document discusses various challenges and methods for causal inference from observational data. It begins with two use cases - estimating the savings from installing heat pumps and the profit increase from placing beer coolers in stores. Both experiments fail standard assumptions as the test and control groups are statistically different. The document then covers methods for estimating average treatment effects such as propensity score matching and regression adjustment. It also discusses estimating individual treatment effects using techniques like honest forests and counterfactual regression that learn balanced representations of the data. The goal is to remove bias from differences between treated and untreated groups to infer valid causal effects.
Hierarchical Clustering is a process by which objects are classified into a number of groups so that they are as much dissimilar as possible from one group to another group and as similar as possible within each group. This technique can help an enterprise organize data into groups to identify similarities and, equally important, dissimilar groups and characteristics, so the business can target pricing, products, services, marketing messages and more.
This document presents an example decision problem to demonstrate decision tree analysis. It describes three potential decisions - expand, maintain status quo, or sell now - under two possible future states, good or poor foreign competitive conditions. It then outlines the steps to analyze the problem: 1) determine the best decision without probabilities using various criteria, 2) determine the best decision with probabilities using expected value and opportunity loss, 3) compute the expected value of perfect information, and 4) develop a decision tree showing expected values at each node.
As part of our team's enrollment for Data Science Super Specialization course under UpX Academy, we submitted many projects for our final assessments, one of them was Telecom Churn Analysis Model.
The input data was provided by UpX academy and language we used is R. As part of the project, our main objective was :-
-> To predict Customer Churn.
-> To Highlight the main variables/factors influencing Customer Churn.
-> To Use various ML algorithms to build prediction models, evaluate the accuracy and performance of these models.
-> Finding out the best model for our business case & providing executive Summary.
To address the mentioned business problem, we tried to follow a thorough approach. We did a detailed level Exploratory Data Analysis which consists of various Box Plots, Bar Plots etc..
Further we tried our best to build as many Classification models possible which fits our business case (Logistic Regression/kNN/Decision Trees/Random Forest/SVM) and also tried to touch Cox Hazard Survival analysis Model. Later for every model we tried to boost their performances by applying various performance tuning techniques.
As we all are still into our learning mode w.r.t these concepts & starting new, please feel free to provide feedback on our work. Any suggestions are most welcome... :)
Thanks!!
The document describes an advanced analytics project using SAS to optimize a collection agency's dialing strategy. The objectives were to 1) identify characteristics of consumers likely to pay debts, 2) determine the best times to call consumers, and 3) allocate resources to maximize ROI. Models were built using variables like age, marital status, number of debts, TU score, and occupation to predict likelihood of payment and phone answering. The results identified important variables and improved segmentation to save 20% of calls while achieving targets.
The importance of this type of research in the telecom market is to help companies make more profit.
It has become known that predicting churn is one of the most important sources of income to Telecom companies.
Hence, this research aimed to build a system that predicts the churn of customers i telecom company.
These prediction models need to achieve high AUC values. To test and train the model, the sample data is divided into 70% for training and 30% for testing.
Decision Tree Analysis for statistical tool. The deck provides understanding on the Decision Analysis.
It provides practical application and limited theory. Will be useful for MBA students.
Predictive modeling is a process used in predictive analytics to create statistical models that can forecast future outcomes based on historical data. Predictive analytics uses techniques from data mining, statistics, modeling, machine learning and AI to analyze current data to predict future events. The predictive modeling process involves collecting data, creating a model, testing and validating the model, and evaluating model performance. Predictive models are commonly used to predict customer behavior, risk levels, and other business outcomes.
Predictive analytics uses past data to forecast future outcomes. The document discusses various predictive analytics techniques including simple forecasting methods, decision trees, and regression. Simple forecasting techniques like moving averages are easiest to implement but lack explanatory power, while decision trees and regression provide more accurate predictions at an individual level but require more complex deployment. The key is selecting the right technique based on the problem, data, and ability to implement predictive models in real-world applications.
Predictive Model for Loan Approval Process using SAS 9.3_M1Akanksha Jain
This is a Predictive Model which uses Logistic Regression to statistically help make better loan approval decisions in future for a German Bank. It uses an historical credit data set with 1000 data points and 20 variables.
Tool used:
SAS 9.3_M1
Steps Involved are:
- Data Quality check using Correlations and VIF Tests
- Analysis of different Variable Selection Methods such as Forward, Backward and Stepwise
- Variable Selection on the basis of Parameter Estimates and Odds Ratio
- Outlier Analysis to identify the outliers and improve the model
- Final Model Selection Decision based on ROC curve, Percent Concordant, PROC Rank and Hosmer Lemeshow Test
Automation of IT Ticket Automation using NLP and Deep LearningPranov Mishra
Overview of Problem Solved: IT leverages Incident Management process to ensure Business Operations is never impacted. The assignment of incidents to appropriate IT groups is still a manual process in many of the IT organizations. Manual assignment of incidents is time consuming and requires human efforts. There may be mistakes due to human errors and resource consumption is carried out ineffectively because of the misaddressing. Manual assignment increases the response and resolution times which result in user satisfaction deterioration / poor customer service.
Solution: Multiple deep learning sequential models with Glove Embeddings were attempted and results compared to arrive at the best model. The two best models are highlighted below through their results.
1. Bi-Directional LSTM attempted on the data set has given an accuracy of 71% and precision of 71%.
2. The accuracy and precision was further improved to 73% and 76% respectively when an ensemble of 7 Bi-LSTM was built.
I built a NLP based Deep Learning model to solve the above problem. Link below
https://github.com/Pranov1984/Application-of-NLP-in-Automated-Classification-of-ticket-routing?fbclid=IwAR3wgofJNMT1bIFxL3P3IoRC3BTuWmhw1SzAyRtHp8vvj9F2sKZdq67SjDA
"Multilayer perceptron (MLP) is a technique of feed
forward artificial neural network using back
propagation learning method to classify the target
variable used for supervised learning. It consists of multiple layers and non-linear activation allowing it to distinguish data that is not linearly separable."
Presentation on Modern Data Science
Data scientists are in high demand. There is simply not enough talent to fill the jobs. Why? Because the sexiest job of 21th century requires a mixture of broad, multidisciplinary skills ranging from an intersection of mathematics, statistics, computer science, communication and business. Finding a data scientist is hard. Finding people who understand who a data scientist is, is equally hard.
Check the video in spanish here :https://www.youtube.com/watch?v=R3jeBHLLiiM
This paper proposes using a "shrinkage" estimator as an alternative to the traditional sample covariance matrix for portfolio optimization. The shrinkage estimator combines the sample covariance matrix with a structured "shrinkage target" using a shrinkage constant to minimize distance from the true covariance matrix. The paper finds this shrinkage estimator significantly increases the realized information ratio of active portfolio managers compared to the sample covariance matrix. An empirical study on historical stock return data confirms the shrinkage method leads to higher ex post information ratios in portfolio optimization. However, the shrinkage target assumes identical pairwise correlations that may not fully reflect market characteristics.
Prediction of customer propensity to churn - Telecom IndustryPranov Mishra
- A logistic regression model was found to best predict customer churn with the highest AUC and accuracy.
- The top variables increasing churn risk were credit class, handset price, average monthly calls, billing adjustments, household subscribers, call waiting ranges, and dropped/blocked calls.
- Cost and billing variables like charges and usage were significant, validating an independent survey.
- A lift chart showed targeting the highest risk 30% of customers could identify 33% of potential churners. The model allows prioritizing retention efforts on the 20% riskiest customers.
The document provides an introduction to decision trees. It defines decision trees as visual representations of choices, consequences, probabilities, and opportunities that break down complicated situations into easier to understand scenarios. The document outlines the steps to create a decision tree, including drawing the diagram, using quantitative data like payoffs and probabilities, and calculating expected values. An example decision tree is provided about a factory owner deciding whether to expand. The key aspects of decision trees are summarized, including how they can help structure sequential decision problems and encourage clear thinking.
BigData Republic teamed up with VodafoneZiggo and hosted an meetup on churn prediction.
Telecom companies like VodafoneZiggo have long benefited from the fine art/science of predicting churn. Currently, in the booming age of subscription based business models (e.g. Netflix, Spotify, HelloFresh), the importance of predicting churn has become widespread. During this event, VodafoneZiggo shared some of its wisdom with the public, after which BDR Data Scientist Tom de Ruijter presented an overview of the modeling tools at hand, both classical, as well as novel approaches. Finally, the participants engaged in a hands-on session showcasing the implementation of different approaches.
PART 1 — Churn Prediction in Practice by Florian Maas
At VodafoneZiggo we are incredibly excited about Advanced Analytics and the enormous potential for progress and innovation. In our state of the art open source platform we store the tremendous amount of data that is generated every single second in our mobile and fixed networks. This means that we have a vast body of rich information, which if unlocked, can lead to something very special. As a company with a primarily subscription-based service model, churn plays a vital role in the daily business. Not only is the churn rate a good indicator of customer (dis)satisfaction, it is also one out of two factors that determines the steady-state level of active customers. During this talk, we will show how data science provides added value in the process of churn prevention at VodafoneZiggo. We will talk about the data and the modeling approach we use, and the pitfalls and shortcomings that we have encountered while building the model. We will also briefly discuss potential improvements to the current approach, which brings us to talk #2.
PART 2 — The Churn Prediction Toolbox by Tom de Ruijter
The second talk will show you the fine intricacies of predicting churn through different approaches. We’ll start off with an overview of different modeling strategies for describing the problem of churn, both in terms of a classification problem as well as a regression problem. Secondly, Tom will give you insights in how you evaluate a churn model in a way such that business stakeholders know how to act upon the model results. Finally, we’ll work towards the hands-on session demonstrating different model approaches for churn prediction, ranging from classical time series prediction to recurrent neural networks.
Python and the Holy Grail of Causal Inference - Dennis Ramondt, Huib KeeminkPyData
The document discusses various challenges and methods for causal inference from observational data. It begins with two use cases - estimating the savings from installing heat pumps and the profit increase from placing beer coolers in stores. Both experiments fail standard assumptions as the test and control groups are statistically different. The document then covers methods for estimating average treatment effects such as propensity score matching and regression adjustment. It also discusses estimating individual treatment effects using techniques like honest forests and counterfactual regression that learn balanced representations of the data. The goal is to remove bias from differences between treated and untreated groups to infer valid causal effects.
Hierarchical Clustering is a process by which objects are classified into a number of groups so that they are as much dissimilar as possible from one group to another group and as similar as possible within each group. This technique can help an enterprise organize data into groups to identify similarities and, equally important, dissimilar groups and characteristics, so the business can target pricing, products, services, marketing messages and more.
This document presents an example decision problem to demonstrate decision tree analysis. It describes three potential decisions - expand, maintain status quo, or sell now - under two possible future states, good or poor foreign competitive conditions. It then outlines the steps to analyze the problem: 1) determine the best decision without probabilities using various criteria, 2) determine the best decision with probabilities using expected value and opportunity loss, 3) compute the expected value of perfect information, and 4) develop a decision tree showing expected values at each node.
As part of our team's enrollment for Data Science Super Specialization course under UpX Academy, we submitted many projects for our final assessments, one of them was Telecom Churn Analysis Model.
The input data was provided by UpX academy and language we used is R. As part of the project, our main objective was :-
-> To predict Customer Churn.
-> To Highlight the main variables/factors influencing Customer Churn.
-> To Use various ML algorithms to build prediction models, evaluate the accuracy and performance of these models.
-> Finding out the best model for our business case & providing executive Summary.
To address the mentioned business problem, we tried to follow a thorough approach. We did a detailed level Exploratory Data Analysis which consists of various Box Plots, Bar Plots etc..
Further we tried our best to build as many Classification models possible which fits our business case (Logistic Regression/kNN/Decision Trees/Random Forest/SVM) and also tried to touch Cox Hazard Survival analysis Model. Later for every model we tried to boost their performances by applying various performance tuning techniques.
As we all are still into our learning mode w.r.t these concepts & starting new, please feel free to provide feedback on our work. Any suggestions are most welcome... :)
Thanks!!
The document describes an advanced analytics project using SAS to optimize a collection agency's dialing strategy. The objectives were to 1) identify characteristics of consumers likely to pay debts, 2) determine the best times to call consumers, and 3) allocate resources to maximize ROI. Models were built using variables like age, marital status, number of debts, TU score, and occupation to predict likelihood of payment and phone answering. The results identified important variables and improved segmentation to save 20% of calls while achieving targets.
The importance of this type of research in the telecom market is to help companies make more profit.
It has become known that predicting churn is one of the most important sources of income to Telecom companies.
Hence, this research aimed to build a system that predicts the churn of customers i telecom company.
These prediction models need to achieve high AUC values. To test and train the model, the sample data is divided into 70% for training and 30% for testing.
Decision Tree Analysis for statistical tool. The deck provides understanding on the Decision Analysis.
It provides practical application and limited theory. Will be useful for MBA students.
Predictive modeling is a process used in predictive analytics to create statistical models that can forecast future outcomes based on historical data. Predictive analytics uses techniques from data mining, statistics, modeling, machine learning and AI to analyze current data to predict future events. The predictive modeling process involves collecting data, creating a model, testing and validating the model, and evaluating model performance. Predictive models are commonly used to predict customer behavior, risk levels, and other business outcomes.
Predictive analytics uses past data to forecast future outcomes. The document discusses various predictive analytics techniques including simple forecasting methods, decision trees, and regression. Simple forecasting techniques like moving averages are easiest to implement but lack explanatory power, while decision trees and regression provide more accurate predictions at an individual level but require more complex deployment. The key is selecting the right technique based on the problem, data, and ability to implement predictive models in real-world applications.
Predictive Model for Loan Approval Process using SAS 9.3_M1Akanksha Jain
This is a Predictive Model which uses Logistic Regression to statistically help make better loan approval decisions in future for a German Bank. It uses an historical credit data set with 1000 data points and 20 variables.
Tool used:
SAS 9.3_M1
Steps Involved are:
- Data Quality check using Correlations and VIF Tests
- Analysis of different Variable Selection Methods such as Forward, Backward and Stepwise
- Variable Selection on the basis of Parameter Estimates and Odds Ratio
- Outlier Analysis to identify the outliers and improve the model
- Final Model Selection Decision based on ROC curve, Percent Concordant, PROC Rank and Hosmer Lemeshow Test
Automation of IT Ticket Automation using NLP and Deep LearningPranov Mishra
Overview of Problem Solved: IT leverages Incident Management process to ensure Business Operations is never impacted. The assignment of incidents to appropriate IT groups is still a manual process in many of the IT organizations. Manual assignment of incidents is time consuming and requires human efforts. There may be mistakes due to human errors and resource consumption is carried out ineffectively because of the misaddressing. Manual assignment increases the response and resolution times which result in user satisfaction deterioration / poor customer service.
Solution: Multiple deep learning sequential models with Glove Embeddings were attempted and results compared to arrive at the best model. The two best models are highlighted below through their results.
1. Bi-Directional LSTM attempted on the data set has given an accuracy of 71% and precision of 71%.
2. The accuracy and precision was further improved to 73% and 76% respectively when an ensemble of 7 Bi-LSTM was built.
I built a NLP based Deep Learning model to solve the above problem. Link below
https://github.com/Pranov1984/Application-of-NLP-in-Automated-Classification-of-ticket-routing?fbclid=IwAR3wgofJNMT1bIFxL3P3IoRC3BTuWmhw1SzAyRtHp8vvj9F2sKZdq67SjDA
"Multilayer perceptron (MLP) is a technique of feed
forward artificial neural network using back
propagation learning method to classify the target
variable used for supervised learning. It consists of multiple layers and non-linear activation allowing it to distinguish data that is not linearly separable."
Presentation on Modern Data Science
Data scientists are in high demand. There is simply not enough talent to fill the jobs. Why? Because the sexiest job of 21th century requires a mixture of broad, multidisciplinary skills ranging from an intersection of mathematics, statistics, computer science, communication and business. Finding a data scientist is hard. Finding people who understand who a data scientist is, is equally hard.
Check the video in spanish here :https://www.youtube.com/watch?v=R3jeBHLLiiM
Analytics: Compitiendo en la era de la información
En años recientes el mundo ha entrado en la era de la información, la evolución de la tecnología y el desarrollo de las redes sociales ha permitido a las compañías obtener más información sobre el comportamiento de sus clientes. Adicionalmente, los sistemas se han vuelto mas eficientes y económicos, dando la oportunidad a las empresas de almacenar gran cantidad de datos. Sin embargo, toda esta información solo almacenada no genera valor agregado para las empresas; entonces la pregunta es como obtener beneficios y tomar decisiones mas informadas usando los datos recolectados? La respuesta es analytics.
Analytics es el uso de métodos y herramientas para entender la información y tomar decisiones más precisas. Permite a las compañías hacer predicciones de comportamientos, identificar potenciales clientes, crear segmentaciones inteligentes, dirigir de una manera eficiente campañas publicitarias, identificar riesgos y anticipar cambios en el mercado. Así, por medio de analytics las compañías logran diferenciarse más de sus competidores y entender mejor las necesidades de sus clientes.
Alejandro Correa Bahnsen
Ingeniero Industrial con Maestría en Ingeniería Industrial de la Universidad de los Andes. Candidato a Doctorado en inteligencia artificial de la Universidad de Luxemburgo. Actualmente se encuentra trabajando en Cetrel, el operador de tarjeta de crédito mas grande de Europa, desarrollando un sistema inteligente para la prevención de fraude.
Experiencia como profesor de analytics y econometría en las universidades de Luxemburgo y de los Andes, respectivamente. Conferencista de analytics en SAS Analytics (Orlando, Las Vegas, Londres), SAS Global Forum (Orlando, San Francisco), IEEE International Conference on Data Mining (Vancouver, Bruselas), IEEE International Conference on Machine Learning and Applications (Miami) y European Conference on Data Analysis (Luxemburgo). Fundador de la comunidad Data Science Luxembourg y organizador del workshop IEEE Data Mining Case Studies (Dallas).
Credit card fraud is a growing problem that affects card holders around the world. Fraud detection has been an interesting topic in machine learning. Nevertheless, current state of the art credit card fraud detection algorithms miss to include the real costs of credit card fraud as a measure to evaluate algorithms. In this paper a new comparison measure that realistically represents the monetary gains and losses due to fraud detection is proposed. Moreover, using the proposed cost measure a cost sensitive method based on Bayes minimum risk is presented. This method is compared with state of the art algorithms and shows improvements up to 23% measured by cost. The results of this paper are based on real life transactional data provided by a large European card processing company.
Online fraud costs the global economy more than $400 billion, with more than 800 million personal records stolen in 2013 alone. Increasingly, fraud has diversified to different digital channels, including mobile and online payments, creating new challenges as innovative fraud patterns emerge. Hence it is still a challenge to find effective methods to mitigate fraud. Existing solutions include simple if-then rules and classical machine learning algorithms.
From an academic perspective, credit card fraud detection is a standard classification problem, in which historical transaction data is used to predict future frauds. However, practical aspects make the problem more complex. Indeed, existent comparison measures lack a realistic representation of monetary gains and losses, which is necessary for effective fraud detection. Moreover, there is an enormous amount of transactions from which only a tiny part are frauds, which implies a huge class imbalance. Additionally, a real fraud detection system is required to give a response in milliseconds. This criterion needs to be taken into account in the modeling process in order for the system to be successfully implemented. To solve these problems, in this presentation two recently proposed algorithms are compared: Bayes minimum risk and example-dependent cost-sensitive decision tree. These methods are compared with state of the art algorithms and shows significant improvements measured by financial savings.
Fraud Analytics: Detección y prevención de fraudes en la era del BigData
Durante el 2012 el nivel de fraude en tarjeta de crédito llego a 11.3 billones de dólares, un aumento de casi un 15% comparado con el 2011, esto demuestra el problema que el fraude representa no solo a las instituciones financieras sino también para la sociedad. Tradicionalmente la prevención del fraude consistía en proteger físicamente la infraestructura, sin embargo con cada vez más medios y canales de pago, la información financiera se ha vuelto cada vez más susceptible a ser hurtada. La siguiente opción para prevenir y controlar el fraude consiste en determinar si una transacción está siendo realizada por el cliente de acuerdo con sus patrones históricos de comportamiento. Este es el enfoque de Fraud Analytics.
En esta presentación se mostrara cómo es posible por medio de Fraud Analytics, determinar la probabilidad que una transacción sea o no realizada por el cliente, utilizando la información de compra de los clientes, sus interacciones con la entidad financiera, y por medio de análisis de redes sociales. Adicionalmente, se discutirán y compararan los resultados de las comúnmente utilizadas reglas de decisión y modelos avanzados de inteligencia artificial.
-------------------------------------
Alejandro Correa Bahnsen
-------------------------------------
Ingeniero Industrial con Maestría en Ingeniería Industrial de la Universidad de los Andes. Candidato a Doctorado en inteligencia artificial de la Universidad de Luxemburgo. Actualmente se encuentra trabajando en SIX, uno de los operadores de tarjeta de crédito más grande de Europa, desarrollando un sistema inteligente para la prevención de fraude.
Experiencia como profesor de analytics y econometría en las universidades de Luxemburgo y de los Andes, respectivamente. Conferencista de analytics en SAS Analytics (Orlando, Las Vegas, Londres, Frankfurt), SAS Global Forum (Orlando, San Francisco), IEEE International Conference on Data Mining (Vancouver, Bruselas, Dallas), IEEE International Conference on Machine Learning and Applications (Miami, Detroit) y European Conference on Data Analysis (Luxemburgo). Fundador de la comunidad Data Science Luxembourg y organizador del workshop IEEE Data Mining Case Studies (Dallas).
As the technical skills and costs associated with the deployment of phishing attacks decrease, we are witnessing an unprecedented level of scams that push the need for better methods to proactively detect phishing threats. In this work, we explored the use of URLs as input for machine learning models applied for phishing site prediction. In this way, we compared a feature-engineering approach followed by a random forest classifier against a novel method based on recurrent neural networks. We determined that the recurrent neural network approach provides an accuracy rate of 98.7% even without the need of manual feature creation, beating by 5% the random forest method. This means it is a scalable and fast-acting proactive detection system that does not require full content analysis.
Online fraud costs the global economy more than $400 billion, with more than 800 million personal records stolen in 2013 alone. Increasingly, fraud has diversified to different digital channels, including mobile and online payments, creating new challenges as innovative fraud patterns emerge. Hence it is still a challenge to find effective methods to mitigate fraud. Existing solutions include simple if-then rules and classical machine learning algorithms.
From an academic perspective, credit card fraud detection is a standard classification problem, in which historical transaction data is used to predict future frauds. However, practical aspects make the problem more complex. Indeed, existent comparison measures lack a realistic representation of monetary gains and losses, which is necessary for effective fraud detection. Moreover, there is an enormous amount of transactions from which only a tiny part are frauds, which implies a huge class imbalance. Additionally, a real fraud detection system is required to give a response in milliseconds. This criterion needs to be taken into account in the modeling process in order for the system to be successfully implemented. To solve these problems, in this presentation two recently proposed algorithms are compared: Bayes minimum risk and example-dependent cost-sensitive decision tree. These methods are compared with state of the art algorithms and shows significant improvements measured by financial savings.
Slides of my Pycon 2017 short talk "Demystifying machine learning using lime"
Jupyter Notebook with code in https://github.com/albahnsen/Talk_Demystifying_Machine_Learning
This is an overview of Tangerine Lab's capabilities and services. Tangerine Lab is a customer experience design agency that employs design thinking and service design methods to create enchanting customer experiences for brands. Our designs are enabled by emerging technologies and fueled by quantitative insights and customer analytics.
A Case Study in Predictive Modeling: How One Firm Achieved Dramatic Results w...Senturus
Case study on how a large U.S. telecomm company used predictive analytics to increase its effectiveness and bottom line. View the webinar video recording and download this deck: http://www.senturus.com/resource-video/predictive-analytics-case-study-demo/?rId=3504.
We also demystify predictive analytics and define predictive models, including how to develop and apply them to create measurable Return on Investment (ROI). This session includes a demonstration of IBM SPSS Modeler.
Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.
The successful analytics organization - Epsilon and Transamerica, LIMRA Data ...Epsilon Marketing
Epsilon and Transamerica recently co-presented The successful analytics organization at the LIMRA Big Data Analytics Conference. The session was well attended and thought-provoking.
In paid search, the majority of campaign processes can be automated with software, scripts, and apps. But which processes are better trusted to machines, and which processes require skilled human analysis for high ROI? Find out here.
Replay here: http://www.roirevolution.com/promos/replay-webinar-mastering-PPC-automation.php
This document provides an overview of predictive analytics for non-programmers. It defines predictive analytics as extracting data from existing datasets to identify trends and patterns which are then used to predict future outcomes. The document discusses why predictive analytics is useful for improving decision making and gaining a competitive advantage. It also outlines common predictive analytic models like classification, clustering, time series forecasting and associative rule mining. Finally, the document lists some popular tools for predictive analytics and concludes with a brief demo.
How to Use Data for Product Decisions by YouTube Product ManagerProduct School
With millions of new data points every moment, how are Product Managers expected to make sense of it all? This talk outlined the steps required to distill and synthesize data to drive actionable product decisions. The most effective Product Managers are those who know their data: they can justify product priority and roadmap changes, calibrate resource asks and manage their own time more effectively. This lecture equipped the audience with the tools necessary to draw insight from unstructured data using Google’s cloud analytics suite.
BDAS-2017 | Maximizing a churn campaign’s profitability with cost sensitive m...Big-Data-Summit
Los modelos predictivos de fuga de clientes churn tratan de predecir la probabilidad de que un cliente sea desertor de la empresa analizando su comportamiento histórico y su información socio económica. Esta herramienta permite maximizar los resultados de las campañas de retención. Los actuales algoritmos de clasificación de última generación no están bien alineados con los objetivos comerciales, en el sentido de que los modelos no incluyen los costos y beneficios financieros reales durante las etapas de entrenamiento y evaluación. En esta presentación, se muestra una nueva metodología sensible al costo para el modelo predictivo de churn de clientes. Primero proponemos una nueva medida financiera para evaluar la efectividad de una campaña de churn teniendo en cuenta la cartera de ofertas disponible, su costo financiero individual y la probabilidad de aceptación de la oferta en función del perfil del cliente. Luego, usando un conjunto de datos de churn del mundo real, comparamos diferentes algoritmos de clasificación y mediremos su efectividad basándonos en su poder predictivo y también en la optimización de costos. Los resultados muestran que el uso de un enfoque sensible al costo produce un aumento en los ahorros de costos de hasta el 26,4%.
The metrics that matter using scalability metrics for project planning of a d...Mary Chan
Have you expanded your organization across multiple locations, or are you a client that utilizes external partners that provide outsourcing services? Both have their "cost savings" challenge where cost savings analysis is often a topic well scrutinized. However, in the grand scheme of your organization, is it a metric that really matters? See actual analytics on multiple game projects and why cost savings isn't as important a metric when making informed decisions about project planning for scalable and distributed development. It's all about the Metrics that Matter.
This complete deck can be used to present to your team. It has PPT slides on various topics highlighting all the core areas of your business needs. This complete deck focuses on Analytics Roadmap Developing Management Platform Automation Framework Technological Business and has professionally designed templates with suitable visuals and appropriate content. This deck consists of total of twelve slides. All the slides are completely customizable for your convenience. You can change the colour, text and font size of these templates. You can add or delete the content if needed. Get access to this professionally designed complete presentation by clicking the download button below. https://bit.ly/2H0jHXR
This presentation given by Think Big's senior data scientist Eliano Marques at Digital Natives conference in Berlin, Germany (November 2015), details how to go from experimentation to productionization for a predictive maintenance use case.
"From Insights to Production with Big Data Analytics", Eliano Marques, Senior...Dataconomy Media
"From Insights to Production with Big Data Analytics", Eliano Marques, Senior Data Scientist at ThinkBig, a Teradata Company
YouTube Link: https://www.youtube.com/watch?v=caTyh1KflsI
Watch more from Data Natives 2015 here: http://bit.ly/1OVkK2J
Visit the conference website to learn more: www.datanatives.io
Follow Data Natives:
https://www.facebook.com/DataNatives
https://twitter.com/DataNativesConf
Stay Connected to Data Natives by Email: Subscribe to our newsletter to get the news first about Data Natives 2016: http://bit.ly/1WMJAqS
About the author:
Eliano is an analytics professional that combines Data Science skills with leadership, vision, creativity, project management, team building/management acquired through academia, personal research, leading internal Data Science/Advanced Modelling teams and providing consulting services to customers in several industries (including manufacturing, utilities, telcos, financial services, hospitality, sports).
Marketing analytics
PREDICTIVE ANALYTICS AND DATA SCIENCECONFERENCE (MAY 27-28)
Surat Teerakapibal, Ph.D.
Lecturer, Department of Marketing
Program Director, Doctor of Philosophy Program in Business Administration
Value Chain Analysis Framework PowerPoint Presentation Slides SlideTeam
Use value chain analysis framework PowerPoint Presentation Slides to deliver best of the products to the consumers. SlideTeam presents you logistics management PPT presentation so that you follow step by step guidance to identify firm’s primary and support activities that add value to its final product. Incorporate value chain analysis framework PPT slideshow to help your organization identify key areas of improvement to create value for the customers. This professionally designed value chain analysis complete presentation deck covers topic like value chain analysis framework, primary activities, support activities, competitive advantage type, cost and differentiation advantage, 3 step value chain analysis process, and more. This PPT presentation has touched all the aspects of the value chain for you to conduct this process easily. Include these content- ready presentation to increase the efficiency and deliver maximum value to the customers. This is an extensive process and this ready-to-use value chain complete presentation explains the framework explicitly. Get on blogs with our Value Chain Analysis Framework PowerPoint Presentation Slides. Everyone will have a comment.
Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...Sagar Deogirkar
Comparing the State-of-the-Art Deep Learning with Machine Learning algorithms performance on TF-IDF vector creation for Sentiment Analysis using Airline Tweeter Data Set.
Organizations are looking to maximize the value and return on their investments in SAP solutions. Value realization is Accenture's approach to helping clients define, manage, and capture business value from transformation projects. It provides ongoing processes to ensure projects are oriented towards business impacts and delivering targeted performance improvements. Accenture has tools like the Business Optimization Seeker for SAP to assess post-implementation opportunities for additional value through areas like inventory, purchasing, sales, and finance optimizations.
Paradigms of trading strategies formulationQuantInsti
The webinar aims to look at trading strategies from different perspectives. The aim has been to provide the audience with the metrics to formulate, evaluate the strategy based on the paradigms that suits one's trading style. We have often seen, when a same strategy is been used by two different traders, results have been quite different. What causes this difference has been the theme for this webinar.
This Workshop Teaches Business Leaders How To Implement AI Technologies To Serve Customers Better Than Anybody Else.
AGENDA
Introduction to Artificial Intelligence
Extracting Value & Delivering Value
Predictive & Preventive maintenance
Marine market, Jet engines
How to prepare & implement AI Playbook
The document discusses the importance of defining an analytics strategy for startups. It explains that analytics can help startups learn faster through feedback loops, provide more clarity on customer behavior, and build consensus on future actions. The document then outlines keys to a great analytics strategy, including tightly integrating analytics with business strategy, using an iterative process, and defining measurable hypotheses. It recommends focusing analytics on segmentation/cohort analysis, retention, funnels, revenue tracking, and marketing effectiveness. Case studies on Airbnb, Khan Academy, and Jawbone demonstrate how analytics provided insights to optimize processes and better understand customer needs.
Similar to Maximizing a churn campaign’s profitability with cost sensitive predictive analytics (20)
This document discusses using machine learning and deep learning techniques to detect malicious URLs and TLS certificates. It describes building recurrent neural network models to classify URLs and certificates based on their content. The models were able to achieve over 98% accuracy on URL classification and can be used to detect phishing URLs and malicious TLS certificates that aim to mimic legitimate ones. A demo of these techniques is also mentioned. The goal is to develop more robust detection of malicious AI that is trying to simulate legitimate behavior.
In this work we describe how threat actors may use AI algorithms to bypass AI phishing detection systems. We analyzed more than a million phishing URLs to understand the different strategies that threat actors use to create phishing URLs. Assuming the role of an attacker, we simulate how different threat actors may leverage Deep Neural Networks to enhance their effectiveness rate. Using Long Short-Term Memory Networks, we created DeepPhish, an algorithm that learns to create better phishing attacks. By training the DeepPhish algorithm for two different threat actors, they were able to increase their effectiveness from 0.69% to 20.9%, and 4.91% to 36.28%, respectively.
This document discusses how artificial intelligence could be used by hackers to improve cyberattacks like phishing. The author details an experiment where threat actors were able to improve their phishing attacks using AI, making them 3000% more successful. The document argues that AI enhances attackers' efficiency and that companies need multi-layered AI and machine learning detection systems as well as deep adversarial learning to monitor threats and fight against adversary AI. It maintains that companies must take action now as AI amplifies the power of attacks like phishing, malware, and weakening authentication controls.
Most people think a successful data product requires just three things: data, the
right algorithm, and good execution. But as anyone who’s tried to create one
knows, an effective product requires much more. In this talk, Dr. Correa Bahnsen
will share his successes—and failures—in building data products for information
security, and why an isolated data science team is a recipe for failure.
Worldwide, billions of euros are lost every year due to credit card fraud. Increasingly, fraud has diversified to different digital channels, including mobile and online payments, creating new challenges as innovative new fraud patterns emerge. Hence, it remains challenging to find effective methods of mitigating fraud. Existing solutions include simple if-then rules and classical machine learning algorithms. Credit card fraud is by definition an example-dependent and cost-sensitive classification problem, in which the costs due to is classification vary between examples and not only within classes, i.e., misclassifying a fraudulent transaction may have a financial impact ranging from a few to thousands of euros. In this paper, we propose an extension to the cost-sensitive decision trees algorithm, by creating an ensemble of such trees, and combining them using a stacking approach with a cost-sensitive logistic regression. We compare our method with standard machine learning algorithms and state-of-the-art cost-sensitive classification methods using a real credit card fraud dataset provided by a large European card processing company. The results show that our method achieves savings of up to 73.3%, more than 2 percentage points more than a single cost-sensitive decision tree.
Navigating the World of Topsoil: A Guide to the Right Choice for Your Gardennerissacampuzano
Are you looking to improve your garden's health but unsure about which topsoil to choose? This PPT provides insights into selecting the right topsoil for your gardening needs. From understanding various types of topsoil to evaluating their benefits, this resource equips you with the essential knowledge to make an informed decision. Explore to learn more.
Click to know more - https://mulchpros.com/blog/navigating-the-world-of-topsoil-a-guide-to-the-right-choice-for-your-garden/
MISS TEEN HYDERABAD 2024 - WINNER RYKA TANDONDK PAGEANT
In the dynamic city of Hyderabad, a youthful and outstandingly skilled person has as of late made waves on the national stage. Ryka Tandon, a 14-year-old understudy, has been honored with the prestigious title of Miss High Schooler India 2024 Victor, Pride of India, from the Dk Show. Her travels to this regarded position are a confirmation of her unflinching devotion, ability, and tireless endeavors. Despite her youthful age, Ryka has, as of now, accomplished momentous points of reference that recognize her as a guide of motivation and pride for her city and her nation. This article digs into the uncommon life and accomplishments of Ryka Tandon, investigating her foundation, achievements, and the qualities that make her a standout individual.
MISS RAIPUR 2024 - WINNER POONAM BHARTI.DK PAGEANT
Poonam Bharti, a guide of ability and diligence, has been chosen as the champ from Raipur for Mrs. India 2024, Pride of India, from the DK Show. Her journey to this prestigious title is a confirmation of her commitment, difficult work, and multifaceted gifts. At fair 23 a long time ago, Poonam has as of now made noteworthy strides in both her proficient and individual lives, encapsulating the soul of present-day Indian ladies who adjust different parts with beauty and competence. This article dives into Poonam Bharti’s foundation, achievements, and qualities that separated her as a meriting champion of this award.
Stag Elevators | Leading Home Elevator Company in IndiaStag Elevators
Discover Stag Elevators, India's premier home elevator company, dedicated to delivering unmatched mobility solutions nationwide. Specializing in certified home elevators, lifts, and platform lifts, Stag Elevators leads with superior safety, quality, and innovation. Partnering with renowned Italian manufacturers ensures every product meets European safety standards and is TUV certified, offering affordability and utmost safety for homes of all sizes and types, from small houses to luxury residences.