Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Model Factory at ING Bank


Published on

At ING Bank, machine learning models are a key factor in making relevant engagements with our customers, empowering them to stay a step ahead in life and in business. In our efforts to make the model building process more rapid, compliant, validated and accessible to roles other than data scientists (such as data analysts or customer journey experts), we have structured it for an easy creation of propensity models.

In this talk, I will present this structure, focusing on pipelining data science models in Apache Spark. In particular, I will show how we use Apache Sqoop & Ranger to comply with GDPR, build a data science workflow on top of python and Jupyter, extend the SparkML libraries on PySpark to create custom standardizers and cross-validators, and show an in-house developed monitoring tool built on top of Elasticsearch for model evaluation.

Finally, I will describe the type of engagement analysts and customer journey experts have with the result set of the models created, and how we refine our dashboards (in IBM Cognos) accordingly.

Speaker: Dor Kedem, Lead Data Scientist
ING Bank

Published in: Technology
  • You can hardly find a student who enjoys writing a college papers. Among all the other tasks they get assigned in college, writing essays is one of the most difficult assignments. Fortunately for students, there are many offers nowadays which help to make this process easier. The best service which can help you is ⇒ ⇐
    Are you sure you want to  Yes  No
    Your message goes here
  • As a single mother every little bit counts! This has been such a great way for me to earn extra money. As a single mother every little bit counts! Finally, a vehicle for making some honest to goodness real money to make life easier and happier now that I don't have to pull my hair out budgeting every penny every day.Thanks for the rainbow in my sky. ♥♥♥
    Are you sure you want to  Yes  No
    Your message goes here

Model Factory at ING Bank

  1. 1. Model Factory @ING Bank Presentation to DataWorks Summit - 2019 2019-03-20Dor Kedem
  2. 2. • Extensive software development career since 2002. • Working on machine learning research & data science applications since 2010. • Today, a lead data scientist and product owner @ ING Bank in Amsterdam. Grab me (or via LinkedIn) during these couple of days to talk about: • CI/CD solutions for a data science project lifecycle. • Impact-driven data science (from POCs to MVPs). • Modelling techniques and machine learning applications. • Transitioning from software development or IT roles to data science. • Boardgames and 3D puzzles. A bit about me 3 Dor.Kedem (at) Image Credit: My wife, HAPPY BIRTHDAY!
  3. 3. ING Bank at a glance Active in more than 40 countries +54.000 employees in ING Group 38M retail customers and 12.5M primary customers in 4Q18 Net Promoter Scores: #1 in 6 out of 13 retail countries Source:
  4. 4. Challenges in European Banking Scene Historically low interest rate = less revenue from lending Source: Source: Historical LIBOR rates (grey – recession) Regulations leads to a more transparent and open banking environment Fintech industries are looking into innovative ways to fill traditional banking roles
  5. 5. How does a bank differentiate itself from the rest? Sources: Empowering people to stay a step ahead in life and in business Our purpose Our strategic priorities
  6. 6. Analytics Efforts in ING "Data is the language of the future. If you don’t speak it yet, we’ll help you master it," says Görkem Köseoğlu, ING’s chief analytics officer. Artificial Intelligence: Currently, ING employs around 80 data scientists, working on various AI-projects: Analytics skillset: Thousands of employees to engage analytical tools and insights:
  7. 7. Our ambition: all customer interactions driven by analytics One-to-One Analytics Maximising number of analytics driven service and sales interactions Data > insight > action is in ING’s DNA Democratize big data usage across ING Users of our services are extremely happy 9
  8. 8. Data Analytics for customer interactions (NL+BE) Customer Journey Experts Data Analysts Data Scientists Data Engineers How many? Over 300 (outside 1:1) Over 100 Roughly 20 Roughly 15 What do we know? • Banking • Marketing theory • Customer engagement • Message framing • BI tools (SAS, IBM Cognos) • Data Privacy • SQL • Statistics & ML • Data Privacy • Programming (i.e. Python, R, Scala) • Big data technologies • CI/CD solutions • Security & Compliance What do we create? • Product specification • Online & offline content • Customer engagement • Reports • Dashboards • A/B Testing • Statistical models • Data Products • ETL systems • Data lake • Model hosting 10 CJE - Christina DA - Arjen DS - Samir DE - Eleanor
  9. 9. The need for a model factory 11
  10. 10. For Black-Friday (Nov 23rd), a customer engagement business unit would want to engage eligible customers (via NBA + emails) with the option to acquire a new credit card. We have two types of offers: regular credit cards & platinum credit card. How to find who to contact with these offerings? 2 approaches: Example case: Credit Card Acquisition 12 DA - Arjen DS - Samir CJE - Christina • Build a likelihood model based on past behavior and engagements. • Rank customers according to this model. • Plot customer engagements on different demographics. • Come up with business rules based on personal understanding.
  11. 11. For Black-Friday (Nov 23rd), a customer engagement business unit would want to engage eligible customers (via NBA + emails) with the option to acquire a new credit card. We have two types of offers: regular credit cards & platinum credit card. How to find who to contact with these offerings? 2 approaches: Example case: Credit Card Acquisition 13 DA - Arjen DS - Samir • Build a likelihood model based on past behavior and engagements. • Rank customers according to this model. • Plot customer engagements on different demographics. • Come up with business rules based on personal understanding. Very vast majority CJE - Christina
  12. 12. Enticements are being offered to the wrong customers: • Customer disengagement (unsubscribes, ad-blindness) • Wasted work by CJE’s and DA’s • Loss of potential revenue. It takes a lot of time to make customer selections: • No structured way to figure out target population. • No structured learning from our past campaigns. • Only a binary selection (to send / not to send) – no ranking. Not leveraging on the full potential of our data: • Not taking into account a large set of features. • Not taking into account engagement with past offerings. • Not taking into account engagement with other products. • You only target what you can code. Problem with manual selection of the customers 14 Purchase No purchase All clients Top 10% One of the added value of models: Ranking customers Unordered Ranked by relevant Selection based on threshold
  13. 13. Here are some responses gathered when asking about current way of work and gaps from best practices: Learning from our past & present… CJE - Christina DA - Arjen “I don’t have time for experiments & evaluation. We have a schedule and need to create the next campaigns” “I get personal satisfaction from weighing in my opinion” “I don’t see how testing everything to death leads to better results” “Management is not critical enough about measuring our performance” “I can’t get clear insights from my DA / CJE colleague” “I know my customers” “There’s a lack of clear guidelines and standards across ING tribes” “I need to be able to contact more customers, even if models disagree” Organizational Personal
  14. 14. Fears needs to be addressed early on (i.e. fear of measurement, of loss of control, of automation) Focusing on empowerment before the revolution: • Helping DAs and CJEs to make better decisions, not to make all decisions for them. • Creating direct link: standardization  gain in personal efficiency. • Incorporating customers in our development squad. Resource: Check out ING PACE: Evidence-based design-driven lean approach. How should we interpret the interviews? 16 PACE PACE Phases Experiment Loop
  15. 15. Our approach – Model Factory 17
  16. 16. Democratizing model building: Enabling DA’s to create models for gaining customers insights. Accelerate best practices: Make it easy & fast to be effective in customer selections. • Model building process “built-in”: Tell us “what” you want – we take care of the “how”. • Evaluation “built-in”: Build a model  Get a free model & campaign evaluation! • Compliance “built-in”: GDPR, archiving, legal, commercial pressure, risk – we got you covered. Our Objective CJE - Christina DA - Arjen DS - Samir DE - Eleanor Saves time Better engagement Making large- scale impact Understand the customer better Saves time Grows in skills Meeting objectives Customer - Claire More relevant offerings ING Bank 17
  17. 17. Building customer models without reinventing the model building process Model Factory 19 Building Blocks Model Recipe Model Building Process Scoring Model 𝑓( ) = 𝑦 Scoring eligible customers Feeding scores to ING processes Creating reports in BI tools for ING business units Somewhat similar open source approach: Uber’s Ludwig, AirBnB’s BigHead & KPN’s model factory
  18. 18. Mandatory ingredients: • Business Objective Selection from: acquisition, deepsell, retention, customer journey. • Business Objective specification Based on the objective. For example: which product to acquire? • Features to include / exclude Selection from a list. Done based on domain expertise. • Customers to include / exclude SQL “where clause”. Based on domain expertise. Optional ingredients (with defaults): • Times specification: (How long does it take to acquire, how long before customer makes decision) • Modelling techniques: (for advanced / data scientists users) Model Recipe Model specification is translated to a 10-15 lines JSON file and is filled by a DA
  19. 19. Analytics features extraction Machine learning monitoring processes Target templates (i.e. acquisition, deepsell) Classifiers Evaluators Hyperparameter / model selection (AutoML) Fairness & bias reduction Building Blocks Data-sets creators Uplift measurement Storage management Scheduling Hosting GDPR applications Interaction with ING services Available to all models built with a recipe specification:
  20. 20. Building blocks concrete examples 22
  21. 21. Building Blocks Example (1): Data Sources Clients (~80) Products (~600) Engagements (~300) Data dumps & streams from ING sources Data Lake Structured Data Sources Analytics Features Table(s) DE - Eleanor DA - Arjen DS - Samir Features in the table are GDPR validated Data scientists & analysts build an analytics repository from data sources. Data engineers build the ETL processes to create data sources. Built on top of: • IBM PureData for Analytics (PDA) • SAS Enterprise Global Creating the model feature sources
  22. 22. Building Blocks Example (1): Data Sources Via Apache Sqoop (diffs / full) Deployed in Ansible Scheduled by IBM Workload Scheduler Hortonworks Data Platform IBM PureData for Analytics Transferring the data to our model building environment 1 2 Managing access via Apache Ranger on both levels: Files are extracted from PDA’s tables to HDFS Hive tables’ metadata is being created / updated. HDFS policies: hdfs://ingestion/raw Hive policies on specific tables and model results. i.e.: • hdfs://access/BEL/models • hdfs://access/NED/models Synchronize users & permissions ldap_user_sync LDAP Container
  23. 23. Building Blocks Example (2): Data Sets Creators Some tips to building datasets: • Selecting different customers in each timestamps  Generalizing to new customers. • Arranging data set in time series accordance  Generalizing better for forecasting. Training set Jan ‘17 Jan ‘18 Valid Mar ‘18 Training set Jan ‘17 May ‘18 Valid Mar ‘18 Training set Jan ‘17 Jul ‘18 Valid May ‘18 Training set Jan ‘17 Jul ‘18 Test Dec ‘18 Time series cross validator Picking best hyper-parameters Train fit the model. Used to Valid learn hyper- parameters. Used to Test Legend evaluate and to pick best mode. Used to Useful resource - Timothy Lin’s Creating a Custom Cross-Validation Function in PySpark
  24. 24. Building Blocks Example (3): Preparing data for training Pipeline approaches ( Elegant way to manage the workflow of your data processing. Each stage simply appends new transformers to the pipeline model stages: Transformers: The vast majority of transformers we use can be found in, as well as some custom transformers we’ve defined ourselves to make sure all preprocessing is managed in the pipeline object. Code snippet: each fit function adds stages to pipeline & transforms data for next stage Code snippet: Example of basic custom transformer
  25. 25. Building Blocks Example (3 - Bonus): Filling missing values Apart from the Imputer class, we also experiment with using Autoencoders for filling missing values. • Encoder: A dimensionality reduction model, from the original features to a lower dimensional code. • Decoder: The reverse action – tries to recreate the original input from the code (not a perfect match). A very good resource for distributed deep learning on top of Spark: dist-keras (by CERN Database group). To learn more about autoencoder’s check out Irhum Shafkat’s Intuitively Understanding Variational Autoencoders. X: 5 10 23 <?> 0 Autoencoder design Autoencoder application: Data with missing feature gets in, filled value comes out X’: 5 10 22 7 0
  26. 26. Building Blocks Example (4): Model Building Relying on open-source Big Data technologies as building blocks Classifiers (the model types): mainly based on the Spark Machine Learning framework and includes: • Linear / Logistic regression • Naïve Bayes • Decision Trees • Ensemble methods (Random Forest, GBRT) • Neural Networks (MLP) Evaluators (the model performance validation): • Everything under the Spark MLLib evaluation metrics. AutoML (finding the best model): • Currently experimenting with auto-sklearn & H2O for faster hyper-parameter tuning. See Georgian Partners’ comparison.
  27. 27. Building Blocks Example (5): Fairness Resource: For easy explanation: Attacking discrimination with smarter machine learning Resource: For approaches on reducing bias: IBM AI Fairness 360
  28. 28. Model Factory Products 30
  29. 29. Engaging with the model factory process & results Customer Journey Expert Data Analyst Data Scientist Data Engineer Validating building blocks Validating model execution Validate model quality Understand the customer better Selecting customers for campaign Post-hoc campaign evaluation Getting the big picture of model usage                  Monitoring Tool (Developed in-house) BI Tools (IBM Cognos Analytics)
  30. 30. ML Monitoring Dashboard 32 Backend Monitoring Dashboard
  31. 31. Designated system for monitoring production ML models This architecture enables access to metrics data from production models 33 User views project metrics Frontend loads the project Logstash Development / production cluster JSON files with project details, metrics, models, etc. When a user creates a model, we create a new model monitoring Project JSON files are pushed Backend Elastic SearchMonitoring Dashboard Test / Prd Test / Prd ING servers
  32. 32. Designated system for monitoring production ML models Useful resource: Google AI’s What’s your ML test score? A rubric for ML production systems (Breck et. al, 2016) Open source alternative: (platform for machine learning lifecycle)
  33. 33. Designated system for monitoring production ML models Useful resource: Google AI’s What’s your ML test score? A rubric for ML production systems (Breck et. al, 2016) Open source alternative: (platform for machine learning lifecycle)
  34. 34. Reporting on the model built Recall Precision AUCs BA G H C D E F F* I F. Customer Segmentation G. Model comparison heat map. H. Compare features distributions. I. Score distribution J. Conversion for feature values. A. Technical quality metrics B. Lift curve C. Cumulative Gains D. Overlap with manual selection E. Feature Importance J 36
  35. 35. A couple of examples to explain visually how good the model is: Credit Card Acquisition – Model Performance Customer percentiles (1% = 54k) 37 B/C Cumulative Gains – how many of our conversions did the model catch? Lift Curve – how much better is our selection than random selection? Customer percentiles (1% = 54k) DA - Arjen “Ok… It’s better than random… But how does it compare to my previous selection?”
  36. 36. What’s the difference between my old selection and the model’s? 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 12000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 Customers not in old selection Ranked customers percentile (left – most relevant) Customersselectedinpercentile 20% Threshold 38 D DA - Arjen Customers in old selection “I feel more confident the model makes meaningful selections” “I see that the model found top customers that I haven’t contacted yet” “I still don’t understand who did I miss…”
  37. 37. Who are the likely customers? (In this example: by age) 6.24% 10.82% 10.55% 10.32% 10.13% 12.06% 12.72% 12.82% 14.34% 1.94% 7.05% 8.38% 8.00% 8.35% 11.59% 15.05% 18.01% 21.64% 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 18-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-60+ How does [age] distribute over the most likely customers? Top 10% versus Bottom 90% Age distribution in each customer group 39 Portionofentireavailablepopulation Who did we miss in the top 20%? Selected versus Not Selected 5.82% 13.77% 12.34% 9.61% 8.68% 10.63% 12.12% 12.35% 14.68% 1.63% 6.81% 8.27% 8.38% 8.72% 12.10% 15.29% 17.99% 20.81% 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 18-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-60+ H
  38. 38. Strategy for features: iterative refinement of the filtered features. Credit Card Acquisition – Important Features (larger = more important, blue color = significant to only one model) Regular credit card acquisitions Platinum credit card acquisitions 40 f_53 f_348 f_127 f_218 f_31 f_38 f_43 f_8 f_857 f_842 f_842 f_12 f_15 f_38 f_31 f_457 f_458 f_857 f_218 f_127 E DA - Arjen “I gain confidence that the models generate meaningful results.” “I can easily troubleshoot issues with model recipe”. “I learn more about our customers.”
  39. 39. Grouping customers together based on the model’s important features. Customer Segmentation Segment size: indication of number of customers. Segment color: average conversion (more yellow = higher conversion). Y,X Axes: Don’t mean much, but the overall distance between segments mean that customers are more different based on important features (closer segments = more similar). Allows for further analysis on customer segments F CJE Christina “This helps me understand who are my customers and to tailor a message for each type of customers.”
  40. 40. X-axis: Ranked customers interested in regular credit card (left - most interested) Y-axis: Ranked customers interested in platinum credit cards (down - most interested). Rectangles – the top 10% of customers in each group. Credit Card Acquisition – Which proposal to who? Bottom 90% Platinum 347k 4.9Mil Top 10% - Platinum 232k 412k Top 10% Regular Bottom 90% Regular # customers in shared percentile (log scale) Brighter = more customers Combined ranking for both credit card acquisition models 42 G DA - Arjen “I can now send the relevant offer to the relevant customers and avoid spamming.
  41. 41. What to expect from the top-scored customers next month? 0.00% 0.50% 1.00% 1.50% 2.00% 2.50% 3.00% 3.50% 0 20 40 60 80 100 Expected next-month conversion rate for each percentile Conversionratenextmonth Customer percentile as ranked by the model (lower = more relevant) Percentile Expected Percentile’s Conversion Expected Total Conversion Expected Total Conversion 5 1.81% 2.25% 1404 10 1.37% 1.88% 2351 20 0.95% 1.50% 3737 50 0.48% 1.00% 6268 100 0.09% 0.65% 8103 43
  42. 42. To wrap up 44
  43. 43. Summary: • Enabling model creation, using data scientists best practices and cumulative efforts. • Simple specification, modular design. • Accelerates DA’s, empowers CJE’s, and makes all of us more relevant to our customers. Model Factory @ING Bank Selected Resources: Driving innovation: • ING PACE: Evidence-based design-driven lean approach Model building: • Uber’s Ludwig – Building models without coding • AirBnB’s BigHead • Georgian Partners’ AutoML comparison • Creating a Custom Cross-Validation Function in PySpark • Distributed deep learning on spark: dist-keras Machine learning in production: • What’s your ML test score? A rubric for ML production systems • MLFlow: machine learning lifecycle Fairness & bias removal: • Google’s “Attacking Discrimination in ML” • IBM’s AI Fairness 360 Dor.Kedem (at)