Heiler

1. © 2013, PAYCHEX, Inc. All rights reserved.
Retention Modeling in
Uncertain Economic
Times
Mark Heiler
Data Scientist
Paychex, Inc.

2. © 2013, PAYCHEX, Inc. All rights reserved.
History of Paychex
1983
1971
Market Cap
$18 Billion
Total Revenue
$2.7 Billion
A leading provider of integrated
human capital management,
providing businesses the
freedom to succeed.
Today

3. Data Science @ Paychex
3
Data
Science
Team
Marketing
Sales
Information
Technology
FinanceRisk
Human
Resources
Service

4. Strategic Client Retention
 2009: developed and deployed first predictive retention model
4
2009 2010 201620152014201320122011
V1 V2 V3 V4
 2012: Clients with retention strategy applied we’re 80% less likely to leave
Controllable Losses
 Price
 Service Issue
 Dissatisfaction w/ Product
Uncontrollable Losses Out of Business
Business Acquired
Business Change
Out of Business
Business Acquired
Business Change
 Out of Business
 Business Acquired
 Business Change
𝑃 𝐶𝑙𝑖𝑒𝑛𝑡 𝐿𝑜𝑠𝑠 𝑿𝑖 ) =
1
1 + 𝑒−(β𝑿 𝑖)

5. Retention Tracking System
F
• Business 1: Annual Est. Revenue
• Business 2: Annual Est. Revenue
• …
D
• Business 3: Annual Est. Revenue
• Business 4: Annual Est. Revenue
• …
- Retention specialists proactively
call clients based on retention
score to mitigate potential issues
- Client payroll liability
- Employee counts
- ...
- Business Type
- Geographic
- Credit Info
- …
Model
Scores
Clients
Monthly
Inputs Output
- Probability
of Client
leaving
Paychex
600K clients; 8 full-time equivalent retention specialists
National Retention Team

6. Problem: Concept Drift
6
The underlying assumptions and distributions change over time
with respect to predicting losses
time
 Population Stability Index (PSI) scores help us identify shifts in the
distributions of the variables in our models
 What happens when the relationship with the target changes?

7. Example of Concept Drift
7
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Date
11/1/2012
12/1/2012
1/1/2013
2/1/2013
3/1/2013
4/1/2013
5/1/2013
6/1/2013
7/1/2013
8/1/2013
9/1/2013
10/1/2013
11/1/2013
12/1/2013
1/1/2014
2/1/2014
3/1/2014
4/1/2014
5/1/2014
6/1/2014
7/1/2014
8/1/2014
9/1/2014
10/1/2014
11/1/2014
12/1/2014
1/1/2015
2/1/2015
3/1/2015
4/1/2015
5/1/2015
6/1/2015
7/1/2015
8/1/2015
9/1/2015
10/1/2015
11/1/2015
12/1/2015
1/1/2016
2/1/2016
3/1/2016
4/1/2016
5/1/2016
6/1/2016
7/1/2016
8/1/2016
9/1/2016
10/1/2016
11/1/2016
12/1/2016
1/1/2017
2/1/2017
3/1/2017
California Accommodation & Food Services:
Year over Year Change in Losses
yoy 6 per. Mov. Avg. (yoy)

8. Solution: Dynamic Model Averaging*
8
Dynamic
 Adapts to account for concept drift
 Allows for the inclusion of new variables as they become available
Model Averaging
 Use an ensemble of historical models to estimate probability of client loss
 Establishing a weighting heuristic to apply each model
* Not Associated w/ Raftery et al. (2010)

9. Model Averaging
9
 Tested four alternatives:
Dynamic Model Averaging
Regression Combiner
Use a weighted average base on ridge regression
,t-12 t-3*
t-12 t-11 t-4 t-3…
Average Model Scores From Each Model
Baseline Dynamic
,t-3*
Use Most Recent Model
One Year Prior Dynamic
,t-12*
Use Model from One Year Prior
 The best performance
achieved by the averaged
scores of a trend model (t-3)
and seasonal model (t-12)

10. Dynamic Variables Inclusion
 Access to data is continually improving. How can we adapt to this?
 Just join new variables to production data set!
 What if variables are discontinued?
 We can identify which variables were used in our model 12 months ago
but aren’t in our current scoring data set.
 Add zero-filled column (not optimal… can retrain if needed)
10
0
0
⋮

11. Model Fitting: Logistic Regression via
LASSO (aka Elastic Net)
 Least Absolute Shrinkage and Selection Operator(LASSO)
 Regularize logistic regression
 Standardize variables (scale them to mean = 0 and sd = 1)
 In traditional modeling, we choose to either include or exclude a variable in a
model (domain knowledge, AIC/BIC)
 LASSO automatically includes all variables in the model, but shrinks
their impact on the model if it is not important
 By shrinking the coefficients in the model, we can drive some
coefficients to zero.
11

12. Example LASSO Plot
12
• A vertical line represents a cross-
sectional view of the coefficients for
a model.
• The red line indicates a chosen
parameter/model. It is a regression
with 3 variables (3 coefficients)
 Lcavol: 0.55
 Weight: 0.17
 Svi : 0.09
• With a very large shrinkage factor,
the sum of usual regression
coefficients are less than the
constraint, so this is exactly the
same as using a regular regression
• With a very small shrinkage factor,
the coefficients must sum to zero, so
there are no coefficients

13. Variables*
13
Continuous Variables
 Trend: Mann-Kendall Trend Test Statistic
 Standard Deviation
 Average
 Demand Index
 Length of Service transformed using a Spline
Categorical Variables
 # of Discount Products > # of Billed Products – {0,1}
 Has Penalty in last 6 months & LOS < 2 years – {0,1}
 Has Case in last 6 months & LOS < 2 years – {0,1}
 Has Return in last 6 months & LOS < 2 years – {0,1}
 Start Month - {1,…,12}
 401k, Taxpay, Time & Attendance Indicators – {0,1}
* Final variables(w/ non Zero Coefficients) will change month by month

14. Performance Tracking
14
Model
Performance
Technical
Discriminative
Ability of Model
 ROC Curve
 Brier score
Diagnostic
 Variable
Importance
 Client Score
Shifts
Business
Strategic
Diagnostics
 Calls / Save
Model Valuation
 $$$
 Proper
resource
allocation

15. Technical Performance Tracking
15

16. Next Steps
16
 Experiment with scoring
 Update weighting heuristic
 Apply strategy to other model developments

17. © 2013, PAYCHEX, Inc. All rights reserved.
Thank You
Mark Heiler
Data Scientist
mheiler@paychex.com