Paychex has been developing predictive retention models since 2009 to identify clients at risk of leaving. They use a dynamic model averaging approach to account for concept drift over time. New variables are included as they become available. Models are fit using logistic regression with LASSO to automatically select important variables. Model performance is tracked using both technical metrics and business outcomes to ensure resources are properly allocated. The next steps are to experiment with different scoring approaches and update the weighting heuristic used for model averaging.

1. © 2013, PAYCHEX, Inc. All rights reserved.
Retention Modeling in
Uncertain Economic
Times
Chip Galusha
Data Scientist
Paychex, Inc.

2. © 2013, PAYCHEX, Inc. All rights reserved.
History of Paychex
1983
1971
Market Cap
$18 Billion
Total Revenue
$2.7 Billion
A leading provider of integrated
human capital management,
providing businesses the
freedom to succeed.
Today

3. Data Science @ Paychex
3
Data
Science
Team
Marketing
Sales
Information
Technology
FinanceRisk
Human
Resources
Service

4. Strategic Client Retention
 2009: developed and deployed first predictive retention model
4
2009 2010 201620152014201320122011
V1 V2 V3 V4
 2012: Clients with retention strategy applied we’re 80% less likely to leave
Controllable Losses
 Price
 Service Issue
 Dissatisfaction w/ Product
Uncontrollable Losses Out of Business
Business Acquired
Business Change
Out of Business
Business Acquired
Business Change
 Out of Business
 Business Acquired
 Business Change
𝑃 𝐶𝑙𝑖𝑒𝑛𝑡 𝐿𝑜𝑠𝑠 𝑿𝑖 ) =
1
1 + 𝑒−(β𝑿 𝑖)

5. Retention Tracking System
F
• Business 1: Annual Est. Revenue
• Business 2: Annual Est. Revenue
• …
D
• Business 3: Annual Est. Revenue
• Business 4: Annual Est. Revenue
• …
- Retention specialists proactively
call clients based on retention
score to mitigate potential issues
- Client payroll liability
- Employee counts
- ...
- Business Type
- Geographic
- Credit Info
- …
Model
Scores
Clients
Monthly
Inputs Output
- Probability
of Client
leaving
Paychex
600K clients; 8 full-time equivalent retention specialists
National Retention Team

6. Problem: Concept Drift
6
The underlying assumptions and distributions change over time
with respect to predicting losses
time
 Population Stability Index (PSI) scores help us identify shifts in the
distributions of the variables in our models
 What happens when the relationship with the target changes?

7. Example of Concept Drift
7
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Date
11/1/2012
12/1/2012
1/1/2013
2/1/2013
3/1/2013
4/1/2013
5/1/2013
6/1/2013
7/1/2013
8/1/2013
9/1/2013
10/1/2013
11/1/2013
12/1/2013
1/1/2014
2/1/2014
3/1/2014
4/1/2014
5/1/2014
6/1/2014
7/1/2014
8/1/2014
9/1/2014
10/1/2014
11/1/2014
12/1/2014
1/1/2015
2/1/2015
3/1/2015
4/1/2015
5/1/2015
6/1/2015
7/1/2015
8/1/2015
9/1/2015
10/1/2015
11/1/2015
12/1/2015
1/1/2016
2/1/2016
3/1/2016
4/1/2016
5/1/2016
6/1/2016
7/1/2016
8/1/2016
9/1/2016
10/1/2016
11/1/2016
12/1/2016
1/1/2017
2/1/2017
3/1/2017
California Accommodation & Food Services:
Year over Year Change in Losses
yoy 6 per. Mov. Avg. (yoy)

8. Solution: Dynamic Model Averaging*
8
Dynamic
 Adapts to account for concept drift
 Allows for the inclusion of new variables as they become available
Model Averaging
 Use an ensemble of historical models to estimate probability of client loss
 Establishing a weighting heuristic to apply each model
* Not Associated w/ Raftery et al. (2010)

9. Model Averaging
9
 Tested four alternatives:
Dynamic Model Averaging
Regression Combiner
Use a weighted average base on ridge regression
,t-12 t-3*
t-12 t-11 t-4 t-3…
Average Model Scores From Each Model
Baseline Dynamic
,t-3*
Use Most Recent Model
One Year Prior Dynamic
,t-12*
Use Model from One Year Prior
 The best performance
achieved by the averaged
scores of a trend model (t-3)
and seasonal model (t-12)

10. Dynamic Variables Inclusion
 Access to data is continually improving. How can we adapt to this?
 Just join new variables to production data set!
 What if variables are discontinued?
 We can identify which variables were used in our model 12 months ago
but aren’t in our current scoring data set.
 Add zero-filled column
10
0
0
⋮

11. Model Fitting: Logistic Regression via
LASSO (aka Elastic Net)
 Least Absolute Shrinkage and Selection Operator(LASSO)
 Regularize logistic regression
 Standardize variables (scale them to mean = 0 and sd = 1)
 In traditional modeling, we choose to either include or exclude a variable in a
model (domain knowledge, AIC/BIC)
 LASSO automatically includes all variables in the model, but shrinks
their impact on the model if it is not important
 By shrinking the coefficients in the model, we can drive some
coefficients to zero.
11

12. Example LASSO Plot
12
• A vertical line represents a cross-
sectional view of the coefficients for
a model.
• The red line indicates a chosen
parameter/model. It is a regression
with 3 variables (3 coefficients)
 Lcavol: 0.55
 Weight: 0.17
 Svi : 0.09
• With a very large shrinkage factor,
the sum of usual regression
coefficients are less than the
constraint, so this is exactly the
same as using a regular regression
• With a very small shrinkage factor,
the coefficients must sum to zero, so
there are no coefficients

13. Variables*
13
Continuous Variables
 Trend: Mann-Kendall Trend Test Statistic
 Standard Deviation
 Average
 Demand Index
 Length of Service transformed using a Spline
Categorical Variables
 # of Discount Products > # of Billed Products – {0,1}
 Has Penalty in last 6 months & LOS < 2 years – {0,1}
 Has Case in last 6 months & LOS < 2 years – {0,1}
 Has Return in last 6 months & LOS < 2 years – {0,1}
 Start Month - {1,…,12}
 401k, Taxpay, Time & Attendance Indicators – {0,1}
* Final variables(w/ non Zero Coefficients) will change month by month

14. Performance Tracking
14
Model
Performance
Technical
Discriminative
Ability of Model
 ROC Curve
 Brier score
Diagnostic
 Variable
Importance
 Client Score
Shifts
Business
Strategic
Diagnostics
 Calls / Save
Model Valuation
 $$$
 Proper
resource
allocation

15. Technical Performance Tracking
15

16. Next Steps
16
 Experiment with scoring
 Update weighting heuristic
 Apply strategy to other model developments

17. © 2013, PAYCHEX, Inc. All rights reserved.
Thank You
Chip Galusha
Data Scientist
fgalusha@paychex.com