Improving the performance of Telco Churn Predictive Model with SPSS & 6 Sigma

SPSS 14SPSS 14thth
Annual ConferenceAnnual Conference
Geetha NadarajanGeetha Nadarajan
Maxis MobileMaxis Mobile
12 Nov 200812 Nov 2008
Eastin HotelEastin Hotel
>> Improving the performance of Telco
Churn Predictive Model
with SPSS & 6 Sigma

© 2008-9 SPSS Malaysia 2
Agenda
 Telco Churn Predictive Model Overview
 Six Sigma DMAIC Framework
 Improving the Performance of Churn
Predictive Model
 Summary

Telco Churn Predictive Model Overview
 Model Business Goal
 Identify potential voluntary churner
Voluntary churn is defined as customers/subscribers terminating their mobile
account with Maxis
• Inclusion Criteria:
Postpaid Consumer Only, Tenure with Maxis more than 3 months
 Behavioural Variables Used in the Model
 Contract Details & Customer Interaction
 Tenure
 No of complaints
 Rate Plans
 Overdue Amount
 Usage
 Outgoing calls behavior (e.g. Voice (min), SMS (count), GPRS (kb), IDD, IR)
 Outgoing calls usages (e.g Voice Usage, SMS usage (RM)
Source: Maxis Postpaid Model

CRISP – DM Methodology used for modelling
 Business Understanding
Project objectives and requirements understanding, Data mining problem
definition
 Data Understanding
Initial data collection and familiarization, Data quality problems
identification
 Data Preparation
Table, record and attribute selection, Data transformation and cleaning
 Modeling
Modeling techniques selection and application, Parameters calibration
 Evaluation
Business objectives & issues achievement evaluation
 Deployment
Result model deployment, Repeatable data mining process
implementation
Source: CRISP-DM, SPSSSource: CRISP-DM, SPSS

Model Build Structure
Churn
Next Month
Last 1
Month
Last 2
Month
Last 3
Month
HistoryHistoryHistory
Looking at past behaviour before subs actually churned!
This
Month
History
Apr-08 May-08 Jun-08 Jul-08 Aug-08
Model BuildModel Build
& Deploy& Deploy
StructureStructure
Sample DataSample Data
( 488K subs- 95%( 488K subs- 95%
Confidence Level, 0.12Confidence Level, 0.12
Confidence Interval)Confidence Interval)
Source: Maxis Postpaid Model

Overall Clementine Stream
2. Data Understanding
1. Business Understanding
3. Data Preparation
4. Modelling 5. Evaluation
For Illustration only

Model Results shows Log Regression Model
is most robust
Model Name Algorithm
Lift
Build
Lift Test
Month 1
Lift Test
Month 2
Lift Test
Month 3
Postpaid_Churn_Prediction_CH CHAID 5.5 4.2 4.3 5.2
Postpaid_Churn_Prediction_NN
Neural
Network
4.5 4.2 4.1 3.9
Postpaid_Churn_Prediction_RR
Log
Regression
5.5 4.6 4.6 5.7

PMML Scripts (part only) to deploy model at
SAS Warehouse
<?xml version="1.0" encoding="UTF-8" ?>
- <PMML xmlns="http://www.dmg.org/PMML-3_1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="3.1"
xsi:schemaLocation="http://www.dmg.org/PMML-3_1 pmml-3-1.xsd">
- <Header copyright="Copyright(c) SPSS Inc. 1989-2007. All rights reserved.">
<Application name="SPSS for Microsoft Windows Release 16.0" version="16.0" />
</Header>
- <DataDictionary numberOfFields="84">
- <DataField dataType="double" displayName="churn_flg" name="churn_flg"
optype="categorical">
<Extension extender="spss.com" name="format" value="5" />
<Extension extender="spss.com" name="width" value="8" />
<Extension extender="spss.com" name="decimals" value="0" />
<Value displayValue="0" property="valid" value="0" />
</DataField>
- <DataField dataType="double" displayName="Out_INTL_SMS_CNT_m3"
name="Out_INTL_SMS_CNT_m3" optype="categorical">
</DataField>
- <DataField dataType="double" displayName="barr_overdue_nonpay_p3m_count"
name="barr_overdue_nonpay_p3m_count" optype="categorical">
</DataField>
- <DataField dataType="double" displayName="contract_tenure_months"
name="contract_tenure_months" optype="continuous">
</DataField>
- <DataField dataType="double" displayName="Out_DOM_MOU_m2"
name="Out_DOM_MOU_m2" optype="continuous">
<Extension extender="spss.com" name="decimals" value="3" /> For Illustration only

Model Operability & Issues
 To maximize returns, models must be quickly deployed for use within current operational systems &
business processes. Delay in deployment leads to model being outdated leading to significantly low
returns on investment especially in volatile markets like the Telecommunication.
 Once the model is deployed, it scores the database on monthly basis.
 Within 3 months in operation, significant variation in the outputs were detected (excluding model decay
process) . These variations range from 20% to 50% difference in lift (apply) raising the following
questions:
• Did the customer change usage behaviour within a short span of time?
• Is the model in production faulty?
• Is an important data prep necessary for the model failed?
• Is there data integrity issue?
• Do we need to rebuild? Will the problem recur with the new model?
Why Did the ModelWhy Did the Model
Performance Vary inPerformance Vary in
Operation Mode?Operation Mode?
Model Name Algorithm Lift Build
Lift test
Month 1
Lift Test
Month 2
Lift Test
Month 3
Operation
Month 1
Operation
Month 2
Postpaid_Churn_Prediction_RR Log Regression 5.5 4.6 4.6 5.7 3.3 3.0

Why Six Sigma?
 There is a gap between the current model and
expected performance.
 The causes of the model performance problem is not
clearly understood
 The solution is not predetermined
Using Six Sigma framework to reduce
variability in churn operational model!!
Six Sigma is proven process improvement in the belowSix Sigma is proven process improvement in the below
situations:situations:
Source: Six Sigma Handbook, Maxis Comm

What is Six Sigma?
 The symbol ‘ ∂ ‘ is taken from the Greek letter. It is
used in statistic as a measurement of variation.
 Six Sigma methodology emphasizes the improvement
of a process towards the purpose of reducing
variability.
 Six ∂ capability means only 3.4 defects will occur per
million parts.
 Invented at Motorola and popularized by Jack Welch,
General Electric

Six Sigma: An Overview of DMAIC
D M A I C
Define
(What’s
Important?)
Measure
(How are we
doing?)
Analyze
(What’s
Wrong?)
Improve
(What Needs
to be done?)
Control
(How do we
guarantee
performance?)
•Define
Project’s
Scope and
Scope.
•Do
Requirement
Gathering
•Gathering
Information on
Current
Situation
•Identify Root
Causes and
Confirm them
with data
•Develop, pilot
•Implement
Solutions
•Evaluate the
solutions
•Maintain
controls
•Standardize
and document
work methods
This churn model improvement adopts only aThis churn model improvement adopts only a
portion of the Six Sigma frameworkportion of the Six Sigma framework

Define Scope & Metric
Current Expected
Problem
Statement
Project Scope
Project Metric 50% average
variability from
Apply Lift Month 1
of Operational
Churn Model
(within 3 months of
implementation)
30% average
variability from
Apply Lift Month 1
of Operational
Churn Model
(within 3 months of
implementation)
D M A I C

D M A I C
Primary Metric (Y):
Y : Model Apply Lift ( at top 5th
percentile)
Measure : Project Metric
Apply Lift at 5th
percentile = Voluntary Churn Rate at 5th
percentile / Base Voluntary Churn Rate
Formulae
Note: The above applies to churn predictive models in operational mode only

What is Lift?
Lift@5th
percentile = Churn Rate at top 5% of base / Base Churn Rate
= 10% / 1%
= 10
Target: 1000 customers
Churners: 10 subscribers
Churn rate: 1%
Base
Base Approach
Target: 50 customers (top 5%)
Churners: 5 subscribers
Churn rate: 10%
Segment
Analytic Approach
Percentage of Population
Lift Chart
Churn rate in
top 5% of list
= 10%
5% 10% 15% 20%
D M A I C

A simple process map was built to identify
key inputs and outputs
D M A I C
Input High Level Process Map Outputs
Model Lift @ 5th percentile
Test Dataset Score List
New Dataset Score List Generation Time
PMML Script
Churn Model Improvement
Input High Level Process Map Outputs
Test Dataset Score List
New Dataset Score List Generation Time
PMML Script
Input u/c Detailed Map Outputs
Test Dataset PMML Script
Score List
New Dataset Score List Generation Time Sheet
PMML Script
Server Capacity
Score List Lift @5th percentile Report
Score List generation Time Sheet Timeliness Report
Churn Model Improvement
Evaluate Model Performance
Deploy Model to operations
Validate model in operation
High LevelHigh Level
Detailed MappingDetailed Mapping

Key inputs were then rated in importance to
primary metric
Rating of Importance 9 3 3 Ranking
Key Requirement
Lift @ 5th
percentile Score List
Score List
Generation Time Total
Process Step Process Input
Evaluate Model Performance Model 9 1 1 87 4
Evaluate Model Performance Test Dataset 3 0 0 27 5
Deploy Model to operations New Dataset 9 9 1 111 2
Deploy Model to operations PMML Script 9 9 9 135 1
Validate model in operation Score List 3 0 3 36 3
Cause & Effect MatrixCause & Effect Matrix
In view of time constraints, aIn view of time constraints, a
decision to focus on 3 keydecision to focus on 3 key
inputs is madeinputs is made
C & E Pareto
0
20
40
60
80
100
120
140
160
PMML Script New Dataset Model Score List Test Dataset Score List
generation
Time Sheet
Deploy Model
to operations
Deploy Model
to operations
Evaluate
Model
Performance
Validate
model in
operation
Evaluate
Model
Performance
Validate
model in
operation
Process Steps & Key Inputs
C&ERating
D M A I C

FMEA Analysis helps identify all possible
shortcomings & potential improvement
opportunities for the key input
Process Step Key Process Input
Potential
Failure Mode
Failure
Effect
S
E
V
Potential
Cause
O
C
C
Current
Controls
D
E
T
R
P
N
What is the Step? What is the Input?
What can go
wrong with the
Input?
What is the
effect on
the Output?
HowBad?
What are the
Causes?
Howoften?
How are
these found
or
prevented?
HowWell?
RiskPriority
Fill-in step 1 2 5 3 6 4 7 8
Deploy Model to operations PMML Script Wrong Script
Low apply
lift
9
Missed/Wrong
Data Prep
1 No Control 9 81
Wrong Script
Low apply
lift
9
Wrong
/Erronous
Score/New
Dataset
5 No Control 9 405
Process
Risk Priority Number = Severity x Occurrence x DetectionRisk Priority Number = Severity x Occurrence x Detection
1.Severity = Importance of effect on primary metric 1=None 10=Very Severe1.Severity = Importance of effect on primary metric 1=None 10=Very Severe
2.Occurrence = Frequency the cause of the effect takes place 1=Not Likely to Occur 10=Very Likely To Occur2.Occurrence = Frequency the cause of the effect takes place 1=Not Likely to Occur 10=Very Likely To Occur
3.Detection = Ability of current control to detect / prevent the occurrence of the cause 1=Likely to Detect 10 = Not Likely to detect3.Detection = Ability of current control to detect / prevent the occurrence of the cause 1=Likely to Detect 10 = Not Likely to detect
D M A I C

Multivari Plan helps to statistically confirm
hypothesis from FMEA
Deploy Model to
operations
Process Input
(X)
PMML Script
Potential
Failure Mode
Wrong Script
Potential Cause
Missed/Wrong Data Prep
Y
Apply Lift @5th perrcentile
Categorical- Continuous
Does wrong/missed data
prep step significantly
affect apply lift@5th
percentile?
Apply lift@5th percentile is
not significantly different
between correct/wrong
data prep step
Apply Lift@5th percentile
of operational model, Data
log
ISD
CI Folder
Graphical Box Plot
Statistical T-Test
Matthew
12th December 08
TBD
Analysis Planned
Person Resp.
Date Due
Result
Theory (Ho)
Data need to collect
Data Source
Recording/Storage Method
Process Step
Relation
Type
Question-whatr
wetryingtofind
out(Ho)
Type of Hypothesis Testing StatsType of Hypothesis Testing Stats
•Independent Samples T –TestIndependent Samples T –Test
•Chi Square TestChi Square Test
•One Way ANOVAOne Way ANOVA
Graphical DisplayGraphical Display
•Box PlotsBox Plots
•HistogramHistogram
•Scatter PlotScatter Plot
D M A I C
Source: Six Sigma
Handbook, Maxis
Comm
SPSS ProceduresSPSS Procedures

Improvement: UAT for PMML Script Testing
Improvement Plan Key X
X1: PMML Script-
Wrong/Missed
Data Prep Step
Measure
1. Test Run PMML script in build
environment & record lift
•Compliance
(Yes/No)
•Lift @5th
percentile
2. Draw a sample of new dataset and test run
the script in build environment & record time
taken & lift
•Compliance
(Yes/No)
•Lift @5th
percentile
•Time take to
complete scoring
3. Test Run the script in operational server
with entire new dataset and record time
taken & lift
•Compliance
(Yes/No)
•Lift @5th
percentile
•Time take to
complete scoring
3. Summary Report on Time & Lift •Compliance
(Yes/No)
•Lift @5th
percentile
•Time take to
complete scoring
D M A I C
This Improvement plan needs to be implemented over a time period and results trackedThis Improvement plan needs to be implemented over a time period and results tracked

Control Plan : PMML Script
D M A I C
Process Step Critical X Y
(relative to the step )
Process Step or
X Owner
Specification/Go
al
What is the
Control
Method?
How will the X
be Measured?
Frequency of
Measurement
Who
Responsible?
(measurement)
Reaction
plan
Who is
Responsible?
(re-action plan)
Document
Storage
related to this
X
Who Audit
(Optional)
Frequency of
Audit
(Optional)
An action step
from a process
map ( Is map or
Should map )
which step of the
process is being
controlled?
Critical X`s you found
in the DMAIC
process.
Which X's need to be
controlled ?
Deliverables and
Performance
variables.
The Y from your
DMAIC process.
Who owns this
step of the
process?
What is the goal
for your X?
Is there an
acceptable range
for your X?
Provide capability
Indices if
applicable.
What are you
going to do to
control the X?
How will it be
measured?
How do we
measure the X in
order to know if
the control is
working?
When or how often
will you measure
the X?
Who measures
the X?
What will
you do if the
X is out of
control?
Who is
responsible for
implementing
the Reaction
Plan?
Where are all
the documents
relating to this
X stored?
When or how
often will audit
be conducted?
Deploy Model
to operations
PMML Script-
Wrong/Missed
Data prep step
Apply Lift @5th
percentile
ISD/CVM
Each PMML
script needs to
be run in UAT
mode (in
operations)
for 2 months &
results tracked
before certified
Fill in UAT
Log
Error/ No
Error
Monthly - Batch
mode
ISD
Delay/Sto
p use of
churn
score till
corrected
ISD CI Folder CI Quarterly
T
T
C
C
T
T
T
C
T
T
C
C
C
T
T
T
T
C
T
High Variability/ Noise Low Variability/ Noise

Improved operationalization of churn models is expected
to generate lower variability in model performance
which leads to an incremental revenue
D M A I C
•Gross Incremental Revenue (Baseline) vs Gross Incremental Revenue (Final Cap)
•Variability in apply lift ( Baseline) vs Variability in Apply Lift ( Final Cap)
Baseline Covers operational churn model performance before project
Improve Covers operational churn model performance during improvement phase
Final Cap Covers operational churn model performance after project
Baseline : Pre Project Phase
Improve : Pilot Implementation Phase
Final Cap : Post Project Phase
Results Tracking MetricResults Tracking Metric

Summary
 Churn models in operation mode needs
monitoring and control to sustain performance
irregardless of build performance
 Six Sigma framework helps establish a
process improvement for operational models
 Hypothesis testing for Six Sigma is easily done
in SPSS Base

Improving the performance of Telco Churn Predictive Model with SPSS & 6 Sigma

Recommended

Recommended

More Related Content

Similar to Improving the performance of Telco Churn Predictive Model with SPSS & 6 Sigma

Similar to Improving the performance of Telco Churn Predictive Model with SPSS & 6 Sigma (20)

Recently uploaded

Recently uploaded (20)

Improving the performance of Telco Churn Predictive Model with SPSS & 6 Sigma

Editor's Notes