The document summarizes a data mining exercise to predict customer interest in caravan insurance. Key steps included understanding the objective, data, and models. First level models on the original data achieved 94% accuracy. Further levels balancing data and adding variables saw no clear improvement. The best model was the original with 94% accuracy, showing additional changes did not enhance predictions.

1. Caravan Insurance Data
Mining Assignment
K6225 Knowledge Discovery and Data Mining
By,
Sesagiri Raamkumar Aravind(G1101761F)
Thangavelu Muthu Kumaar(G1101765E)
Page 1 of 11

2. Table of Contents
1.0 Objective ........................................................................................................................................... 3
2.0 Summary of Final Results .................................................................................................................. 3
3.0 Exercise Lifecycle............................................................................................................................... 3
3.1 Understanding the objective of the exercise and its expectations .............................................. 4
3.2 Understanding the data dictionary of the data set ...................................................................... 4
3.3 Assigning appropriate measure values (Set/Range) for data fields.............................................. 4
3.4 Constructing first level models with Training dataset .................................................................. 4
3.4.1 Logistic Regression ................................................................................................................. 4
3.4.2 Decision Trees ........................................................................................................................ 5
3.11.3 Neural Networks .................................................................................................................. 5
3.5 Running the first level Models with Test data .............................................................................. 6
3.6 Performing bivariate analysis on training dataset ........................................................................ 7
3.7 Creating interaction variables based on results of Step 5 ............................................................ 7
3.8 Balancing the training data ........................................................................................................... 8
3.9 Constructing second level models with Training dataset ............................................................. 9
3.10 Running the second level Models with Test dataset .................................................................. 9
3.11 Constructing third level models by adding new interaction variables ..................................... 10
3.12 Running the third level models with Test dataset .................................................................... 10
3.13 Final Results Interpretation ...................................................................................................... 11
Page 2 of 11

3. 1.0 Objective
The objective of this data mining exercise is to find the best possible model to predict whether
customer signature will opt for caravan insurance (mobile home policy) or not. The techniques used
are logistic regression, decision tree and neural network.
2.0 Summary of Final Results
The model built using Logistic Regression and Decision Tree came out with the highest accuracy on
comparison with the models built using Neural Network. The best model had an accuracy of 94%.
The most interesting part of the exercise is that base model (as provided originally) without any
interaction variables and balancing, gave the best results. It has been expectedly observed that most
models had higher accuracy with training data set but the accuracy rate reduced when run with test
dataset .Cross-validation techniques such as 10-step validation was not done in this exercise which
could have delineated the results even more.
3.0 Exercise Lifecycle
The lifecycle of the complete data mining exercise comprises of the following steps:-
1. Understanding the objective of the exercise and its expectations
2. Understanding the data dictionary of the data set
3. Assigning appropriate measure values (Set/Range) for data fields
4. Constructing first level Models with Training dataset
5. Running the first level Models with Test dataset
6. Performing bivariate analysis
7. Balancing the training data
8. Constructing second level Models with Training dataset
9. Dataset modification of Training dataset
10. Running the second level Models with Test dataset
11. Creating interaction variables based on results of Step 6
12. Constructing third level Models with Training dataset
13. Running the third level Models with Test dataset
14. Final Results Interpretation
Page 3 of 11

4. 3.1 Understanding the objective of the exercise and its
expectations
The first and foremost step in a data mining exercise is to understand the business objective. The
business wants to use their existing customer signatures to build a predictive model for predicting the
number of mobile home policies. The model construction and its inference will be a precursor for a
potential marketing campaign to target specific customer groups. The data mining techniques that are
in the scope of this exercise are logistic regression, decision trees and neural networks.
3.2 Understanding the data dictionary of the data set
The data dictionary consists of 86 variables with an equal mix of socio-demographic and product
ownership data. There are few ordinal variables that need to be changed to numeric variables for build
efficiency. The socio-demographic variables are captured at zip-code level.
3.3 Assigning appropriate measure values (Set/Range) for data
fields
The measure of the below variables were manually changed to ‘Range’ in Clementine, apart from the
automatically assigned measures:-
MAANTHUI Number of houses
MGEMOMV Avg size household
MGEMLEEF Avg age
MGODRK Roman catholic
PWAPART Contribution private third party insurance
There is an academic insight that socio-demographic variables are to be converted to ‘Range’
variables so that it would be convenient to plot the values in logistic regression graph curve. The
authors retained the variables as ‘Set’ variables initially to test the postulation at a later stage.
3.4 Constructing first level models with Training dataset
The authors made a plan of arriving at the best model by using a three level approach. The models
built in first level will be crude models constructed on the data set directly without any new
interaction variables or data balancing. These models will be the first benchmark to gauge subsequent
improvements. Models were built using the Logistic, C5.0 and Neural Net nodes.
3.4.1 Logistic Regression
No changes were done for the Logistic Regression as all attributes were seemingly optimal.
Page 4 of 11

5. 3.4.2 Decision Trees
The changes done for C5 node under the Export mode are
Pruning Severity was set to 5
‘Minimum records per child branch’ was changed to 5 from 2 as it was found to be optimal number.
Value 1 impaired the results and the same could be said for values greater than 2
‘Use Boosting’ option was enabled so that more classifiers are created. The value was set to15 for
first level and changed to 5 for second and third level.
Fig 7: C5 Model Attributes
3.4.3 Neural Networks
For the Neural networks, the RBFN method was selected first but the model did not produce better
results. The final method selected was ‘Quick’. The number of hidden layers was set as 3 so that more
transformations can take place. The learning rates were initially increased marginally to check for
performance improvements assuming that the results are converging towards the globally consistent
depression in the learning curve of the networks. But as marginal increase of alpha learning rate didn’t
get produce significant results, it was increased dramatically to 0.9 for overcoming the possibly
assumed local depression. The final values are available in the screenshot below.
Page 5 of 11

6. Fig 8: Neural Network Attributes
3.5 Running the first level Models with Test data
The trained first level models were run with the test dataset and the results of the different modelling
techniques were compared with the Analysis and Evaluation node. Logistic Regression and Decision
Tree both had the best accuracy rate of 94%. The Nagelkerke Rsquare value with training data set was
16.7%. These results will be maintained as the first level benchmark. Screenshots provided below
Fig1: First Level Models Analysis Node Results
Page 6 of 11

7. Fig 2: First Level Gain and Lift Chart
3.6 Performing bivariate analysis on training dataset
This step marks the start of the second level model building process. Bivariate analysis in Clementine
can be done using the Web node that represents the relationships between the values of variables
using thick and thin lines. The authors performed the analysis using both the normal web and directed
web option in the web node. The directed web had the target as Caravan variable and all the other
variables were put in dependant section. This analysis wasn’t helpful as the relationships were present
among different values in independent variables and CARAVAN therefore no significant inferences
were made. However, the normal web analysis indicated strong relationships between the customer
type and customer subtype, a potential candidate for interaction variable.
3.7 Creating interaction variables based on results of Step 5
The indication from last step was implemented in this step by creating two interaction variables. The
first interaction variable Derive1(aka customer lifestyle reflector) contains the parent variables
Customer Type and Subtype. The second interaction variable Derive3(aka Combined Age-Income
Factor ) contains the parent variables Avg age and Avg Income. This variable was created based on
the author’s intuition that it would help build a better model. Screenshots provided below for
reference
Page 7 of 11

8. Fig 3: Derived Variables
3.8 Balancing the training data
It has been noticed that the training dataset is not highly representative of positive cases
i.e.CARAVAN=1. Therefore, models constructed using this data set may not be the best predictor for
positive cases. Clementine provides a feature called as Balancing to create more signatures based on
conditions. The overall positivity is increased in the data set. The authors chose a factor of 6 to make
the dataset slightly better looking in terms of value share (72%:28%)
Fig 4: Balancing
Page 8 of 11

9. 3.9 Constructing second level models with Training dataset
The second level models were built with the balanced dataset. The attributes of the nodes were
maintained from the first level except for C5 node in which the boosting interval was changed to 5 as
the software did not have enough memory to run with value 15.
3.10 Running the second level Models with Test dataset
The trained second level models were run with the test dataset and the results of the different
modelling techniques were compared with the Analysis and Evaluation node. Decision Tree model
came out with the highest accuracy of 90.48%. These results were maintained as the second level
benchmark. Screenshots provided below
Fig 5: Second Level Models Analysis Node Results
Page 9 of 11

10. Fig 6: Second Level Models Gain and Lift Charts
3.11 Constructing third level models by adding new interaction
variables
The third level model building step is not the same as the second level in terms of data fields. The two
new interaction variables Derive 1and Derive 2 were created. No additional balancing was done.
3.12 Running the third level models with Test dataset
The trained third level models were run with the test dataset and the results of the different modelling
techniques were compared with the Analysis and Evaluation node. Neural Network model gives the
best accuracy rate at 90.1%.
Fig 9: Third Level Models Analysis Node Results
Page 10 of 11

11. Fig 10: Third Level Models Gain and Lift Charts
3.13 Final Results Interpretation
The below table compares the output of the Analysis node from all three levels. There is no marked
improvement in each level. It has been inferred that building the model after balancing the training
data set, doesn’t produce a better model.
In Level1 (base dataset): Highest accuracy is generated by both Decision Tree and Logistic
Regression
In Level2 (model build with balanced dataset): Highest accuracy is generated by Decision Tree
In Level 3(model build with balanced dataset and interaction variables): Highest accuracy is
generated by Neural Network
1st 2nd 3rd
Technique Factor level level level
Logistic
Regression Test dataset Accuracy 94.00% 87.50% 87.50%
Decision Tree Test dataset Accuracy 94.00% 90.48% 89.75%
Neural Network Test dataset Accuracy 92.05% 90.12% 90.10%
Combined Agreement with CARAVAN 94.52% 95.11% 94.80%
Table 1: Level Comparison
Page 11 of 11