This document summarizes research on using machine learning algorithms to classify potential donors for fundraising campaigns. The researchers built a binary classification model using neural networks to identify likely donors. They found that a neural network approach had the lowest false positive rate compared to other models. Testing different thresholds, they determined that a threshold of -0.1 achieved the most cost-effective balance between identifying donors and minimizing mailing costs.

1. Accurate Campaign Targeting Using Classification
Algorithms
CS 229 Project: Final Report
Jieming Wei Sharon Zhang
Introduction
Contact
<your name>
<your organization>
Email:
Website:
Phone:
Methods
misclassification errors with different nets
misclassification errors with different decay level
1.
2.
3.
4.
5.
6.
7.
8.
9.
Many organizations prospect for loyal supporters and
donors by sending direct mail appeals. This is an
effective way to build a large base, but can be very
expensive and have a low efficiency. Eliminating likely
non-donors is the key to running an efficient
Prospecting program and ultimately to pursuing the
mission of the organization, especially when there’s
limited funding (which is common in most campaigns).
We built a binary classification model to help
organizations to build their donor bases in the most
efficient way possible. We aimed to help them separate
the likely donors from the non-likely donors and target
at the most receptive donors by sending direct mail
campaigns to the likely donors.
We choose to use Neural Network for this experiment,
since it has the lowest false positive rate. There are three
steps – first choose the number of neural nets and the
threshold for the structural model, then choose the optimal
regularization parameter under this optimized structural
model, and lastly choose the threshold for false positive
minimization.
We did a looping experiment with 10 nets, with 20 threshold
values ranging from -0.2 to 0.2. Applying averages to rows
and columns, we choose the lowest average in both
column and row to be the number of nets and threshold
value. We chose optimal threshold of -0.14, 8 layers, decay
parameter of 0.5.
Data Preparation
We tried the following approaches to classify mailings into
two classes, namely donors who are likely to donate and
those who are not: Multivariate Regression, SVM, Decision
Tree, Neural Network, and Random Forest. Training errors ,
test errors, false positive error rates, false negative error
rates are calculated for each model.
We noticed that test errors for most models are relatively
high, with MVR and Random Forest models having high
variance problems, and SVM having high bias problems.
False positive errors are high for most models, with Neural
Net model having the least false positive error. We want to
minimize false positive error so as to save costs for
campaign companies.
To find a threshold between false positice and false
negative errors, we use the following formula:
Net Collected $=Donation $*Donors M-Package Cost
$*Packages Sent (N)
Where Donation ($) and Package Cost($) are assumptions,
Donors (M) corresponds to the number of positive
occurrences, and Packages Sent (N) corresponds to the
sum of positive and false-positive occurrences since the
model predicts those prospects will donate. When we use
the assumptions Donation = $50 and Package Cost = $20,
we observe the following Net Collected amount trend in
chart #.
Thus, the most cost-effective threshold is r=-0.1, with false-positive
rate 18%.
Result Savings
The original dataset is 13-dimensional with 1048575
observations on demographic features of prospects and
campaign attributes. In order to serve the purpose of this
study, we adapted the dataset, reducing it to a 10-
dimensional dataset to keep the relevant features and
selected training set and test set.
Feature selection
We selected features based on their relevance to the result
of the campaign, estimated by the correlation of a given
feature to the result. We selected 7 features with relatively
large correlation with the result.
Neural Nets Algorithm
Chart 1.
REPLACE THIS BOX
WITH YOUR
ORGANIZATION’S
HIGH RESOLUTION
LOGO
REPLACE THIS BOX
WITH YOUR
ORGANIZATION’S
HIGH RESOLUTION
LOGO
Training
error
Overall
test error
False
positive
False
negative
MVR 0 0.31 0.25 0.06
SVM 0.3233 0.54 0.18 0.36
Neural
Nets
-- 0.3787 0.1742 0.2045
Random
Forest
0.0033 0.47 0.23 0.24
company
type
project
type
cause1 cause2 phase zip donor
income
result
-0.016 -0.0008 -0.0765 -0.0416 0.0149 0.071 0.248 1
Company Type Histogram
company type
Frequency
1 2 3 4 5 6
0 20 40 60 80 100
Cause1 Histogram
candidate/non-profit
Frequency
1.0 1.2 1.4 1.6 1.8 2.0
0 50 100 150 200
Project Type Histogram
project type
Frequency
0 2 4 6 8 10
0 10 30 50
Cause2 Histogram
causes
Frequency
2 4 6 8 10
0 20 40 60 80 100
Phase Histogram
phase
Frequency
0 5 10 15 20
0 20 40 60 80 100
Income Histogram
natural log of income
Frequency
10 11 12 13 14 15 16 17
0 10 20 30 40 50 60
-0.2 -0.1 0.0 0.1
40 45 50 55 60
misclassification errors with different thresholds
thresholds
percentage of misclassification error
threshold selected
2 4 6 8 10
44 46 48 50
nets
percentage of misclassification error
nets selected
0.0 0.2 0.4 0.6 0.8 1.0
38 40 42 44
decay
percentage of misclassification error
decay level selected
70
60
50
40
30
20
10
100
80
60
40
20
0
-20
-40
-60
Classification Results with Different Threshold Values
r=-0.3, r=-0.28, r=-0.26, r=-0.24, r=-0.22, r=-0.2, r=-0.18, r=-0.16, r=-0.14, r=-0.12, r=-0.1, r=-0.08, r=-0.06, r=-0.04, r=-0.02, r=0
Net Collected $ with Different Threshold Values
0
2
7
10 10 11
13 14
17 17 18 19
21 22 23 23
0
r=-0.3, r=-0.28, r=-0.26, r=-0.24, r=-0.22, r=-0.2, r=-0.18, r=-0.16, r=-0.14, r=-0.12, r=-0.1, r=-0.08, r=-0.06, r=-0.04, r=-0.02, r=0
positive false-negative false-positive negative