Machine Learning:  Charity Donor Analysis

Introduction

A charitable organization wishes to develop a machine learning model to improve the cost effectiveness of their direct marketing campaigns to previous donors. The recent mailing records reflect an overall 10% response rate with an average donation of $14.50. The cost to produce and send each mail is $2. The data consists of 3,984 training observations, 2,018 validation observations and 2,007 test observations. Weighted sampling has been used which over-represents the responders so that the training and validation samples have approximately equal numbers of donors and non-donors. Additionally, a prediction model to predict the expected gift amounts from donors should also be built based on the records of donors only.

Analysis

Exploratory Data Analysis

Based on R’s describe function, there are 22 variables and 8,009 observations in the initial dataset with no missing data. The observations reflect prior mailing data including the whether the mailing was successful and a donation was received, the amount of the donation along with numerous other aspects that may or may not be helpful in determining who to send future donation requests to in order to maximize the amount of donations. Categorical and numerical variables along with a brief description and statistical information are shown below.

Categorical Variables:

Numerical Variables:

Machine Learning: Number of Children

Machine Learning: Donation Times

Machine Learning: Wealth Rating Bar Chart

Machine Learning: Gender Bar Chart

Machine Learning: 5 variables

Machine Learning: Kernel Density of AVHV

 Correlations

Reviewing the correlation between variables and donation amount (damt) some of the larger correlations are as follows:

Logistic Regression

Some of the significant variables are reflected below but additional insight into which variables are more significant based on the model will be provided in the discussion of the related model.

Classification Modeling

Goal:  Maximize the gross margin (expected revenue less the costs) on the marketing campaign

Utilizing the data, various machine learning modelling techniques were applied to determine the number of households to receive the marketing materials that are most likely to be a donor.  The outcome of each technique results in a confusion matrix that is used to determine the percentage of mailers that are successful versus those that do not ultimately result in a donation.  For example, in Quadratic Discriminant Analysis (QDA) the resulting confusion matrix is:

The matrix reflects those campaign mailing targets that are likely to donate.  Unfortunately, we won’t always be correct and thus, will expend funds without a return.  Targeting 1,469 households that are thought to be donors will only result with 976 donations. On the flipside, a cost savings of 549 mailouts will be achieved but not sending the campaign to targets who will not provide a donation.  Thus, accuracy is important by correctly knowing who is or who is not a donor and is based on correctly determining those are not a donor and do not donate and those who are a donor and do donate is valuable and is calculated in as (526+976)/(549+1469).  Thus, our accuracy is 74.43%.

The second part will deal with the size of the donation but for now, we are using past experience which reflects an average donation of $14.50 and a cost of $2.00 each.  Thus, our margin is ($14.50 * 976) – ($2.00 * 1,469) = $11,214.  It is easy to understand why it is important to mail to those donors who are most likely to provide a donation.

A summary by each machine learning method is as follows:

Logistic Regression

Logistic regression models the probability of a binary response based on one or more predictor variables.  For instance, whether someone will donate to the charity or will not donate.

Log models were created based on earlier models.  The first model reflecting 21 variables and the second reflecting 9 variables that were selected based on their significance.  The plot reflects the log function based on the profit.

Machine Learning: Logistic Regression

Variables Log2:  chld + home + reg2 + I(hinc^2) + wrat + tdon + incm + tlag + tgif

Linear Discriminant Analysis (LDA)

LDA uses as Bayes classifier and a threshold of the posterior probability to assign observations to a default class by plugging in estimates for several parameters including the weighted average of the sample variances for each class and the number of observations.  The first LDA model uses 21 variables including a transformation on hinc.  LDA2 uses a subset of the most significant variables.

Variables in LDA2:  reg1, reg2, reg3, chld, home, I(hinc^2), wrat, npro, tdon, tlag

Quadratic Discriminant Analysis (QDA)

Quadratic Discriminant Analysis is similar to LDA as it assumes that the observations from each class are drawn from a Gaussian distribution creates a prediction based on plugging in parameter estimates into Bayes’ theorem.  However, QDA uses a quadratic function and creates its own covariance matrix unlike LDA making QDA more flexible with a lower variance.

In each of the QDA models, a different set of variables were used with QDA1 using nine variables, QDA using all 21 variables and QDA3 using seven variables.

Variables:  QDA1:  reg1, reg2, home, chld, I(hinc^2), wrat, npro, tdon, tlag;     QDA3:  chld, home, wrat, hinc, reg2, tdon, incm

K=Nearest Neighbors (KNN)

KNN assigns a weighting so the nearer ‘neighbors’ contribute more to the average than observations further away.  A small K value will provide the most flexible fit and have a low bias but high variance due to the prediction relying on one observation and vice versa, a larger K value will provide a smoother and less variable fit but may cause bias by masking some of the structure.

Generalized Additive Models (GAM)

GAM models go beyond linear models by allowing non-linear functions of each variable while being additive, meaning that the effect of changes in a predictor on the response is independent of the values of the other predictors.  Thus, one of the assumptions is that interaction amongst various marketing activities are not creating some type of synergy whereby we need to consider an interaction term for example, when there is a mailing campaign combined with radio, television or other marketing activities.  GAM uses smoothing splines via backfitting which allows multiple predictors by repeatedly updating the fit for each predictor.

Machine Learning: Decision Trees Nodes

Variables: reg1, reg2, home, chld, I(hinc^2), wrat, npro, tdon , tlag , tdon

Decision Trees

Decision trees can be applied to both regression and classification problems and are commonly used for their ease of interpretability.  The top of the tree is split into branches based of the best split at that particular junction without looking forward to determine if a more optimized split should be completed which is considered to be a ‘greedy’ approach.  At each split the predictor space is split based on the greatest possible reduction in the RSS.  As the training set is likely to create a too complex tree and overfit the data, an alternative is to build the tree as long as the decrease in the RSS exceeds a threshold.

Various decision trees were also modeled including 16, 9 and 5 terminal nodes.  The results are as follows:

Machine Learning: Decision Trees

Bagging

Bagging focuses on reducing the variance of a statistical learning method such as decision trees.  Bagging can reduce the high variance and low bias of decision trees by combining hundreds or thousands of trees into one procedure by taking repeated samples from the training set and averaging all of the predictions.

Random Forest

Improving bagging by tweaking the splits in the decision tree is the basis for Random Forests.  Based on a random sample of predictors is chosen from the full set of predictors with a fresh sample of predictors being taken at each split and thus, makes the average of the resulting trees less variable and more reliable.

Additionally, the Importance function in R, provides a list of the most important variables.  As shown below, this list was used in the choice of variables in each of the Random Forest Models as reflected with the ‘mtry’ selection.  For instance, mtry=2 uses the two best variables, chld and home.

Machine Learning: Random Forest

Neural Network

A neural network mimics the learning pattern of natural biological neural networks.  They begin with a single perceptron that receives inputs, applies a weighting then passes into a activation function to produce an output which is then layered to create a network.  Hidden layers are the layers between the input and output layers where you cannot see the inputs or outputs (Portilla, Jose. 2016.   A Beginner’s Guide to Neural Networks with R!  Retrieved from kdnuggets.com).

Machine Learning: Neural Network 10 10

Machine Learning: Neural Network 10 10 50

Support Vector Machine

Support vector machines are intended for binary classifications such as donr and when there are non-linear class boundaries which is addresses by enlarging the feature space using polynomial functions as predictors using kernels.  A kernel is a function that quantifies the similarity of two observations.  A radial kernel is effected by nearby training observations but there are also other types including linear, polynomial and sigmoid.

Prediction Modeling

Goal:  Minimize the prediction error based on the donation amount

Quantify the extent to which the predicted response value for a given observation is close to the true response value for that observation.  This is measured using the mean squared error (MSE).  The MSE will be small if the predicted responses are close to the true responses.

Least Squares Regression

Least squares regression measures closeness by minimizing the sum of squared errors by minimizing the residuals

Three models using PLS regression with LS1 using all of the variables, LS2 using all except genf and wrat but added the transformed hinc2 and LS3 used the most significant factors as determined by the Random Forest model.

Variables: LS3 chld, hinc, reg2, home, wrat, tdon, tgif, incm, npro, tlag, avhv, inca, plow, agif, lgif, rgif, reg1, reg4, reg3).

Principal Component Regression

The principal Component Regression reflects the trade between the number of components that explain most of the variability in the data that is fit using least squares.

Training Variance Explained by the number of components is shown below which reflects the percentage of variance explained by the number of different components i.e. using half of the components explains 77.67% of all variance.  The plot also reflects ‘elbows’ where the variance is:

Machine Learning: Principal Components Regression

The table above reflects standardized predictors with a ten-fold cross-validation error for each possible value.

Based on the MPE, a PCR with 15 components results in the lowest MPE.

Partial Least Squares (PLS) Regression

Partial Lease Squares is a dimension reduction method that first identifies a new set of features that are linear combination of the original features and then using least squares fits a linear model.  It considers the predictor variables that are most strongly related to the response variable.

Best Subset Selection with K-fold Cross-Validation

Best subset selection fits a separate least squares regression to each possible combination of the predictors and chooses the best model based on the smallest RSS.

Results in a ten-variable model consisting of: reg3, reg4, home, chld, hinc, incm, plow, npro, rgif, agif.

The best subset selection plot, as shown below, reflects the top row with all of the variables according to the optimal model.  The optimal models are based on four different statistical methods, R-squared, Adjusted r-squared, Mallow’s Cp and Bayesian Information Criterion (BIC).

Machine Learning: Best Subset Selection

Ridge Regression

Expanded the grid from  to cover a whole range of scenarios. The variables are standardized by default.  Using the best function, the best value of lambda is applied using cross-validation.

The Lasso

As ridge regression will always build a model with all of the predictors, Lasso overcomes this by performing variable selection that lead to the smallest RSS subject to a set constraint.  Thus, for every value of   there is a lasso coefficient estimate. It is closely related to the best subset selection.

Fitting a lasso model, we can see that depending on the choice of the tuning parameter, some of the coefficients will be exactly zero.

Lasso

Results

The best models for classification and prediction based are addressed in two separate parts.

Classification

As reflected above in the ten classification models (Logistic Regression, GAM, LDA, Log, Decision Trees, QDA, Bagging, Random Forest, KNN, Neural Network and Support Vector Machine), the results on the training and test / validation data provided different results in determining how to maximize the gross margin on the marketing campaign.

Based on the accuracy on determine which donors are most likely to donate, the top three models are:

• K-Nearest Neighbors with k=7 resulting in 90.58% accuracy on the test data
• Bagging using 6 variables at each split resulting in 89.54% accuracy on the test data
• Random Forest with 5 variables at each split resulting in 88.80% accuracy on the test data

The best classification models based on Gross Margin are as follows:

• K-Nearest Neighbors with k= 9 resulting in $11,852.00
• Logarithmic resulting in $11,642.50
• Linear Discriminant Analysis resulting in $11,624.50

For future data collection, some variables have proven to be more important than other variables which may assist in reducing costs and /or replacing with other variables that may be more beneficial to the marketing efforts.  For example, Random Forest determined that the number of children a donor has and whether they are a home owner were important in determining if they would donate.  Additionally, while most of the regions did not play a huge role in determining whether a donor would donate, region 2 was more important than the other three regional variables included in the data set.

Prediction

As reflected above in the six prediction models (Partial Least Squares, Principal Component Regression, Best Subset, Ridge and Lasso), the Mean Prediction Error (MPE) and Standard Error (SE) on the different data sets provided different results in determining the amount of the donation.  The top three prediction models were:

• Best Subset with a MPE of 1.8126 and a SE of .168675
• Partial Least Squares with 3 with a MPE of 1.8150 and a SE of .1701451
• Lasso with a MPE of 1.86133 and a SE of .1694185

In order to maximize the value of the list of people who have donated to the charity we need to determine the two parts of the equation a) who will donate and b) how much.  This is accomplished by applying various models to determine which model predicts who will donate which is reflected in the gross margin value of the model and the amount of the donation using the mean prediction error.  Based on the above, the best models to use are a) K-Nearest Neighbors with k=7 to determine who will donate and b) the best model for the donation amount which is the Best Subset Selection based on ten variables and also has the lowest standard error which tells us the average amount of the differences from the actual values.

Machine Learning:  Charity Donor Analysis Conclusions

Utilizing the various training, validation and test datasets has provided insight into the type of data that is collected by Charity Inc. and how it can be utilized to benefit the company’s marketing campaign.  The exploratory data analysis reflected the different types of data that were contained within the datasets along with their statistical information and plots for ease of reference.

Next, a variety of machine learning and regular models were produced for both classification and prediction modeling.  The majority of the models detailed various iterations along with the number of mailouts, profit, confusion matrix and accuracy ratio.  While there are numerous other models that could also be included along with additional iterations and transformations of variables this version has tried to remain easily interpreted.

Based on the analysis, the K-Nearest Neighbor is the best model for prediction error while the Best Subset Selection model is best for determining which people on the mailing list are most likely to donate.  Based on the test data, the marketing campaign should be focused on 354 donors who will on average donate $14.50 for a total predicted donation amount of $5,132.  Additionally, through the modelling efforts contained within, Charity Inc. can be more confident in determining who should be targeted to maximize revenues while minimizes costs.

In future iterations, segregating donors based on their projected contributions could be helpful in customizing mailout and mailout materials (and related costs) to future maximize the charities marketing efforts and resulting donations.