Customer churn is a problem that all companies need to monitor, especially those that depend on subscription-based revenue streams. The simple fact is that most organizations have data that can be used to target these individuals and to understand the key drivers of churn, and we now have Keras for Deep Learning available in R (Yes, in R!!), which predicted customer churn with 82% accuracy. We’re super excited for this article because we are using the new keras package to produce an Artificial Neural Network (ANN) model on the IBM Watson Telco Customer Churn Data Set! As for most business problems, it’s equally important to explain what features drive the model, which is why we’ll use the lime package for explainability. We cross-checked the LIME results with a Correlation Analysis using the corrr package. We’re not done yet. In addition, we use three new packages to assist with Machine Learning (ML): recipes for preprocessing, rsample for sampling data and yardstick for model metrics. These are relatively new additions to CRAN developed by Max Kuhn at RStudio (creator of the caret package). It seems that R is quickly developing ML tools that rival Python. Good news if you’re interested in applying Deep Learning in R! We are so let’s get going!!
Customer Churn: Hurts Sales, Hurts Company
Customer churn refers to the situation when a customer ends their relationship with a company, and it’s a costly problem. Customers are the fuel that powers a business. Loss of customers impacts sales. Further, it’s much more difficult and costly to gain new customers than it is to retain existing customers. As a result, organizations need to focus on reducing customer churn.
The good news is that machine learning can help. For many businesses that offer subscription based services, it’s critical to both predict customer churn and explain what features relate to customer churn. Older techniques such as logistic regression can be less accurate than newer techniques such as deep learning, which is why we are going to show you how to model an ANN in R with the keras package.
Churn Modeling With Artificial Neural Networks (Keras)
Artificial Neural Networks (ANN) are now a staple within the sub-field of Machine Learning called Deep Learning. Deep learning algorithms can be vastly superior to traditional regression and classification methods (e.g. linear and logistic regression) because of the ability to model interactions between features that would otherwise go undetected. The challenge becomes explainability, which is often needed to support the business case. The good news is we get the best of both worlds with keras and lime.
IBM Watson Dataset (Where We Got The Data)
The dataset used for this tutorial is IBM Watson Telco Dataset. According to IBM, the business challenge is…
A telecommunications company [Telco] is concerned about the number of customers leaving their landline business for cable competitors. They need to understand who is leaving. Imagine that you’re an analyst at this company and you have to find out who is leaving and why.
The dataset includes information about:
Customers who left within the last month: The column is called Churn
Services that each customer has signed up for: phone, multiple lines, internet, online security, online backup, device protection, tech support, and streaming TV and movies
Customer account information: how long they’ve been a customer, contract, payment method, paperless billing, monthly charges, and total charges
Demographic info about customers: gender, age range, and if they have partners and dependents
Deep Learning With Keras (What We Did With The Data)
In this example we show you how to use keras to develop a sophisticated and highly accurate deep learning model in R. We walk you through the preprocessing steps, investing time into how to format the data for Keras. We inspect the various classification metrics, and show that an un-tuned ANN model can easily get 82% accuracy on the unseen data. Here’s the deep learning training history visualization.
We have some fun with preprocessing the data (yes, preprocessing can actually be fun and easy!). We use the new recipes package to simplify the preprocessing workflow.
We end by showing you how to explain the ANN with the lime package. Neural networks used to be frowned upon because of the “black box” nature meaning these sophisticated models (ANNs are highly accurate) are difficult to explain using traditional methods. Not no more with LIME! Here’s the feature importance visualization.
We also cross-checked the LIME results with a Correlation Analysis using the corrr package. Here’s the correlation visualization.
We even built an ML-Powered Interactive PowerBI Web Application with a Customer Scorecard to monitor customer churn risk and to make recommendations on how to improve customer health! Feel free to take it for a spin.
View in Full Screen Mode for best experience
Credits
We saw that just last week the same Telco customer churn dataset was used in the article, Predict Customer Churn – Logistic Regression, Decision Tree and Random Forest. We thought the article was excellent.
This article takes a different approach with Keras, LIME, Correlation Analysis, and a few other cutting edge packages. We encourage the readers to check out both articles because, although the problem is the same, both solutions are beneficial to those learning data science and advanced modeling.
Prerequisites
We use the following libraries in this tutorial:
keras: Library that ports Keras from Python enabling deep learning in R. Visit the documentation for more information.
lime: Used to explain the predictions of black box classifiers. Deep Learning falls into this category.
tidyquant: Loads the tidyverse (dplyr, ggplot2, etc) and has nice visualization functions with theme_tq(). Visit the tidyquant documentation and the tidyverse documentation for more information on the individual packages.
rsample: New package for generating resamples. Visit the documentation for more information.
recipes: New package for preprocessing machine learning data sets. Visit the documentation for more information.
yardstick: Tidy methods for measuring model performance. Visit the GitHub Page for more information.
corrr: Tidy methods for correlation. Visit the GitHub Page for more information.
Install the following packages with install.packages().
pkgs %) chain.
# Remove unnecessary data
churn_data_tbl %
select(-customerID) %__%
drop_na() %__%
select(Churn, everything())
glimpse(churn_data_tbl)## Observations: 7,032
## Variables: 20
## $ Churn &quot;No&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;Yes&quot;, &quot;...
## $ gender &quot;Female&quot;, &quot;Male&quot;, &quot;Male&quot;, &quot;Male&quot;, &quot;Femal...
## $ SeniorCitizen 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
## $ Partner &quot;Yes&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No...
## $ Dependents &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;Yes...
## $ tenure 1, 34, 2, 45, 2, 8, 22, 10, 28, 62, 13, ...
## $ PhoneService &quot;No&quot;, &quot;Yes&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;Yes&quot;, ...
## $ MultipleLines &quot;No phone service&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No phon...
## $ InternetService &quot;DSL&quot;, &quot;DSL&quot;, &quot;DSL&quot;, &quot;DSL&quot;, &quot;Fiber optic...
## $ OnlineSecurity &quot;No&quot;, &quot;Yes&quot;, &quot;Yes&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;No&quot;, &quot;...
## $ OnlineBackup &quot;Yes&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;Y...
## $ DeviceProtection &quot;No&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;...
## $ TechSupport &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No...
## $ StreamingTV &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;Ye...
## $ StreamingMovies &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;No...
## $ Contract &quot;Month-to-month&quot;, &quot;One year&quot;, &quot;Month-to-...
## $ PaperlessBilling &quot;Yes&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;Yes&quot;, ...
## $ PaymentMethod &quot;Electronic check&quot;, &quot;Mailed check&quot;, &quot;Mai...
## $ MonthlyCharges 29.85, 56.95, 53.85, 42.30, 70.70, 99.65...
## $ TotalCharges 29.85, 1889.50, 108.15, 1840.75, 151.65,...Split Into Train/Test Sets
We have a new package, rsample, which is very useful for sampling methods. It has the initial_split() function for splitting data sets into training and testing sets. The return is a special rsplit object.
# Split test/training sets
set.seed(100)
train_test_split % as.factor() %__% as.numeric(),
LogTotalCharges = log(TotalCharges)
) %__%
correlate() %__%
focus(Churn) %__%
fashion()## rowname Churn
## 1 TotalCharges -.20
## 2 LogTotalCharges -.25The correlation between “Churn” and “LogTotalCharges” is greatest in magnitude indicating the log transformation should improve the accuracy of the ANN model we build. Therefore, we should perform the log transformation.
One-Hot Encoding
One-hot encoding is the process of converting categorical data to sparse data, which has columns of only zeros and ones (this is also called creating “dummy variables” or a “design matrix”). All non-numeric data will need to be converted to dummy variables. This is simple for binary Yes/No data because we can simply convert to 1’s and 0’s. It becomes slightly more complicated with multiple categories, which requires creating new columns of 1’s and 0`s for each category (actually one less). We have four features that are multi-category: Contract, Internet Service, Multiple Lines, and Payment Method.
Feature Scaling
ANN’s typically perform faster and often times with higher accuracy when the features are scaled and/or normalized (aka centered and scaled, also known as standardizing). Because ANNs use gradient descent, weights tend to update faster. According to Sebastian Raschka, an expert in the field of Deep Learning, several examples when feature scaling is important are:
k-nearest neighbors with an Euclidean distance measure if want all features to contribute equally
k-means (see k-nearest neighbors)
logistic regression, SVMs, perceptrons, neural networks etc. if you are using gradient descent/ascent-based optimization, otherwise some weights will update much faster than others
linear discriminant analysis, principal component analysis, kernel principal component analysis since you want to find directions of maximizing the variance (under the constraints that those directions/eigenvectors/principal components are orthogonal); you want to have features on the same scale since you’d emphasize variables on “larger measurement scales” more. There are many more cases than I can possibly list here … I always recommend you to think about the algorithm and what it’s doing, and then it typically becomes obvious whether we want to scale your features or not.
The interested reader can read Sebastian Raschka’s article for a full discussion on the scaling/normalization topic. Pro Tip: When in doubt, standardize the data.
Preprocessing With Recipes
Let’s implement the preprocessing steps/transformations uncovered during our exploration. Max Kuhn (creator of caret) has been putting some work into Rlang ML tools lately, and the payoff is beginning to take shape. A new package, recipes, makes creating ML data preprocessing workflows a breeze! It takes a little getting used to, but I’ve found that it really helps manage the preprocessing steps. We’ll go over the nitty gritty as it applies to this problem.
Step 1: Create A Recipe
A “recipe” is nothing more than a series of steps you would like to perform on the training, testing and/or validation sets. Think of preprocessing data like baking a cake (I’m not a baker but stay with me). The recipe is our steps to make the cake. It doesn’t do anything other than create the playbook for baking.
We use the recipe() function to implement our preprocessing steps. The function takes a familiar object argument, which is a modeling function such as object = Churn ~ . meaning “Churn” is the outcome (aka response, predictor, target) and all other features are predictors. The function also takes the data argument, which gives the “recipe steps” perspective on how to apply during baking (next).
A recipe is not very useful until we add “steps”, which are used to transform the data during baking. The package contains a number of useful “step functions” that can be applied. The entire list of Step Functions can be viewed here. For our model, we use:
step_discretize() with the option = list(cuts = 6) to cut the continuous variable for “tenure” (number of years as a customer) to group customers into cohorts.
step_log() to log transform “TotalCharges”.
step_dummy() to one-hot encode the categorical data. Note that this adds columns of one/zero for categorical data with three or more categories.
step_center() to mean-center the data.
step_scale() to scale the data.
The last step is to prepare the recipe with the prep() function. This step is used to “estimate the required parameters from a training set that can later be applied to other data sets”. This is important for centering and scaling and other functions that use parameters defined from the training set.
Here’s how simple it is to implement the preprocessing steps that we went over!
# Create recipe
rec_obj %
step_discretize(tenure, options = list(cuts = 6)) %__%
step_log(TotalCharges) %__%
step_dummy(all_nominal(), -all_outcomes()) %__%
step_center(all_predictors(), -all_outcomes()) %__%
step_scale(all_predictors(), -all_outcomes()) %__%
prep(data = train_tbl)## step 1 discretize training ## step 2 log training ## step 3 dummy training ## step 4 center training ## step 5 scale trainingWe can print the recipe object if we ever forget what steps were used to prepare the data. Pro Tip: We can save the recipe object as an RDS file using saveRDS(), and then use it to bake() (discussed next) future raw data into ML-ready data in production!
# Print the recipe object
rec_obj## Data Recipe
## ## Inputs:
## ## role #variables
## outcome 1
## predictor 19
## ## Training data contained 5626 data points and no missing data.
## ## Steps:
## ## Dummy variables from tenure [trained]
## Log transformation on TotalCharges [trained]
## Dummy variables from ~gender, ~Partner, ... [trained]
## Centering for SeniorCitizen, ... [trained]
## Scaling for SeniorCitizen, ... [trained]Step 2: Baking With Your Recipe
Now for the fun part! We can apply the “recipe” to any data set with the bake() function, and it processes the data following our recipe steps. We’ll apply to our training and testing data to convert from raw data to a machine learning dataset. Check our training set out with glimpse(). Now that’s an ML-ready dataset prepared for ANN modeling!!
# Predictors
x_train_tbl %
# Output layer
layer_dense(
units = 1, kernel_initializer = &quot;uniform&quot;, activation = &quot;sigmoid&quot;) %__% # Compile ANN
compile(
optimizer = 'adam',
loss = 'binary_crossentropy',
metrics = c('accuracy')
)
model_keras## Model
## ______________________________________________________________________
## Layer (type) Output Shape Param # ## ======================================================================
## dense_1 (Dense) (None, 16) 576 ## ______________________________________________________________________
## dropout_1 (Dropout) (None, 16) 0 ## ______________________________________________________________________
## dense_2 (Dense) (None, 16) 272 ## ______________________________________________________________________
## dropout_2 (Dropout) (None, 16) 0 ## ______________________________________________________________________
## dense_3 (Dense) (None, 1) 17 ## ======================================================================
## Total params: 865
## Trainable params: 865
## Non-trainable params: 0
## ______________________________________________________________________We use the fit() function to run the ANN on our training data. The object is our model, and x and y are our training data in matrix and numeric vector forms, respectively. The batch_size = 50 sets the number samples per gradient update within each epoch. We set epochs = 35 to control the number training cycles. Typically we want to keep the batch size high since this decreases the error within each training cycle (epoch). We also want epochs to be large, which is important in visualizing the training history (discussed below). We set validation_split = 0.30 to include 30% of the data for model validation, which prevents overfitting. The training process should complete in 15 seconds or so.
# Fit the keras model to the training data
fit_keras % fct_recode(yes = &quot;1&quot;, no = &quot;0&quot;),
class_prob = yhat_keras_prob_vec
)
estimates_keras_tbl## # A tibble: 1,406 x 3
## truth estimate class_prob
## ## 1 yes no 0.328355074
## 2 yes yes 0.633630514
## 3 no no 0.004589651
## 4 no no 0.007402068
## 5 no no 0.049968336
## 6 no no 0.116824441
## 7 no yes 0.775479317
## 8 no no 0.492996633
## 9 no no 0.011550998
## 10 no no 0.004276015
## # ... with 1,396 more rowsNow that we have the data formatted, we can take advantage of the yardstick package.
Confusion Table
We can use the conf_mat() function to get the confusion table. We see that the model was by no means perfect, but it did a decent job of identifying customers likely to churn.
# Confusion Table
estimates_keras_tbl %__% conf_mat(truth, estimate)## Truth
## Prediction no yes
## no 950 161
## yes 99 196Accuracy
We can use the metrics() function to get an accuracy measurement from the test set. We are getting roughly 82% accuracy.
# Accuracy
estimates_keras_tbl %__% metrics(truth, estimate)## # A tibble: 1 x 1
## accuracy
## ## 1 0.8150782AUC
We can also get the ROC Area Under the Curve (AUC) measurement. AUC is often a good metric used to compare different classifiers and to compare to randomly guessing (AUC_random = 0.50). Our model has AUC = 0.85, which is much better than randomly guessing. Tuning and testing different classification algorithms may yield even better results.
# AUC
estimates_keras_tbl %__% roc_auc(truth, class_prob)## [1] 0.8523951Precision And Recall
Precision is when the model predicts “yes”, how often is it actually “yes”. Recall (also true positive rate or specificity) is when the actual value is “yes” how often is the model correct. We can get precision() and recall() measurements using yardstick.
# Precision
tibble(
precision = estimates_keras_tbl %__% precision(truth, estimate),
recall = estimates_keras_tbl %__% recall(truth, estimate)
)## # A tibble: 1 x 2
## precision recall
## ## 1 0.8550855 0.9056244Precision and recall are very important to the business case: The organization is concerned with balancing the cost of targeting and retaining customers at risk of leaving with the cost of inadvertently targeting customers that are not planning to leave (and potentially decreasing revenue from this group). The threshold above which to predict Churn = “Yes” can be adjusted to optimize for the business problem. This becomes an Customer Lifetime Value optimization problem that is discussed further in Next Steps.
F1 Score
We can also get the F1-score, which is a weighted average between the precision and recall. Machine learning classifier thresholds are often adjusted to maximize the F1-score. However, this is often not the optimal solution to the business problem.
# F1-Statistic
estimates_keras_tbl %__% f_meas(truth, estimate, beta = 1)## [1] 0.8796296Explain The Model With LIME
LIME stands for Local Interpretable Model-agnostic Explanations, and is a method for explaining black-box machine learning model classifiers. For those new to LIME, this YouTube video does a really nice job explaining how LIME helps to identify feature importance with black box machine learning models (e.g. deep learning, stacked ensembles, random forest).
Setup
The lime package implements LIME in R. One thing to note is that it’s not setup out-of-the-box to work with keras. The good news is with a few functions we can get everything working properly. We’ll need to make two custom functions:
model_type: Used to tell lime what type of model we are dealing with. It could be classification, regression, survival, etc.
predict_model: Used to allow lime to perform predictions that its algorithm can interpret.
The first thing we need to do is identify the class of our model object. We do this with the class() function.
class(model_keras)## [1] &quot;keras.models.Sequential&quot; ## [2] &quot;keras.engine.training.Model&quot; ## [3] &quot;keras.engine.topology.Container&quot;
## [4] &quot;keras.engine.topology.Layer&quot; ## [5] &quot;python.builtin.object&quot;Next we create our model_type() function. It’s only input is x the keras model. The function simply returns “classification”, which tells LIME we are classifying.
# Setup lime::model_type() function for keras
model_type.keras.models.Sequential %
arrange(abs(Churn)) %__%
mutate(feature = as_factor(feature)) corrr_analysis## # A tibble: 35 x 2
## feature Churn
## ## 1 gender_Male -0.006690899
## 2 tenure_bin3 -0.009557165
## 3 MultipleLines_No.phone.service -0.016950072
## 4 PhoneService_Yes 0.016950072
## 5 MultipleLines_Yes 0.032103354
## 6 StreamingTV_Yes 0.066192594
## 7 StreamingMovies_Yes 0.067643871
## 8 DeviceProtection_Yes -0.073301197
## 9 tenure_bin4 -0.073371838
## 10 PaymentMethod_Mailed.check -0.080451164
## # ... with 25 more rowsThe correlation visualization helps in distinguishing which features are relavant to Churn.
# Correlation visualization
corrr_analysis %__%
ggplot(aes(x = Churn, y = fct_reorder(feature, desc(Churn)))) +
geom_point() +
# Positive Correlations - Contribute to churn
geom_segment(aes(xend = 0, yend = feature), color = palette_light()[[2]], data = corrr_analysis %__% filter(Churn __ 0)) +
geom_point(color = palette_light()[[2]], data = corrr_analysis %__% filter(Churn __ 0)) +
# Negative Correlations - Prevent churn
geom_segment(aes(xend = 0, yend = feature), color = palette_light()[[1]], data = corrr_analysis %__% filter(Churn % filter(Churn 0.25):
Increases Likelihood of Churn (Red):
Tenure = Bin 1 (