Sales, customer service, supply chain and logistics, manufacturing… no matter which department you’re in, you more than likely care about backorders. Backorders are products that are temporarily out of stock, but a customer is permitted to place an order against future inventory. Back orders are both good and bad: Strong demand can drive back orders, but so can suboptimal planning. The problem is when a product is not immediately available, customers may not have the luxury or patience to wait. This translates into lost sales and low customer satisfaction. The good news is that machine learning (ML) can be used to identify products at risk of backorders. In this article we use the new H2O automated ML algorithm to implement Kaggle-quality predictions on the Kaggle dataset, “Can You Predict Product Backorders?”. This is an advanced tutorial, which can be difficult for learners. We have good news, see our announcement below if you are interested in a machine learning course from Business Science. If you love this tutorial, please connect with us on social media to stay up on the latest Business Science news, events and information! Good luck and happy analyzing!
Challenges and Benefits
This tutorial covers two challenges. One challenge with this problem is dataset imbalance, when the majority class significantly outweighs the minority class. We implement a special technique for dealing with unbalanced data sets called SMOTE (synthetic minority over-sampling technique) that improves modeling accuracy and efficiency (win-win). The second challenge is optimizing for the business case. To do so, we explore cutoff (threshold) optimization which can be used to find the cutoff that maximizes expected profit.
One of the important visualizations (benefits to your understanding) is the effect of precision and recall on inventory strategy. A lever the analyst controls is the cutoff (threshold). The threshold has an effect on precision and recall. By selecting the optimal threshold we can maximize expected profit.
Quick Navigation
This is a longer analysis, and certain sections may be more interesting based on your role. Are you an Executive or a Data Scientist?
Executive: Read these:
Background on Backorder Prediction
The Challenges with Predicting Backorders
Data Scientist: Read these:
Case Study: Predicting Backorder Risk and Modeling Profit
Using Machine Learning to Predict Backorders
Optimizing the Model for the Expected Profit
Others should just read the whole thing. ;)
Background on Backorder Prediction What is a backorder?
According to Investopedia, a backorder is…
A customer order that has not been fulfilled. A backorder generally indicates that customer demand for a product or service exceeds a company’s capacity to supply it. Total backorders, also known as backlog, may be expressed in terms of units or dollar amount.
Why do we care?
Source: www.trustedreviews.com
A backorder can be both a good thing and a bad thing. Consider a company like Apple that recently rolled out several new iPhone models including iPhone 8 and iPhone X. Because the initial supply cannot possibly keep up with the magnitude of demand, Apple is immediately in a “backorder” situation that requires customer service management.
On the positive side, backorders indicate healthy demand for a product. For Apple, this is a credit to strong brand recognition. Customers are likely to wait several months to get the latest model because of the new and innovative technology and incredible branding. Conversely, Apple’s competitors don’t have this luxury. If new models cannot be provided immediately, their customers cancel orders and go elsewhere.
Companies constantly strive for a balance in managing backorders. It’s a fine line: Too much supply increases inventory costs while too little supply increases the risk that customers may cancel orders. Why not just keep 100,000 of everything on the shelves at all time? For most retailers and manufacturers, this strategy will drive inventory costs through the roof considering they likely have a large number of SKUs (unique product IDs).
A predictive analytics program can identify which products are most likely to experience backorders giving the organization information and time to adjust. Machine learning can identify patterns related to backorders before customers order. Production can then adjust to minimize delays while customer service can provide accurate dates to keep customers informed and happy. The predictive analytics approach enables the maximum product to get in the hands of customers at the lowest cost to the organization. The result is a win-win: organizations increase sales while customers get to enjoy the products they demand.
Challenges with Predicting Backorders One of the first steps in developing a backorder strategy is to understand which products are at risk of being backordered. Sounds easy right? Wrong. A number of factors can make this a challenging business problem.
Demand-Event Dependencies
Demand for existing products are constantly changing, often dependent on non-business data (holidays, weather, and other events). Consider demand for snow blowers at Home Depot. We know that geographically-speaking in Northern USA the winter is when demand will rise and the spring is when demand will fall. However, this demand is highly dependent on the level of snowfall. If snowfall is minimal, demand will be low regardless of it being winter. Therefore, Home Depot will need to know the likelihood of snowfall to best predict demand surges and shortfalls in order to optimize inventory.
Small Numbers (Anomalies)
If backorders are very infrequent but highly important, it can be very difficult to predict the minority class accurately because of the imbalance between backorders to non-backorders within the data set. Accuracy for the model will look great, but the actual predictive quality may be very poor. Consider the NULL error rate, or the rate of just picking the majority class. Just picking “non-backorder” may be the same or more accurate than the model. Special strategies exist for dealing with unbalanced data, and we’ll implement SMOTE.
Time Dependency
Often this problem is viewed as a cross-sectional problem rather than as a temporal (or time-based) problem. The problem is time can play a critical role. Just because a product sold out this time last year, will it sell out again this year? Maybe. Maybe not. Time-based demand forecasting can be necessary to augment cross-sectional binary classification.
Case Study: Predicting Backorder Risk and Optimizing Profit The Problem
A hypothetical manufacturer has a data set that identifies whether or not a backorder has occurred. The challenge is to accurately predict future backorder risk using predictive analytics and machine learning and then to identify the optimal strategy for inventorying products with high backorder risk.
The Data
The data comes from Kaggle’s Can You Predict Product Backorders? dataset. If you have a Kaggle account, you can download the data, which includes both a training and a test set. We’ll use the training set for developing our model and the test set for determining the final accuracy of the best model.
The data file contains the historical data for the 8 weeks prior to the week we are trying to predict. The data were taken as weekly snapshots at the start of each week. The target (or response) is the went_on_backorder variable. To model and predict the target, we’ll use the other features, which include:
sku - Random ID for the product
national_inv - Current inventory level for the part
lead_time - Transit time for product (if available)
in_transit_qty - Amount of product in transit from source
forecast_3_month - Forecast sales for the next 3 months
forecast_6_month - Forecast sales for the next 6 months
forecast_9_month - Forecast sales for the next 9 months
sales_1_month - Sales quantity for the prior 1 month time period
sales_3_month - Sales quantity for the prior 3 month time period
sales_6_month - Sales quantity for the prior 6 month time period
sales_9_month - Sales quantity for the prior 9 month time period
min_bank - Minimum recommend amount to stock
potential_issue - Source issue for part identified
pieces_past_due - Parts overdue from source
perf_6_month_avg - Source performance for prior 6 month period
perf_12_month_avg - Source performance for prior 12 month period
local_bo_qty - Amount of stock orders overdue
deck_risk - Part risk flag
oe_constraint - Part risk flag
ppap_risk - Part risk flag
stop_auto_buy - Part risk flag
rev_stop - Part risk flag
went_on_backorder - Product actually went on backorder. This is the target value.
Using Machine Learning to Predict Backorders Libraries
We’ll use the following libraries:
tidyquant: Used to quickly load the “tidyverse” (dplyr, tidyr, ggplot, etc) along with custom, business-report-friendly ggplot themes. Also great for time series analysis (not featured)
unbalanced: Contains various methods for working with unbalanced data. We’ll use the ubSMOTE() function.
h2o: Go-to package for implementing professional grade machine learning.
Note on H2O: You’ll want to install the latest stable R version from H2O.ai. If you have issues in this post, you probably did not follow these steps:
Go to H2O.ai’s download page
Under H2O, select “Latest Stable Release”
Click on the “Install in R” tab
Follow instructions exactly.
Load the libraries.
# You can install these from CRAN using install.packages()
library(tidyquant) # Loads tidyverse and custom ggplot themes
library(unbalanced) # Methods for dealing with unbalanced data sets
# Follow instructions for latest stable release, don't just install from CRAN
library(h2o) # Professional grade MLLoad Data
Use read_csv() to load the training and test data.
# Load training and test sets train_raw_df % knitr::kable()
sku
national_inv
lead_time
in_transit_qty
forecast_3_month
forecast_6_month
forecast_9_month
sales_1_month
sales_3_month
sales_6_month
sales_9_month
min_bank
potential_issue
pieces_past_due
perf_6_month_avg
perf_12_month_avg
local_bo_qty
deck_risk
oe_constraint
ppap_risk
stop_auto_buy
rev_stop
went_on_backorder
1026827
0
NA
0
0
0
0
0
0
0
0
0
No
0
-99.00
-99.00
0
No
No
No
Yes
No
No
1043384
2
9
0
0
0
0
0
0
0
0
0
No
0
0.99
0.99
0
No
No
No
Yes
No
No
1043696
2
NA
0
0
0
0
0
0
0
0
0
No
0
-99.00
-99.00
0
Yes
No
No
Yes
No
No
1043852
7
8
0
0
0
0
0
0
0
0
1
No
0
0.10
0.13
0
No
No
No
Yes
No
No
1044048
8
NA
0
0
0
0
0
0
0
4
2
No
0
-99.00
-99.00
0
Yes
No
No
Yes
No
No
1044198
13
8
0
0
0
0
0
0
0
0
0
No
0
0.82
0.87
0
No
No
No
Yes
No
No
The tail shows another issue, the last row is NA values.
# Tail
train_raw_df %__% tail() %__% knitr::kable()
sku
national_inv
lead_time
in_transit_qty
forecast_3_month
forecast_6_month
forecast_9_month
sales_1_month
sales_3_month
sales_6_month
sales_9_month
min_bank
potential_issue
pieces_past_due
perf_6_month_avg
perf_12_month_avg
local_bo_qty
deck_risk
oe_constraint
ppap_risk
stop_auto_buy
rev_stop
went_on_backorder
1407754
0
2
0
10
10
10
0
5
7
7
0
No
0
0.69
0.69
5
Yes
No
No
Yes
No
No
1373987
-1
NA
0
5
7
9
1
3
3
8
0
No
0
-99.00
-99.00
1
No
No
No
Yes
No
No
1524346
-1
9
0
7
9
11
0
8
11
12
0
No
0
0.86
0.84
1
Yes
No
No
No
No
Yes
1439563
62
9
16
39
87
126
35
63
153
205
12
No
0
0.86
0.84
6
No
No
No
Yes
No
No
1502009
19
4
0
0
0
0
2
7
12
20
1
No
0
0.73
0.78
1
No
No
No
Yes
No
No
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
If we take a look at the distribution of the target, train_raw_df$went_on_backorder, we see that the data set is severely imbalanced. We’ll need a strategy to balance the data set if we want to get maximum model performance and efficiency.
# Unbalanced data set
train_raw_df$went_on_backorder %__% table() %__% prop.table()## .
## No Yes ## 0.993309279 0.006690721We can also inspect missing values. Because both the train and test sets have missing values, we’ll need to come up with a strategy to deal with them.
# train set: Percentage of complete cases
train_raw_df %__% complete.cases() %__% sum() / nrow(train_raw_df)## [1] 0.9402238The test set has a similar distribution.
# test set: Percentage of complete cases
test_raw_df %__% complete.cases() %__% sum() / nrow(test_raw_df)## [1] 0.939172Finally, it’s also worth taking a glimpse of the data. We can quickly see:
We have some character columns with Yes/No values.
The “perf_” columns have -99 values, which reflect missing data.
glimpse(train_raw_df)## Observations: 1,687,861
## Variables: 23
## $ sku 1026827, 1043384, 1043696, 1043852, 104...
## $ national_inv 0, 2, 2, 7, 8, 13, 1095, 6, 140, 4, 0, ...
## $ lead_time NA, 9, NA, 8, NA, 8, NA, 2, NA, 8, 2, N...
## $ in_transit_qty 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ forecast_3_month 0, 0, 0, 0, 0, 0, 0, 0, 15, 0, 0, 0, 0,...
## $ forecast_6_month 0, 0, 0, 0, 0, 0, 0, 0, 114, 0, 0, 0, 0...
## $ forecast_9_month 0, 0, 0, 0, 0, 0, 0, 0, 152, 0, 0, 0, 0...
## $ sales_1_month 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ sales_3_month 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ sales_6_month 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ sales_9_month 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ min_bank 0, 0, 0, 1, 2, 0, 4, 0, 0, 0, 0, 0, 0, ...
## $ potential_issue &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No...
## $ pieces_past_due 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ perf_6_month_avg -99.00, 0.99, -99.00, 0.10, -99.00, 0.8...
## $ perf_12_month_avg -99.00, 0.99, -99.00, 0.13, -99.00, 0.8...
## $ local_bo_qty 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ deck_risk &quot;No&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;Yes&quot;, &quot;No&quot;, &quot;...
## $ oe_constraint &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No...
## $ ppap_risk &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No...
## $ stop_auto_buy &quot;Yes&quot;, &quot;Yes&quot;, &quot;Yes&quot;, &quot;Yes&quot;, &quot;Yes&quot;, &quot;Yes...
## $ rev_stop &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No...
## $ went_on_backorder &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No&quot;, &quot;No...Data Setup
Validation Set
We have a training and a test set, but we’ll need a validation set to assist with modeling. We can perform a random 85/15 split of the training data.
# Train/Validation Set Split
split_pct =0.5
geom_vline(xintercept = 0.5, color = palette_light()[[3]], size = 1) +
annotate(&quot;text&quot;, x = 0.62, y = 0.75, size = 3,
label = 'p1 __= 0.50: &quot;Yes&quot;\nInventory\nProbability\nSplit 50/50') +
geom_segment(x = 0.5, y = 0.70, xend = 0.52, yend= 0.72, color = palette_light()[[3]], size = 1) +
# p__=0.75
geom_vline(xintercept = 0.75, color = palette_light()[[3]], size = 1) +
annotate(&quot;text&quot;, x = 0.87, y = 0.75, size = 3,
label = 'p1 __= 0.75: &quot;Yes&quot;\nInventory Very\nConservatively\n(Most Likely Backorder)') +
geom_segment(x = 0.75, y = 0.70, xend = 0.77, yend= 0.72, color = palette_light()[[3]], size = 1) +
# p__=1
geom_vline(xintercept = 1, color = palette_light()[[3]], size = 1) +
annotate(&quot;text&quot;, x = 0.87, y = 0.22, size = 3,
label = 'p1 __= 1.00: &quot;Yes&quot;\nInventory Nothing') +
geom_segment(x = 1.00, y = 0.23, xend = 0.98, yend= 0.21, color = palette_light()[[3]], size = 1) Expected Value Framework
We need another function to help us determine which values of cutoff to use, and this comes from the Expected Value Framework described in Data Science for Business.
Source: Data Science for Business
To implement this framework we need two things:
Expected Rates (matrix of probabilities): Needed for each threshold
Cost/Benefit Information: Needed for each observation
Expected Rates
We have the class probabilities and rates from the confusion matrix, and we can retrieve this using the h2o.confusionMatrix() function.
# Confusion Matrix, p1_cutoff = 0.63
h2o.confusionMatrix(perf_h2o)## Confusion Matrix (vertical: actual; across: predicted) for max f1 @ threshold = 0.630068453397666:
## 0 1 Error Rate
## 0 235796 3591 0.015001 =3591/239387
## 1 1647 1041 0.612723 =1647/2688
## Totals 237443 4632 0.021638 =5238/242075The problems are:
The confusion matrix is for just one specific cutoff. We want to evaluate for all of the potential cutoffs to determine which strategy is best.
We need to convert to expected rates, as shown in the Figure 7-2 diagram.
Luckily, we can retrieve the rates by cutoff conveniently using h2o.metric(). Huzzah!
# Get expected rates by cutoff
expected_rates %
as.tibble() %__%
select(threshold, tpr, fpr, fnr, tnr)
expected_rates## # A tibble: 400 x 5
## threshold tpr fpr fnr tnr
## ## 1 0.9933572 0.003348214 2.506402e-05 0.9966518 0.9999749
## 2 0.9855898 0.004464286 4.177336e-05 0.9955357 0.9999582
## 3 0.9706335 0.005952381 8.354673e-05 0.9940476 0.9999165
## 4 0.9604183 0.010416667 1.420294e-04 0.9895833 0.9998580
## 5 0.9513270 0.013020833 1.879801e-04 0.9869792 0.9998120
## 6 0.9423699 0.014508929 2.339308e-04 0.9854911 0.9997661
## 7 0.9320206 0.017113095 3.216549e-04 0.9828869 0.9996783
## 8 0.9201693 0.020833333 4.052016e-04 0.9791667 0.9995948
## 9 0.9113555 0.023437500 4.511523e-04 0.9765625 0.9995488
## 10 0.9061218 0.025297619 5.472311e-04 0.9747024 0.9994528
## # ... with 390 more rowsCost/Benefit Information
The cost-benefit matrix is a business assessment of the cost and benefit for each of four potential outcomes:
True Negative (benefit): This is the benefit from a SKU that is correctly predicted as not backordered. The benefit is zero because a customer simply does not buy this item (lack of demand).
True Positive (benefit): This is the benefit from a SKU that is correctly predicted as backorder and the customer needs this now. The benefit is the profit from revenue. For example, if the item price is $1000 and the cost is $600, the unit profit is $400.
False Positive (cost): This is the cost associated with incorrectly classifying a SKU as backordered when demand is not present. The is the warehousing and inventory related cost. For example, if the item costs $10/unit to inventory, which would have otherwise not have occurred.
False Negative (cost): This is the cost associated with incorrectly classifying a SKU when demand is present. In our case, no money was spent and nothing was gained.
The cost-benefit information is needed for each decision pair. If we take one SKU, say the first prediction from the test set, we have an item that was predicted to NOT be backordered (went_on_backorder == &quot;No&quot;). If hypothetically the value for True Positive (benefit) is $400/unit in profit from correctly predict a backorder and the False Positive (cost) of accidentally inventorying and item that was not backordered is $10/unit then a data frame can be be structured like so.
# Cost/benefit info codified for first item
first_item %
as.tibble() %__%
slice(1) %__%
add_column(
cb_tn = 0,
cb_tp = 400,
cb_fp = -10,
cb_fn = 0
)
first_item## # A tibble: 1 x 7
## predict p0 p1 cb_tn cb_tp cb_fp cb_fn
## ## 1 0 0.9838573 0.01614266 0 400 -10 0Optimization: Single Item
The expected value equation generalizes to:
Where,
pi is the probability of the outcome for one observation
vi is the value of the outcome for one observation
The general form isn’t very useful, but from it we can create an Expected Profit equation using a basic rule of probability p(x,y) = p(y)*p(x|y) that combines both the Expected Rates (2x2 matrix of probabilities after normalization of Confusion Matrix) and a Cost/Benefit Matrix (2x2 matrix with expected costs and benefits). Refer to Data Science for Business for the proof.
Where,
p(p) is the confusion matrix positive class prior (probability of actual yes / total)
p(n) is the confusion matrix positive class prior (probability of actual no / total = 1 - positive class prior)
p(Y|p) is the True Positive Rate (tpr)
p(N|p) is the False Negative Rate (fnr)
p(N|n) is the True Negative Rate (tnr)
p(Y|n) is the False Positive Rate (fpr)
b(Y,p) is the benefit from true positive (cb_tp)
c(N,p) is the cost from false negative (cb_fn)
b(N,n) is the benefit from true negative (cb_tn)
c(Y,n) is the cost from false positive (cb_fp)
This equation simplifies even further since we have a case where cb_tn and cb_fn are both zero.
We create a function to calculate the expected profit using the probability of a positive case (positive prior, p1), the cost/benefit of a true positive (cb_tp), and the cost/benefit of a false positive (cb_fp). We’ll take advantage of the expected_rates data frame we previously created, which contains the true positive rate and false positive rate for each threshold (400 thresholds in the range of 0 and 1).
# Function to calculate expected profit
calc_expected_profit %
# Add in expected rates
mutate(expected_rates = list(expected_rates)) %__%
unnest() %__%
mutate(
expected_profit = p1 * (tpr * cb_tp) + (1 - p1) * (fpr * cb_fp)
) %__%
select(threshold, expected_profit)
}We can test the function for a hypothetical prediction that is unlikely to have a backorder (most common class). Setting p1 = 0.01 indicates the hypothetical SKU has a low probability. Continuing with the example case of $400/unit profit and $10/unit inventory cost, we can see that optimal threshold is approximately 0.4. Note that an inventory all items strategy (threshold = 0) would cause the company to lose money on low probability of backorder items (-$6/unit) and an inventory nothing strategy would result in no benefit but no loss ($0/unit).
# Investigate a expected profit of item with low probability of backorder
hypothetical_low % round (2)),
x = &quot;Cutoff (Threshold)&quot;,
y = &quot;Expected Profit per Unit&quot;
)
Conversely if we investigate a hypothetical item with high probability of backorder, we can see that it’s much more advantageous to have a loose strategy with respect to inventory conservatism. Notice the profit per-unit is 80% of the theoretical maximum profit (80% of $400/unit = $320/unit if “p1” = 0.8). The profit decreases to zero as the inventory strategy becomes more conservative.
# Investigate a expected profit of item with high probability of backorder
hypothetical_high % round (2))
)
Let’s take a minute to digest what’s going on in both the high and low expected profit curves. Units with low probability of backorder (the majority class) will tend to increase the cutoff while units with high probability will tend to lower the cutoff. It’s this balance or tradeoff that we need to scale to understand the full picture.
Optimization: Multi-Item
Now the fun part, scaling the optimization to multiple products. Let’s analyze a simplified case: 10 items with varying backorder probabilities, benefits, costs, and safety stock levels. In addition, we’ll include a backorder purchase quantity with logic of 100% safety stock (meaning items believed to be backordered will have an additional quantity purchased equal to that of the safety stock level). We’ll investigate optimal stocking level for this subset of items to illustrate scaling the analysis to find the global optimized cutoff (threshold).
# Ten Hypothetical Items
ten_items %
unnest() %__%
rename(expected_profit_per_unit = expected_profit) %__%
# Calculate 100% safety stock repurchase and sell
mutate(expected_profit_extended = expected_profit_per_unit * 1 * safety_stock) %__%
select(item, p1, threshold, expected_profit_extended) extended_expected_profit_ten_items## # A tibble: 4,000 x 4
## item p1 threshold expected_profit_extended
## ## 1 1 0.02 0.9933572 0.06512208
## 2 1 0.02 0.9855898 0.08621537
## 3 1 0.02 0.9706335 0.11290693
## 4 1 0.02 0.9604183 0.19789417
## 5 1 0.02 0.9513270 0.24660013
## 6 1 0.02 0.9423699 0.27298466
## 7 1 0.02 0.9320206 0.31862027
## 8 1 0.02 0.9201693 0.38688435
## 9 1 0.02 0.9113555 0.43559030
## 10 1 0.02 0.9061218 0.46573090
## # ... with 3,990 more rowsWe can visualize the expected profit curves for each item extended for backorder-prevention quantity to be purchased and sold (note that selling 100% is a simplifying assumption).
# Visualizing Expected Profit extended_expected_profit_ten_items %__%
ggplot(aes(threshold, expected_profit_extended, color = factor(item)), group = item) +
geom_line(size = 1) +
theme_tq() +
scale_color_tq() +
labs(
title = &quot;Expected Profit Curves, Modeling Extended Expected Profit&quot;,
subtitle = &quot;Taking backorder-prevention purchase quantity into account weights curves&quot;,
color = &quot;Item No.&quot; )
Finally, we can aggregate the expected profit using a few dplyr operations. Visualizing the final curve exposes the optimal threshold.
# Aggregate extended expected profit by threshold total_expected_profit_ten_items %
group_by(threshold) %__%
summarise(expected_profit_total = sum(expected_profit_extended)) # Get maximum (optimal) threshold
max_expected_profit %
filter(expected_profit_total == max(expected_profit_total))
# Visualize the total expected profit curve
total_expected_profit_ten_items %__%
ggplot(aes(threshold, expected_profit_total)) +
geom_line(size = 1, color = palette_light()[[1]]) +
geom_vline(xintercept = max_expected_profit$threshold, color = palette_light()[[1]]) +
theme_tq() +
scale_color_tq() +
labs(
title = &quot;Expected Profit Curve, Modeling Total Expected Profit&quot;,
subtitle = &quot;Summing up the curves by threshold yields optimal strategy&quot;,
caption = paste0('threshold @ max = ', max_expected_profit$threshold %__% round (2))
)
Conclusions
This was a very technical and detailed post, and if you made it through congratulations! We covered automated machine learning with H2O, an efficient and high accuracy tool for prediction. We worked with an extremely unbalanced data set, showing how to use SMOTE to synthetically improve dataset balance and ultimately model performance. We spent a considerable amount of effort optimizing the cutoff (threshold) selection to maximize expected profit, which ultimately matters most to the bottom line. Hopefully you can see how data science and machine learning can be very beneficial to the business, enabling better decisions and ROI.
Announcements Based on recent demand, we are considering offering application-specific machine learning courses for Data Scientists. The content will be business problems similar to this article and our article, HR Analytics: Using Machine Learning to Predict Employee Turnover. The student will learn from Business Science how to implement cutting edge data science to solve business problems. Please let us know if you are interested. You can leave comments as to what you would like to see at the bottom of the post in Disqus.
About Business Science Business Science specializes in “ROI-driven data science”. Our focus is machine learning and data science in business applications. We help businesses that seek to add this competitive advantage but may not have the resources currently to implement predictive analytics. Business Science works with clients primarily in small to medium size businesses, guiding these organizations in expanding predictive analytics while executing on ROI generating projects. Visit the Business Science website or contact us to learn more!
Follow Business Science on Social Media @bizScienc is on twitter!
Check us out on Facebook page!
Check us out on LinkedIn!
Sign up for our insights blog to stay updated!
If you like our software, star our GitHub packages!