As advanced machine learning algorithms are gaining acceptance across many organizations and domains, machine learning interpretability is growing in importance to help extract insight and clarity regarding how these algorithms are performing and why one prediction is made over another. There are many methodologies to interpret machine learning results (i.e. variable importance via permutation, partial dependence plots, local interpretable model-agnostic explanations), and many machine learning R packages implement their own versions of one or more methodologies. However, some recent R packages that focus purely on ML interpretability agnostic to any specific ML algorithm are gaining popularity. One such package is DALEX and this post covers what this package does (and does not do) so that you can determine if it should become part of your preferred machine learning toolbox.
We implement machine learning models using H2O, a high performance ML toolkit. Let’s see how DALEX and H2O work together to get the best of both worlds with high performance and feature explainability!
Similar Articles You Might Enjoy
Data science for business tutorials:
HR Analytics: Using Machine Learning To Predict Employee Turnover
Customer Analytics: Using Deep Learning With Keras To Predict Customer Churn
Sales Analytics: How To Use Machine Learning To Predict And Optimize Product Backorders
LIME for black-box model interpretability and feature:
Using Machine Learning With LIME To Understand Employee Churn
Expected Value Framework for ML Classification in Business:
3 Reasons You Need To Learn The Expected Value Framework
Learning Trajectory
We’ll cover the following topics on DALEX in this article:
Advantages &amp; disadvantages: a quick breakdown of what DALEX does and does not do.
Replication requirements: what you’ll need to reproduce the analysis.
DALEX procedures: necessary functions for downstream explainers.
Residual diagnostics: understanding and comparing errors.
Variable importance: permutation based importance score.
Predictor-response relationship: PDP and ALE plots.
Local interpretation: explanations for a single prediction.
Get The Best Resources In Data Science. Every Friday!
Sign up for our free &quot;5 Topic Friday&quot; Newsletter. Every week, I'll send you the five coolest topics in data science for business that I've found that week. These could be new R packages, free books, or just some fun to end the week on. Sign Up For Five-Topic-Friday! DALEX and H2O: Machine Learning Model Interpretability And Feature Explanation
By Brad Boehmke, Director of Data Science at 84.51°
1.0 Advantages &amp; disadvantages
DALEX is an R package with a set of tools that help to provide Descriptive mAchine Learning EXplanations ranging from global to local interpretability methods. In particular, it makes comparing performance across multiple models convenient. However, as is, there are some problems with this package scaling to wider data sets commonly used by organizations. The following provides a quick list of its pros and cons:
Advantages
ML model and package agnostic: can be used for any supervised regression and binary classification ML model where you can customize the format of the predicted output.
Provides convenient approaches to compare results across multiple models.
Residual diagnostics: allows you to compare residual distributions.
Variable importance: uses a permutation-based approach for variable importance, which is model agnostic, and accepts any loss function to assess importance.
Partial dependence plots: leverages the pdp package.
Provides an alternative to PDPs for categorical predictor variables (merging path plots).
Includes a unique and intuitive approach for local intepretation.
Disadvantages
Some functions do not scale well to wide data (many predictor variables)
Currently only supports regression and binary classification problems (i.e. no multinomial support).
Only provides permutation-based variable importance scores (which become slow as number of features increase).
PDP plots can only be performed one variable at a time (options for two-way interaction PDP plots).
Does not provide ICE curves.
Does not provide alternative local interpretation algorithms (i.e. LIME, SHAP values).
2.0 Replication requirements
We leverage the following packages:
# load required packages
library(rsample)
library(dplyr)
library(purrr)
library(ggplot2)
library(h2o)
library(DALEX)
# initialize h2o session
h2o.no_progress()
h2o.init()## ## H2O is not running yet, starting it now...
## ## Note: In case of errors look at the following log files:
## C:\Users\mdancho\AppData\Local\Temp\Rtmp6f3d5n/h2o_mdancho_started_from_r.out
## C:\Users\mdancho\AppData\Local\Temp\Rtmp6f3d5n/h2o_mdancho_started_from_r.err
## ## ## Starting H2O JVM and connecting: . Connection successful!
## ## R is connected to the H2O cluster: ## H2O cluster uptime: 7 seconds 710 milliseconds ## H2O cluster timezone: America/New_York ## H2O data parsing timezone: UTC ## H2O cluster version: 3.19.0.4208 ## H2O cluster version age: 5 months and 3 days !!! ## H2O cluster name: H2O_started_from_R_mdancho_dpc744 ## H2O cluster total nodes: 1 ## H2O cluster total memory: 3.51 GB ## H2O cluster total cores: 4 ## H2O cluster allowed cores: 4 ## H2O cluster healthy: TRUE ## H2O Connection ip: localhost ## H2O Connection port: 54321 ## H2O Connection proxy: NA ## H2O Internal Security: FALSE ## H2O API Extensions: Algos, AutoML, Core V3, Core V4 ## R Version: R version 3.4.4 (2018-03-15)To demonstrate model visualization techniques we’ll use the employee attrition data that has been included in the rsample package. This demonstrates a binary classification problem (“Yes” vs. “No”) but the same process that you’ll observe can be used for a regression problem.
To demonstrate DALEX’s capabilities we’ll use the employee attrition data that has been included in the rsample package. This demonstrates a binary classification problem (“Yes” vs. “No”) but the same process that you’ll observe can be used for a regression problem.
I perform a few house cleaning tasks on the data prior to converting to an h2o object and splitting.
NOTE: To use some of DALEX’s functions, categorical predictor variables need to be converted to factors. Also, I force ordered factors to be unordered as h2o does not support ordered categorical variables.
# classification data
df % mutate_if(is.ordered, factor, ordered = FALSE) %__%
mutate(Attrition = recode(Attrition, &quot;Yes&quot; = &quot;1&quot;, &quot;No&quot; = &quot;0&quot;) %__% factor(levels = c(&quot;1&quot;, &quot;0&quot;)))
# convert to h2o object
df.h2o