<p><a href="https://twitter.com/drsimonj">@drsimonj</a> here to show you how to use xgboost (extreme gradient boosting) models in pipelearner.</p>
<h2><a href="https://drsimonj.svbtle.com/#why-a-post-on-xgboost-and-pipelearner_2"> </a>Why a post on xgboost and pipelearner?</h2>
<p>xgboost is one of the most powerful machine-learning libraries, so there’s a good reason to use it. pipelearner helps to create machine-learning pipelines that make it easy to do cross-fold validation, hyperparameter grid searching, and more. So bringing them together will make for an awesome combination!</p>
<p>The only problem - out of the box, xgboost doesn’t play nice with pipelearner. Let’s work out how to deal with this.</p>
<h2><a href="https://drsimonj.svbtle.com/#setup_2"> </a>Setup</h2>
<p>To follow this post you’ll need the following packages:</p>
<pre><code># Install (if necessary)
install.packages(c(&quot;xgboost&quot;, &quot;tidyverse&quot;, &quot;devtools&quot;))
devtools::install_github(&quot;drsimonj/pipelearner&quot;)

# Attach
library(tidyverse)
library(xgboost)
library(pipelearner)
library(lazyeval)
</code></pre>
<p>Our example will be to try and predict whether tumours are cancerous or not using the <a href="https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)">Breast Cancer Wisconsin (Diagnostic) Data Set</a>. Set up as follows:</p>
<pre><code>data_url &lt;- 'https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data'

d &lt;- read_csv(
  data_url,
  col_names = c('id', 'thinkness', 'size_uniformity',
                'shape_uniformity', 'adhesion', 'epith_size',
                'nuclei', 'chromatin', 'nucleoli', 'mitoses', 'cancer')) %>% 
  select(-id) %>%            # Remove id; not useful here
  filter(nuclei != '?') %>%  # Remove records with missing data
  mutate(cancer = cancer == 4) %>% # one-hot encode 'cancer' as 1=malignant;0=benign
  mutate_all(as.numeric)     # All to numeric; needed for XGBoost

d
#> # A tibble: 683 × 10
#>    thinkness size_uniformity shape_uniformity adhesion epith_size nuclei
#>        &lt;dbl>           &lt;dbl>            &lt;dbl>    &lt;dbl>      &lt;dbl>  &lt;dbl>
#> 1          5               1                1        1          2      1
#> 2          5               4                4        5          7     10
#> 3          3               1                1        1          2      2
#> 4          6               8                8        1          3      4
#> 5          4               1                1        3          2      1
#> 6          8              10               10        8          7     10
#> 7          1               1                1        1          2     10
#> 8          2               1                2        1          2      1
#> 9          2               1                1        1          2      1
#> 10         4               2                1        1          2      1
#> # ... with 673 more rows, and 4 more variables: chromatin &lt;dbl>,
#> #   nucleoli &lt;dbl>, mitoses &lt;dbl>, cancer &lt;dbl>
</code></pre>
<h2><a href="https://drsimonj.svbtle.com/#pipelearner_2"> </a>pipelearner</h2>
<p>pipelearner makes it easy to do lots of routine machine learning tasks, many of which you can check out <a href="https://drsimonj.svbtle.com/easy-machine-learning-pipelines-with-pipelearner-intro-and-call-for-contributors">in this post</a>. For this example, we’ll use pipelearner to perform a grid search of some xgboost hyperparameters.</p>
<p>Grid searching is easy with pipelearner. For detailed instructions, check out my previous post: <a href="https://drsimonj.svbtle.com/how-to-grid-search-with-pipelearner">tidy grid search with pipelearner</a>. As a quick reminder, we declare a data frame, machine learning function, formula, and hyperparameters as vectors. Here’s an example that would grid search multiple values of <code>minsplit</code> and <code>maxdepth</code> for an rpart decision tree:</p>
<pre><code>pipelearner(d, rpart::rpart, cancer ~ .,
            minsplit = c(2, 4, 6, 8, 10),
            maxdepth = c(2, 3, 4, 5))
</code></pre>
<p>The challenge for xgboost:</p>
<blockquote><p>pipelearner expects a model function that has two arguments: <code>data</code> and <code>formula</code></p>
</blockquote><h2><a href="https://drsimonj.svbtle.com/#xgboost_2"> </a>xgboost</h2>
<p>Here’s an xgboost model:</p>
<pre><code># Prep data (X) and labels (y)
X &lt;- select(d, -cancer) %>% as.matrix()
y &lt;- d$cancer

# Fit the model
fit &lt;- xgboost(X, y, nrounds = 5, objective = &quot;reg:logistic&quot;)
#> [1]  train-rmse:0.372184 
#> [2]  train-rmse:0.288560 
#> [3]  train-rmse:0.230171 
#> [4]  train-rmse:0.188965 
#> [5]  train-rmse:0.158858

# Examine accuracy
predicted &lt;- as.numeric(predict(fit, X) >= .5)
mean(predicted == y)
#> [1] 0.9838946
</code></pre>
<p>Look like we have a model with 98.39% accuracy on the training data!</p>
<p>Regardless, notice that first two arguments to xgboost() are a numeric data matrix and a numeric label vector. This is not what pipelearner wants!</p>
<h2><a href="https://drsimonj.svbtle.com/#wrapper-function-to-parse-code-classprettypri_2"> </a>Wrapper function to parse <code>data</code> and <code>formula</code></h2>
<p>To make xgboost compatible with pipelearner we need to write a wrapper function that accepts <code>data</code> and <code>formula</code>, and uses these to pass a feature matrix and label vector to <code>xgboost</code>:</p>
<pre><code>pl_xgboost &lt;- function(data, formula, ...) {
  data &lt;- as.data.frame(data)

  X_names &lt;- as.character(f_rhs(formula))
  y_name  &lt;- as.character(f_lhs(formula))

  if (X_names == '.') {
    X_names &lt;- names(data)[names(data) != y_name]
  }

  X &lt;- data.matrix(data[, X_names])
  y &lt;- data[[y_name]]

  xgboost(data = X, label = y, ...)
}
</code></pre>
<p>Let’s try it out:</p>
<pre><code>pl_fit &lt;- pl_xgboost(d, cancer ~ ., nrounds = 5, objective = &quot;reg:logistic&quot;)
#> [1]  train-rmse:0.372184 
#> [2]  train-rmse:0.288560 
#> [3]  train-rmse:0.230171 
#> [4]  train-rmse:0.188965 
#> [5]  train-rmse:0.158858

# Examine accuracy
pl_predicted &lt;- as.numeric(predict(pl_fit, as.matrix(select(d, -cancer))) >= .5)
mean(pl_predicted == y)
#> [1] 0.9838946
</code></pre>
<p>Perfect!</p>
<h2><a href="https://drsimonj.svbtle.com/#bringing-it-all-together_2"> </a>Bringing it all together</h2>
<p>We can now use <code>pipelearner</code> and <code>pl_xgboost()</code> for easy grid searching:</p>
<pre><code>pl &lt;- pipelearner(d, pl_xgboost, cancer ~ .,
                  nrounds = c(5, 10, 25),
                  eta = c(.1, .3),
                  max_depth = c(4, 6))

fits &lt;- pl %>% learn()
#> [1]  train-rmse:0.453832 
#> [2]  train-rmse:0.412548 
#> ...

fits
#> # A tibble: 12 × 9
#>    models.id cv_pairs.id train_p               fit target      model
#>        &lt;chr>       &lt;chr>   &lt;dbl>            &lt;list>  &lt;chr>      &lt;chr>
#> 1          1           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 2         10           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 3         11           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 4         12           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 5          2           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 6          3           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 7          4           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 8          5           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 9          6           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 10         7           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 11         8           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> 12         9           1       1 &lt;S3: xgb.Booster> cancer pl_xgboost
#> # ... with 3 more variables: params &lt;list>, train &lt;list>, test &lt;list>
</code></pre>
<p>Looks like all the models learned OK. Let’s write a custom function to extract model accuracy and examine the results:</p>
<pre><code>accuracy &lt;- function(fit, data, target_var) {
  # Convert resample object to data frame
  data &lt;- as.data.frame(data)
  # Get feature matrix and labels
  X &lt;- data %>%
    select(-matches(target_var)) %>% 
    as.matrix()
  y &lt;- data[[target_var]]
  # Obtain predicted class
  y_hat &lt;- as.numeric(predict(fit, X) > .5)
  # Return accuracy
  mean(y_hat == y)
}

results &lt;- fits %>% 
  mutate(
    # hyperparameters
    nrounds   = map_dbl(params, &quot;nrounds&quot;),
    eta       = map_dbl(params, &quot;eta&quot;),
    max_depth = map_dbl(params, &quot;max_depth&quot;),
    # Accuracy
    accuracy_train = pmap_dbl(list(fit, train, target), accuracy),
    accuracy_test  = pmap_dbl(list(fit, test,  target), accuracy)
  ) %>% 
  # Select columns and order rows
  select(nrounds, eta, max_depth, contains(&quot;accuracy&quot;)) %>% 
  arrange(desc(accuracy_test), desc(accuracy_train))

results
#> # A tibble: 12 × 5
#>    nrounds   eta max_depth accuracy_train accuracy_test
#>      &lt;dbl> &lt;dbl>     &lt;dbl>          &lt;dbl>         &lt;dbl>
#> 1       25   0.3         6      1.0000000     0.9489051
#> 2       25   0.3         4      1.0000000     0.9489051
#> 3       10   0.3         6      0.9981685     0.9489051
#> 4        5   0.3         6      0.9945055     0.9489051
#> 5       10   0.1         6      0.9945055     0.9489051
#> 6       25   0.1         6      0.9945055     0.9489051
#> 7        5   0.1         6      0.9926740     0.9489051
#> 8       25   0.1         4      0.9890110     0.9489051
#> 9       10   0.3         4      0.9871795     0.9489051
#> 10       5   0.3         4      0.9853480     0.9489051
#> 11      10   0.1         4      0.9853480     0.9416058
#> 12       5   0.1         4      0.9835165     0.9416058
</code></pre>
<p>Our top model, which got 94.89% on a test set, had <code>nrounds</code> = 25, <code>eta</code> = 0.3, and <code>max_depth</code> = 6.</p>
<p>Either way, the trick was the wrapper function <code>pl_xgboost()</code> that let us bridge xgboost and pipelearner. Note that this same principle can be used for any other machine learning functions that don’t play nice with pipelearner.</p>
<h2><a href="https://drsimonj.svbtle.com/#bonus-bootstrapped-cross-validation_2"> </a>Bonus: bootstrapped cross validation</h2>
<p>For those of you who are comfortable, below is a bonus example of using 100 boostrapped cross validation samples to examine consistency in the accuracy. It doesn’t get much easier than using pipelearner!</p>
<pre><code>results &lt;- pipelearner(d, pl_xgboost, cancer ~ ., nrounds = 25) %>% 
  learn_cvpairs(n = 100) %>% 
  learn() %>% 
  mutate(
    test_accuracy  = pmap_dbl(list(fit, test,  target), accuracy)
  )
#> [1]  train-rmse:0.357471 
#> [2]  train-rmse:0.256735 
#> ...

results %>% 
  ggplot(aes(test_accuracy)) +
    geom_histogram(bins = 30) +
    scale_x_continuous(labels = scales::percent) +
    theme_minimal() +
    labs(x = &quot;Accuracy&quot;, y = &quot;Number of samples&quot;,
         title = &quot;Test accuracy distribution for\n100 bootstrapped samples&quot;)
</code></pre>
<p><a href="https://svbtleusercontent.com/jyv67ubdt2alxa.jpg"></a></p>
<h2><a href="https://drsimonj.svbtle.com/#sign-off_2"> </a>Sign off</h2>
<p>Thanks for reading and I hope this was useful for you.</p>
<p>For updates of recent blog posts, follow <a href="https://twitter.com/drsimonj">@drsimonj</a> on Twitter, or email me at <a href="mailto:drsimonjackson@gmail.com">drsimonjackson@gmail.com</a> to get in touch.</p>
<p>If you’d like the code that produced this blog, check out the <a href="https://github.com/drsimonj/blogR">blogR GitHub repository</a>.</p>