<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/tuning-xgboost-in-r-part-ii/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://insightr.wordpress.com/2018/07/28/tuning-xgboost-in-r-part-ii/"> R – insightR</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<div><strong>By Gabriel Vasconcelos</strong></div>
</div>
<div></div>
<div></div>
<p>In <a rel="nofollow" href="https://insightr.wordpress.com/2018/05/17/tuning-xgboost-in-r-part-i/" target="_blank">this previous post</a> I discussed some of the parameters we have to tune to estimate a boosting model using the xgboost package. In this post I will discuss the two parameters that were left out in part I, which are the gamma and the min_child_weight. These two parameters are much less obvious to understand but they can significantly change the results. Unfortunately, the best way to set them changes from dataset to dataset and we have to test a few values to select the best model. Note that there are many other parameters in the xgboost package. I am only showing the ones I use more.</p>
<p><span></span></p>
<h2>Parameters</h2>
<ul>
<li>
<p><strong>min_child_weigth:</strong> This parameter puts a lower bound on the number of instances in a terminal node. Bigger values require more instances in terminal nodes, which makes each tree in the boosting smaller and the algorithm more conservative.</p>
</li>
<li>
<p><strong>gamma:</strong> Controls the minimum reduction in the loss function required to grow a new node in a tree. This parameter is sensitive to the scale of the loss function, which will be linked to the scale of your response variable. The xgboost documentation defines general loss functions as <img alt="L(theta) = sum_i ell(theta)" title="L(theta) = sum_i ell(theta)" src="https://s0.wp.com/latex.php?latex=L%28%5Ctheta%29+%3D+%5Csum_i+%5Cell%28%5Ctheta%29&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex">. In the quadratic loss, for example, we have <img alt="ell(theta) = (y_i - hat{y}_i)^2" title="ell(theta) = (y_i - hat{y}_i)^2" src="https://s0.wp.com/latex.php?latex=%5Cell%28%5Ctheta%29+%3D+%28y_i+-+%5Chat%7By%7D_i%29%5E2&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex">. If we want to use a gamma that requires a RMSE reduction in a split of at least <img alt="m" title="m" src="https://s0.wp.com/latex.php?latex=m&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex"> our gamma must be in the order of <img alt="(m*N)^2" title="(m*N)^2" src="https://s0.wp.com/latex.php?latex=%28m%2AN%29%5E2&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex">, where <img alt="N" title="N" src="https://s0.wp.com/latex.php?latex=N&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex"> is the sample size.</p>
</li>
</ul>
<h2>Examples</h2>
<p>I will use a housing dataset from the Ecdat package, which includes 546 observations of sold houses with their prices and characteristics. The objective is to change the two parameters keeping everything else constant and see what happens.</p>
<pre>library(xgboost)
library(ggplot2)
library(reshape2)
library(Ecdat)

data = Housing
N = nrow(data)

# = divided price by 10000 to use smaller values of gamma = #
y = data$price/10000
x = data[,-1]
# = Transform categorical variables into dummies = #
for(i in 1:ncol(x)){
  if(is.factor(x[,i])){
    x[,i] = ifelse(x[,i] == &quot;yes&quot;,1,0)
  }
}
x = as.matrix(x)

# = select train and test indexes = #
set.seed(1)
train=sample(1:N,456)
test=setdiff(1:N,train)

# = min_child_weights candidates = #
mcw=c(1,10,100,400)
# = gamma candidates = #
gamma=c(0.1,1,10,100)

# = train and test data = #
xtrain = x[train,]
ytrain = y[train]
xtest = x[test,]
ytest = y[test]
</pre>
<h3>min_child_weight</h3>
<p>The code below runs the boosting for the four min_child_weights in the mcw vector. You can see the convergence and the test RMSE for each setup below the code. The results show a significant change in the model as we move on the min_child_weight. Values of 1 and 10 seems to overfit the data a little while and a value of 400 ignores a lot of information and returns a poor predictive model. The best solution in this case was for min_child_weight = 100.</p>
<pre>set.seed(1)
conv_mcw = matrix(NA,500,length(mcw))
pred_mcw = matrix(NA,length(test), length(mcw))
colnames(conv_mcw) = colnames(pred_mcw) = mcw
for(i in 1:length(mcw)){
  params = list(eta = 0.1, colsample_bylevel=2/3,
              subsample = 1, max_depth = 6,
              min_child_weight = mcw[i], gamma = 0)
  xgb = xgboost(xtrain, label = ytrain, nrounds = 500, params = params)
  conv_mcw[,i] = xgb$evaluation_log$train_rmse
  pred_mcw[,i] = predict(xgb, xtest)
}

conv_mcw = data.frame(iter=1:500, conv_mcw)
conv_mcw = melt(conv_mcw, id.vars = &quot;iter&quot;)
ggplot(data = conv_mcw) + geom_line(aes(x = iter, y = value, color = variable))
</pre>
<p><img alt="plot of chunk unnamed-chunk-2" src="https://i0.wp.com/i.imgur.com/wZDmKv1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<pre>(RMSE_mcw = sqrt(colMeans((ytest-pred_mcw)^2)))
</pre>
<pre>##        1       10      100      400
## 1.744679 1.751487 1.690150 2.607713
</pre>
<h3>gamma</h3>
<p>The results for gamma are similar to the min_child_weight. The models in the middle (gamma = 1 and gamma = 10) are superior in terms of predictive accuracy. Unfortunately, the convergence plot does not give us any clue on which model is the best. We have to test the model in a test sample or in a cross-validation scheme to select the most accurate. Both the min_child_weight and the gamma are applying shrinkage to the trees by limiting their size, bu they look at different measures to do so.</p>
<pre>set.seed(1)
conv_gamma = matrix(NA,500,length(gamma))
pred_gamma = matrix(NA,length(test), length(gamma))
colnames(conv_gamma) = colnames(pred_gamma) = gamma
for(i in 1:length(gamma)){
  params = list(eta = 0.1, colsample_bylevel=2/3,
              subsample = 1, max_depth = 6, min_child_weight = 1,
              gamma = gamma[i])
  xgb = xgboost(xtrain, label = ytrain, nrounds = 500, params = params)
  conv_gamma[,i] = xgb$evaluation_log$train_rmse
  pred_gamma[,i] = predict(xgb, xtest)
}

conv_gamma = data.frame(iter=1:500, conv_gamma)
conv_gamma = melt(conv_gamma, id.vars = &quot;iter&quot;)
ggplot(data = conv_gamma) + geom_line(aes(x = iter, y = value, color = variable))
</pre>
<p><img alt="plot of chunk unnamed-chunk-4" src="https://i0.wp.com/i.imgur.com/LLRJCac.png?w=456&ssl=1" data-recalc-dims="1"></p>
<pre>(RMSE_gamma = sqrt(colMeans((ytest-pred_gamma)^2)))
</pre>
<pre>##      0.1        1       10      100
## 1.810786 1.737987 1.739655 1.909785
</pre>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://insightr.wordpress.com/2018/07/28/tuning-xgboost-in-r-part-ii/"> R – insightR</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/elwKE0GOKpY" height="1"></a></div>