<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/using-caret-to-compare-models/"></div></div><div><a data-url="http://www.r-bloggers.com/using-caret-to-compare-models/" data-counturl="http://www.r-bloggers.com/using-caret-to-compare-models/" data-count="horizontal" data-text="Using caret to compare models" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/05/using-caret-to-compare-models.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>by Joseph Rickert</p>
<p>The model table on the <a target="_blank" href="http://topepo.github.io/caret/index.html">caret package website</a> lists more that 200 variations of predictive analytics models that are available withing the caret framework. All of these models may be prepared, tuned, fit and evaluated with a common set of caret functions. All on its own, the table is an impressive testament to the utility and scope of the R language as data science tool. </p>
<p><iframe height="150" allowfullscreen="" width="525" src="http://topepo.github.io/caret/modelList.html"></iframe></p>
<p>For the past year or so <a target="_blank" href="https://mran.revolutionanalytics.com/package/xgboost/">xgboost</a>, the extreme gradient boosting algorithm, has been getting a lot of attention. The code below compares <a target="_blank" href="https://mran.revolutionanalytics.com/package/gbm/">gbm</a> with xgboost using the segmentationData set that comes with caret. The analysis presented here is far from the last word on comparing these models, but it does show how one might go about setting up a serious comparison using caret's functions to sweep through parameter space using parallel programming, and then used synchronized bootstrap samples to make a detailed comparison.</p>
<p>After reading in the data and dividing it into training and test data sets, caret's <span>trainControl()</span> and <span>expand.grid()</span> functions are used to set up to train the gbm model on all of the combinations of represented in the data frame built by <span>expand.grid()</span>. Then <span>train()</span> function does the actual training and fitting of the model. Notice that all of this happens in parallel. The Task Manager on my Windows 10 laptop shows all four cores maxed out at 100%.</p>
<p>After model fitting, predictions on the test data are computed and an ROC curve is drawn in the usual way. The AUC for gbm was computed to be 0.8731. Here is the ROC curve.</p>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c86693ec970b-pi"><img title="GBM_ROC" alt="GBM_ROC" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c86693ec970b-800wi" border="0"></a></p>
<p>Next, a similar process for xgboost computes the AUC to be 0.8857, a fair improvement. The following plot shows how the ROC measure behaves with increasing tree depth for the two different values of the shrinkage parameter.</p>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d1f0953d970c-pi"><img title="Xgboost_treevsshrink" alt="Xgboost_treevsshrink" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d1f0953d970c-800wi" border="0"></a></p>
<p> </p>
<p>The final section of code shows how to caret can be used to compare the two models using the bootstrap samples that were created in the process of constructing the two models. The boxplots show xgboost has the edge although the gbm has a tighter distribution.<br> </p>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb090a11a5970d-pi"><img title="Boxplot_gbmvsxgboost" alt="Boxplot_gbmvsxgboost" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb090a11a5970d-800wi" border="0"></a></p>
<p>The next step, which I hope to take soon, is to rerun the analysis with more complete grids of tuning parameters. For a very accessible introduction to caret have a look at <a target="_blank" href="https://www.r-project.org/nosvn/conferences/useR-2013/Tutorials/kuhn/user_caret_2up.pdf">Max Kuhn's 2013 useR! tutorial</a>.</p>
<div>
<div>
<pre><span>#COMPARE XGBOOST with GBM</span>
<span>### Packages Required</span>
 
<a target="_blank" href="http://inside-r.org/r-doc/base/library"><span>library</span></a><span>(</span><a target="_blank" href="http://inside-r.org/packages/cran/caret">caret</a><span>)</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/library"><span>library</span></a><span>(</span><a target="_blank" href="http://inside-r.org/packages/cran/corrplot">corrplot</a><span>)</span>			<span># plot correlations</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/library"><span>library</span></a><span>(</span>doParallel<span>)</span>		<span># parallel processing</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/library"><span>library</span></a><span>(</span>dplyr<span>)</span>        <span># Used by caret</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/library"><span>library</span></a><span>(</span><a target="_blank" href="http://inside-r.org/packages/cran/gbm">gbm</a><span>)</span>				  <span># GBM Models</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/library"><span>library</span></a><span>(</span>pROC<span>)</span>				  <span># plot the ROC curve</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/library"><span>library</span></a><span>(</span>xgboost<span>)</span>      <span># Extreme Gradient Boosting</span>
 
 
<span>### Get the Data</span>
<span># Load the data and construct indices to divied it into training and test data sets.</span>
 
<a target="_blank" href="http://inside-r.org/r-doc/utils/data"><span>data</span></a><span>(</span>segmentationData<span>)</span>  	<span># Load the segmentation data set</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/dim"><span>dim</span></a><span>(</span>segmentationData<span>)</span>
<a target="_blank" href="http://inside-r.org/r-doc/utils/head"><span>head</span></a><span>(</span>segmentationData<span>,</span><span>2</span><span>)</span>
<span>#</span>
trainIndex &lt;- createDataPartition<span>(</span>segmentationData$Case<span>,</span>p=<span>.5</span><span>,</span><a target="_blank" href="http://inside-r.org/r-doc/base/list"><span>list</span></a>=<span>FALSE</span><span>)</span>
trainData &lt;- segmentationData<span>[</span>trainIndex<span>,</span>-<a target="_blank" href="http://inside-r.org/r-doc/base/c"><span>c</span></a><span>(</span><span>1</span><span>,</span><span>2</span><span>)</span><span>]</span>
testData  &lt;- segmentationData<span>[</span>-trainIndex<span>,</span>-<a target="_blank" href="http://inside-r.org/r-doc/base/c"><span>c</span></a><span>(</span><span>1</span><span>,</span><span>2</span><span>)</span><span>]</span>
<span>#</span>
trainX &lt;-trainData<span>[</span><span>,</span>-<span>1</span><span>]</span>        <span># Pull out the dependent variable</span>
testX &lt;- testData<span>[</span><span>,</span>-<span>1</span><span>]</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/sapply"><span>sapply</span></a><span>(</span>trainX<span>,</span><a target="_blank" href="http://inside-r.org/r-doc/base/summary"><span>summary</span></a><span>)</span> <span># Look at a summary of the training data</span>
 
<span>## GENERALIZED BOOSTED RGRESSION MODEL (BGM) </span>
 
<span># Set up training control</span>
ctrl &lt;- trainControl<span>(</span>method = <span>&quot;repeatedcv&quot;</span><span>,</span>   <span># 10fold cross validation</span>
                     number = <span>5</span><span>,</span>							<span># do 5 repititions of cv</span>
                     summaryFunction=twoClassSummary<span>,</span>	<span># Use AUC to pick the best model</span>
                     classProbs=<span>TRUE</span><span>,</span>
                     allowParallel = <span>TRUE</span><span>)</span>
 
<span># Use the expand.grid to specify the search space	</span>
<span># Note that the default search grid selects multiple values of each tuning parameter</span>
 
<a target="_blank" href="http://inside-r.org/r-doc/graphics/grid"><span>grid</span></a> &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/expand.grid"><span>expand.grid</span></a><span>(</span>interaction.depth=<a target="_blank" href="http://inside-r.org/r-doc/base/c"><span>c</span></a><span>(</span><span>1</span><span>,</span><span>2</span><span>)</span><span>,</span> <span># Depth of variable interactions</span>
                    n.trees=<a target="_blank" href="http://inside-r.org/r-doc/base/c"><span>c</span></a><span>(</span><span>10</span><span>,</span><span>20</span><span>)</span><span>,</span>	        <span># Num trees to fit</span>
                    shrinkage=<a target="_blank" href="http://inside-r.org/r-doc/base/c"><span>c</span></a><span>(</span><span>0.01</span><span>,</span><span>0.1</span><span>)</span><span>,</span>		<span># Try 2 values for learning rate </span>
                    n.minobsinnode = <span>20</span><span>)</span>
<span>#	</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/set.seed"><span>set.seed</span></a><span>(</span><span>1951</span><span>)</span>  <span># set the seed</span>
 
<span># Set up to do parallel processing </span>
registerDoParallel<span>(</span><span>4</span><span>)</span>		<span># Registrer a parallel backend for train</span>
getDoParWorkers<span>(</span><span>)</span>
 
gbm.tune &lt;- train<span>(</span>x=trainX<span>,</span>y=trainData$Class<span>,</span>
                              method = <span>&quot;gbm&quot;</span><span>,</span>
                              metric = <span>&quot;ROC&quot;</span><span>,</span>
                              trControl = ctrl<span>,</span>
                              tuneGrid=<a target="_blank" href="http://inside-r.org/r-doc/graphics/grid"><span>grid</span></a><span>,</span>
                              verbose=<span>FALSE</span><span>)</span>
 
 
<span># Look at the tuning results</span>
<span># Note that ROC was the performance criterion used to select the optimal model. </span>
 
gbm.tune$bestTune
<a target="_blank" href="http://inside-r.org/r-doc/graphics/plot"><span>plot</span></a><span>(</span>gbm.tune<span>)</span>  		<span># Plot the performance of the training models</span>
res &lt;- gbm.tune$results
res
 
<span>### GBM Model Predictions and Performance</span>
<span># Make predictions using the test data set</span>
gbm.pred &lt;- <a target="_blank" href="http://inside-r.org/r-doc/stats/predict"><span>predict</span></a><span>(</span>gbm.tune<span>,</span>testX<span>)</span>
 
<span>#Look at the confusion matrix </span>
confusionMatrix<span>(</span>gbm.pred<span>,</span>testData$Class<span>)</span>   
 
<span>#Draw the ROC curve </span>
gbm.probs &lt;- <a target="_blank" href="http://inside-r.org/r-doc/stats/predict"><span>predict</span></a><span>(</span>gbm.tune<span>,</span>testX<span>,</span>type=<span>&quot;prob&quot;</span><span>)</span>
<a target="_blank" href="http://inside-r.org/r-doc/utils/head"><span>head</span></a><span>(</span>gbm.probs<span>)</span>
 
gbm.ROC &lt;- roc<span>(</span>predictor=gbm.probs$PS<span>,</span>
               response=testData$Class<span>,</span>
               <a target="_blank" href="http://inside-r.org/r-doc/base/levels"><span>levels</span></a>=<a target="_blank" href="http://inside-r.org/r-doc/base/rev"><span>rev</span></a><span>(</span><a target="_blank" href="http://inside-r.org/r-doc/base/levels"><span>levels</span></a><span>(</span>testData$Class<span>)</span><span>)</span><span>)</span>
gbm.ROC$auc
<span>#Area under the curve: 0.8731</span>
<a target="_blank" href="http://inside-r.org/r-doc/graphics/plot"><span>plot</span></a><span>(</span>gbm.ROC<span>,</span>main=<span>&quot;GBM ROC&quot;</span><span>)</span>
 
<span># Plot the propability of poor segmentation</span>
<a target="_blank" href="http://inside-r.org/r-doc/lattice/histogram"><span>histogram</span></a><span>(</span>~gbm.probs$PS|testData$Class<span>,</span>xlab=<span>&quot;Probability of Poor Segmentation&quot;</span><span>)</span>
<span>##----------------------------------------------</span>
<span>## XGBOOST</span>
<span># Some stackexchange guidance for xgboost</span>
<span># http://stats.stackexchange.com/questions/171043/how-to-tune-hyperparameters-of-xgboost-trees</span>
 
<span># Set up for parallel procerssing</span>
<a target="_blank" href="http://inside-r.org/r-doc/base/set.seed"><span>set.seed</span></a><span>(</span><span>1951</span><span>)</span>
registerDoParallel<span>(</span><span>4</span><span>,</span>cores=<span>4</span><span>)</span>
getDoParWorkers<span>(</span><span>)</span>
 
<span># Train xgboost</span>
xgb.grid &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/expand.grid"><span>expand.grid</span></a><span>(</span>nrounds = <span>500</span><span>,</span> <span>#the maximum number of iterations</span>
                        eta = <a target="_blank" href="http://inside-r.org/r-doc/base/c"><span>c</span></a><span>(</span><span>0.01</span><span>,</span><span>0.1</span><span>)</span><span>,</span> <span># shrinkage</span>
                        max_depth = <a target="_blank" href="http://inside-r.org/r-doc/base/c"><span>c</span></a><span>(</span><span>2</span><span>,</span><span>6</span><span>,</span><span>10</span><span>)</span><span>)</span>
 
xgb.tune &lt;-train<span>(</span>x=trainX<span>,</span>y=trainData$Class<span>,</span>
                 method=<span>&quot;xgbTree&quot;</span><span>,</span>
                 metric=<span>&quot;ROC&quot;</span><span>,</span>
                 trControl=ctrl<span>,</span>
                 tuneGrid=xgb.grid<span>)</span>
 
 
xgb.tune$bestTune
<a target="_blank" href="http://inside-r.org/r-doc/graphics/plot"><span>plot</span></a><span>(</span>xgb.tune<span>)</span>  		<span># Plot the performance of the training models</span>
res &lt;- xgb.tune$results
res
 
<span>### xgboostModel Predictions and Performance</span>
<span># Make predictions using the test data set</span>
xgb.pred &lt;- <a target="_blank" href="http://inside-r.org/r-doc/stats/predict"><span>predict</span></a><span>(</span>xgb.tune<span>,</span>testX<span>)</span>
 
<span>#Look at the confusion matrix </span>
confusionMatrix<span>(</span>xgb.pred<span>,</span>testData$Class<span>)</span>   
 
<span>#Draw the ROC curve </span>
xgb.probs &lt;- <a target="_blank" href="http://inside-r.org/r-doc/stats/predict"><span>predict</span></a><span>(</span>xgb.tune<span>,</span>testX<span>,</span>type=<span>&quot;prob&quot;</span><span>)</span>
<span>#head(xgb.probs)</span>
 
xgb.ROC &lt;- roc<span>(</span>predictor=xgb.probs$PS<span>,</span>
               response=testData$Class<span>,</span>
               <a target="_blank" href="http://inside-r.org/r-doc/base/levels"><span>levels</span></a>=<a target="_blank" href="http://inside-r.org/r-doc/base/rev"><span>rev</span></a><span>(</span><a target="_blank" href="http://inside-r.org/r-doc/base/levels"><span>levels</span></a><span>(</span>testData$Class<span>)</span><span>)</span><span>)</span>
xgb.ROC$auc
<span># Area under the curve: 0.8857</span>
 
<a target="_blank" href="http://inside-r.org/r-doc/graphics/plot"><span>plot</span></a><span>(</span>xgb.ROC<span>,</span>main=<span>&quot;xgboost ROC&quot;</span><span>)</span>
<span># Plot the propability of poor segmentation</span>
<a target="_blank" href="http://inside-r.org/r-doc/lattice/histogram"><span>histogram</span></a><span>(</span>~xgb.probs$PS|testData$Class<span>,</span>xlab=<span>&quot;Probability of Poor Segmentation&quot;</span><span>)</span>
 
 
<span># Comparing Multiple Models</span>
<span># Having set the same seed before running gbm.tune and xgb.tune</span>
<span># we have generated paired samples and are in a position to compare models </span>
<span># using a resampling technique.</span>
<span># (See Hothorn at al, &quot;The design and analysis of benchmark experiments</span>
<span># -Journal of Computational and Graphical Statistics (2005) vol 14 (3) </span>
<span># pp 675-699) </span>
 
rValues &lt;- resamples<span>(</span><a target="_blank" href="http://inside-r.org/r-doc/base/list"><span>list</span></a><span>(</span>xgb=xgb.tune<span>,</span><a target="_blank" href="http://inside-r.org/packages/cran/gbm">gbm</a>=gbm.tune<span>)</span><span>)</span>
rValues$values
<a target="_blank" href="http://inside-r.org/r-doc/base/summary"><span>summary</span></a><span>(</span>rValues<span>)</span>
 
<a target="_blank" href="http://inside-r.org/r-doc/lattice/bwplot"><span>bwplot</span></a><span>(</span>rValues<span>,</span>metric=<span>&quot;ROC&quot;</span><span>,</span>main=<span>&quot;GBM vs xgboost&quot;</span><span>)</span>	<span># boxplot</span>
<a target="_blank" href="http://inside-r.org/r-doc/lattice/dotplot"><span>dotplot</span></a><span>(</span>rValues<span>,</span>metric=<span>&quot;ROC&quot;</span><span>,</span>main=<span>&quot;GBM vs xgboost&quot;</span><span>)</span>	<span># dotplot</span>
<span>#splom(rValues,metric=&quot;ROC&quot;)</span></pre>
</div>
</div>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/using-caret-to-compare-models/"></div></div><div><a data-url="http://www.r-bloggers.com/using-caret-to-compare-models/" data-counturl="http://www.r-bloggers.com/using-caret-to-compare-models/" data-count="vertical" data-text="Using caret to compare models" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/05/using-caret-to-compare-models.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/aMndVjgn1qY"></a></div>