<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/bayesian-optimization-of-machine-learning-models/"></div></div><div><a data-url="http://www.r-bloggers.com/bayesian-optimization-of-machine-learning-models/" data-counturl="http://www.r-bloggers.com/bayesian-optimization-of-machine-learning-models/" data-count="horizontal" data-text="Bayesian Optimization of Machine Learning Models" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/06/bayesian-optimization-of-machine-learning-models.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>by Max Kuhn: Director, Nonclinical Statistics, Pfizer</p>
<div>
<p>Many predictive and machine learning models have structural or <em>tuning</em> parameters that cannot be directly estimated from the data. For example, when using <em>K</em>-nearest neighbor model, there is no analytical estimator for <em>K</em> (the number of neighbors). Typically, resampling is used to get good performance estimates of the model for a given set of values for <em>K</em> and the one associated with the best results is used. This is basically a grid search procedure. However, there are other approaches that can be used. I’ll demonstrate how Bayesian optimization and Gaussian process models can be used as an alternative.</p>
<p>To demonstrate, I’ll use the regression simulation system of <a target="_blank" href="https://scholar.google.com/scholar?cluster=18280745619000490842&hl=en&as_sdt=0,5">Sapp et al. (2014)</a> where the predictors (i.e. <code>x</code>’s) are independent Gaussian random variables with mean zero and a variance of 9. The prediction equation is:</p>
<pre>x_1 + sin(x_2) + log(abs(x_3)) + x_4^2 + x_5*x_6 + 
I(x_7*x_8*x_9 &lt; 0) + I(x_10 > 0) + x_11*I(x_11 > 0) + 
sqrt(abs(x_12)) + cos(x_13) + 2*x_14 + abs(x_15) + 
I(x_16 &lt; -1) + x_17*I(x_17 &lt; -1) - 2 * x_18 - x_19*x_20
</pre>
<p>The random error here is also Gaussian with mean zero and a variance of 9. This simulation is available in the <a target="_blank" href="https://cran.r-project.org/package=caret"><code>caret</code></a> package via a function called <code>SLC14_1</code>. First, we’ll simulate a training set of 250 data points and also a larger set that we will use to elucidate the true parameter surface:</p>
<pre><code>> library(caret)
> set.seed(7210)
> train_dat &lt;- SLC14_1(250)
> large_dat &lt;- SLC14_1(10000)</code></pre>
<p>We will use a radial basis function support vector machine to model these data. For a fixed epsilon, the model will be tuned over the cost value and the radial basis kernel parameter, commonly denotes as <code>sigma</code>. Since we are simulating the data, we can figure out a good approximation to the relationship between these parameters and the root mean squared error (RMSE) or the model. Given our specific training set and the larger simulated sample, here is the RMSE surface for a wide range of values:</p>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d1f3992b970c-pi"><img title="Global" alt="Global" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d1f3992b970c-800wi" border="0"></a><br>
<br>
<br>
<br>
<br>
<br>
<br>

</p>
<div>
<p>There is a wide range of parameter values that are associated with very low RMSE values in the northwest.</p>
</div>
<p>A simple way to get an initial assessment is to use <a target="_blank" href="http://www.jmlr.org/papers/v13/bergstra12a.html">random search</a> where a set of random tuning parameter values are generated across a “wide range”. For a RBF SVM, <a target="_blank" href="https://cran.r-project.org/package=caret"><code>caret</code></a>’s <code>train</code> function defines wide as cost values between <code>2^c(-5, 10)</code> and <code>sigma</code> values inside the range produced by the <code>sigest</code> function in the <a target="_blank" href="https://cran.r-project.org/package=kernlab"><code>kernlab</code></a> package. This code will do 20 random sub-models in this range:</p>
<pre><code>> rand_ctrl &lt;- trainControl(method = &quot;repeatedcv&quot;, repeats = 5,
+                           search = &quot;random&quot;)
> 
> set.seed(308) 
> rand_search &lt;- train(y ~ ., data = train_dat,
+                      method = &quot;svmRadial&quot;,
+                      ## Create 20 random parameter values
+                      tuneLength = 20,
+                      metric = &quot;RMSE&quot;,
+                      preProc = c(&quot;center&quot;, &quot;scale&quot;),
+                      trControl = rand_ctrl)</code></pre>
<pre><code>> rand_search</code></pre>
<pre><code>Support Vector Machines with Radial Basis Function Kernel 

250 samples
 20 predictor

Pre-processing: centered (20), scaled (20) 
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 226, 224, 224, 225, 226, 224, ... 
Resampling results across tuning parameters:

  sigma       C             RMSE      Rsquared 
  0.01161955   42.75789360  10.50838  0.7299837
  0.01357777   67.97672171  10.71276  0.7212605
  0.01392676  828.08072944  10.75235  0.7195869
  0.01394119    0.18386619  18.56921  0.2109284
  0.01538656    0.05224914  19.33310  0.1890599
  0.01711920  228.59215128  11.09522  0.7047713
  0.01790202    0.78835920  16.78597  0.3217203
  0.01936110    0.91401289  16.45485  0.3492278
  0.02023763    0.07658831  19.03987  0.2081059
  0.02690269    0.04128731  19.33974  0.2126950
  0.02780880    0.64865483  16.52497  0.3545042
  0.02920113  974.08943821  12.22906  0.6508754
  0.02963586    1.19350198  15.46690  0.4407725
  0.03370625   31.45179445  12.60653  0.6314384
  0.03561750    0.04970422  19.23564  0.2306298
  0.03752561    0.06592800  19.07130  0.2375616
  0.03783570  398.44599747  12.92958  0.6143790
  0.04534046    3.91017571  13.56612  0.5798001
  0.05171719  296.65916049  13.88865  0.5622445
  0.06482201   47.31716568  14.66904  0.5192667

RMSE was used to select the optimal model using  the smallest value.
The final values used for the model were sigma = 0.01161955 and C
 = 42.75789. </code></pre>
<pre><code>> ggplot(rand_search) + scale_x_log10() + scale_y_log10()<br> <a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb090d3c10970d-pi"><img title="Random" alt="Random" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb090d3c10970d-800wi" border="0"><br></a><br></code><span>></span> <span>getTrainPerf</span><span>(</span><span>rand_search</span><span>)</span></pre>
<pre><code>  TrainRMSE TrainRsquared    method
1  10.50838     0.7299837 svmRadial</code></pre>
<p>There are other approaches that we can take, including a more comprehensive grid search or using a nonlinear optimizer to find better values of cost and <code>sigma</code>. Another approach is to use <a target="_blank" href="https://scholar.google.com/scholar?hl=en&q=%22Bayesian+optimization%22&btnG=&as_sdt=1%2C22&as_sdtp=">Bayesian optimization</a> to find good values for these parameters. This is an optimization scheme that uses Bayesian models based on <a target="_blank" href="https://en.wikipedia.org/wiki/Gaussian_process">Gaussian processes</a> to predict good tuning parameters.</p>
<p><a target="_blank" href="http://www.gaussianprocess.org/gpml/">Gaussian Process (GP) regression</a> is used to facilitate the Bayesian analysis. If creates a regression model to formalize the relationship between the outcome (RMSE, in this application) and the SVM tuning parameters. The standard assumption regarding normality of the residuals is used and, being a Bayesian model, the regression parameters also gain a prior distribution that is multivariate normal. The GP regression model uses a kernel basis expansion (much like the SVM model does) in order to allow the model to be nonlinear in the SVM tuning parameters. To do this, a radial basis function kernel is used for the covariance function of the multivariate normal prior and maximum likelihood is used to estimate the kernel parameters of the GP.</p>
<p>In the end, the GP regression model can take the current set of resampled RMSE values and make predictions over the entire space of potential cost and <code>sigma</code> parameters. The Bayesian machinery allows of this prediction to have a <em>distribution</em>; for a given set of tuning parameters, we can obtain the estimated mean RMSE values as well as an estimate of the corresponding prediction variance. For example, if we were to use our data from the random search to build a GP model, the predicted mean RMSE would look like:</p>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb090d3c9c970d-pi"><img title="Gp_mean" alt="Gp_mean" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb090d3c9c970d-800wi" border="0"></a><br>
<br>
<br>
<br>
<br>
<br>
<br>

</p>
<div>
<p>The darker regions indicate smaller RMSE values given the current resampling results. The predicted standard deviation of the RMSE is:</p>
</div>
</div>
<div> </div>
<div><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c869bd94970b-pi"><img title="Gp_stdev" alt="Gp_stdev" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c869bd94970b-800wi" border="0"></a>
<div>
<p>The prediction noise becomes larger (e.g. darker) as we move away from the current set of observed values.</p>
</div>
<p>(The <a target="_blank" href="https://www.jstatsoft.org/article/view/v064i12"><code>GPfit</code></a> package was used to create these models.)</p>
<p>To find good parameters to test, there are several approaches. <a target="_blank" href="http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf">This paper (pdf)</a> outlines several but we will use the <a target="_blank" href="https://arxiv.org/abs/0912.3995"><em>confidence bound</em> approach</a>. For any combination of cost and <code>sigma</code>, we can compute the lower confidence bound of the predicted RMSE. Since this takes the uncertainty of prediction into account it has the potential to produce better directions to take the optimization. Here is a plot of the confidence bound using a single standard deviation of the predicted mean:</p>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c869bdd9970b-pi"><img title="Cb" alt="Cb" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c869bdd9970b-800wi" border="0"></a><br>
<br>
<br>
<br>
<br>
<br>
<br>

</p>
<div>
<p>Darker values indicate better conditions to explore. Since we know the true RMSE surface, we can see that the best region (the northwest) is estimated to be an interesting location to take the optimization. The optimizer would pick a good location based on this model and evaluate this as the next parameter value. This most recent configuration is added to the GP’s training set and the process continues for a pre-specified number of iterations.</p>
</div>
<p>Yachen Yan created an <a target="_blank" href="https://cran.r-project.org/package=rBayesianOptimization">R package</a> for Bayesian optimization. He also made <a target="_blank" href="https://github.com/yanyachen/rBayesianOptimization/issues/1">a modification</a> so that we can use our initial random search as the substrate to the first GP used. To search a much wider parameter space, our code looks like:</p>
<pre><code>> ## Define the resampling method
> ctrl &lt;- trainControl(method = &quot;repeatedcv&quot;, repeats = 5)
> 
> ## Use this function to optimize the model. The two parameters are 
> ## evaluated on the log scale given their range and scope. 
> svm_fit_bayes &lt;- function(logC, logSigma) {
+   ## Use the same model code but for a single (C, sigma) pair. 
+   txt &lt;- capture.output(
+     mod &lt;- train(y ~ ., data = train_dat,
+                  method = &quot;svmRadial&quot;,
+                  preProc = c(&quot;center&quot;, &quot;scale&quot;),
+                  metric = &quot;RMSE&quot;,
+                  trControl = ctrl,
+                  tuneGrid = data.frame(C = exp(logC), sigma = exp(logSigma)))
+   )
+   ## The function wants to _maximize_ the outcome so we return 
+   ## the negative of the resampled RMSE value. `Pred` can be used
+   ## to return predicted values but we'll avoid that and use zero
+   list(Score = -getTrainPerf(mod)[, &quot;TrainRMSE&quot;], Pred = 0)
+ }
> 
> ## Define the bounds of the search. 
> lower_bounds &lt;- c(logC = -5, logSigma = -9)
> upper_bounds &lt;- c(logC = 20, logSigma = -0.75)
> bounds &lt;- list(logC = c(lower_bounds[1], upper_bounds[1]),
+                logSigma = c(lower_bounds[2], upper_bounds[2]))
> 
> ## Create a grid of values as the input into the BO code
> initial_grid &lt;- rand_search$results[, c(&quot;C&quot;, &quot;sigma&quot;, &quot;RMSE&quot;)]
> initial_grid$C &lt;- log(initial_grid$C)
> initial_grid$sigma &lt;- log(initial_grid$sigma)
> initial_grid$RMSE &lt;- -initial_grid$RMSE
> names(initial_grid) &lt;- c(&quot;logC&quot;, &quot;logSigma&quot;, &quot;Value&quot;)
> 
> ## Run the optimization with the initial grid and do
> ## 30 iterations. We will choose new parameter values
> ## using the upper confidence bound using 1 std. dev. 
> 
> library(rBayesianOptimization)
> 
> set.seed(8606)
> ba_search &lt;- BayesianOptimization(svm_fit_bayes,
+                                   bounds = bounds,
+                                   init_grid_dt = initial_grid, 
+                                   init_points = 0, 
+                                   n_iter = 30,
+                                   acq = &quot;ucb&quot;, 
+                                   kappa = 1, 
+                                   eps = 0.0,
+                                   verbose = TRUE)</code></pre>
<pre><code>20 points in hyperparameter space were pre-sampled
elapsed = 1.53  	Round = 21  logC = 5.4014   logSigma = -5.8974  Value = -10.8148 
elapsed = 1.54  	Round = 22  logC = 4.9757   logSigma = -5.0449  Value = -9.7936 
elapsed = 1.42  	Round = 23  logC = 5.7551   logSigma = -5.0244  Value = -9.8128 
elapsed = 1.30  	Round = 24  logC = 5.2754   logSigma = -4.9678  Value = -9.7530 
elapsed = 1.39  	Round = 25  logC = 5.3009   logSigma = -5.0921  Value = -9.5516 
elapsed = 1.48  	Round = 26  logC = 5.3240   logSigma = -5.2313  Value = -9.6571 
elapsed = 1.39  	Round = 27  logC = 5.3750   logSigma = -5.1152  Value = -9.6619 
elapsed = 1.44  	Round = 28  logC = 5.2356   logSigma = -5.0969  Value = -9.4167 
elapsed = 1.38  	Round = 29  logC = 11.8347  logSigma = -5.1074  Value = -9.6351 
elapsed = 1.42  	Round = 30  logC = 15.7494  logSigma = -5.1232  Value = -9.4243 
elapsed = 25.24 	Round = 31  logC = 14.6657  logSigma = -7.9164  Value = -8.8410 
elapsed = 32.60 	Round = 32  logC = 18.3793  logSigma = -8.1083  Value = -8.7139 
elapsed = 1.86  	Round = 33  logC = 20.0000  logSigma = -5.6297  Value = -9.0580 
elapsed = 0.97  	Round = 34  logC = 20.0000  logSigma = -1.5768  Value = -19.2183 
elapsed = 5.92  	Round = 35  logC = 17.3827  logSigma = -6.6880  Value = -9.0224 
elapsed = 18.01 	Round = 36  logC = 20.0000  logSigma = -7.6071  Value = -8.5728 
elapsed = 114.49	Round = 37  logC = 16.0079  logSigma = -9.0000  Value = -8.7058 
elapsed = 89.31		Round = 38  logC = 12.8319  logSigma = -9.0000  Value = -8.6799 
elapsed = 99.29		Round = 39  logC = 20.0000  logSigma = -9.0000  Value = -8.5596 
elapsed = 106.88	Round = 40  logC = 14.1190  logSigma = -9.0000  Value = -8.5150 
elapsed = 4.84		Round = 41  logC = 13.4694  logSigma = -6.5271  Value = -8.9728 
elapsed = 108.37	Round = 42  logC = 19.0216  logSigma = -9.0000  Value = -8.7461 
elapsed = 52.43 	Round = 43  logC = 13.5273  logSigma = -8.5130  Value = -8.8728 
elapsed = 39.69 	Round = 44  logC = 20.0000  logSigma = -8.3288  Value = -8.4956 
elapsed = 5.99  	Round = 45  logC = 20.0000  logSigma = -6.7208  Value = -8.9455 
elapsed = 113.01	Round = 46  logC = 14.9611  logSigma = -9.0000  Value = -8.7576 
elapsed = 27.45 	Round = 47  logC = 19.6181  logSigma = -7.9872  Value = -8.6186 
elapsed = 116.00	Round = 48  logC = 17.3060  logSigma = -9.0000  Value = -8.6820 
elapsed = 2.26  	Round = 49  logC = 14.2698  logSigma = -5.8297  Value = -9.1837 
elapsed = 64.50 	Round = 50  logC = 20.0000  logSigma = -8.6438  Value = -8.6914 

Best Parameters Found: 
Round = 44  logC = 20.0000  logSigma = -8.3288  Value = -8.4956 </code></pre>
<p>Animate the search!</p>
</div>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c869bebc970b-pi"><img title="Bopt" alt="Bopt" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c869bebc970b-800wi" border="0"></a><br>
<br>
<br>
<br>
<br>
<br>
<br>

</p>
<div>
<p>The final settings were found at iteration 44 with a cost setting of 485,165,195 and <code>sigma</code>=0.0002043. I would have never thought to evaluate a cost parameter so large and the algorithm wants to make it even larger. Does it really work?</p>
</div>
<p>We can fit a model based on the new configuration and compare it to random search in terms of the resampled RMSE and the RMSE on the test set:</p>
<pre><code>> set.seed(308)
> final_search &lt;- train(y ~ ., data = train_dat,
+                       method = &quot;svmRadial&quot;,
+                       tuneGrid = data.frame(C = exp(ba_search$Best_Par[&quot;logC&quot;]), 
+                                             sigma = exp(ba_search$Best_Par[&quot;logSigma&quot;])),
+                       metric = &quot;RMSE&quot;,
+                       preProc = c(&quot;center&quot;, &quot;scale&quot;),
+                       trControl = ctrl)
> compare_models(final_search, rand_search)</code></pre>
<pre><code>
    One Sample t-test

data:  x
t = -9.0833, df = 49, p-value = 4.431e-12
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -2.34640 -1.49626
sample estimates:
mean of x 
 -1.92133 </code></pre>
<pre><code>> postResample(predict(rand_search, large_dat), large_dat$y)</code></pre>
<pre><code>      RMSE   Rsquared 
10.1112280  0.7648765 </code></pre>
<pre><code>> postResample(predict(final_search, large_dat), large_dat$y)</code></pre>
<pre><code>     RMSE  Rsquared 
8.2668843 0.8343405 </code></pre>
<p>Much Better!</p>
<p>Thanks to Yachen Yan for making the <a target="_blank" href="https://cran.r-project.org/package=rBayesianOptimization"><code>rBayesianOptimization</code></a> package.</p>
<p><br>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/bayesian-optimization-of-machine-learning-models/"></div></div><div><a data-url="http://www.r-bloggers.com/bayesian-optimization-of-machine-learning-models/" data-counturl="http://www.r-bloggers.com/bayesian-optimization-of-machine-learning-models/" data-count="vertical" data-text="Bayesian Optimization of Machine Learning Models" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/06/bayesian-optimization-of-machine-learning-models.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/-iM0-LK-XqU"></a></div>