<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/first-release-of-mlrmbo-the-toolbox-for-bayesian-black-box-optimization/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://mlr-org.github.io/First_release_of_mlrMBO_the_toolbox_for_Bayesian_Black_Box_Optimization/"> mlr-org</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>We are happy to finally announce the first release of <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=mlrMBO"><strong>mlrMBO</strong> on cran</a> after a quite long development time.<br>
For the theoretical background and a nearly complete overview of mlrMBOs capabilities you can check our <a target="_blank" rel="nofollow" href="https://arxiv.org/abs/1703.03373">paper on <strong>mlrMBO</strong> that we presubmitted to arxiv</a>.</p>
<p>The key features of <strong>mlrMBO</strong> are:</p>
<ul>
<li>Global optimization of expensive Black-Box functions.</li>
<li>Mulit-Criteria Optimization.</li>
<li>Parallelization through multi-point proposals.</li>
<li>Support for optimization over categorical variables using random forests as a surrogate.</li>
</ul>
<p>For examples covering different scenarios we have Vignettes that are also available as an <a target="_blank" rel="nofollow" href="https://mlr-org.github.io/mlrMBO/">online documentation</a>.<br>
For <strong>mlr</strong> users <strong>mlrMBO</strong> is especially interesting for hyperparameter optimization.</p>
<p><span></span></p>
<p><strong>mlrMBO</strong> for <strong>mlr</strong> hyperparameter tuning was already used in <a target="_blank" rel="nofollow" href="https://mlr-org.github.io/How-to-win-a-drone-in-20-lines-of-R-code">an earlier blog post</a>.<br>
Nonetheless we want to provide a small toy example to demonstrate the work flow of <strong>mlrMBO</strong> in this post.</p>
<h3>Example</h3>
<p>First, we define an objective function that we are going to minimize:</p>
<figure>
<pre><code data-lang="r"><span>set.seed</span><span>(</span><span>1</span><span>)</span><span>
</span><span>library</span><span>(</span><span>mlrMBO</span><span>)</span><span>
</span><span>fun</span><span> </span><span>=</span><span> </span><span>makeSingleObjectiveFunction</span><span>(</span><span>
</span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;SineMixture&quot;</span><span>,</span><span>
</span><span>fn</span><span> </span><span>=</span><span> </span><span>function</span><span>(</span><span>x</span><span>)</span><span> </span><span>sin</span><span>(</span><span>x</span><span>[</span><span>1</span><span>])</span><span>*</span><span>cos</span><span>(</span><span>x</span><span>[</span><span>2</span><span>])</span><span>/</span><span>2</span><span> </span><span>+</span><span> </span><span>0.04</span><span> </span><span>*</span><span> </span><span>sum</span><span>(</span><span>x</span><span>^</span><span>2</span><span>),</span><span>
</span><span>par.set</span><span> </span><span>=</span><span> </span><span>makeNumericParamSet</span><span>(</span><span>id</span><span> </span><span>=</span><span> </span><span>&quot;x&quot;</span><span>,</span><span> </span><span>len</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span> </span><span>lower</span><span> </span><span>=</span><span> </span><span>-5</span><span>,</span><span> </span><span>upper</span><span> </span><span>=</span><span> </span><span>5</span><span>)</span><span>
</span><span>)</span></code></pre>
</figure>
<p>To define the objective function we use <code>makeSingleObjectiveFunction</code> from the neat package <a target="_blank" rel="nofollow" href="https://github.com/jakobbossek/smoof"><strong>smoof</strong></a>, which gives us the benefit amongst others to be able to directly visualize the function.<br>
<em>If you happen to be in need of functions to optimize and benchmark your optimization algorithm I recommend you to have a look at the package!</em></p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>plot3D</span><span>)</span><span>
</span><span>plot3D</span><span>(</span><span>fun</span><span>,</span><span> </span><span>contour</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>lightning</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk plotObjectiveFunction" src="https://mlr-org.github.io/figures/2017-03-13-First_release_of_mlrMBO_the_toolbox_for_Bayesian_Black_Box_Optimization/plotObjectiveFunction-1.svg"></p>
<p>Let’s start with the configuration of the optimization:</p>
<figure>
<pre><code data-lang="r"><span># In this simple example we construct the control object with the defaults:
</span><span>ctrl</span><span> </span><span>=</span><span> </span><span>makeMBOControl</span><span>()</span><span>
</span><span># For this numeric optimization we are going to use the Expected Improvement as infill criterion:
</span><span>ctrl</span><span> </span><span>=</span><span> </span><span>setMBOControlInfill</span><span>(</span><span>ctrl</span><span>,</span><span> </span><span>crit</span><span> </span><span>=</span><span> </span><span>crit.ei</span><span>)</span><span>
</span><span># We will allow for exactly 25 evaluations of the objective function:
</span><span>ctrl</span><span> </span><span>=</span><span> </span><span>setMBOControlTermination</span><span>(</span><span>ctrl</span><span>,</span><span> </span><span>max.evals</span><span> </span><span>=</span><span> </span><span>25L</span><span>)</span></code></pre>
</figure>
<p>The optimization has to so start with an initial design.<br>
<strong>mlrMBO</strong> can automatically create one but here we are going to use a randomly sampled LHS design of our own:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>des</span><span> </span><span>=</span><span> </span><span>generateDesign</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>8L</span><span>,</span><span> </span><span>par.set</span><span> </span><span>=</span><span> </span><span>getParamSet</span><span>(</span><span>fun</span><span>),</span><span> </span><span>fun</span><span> </span><span>=</span><span> </span><span>lhs</span><span>::</span><span>randomLHS</span><span>)</span><span>
</span><span>autoplot</span><span>(</span><span>fun</span><span>,</span><span> </span><span>render.levels</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>+</span><span> </span><span>geom_point</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>des</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Warning: Ignoring unknown aesthetics: fill</code></pre>
</figure>
<p><img alt="plot of chunk design" src="https://mlr-org.github.io/figures/2017-03-13-First_release_of_mlrMBO_the_toolbox_for_Bayesian_Black_Box_Optimization/design-1.svg"></p>
<p>The points demonstrate how the initial design already covers the search space but is missing the area of the global minimum.<br>
Before we can start the Bayesian optimization we have to set the surrogate learner to <em>Kriging</em>.<br>
Therefore we use an <em>mlr</em> regression learner.<br>
In fact, with <em>mlrMBO</em> you can use any regression learner integrated in <em>mlr</em> as a surrogate allowing for many special optimization applications.</p>
<figure>
<pre><code data-lang="r"><span>sur.lrn</span><span> </span><span>=</span><span> </span><span>makeLearner</span><span>(</span><span>&quot;regr.km&quot;</span><span>,</span><span> </span><span>predict.type</span><span> </span><span>=</span><span> </span><span>&quot;se&quot;</span><span>,</span><span> </span><span>config</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>show.learner.output</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span></code></pre>
</figure>
<p><em>Note:</em> <strong>mlrMBO</strong> can automatically determine a good surrogate learner based on the search space defined for the objective function.<br>
For a purely numeric domain it would have chosen <em>Kriging</em> as well with some slight modifications to make it a bit more stable against numerical problems that can occur during optimization.</p>
<p>Finally, we can start the optimization run:</p>
<figure>
<pre><code data-lang="r"><span>res</span><span> </span><span>=</span><span> </span><span>mbo</span><span>(</span><span>fun</span><span> </span><span>=</span><span> </span><span>fun</span><span>,</span><span> </span><span>design</span><span> </span><span>=</span><span> </span><span>des</span><span>,</span><span> </span><span>learner</span><span> </span><span>=</span><span> </span><span>sur.lrn</span><span>,</span><span> </span><span>control</span><span> </span><span>=</span><span> </span><span>ctrl</span><span>,</span><span> </span><span>show.info</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Computing y column(s) for design. Not provided.</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x=-0.0101,-4.52 : y = 0.817 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x=-4.52,-2.48 : y = 0.677 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x=-2.78,-3.27 : y = 0.913 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x=4.92,1.09 : y = 0.787 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x=2.77,2.93 : y = 0.469 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x=0.815,-0.647 : y = 0.333 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x=-2.34,4.5 : y = 1.11 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x=1.58,1.87 : y = 0.0939 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 1: x=1.48,5 : y = 1.23 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 2: x=-3.77,2.2 : y = 0.589 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 3: x=0.429,1.49 : y = 0.113 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 4: x=0.776,1.98 : y = 0.0413 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 5: x=0.126,1.93 : y = 0.127 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 6: x=1.01,2.15 : y = -0.00662 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 7: x=0.963,2.36 : y = -0.0317 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 8: x=0.922,0.539 : y = 0.388 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 9: x=-2.7,-0.524 : y = 0.119 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 10: x=-5,-0.253 : y = 1.47 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 11: x=-1.46,-0.613 : y = -0.306 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 12: x=-1.39,-1.1 : y = -0.098 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 13: x=-1.29,-0.228 : y = -0.4 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 14: x=-1.57,0.256 : y = -0.382 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 15: x=-1.43,-0.0423 : y = -0.413 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 16: x=-1.27,0.0745 : y = -0.412 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 17: x=5,-3.84 : y = 1.96 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>res</span><span>$</span><span>x</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## $x
## [1] -1.42836803 -0.04234841</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>res</span><span>$</span><span>y</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [1] -0.4128122</code></pre>
</figure>
<p>We can see that we have found the global optimum of $y = -0.414964$ at $x = (-1.35265,0)$ quite sufficiently.<br>
Let’s have a look at the points mlrMBO evaluated.<br>
Therefore we can use the <code>OptPath</code> which stores all information about all evaluations during the optimization run:</p>
<figure>
<pre><code data-lang="r"><span>opdf</span><span> </span><span>=</span><span> </span><span>as.data.frame</span><span>(</span><span>res</span><span>$</span><span>opt.path</span><span>)</span><span>
</span><span>autoplot</span><span>(</span><span>fun</span><span>,</span><span> </span><span>render.levels</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>render.contours</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span>+</span><span> </span><span>geom_text</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>opdf</span><span>,</span><span> </span><span>aes</span><span>(</span><span>label</span><span> </span><span>=</span><span> </span><span>dob</span><span>))</span></code></pre>
</figure>
<p><img alt="plot of chunk mboPoints" src="https://mlr-org.github.io/figures/2017-03-13-First_release_of_mlrMBO_the_toolbox_for_Bayesian_Black_Box_Optimization/mboPoints-1.svg"></p>
<p>It is interesting to see, that for this run the algorithm first went to the local minimum on the top right in the 6th and 7th iteration but later, thanks to the explorative character of the <em>Expected Improvement</em>, found the real global minimum.</p>
<h3>Comparison</h3>
<p>That is all good, but how do other optimization strategies perform?</p>
<h4>Grid Search</h4>
<p>Grid search is seldom a good idea.<br>
But especially for hyperparameter tuning it is still used.<br>
Probably because it kind of gives you the feeling that you know what is going on and have not left out any important area of the search space.<br>
In reality the grid is usually so sparse that it leaves important areas untouched as you can see in this example:</p>
<figure>
<pre><code data-lang="r"><span>grid.des</span><span> </span><span>=</span><span> </span><span>generateGridDesign</span><span>(</span><span>par.set</span><span> </span><span>=</span><span> </span><span>getParamSet</span><span>(</span><span>fun</span><span>),</span><span> </span><span>resolution</span><span> </span><span>=</span><span> </span><span>5</span><span>)</span><span>
</span><span>grid.des</span><span>$</span><span>y</span><span> </span><span>=</span><span> </span><span>apply</span><span>(</span><span>grid.des</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>fun</span><span>)</span><span>
</span><span>grid.des</span><span>[</span><span>which.min</span><span>(</span><span>grid.des</span><span>$</span><span>y</span><span>),]</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##      x1 x2           y
## 12 -2.5  0 -0.04923607</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>autoplot</span><span>(</span><span>fun</span><span>,</span><span> </span><span>render.levels</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>render.contours</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span>+</span><span> </span><span>geom_point</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>grid.des</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk gridSeach" src="https://mlr-org.github.io/figures/2017-03-13-First_release_of_mlrMBO_the_toolbox_for_Bayesian_Black_Box_Optimization/gridSeach-1.svg"></p>
<p>It is no surprise, that the grid search could not cover the search space well enough and we only reach a bad result.</p>
<h4>What about a simple random search?</h4>
<figure>
<pre><code data-lang="r"><span>random.des</span><span> </span><span>=</span><span> </span><span>generateRandomDesign</span><span>(</span><span>par.set</span><span> </span><span>=</span><span> </span><span>getParamSet</span><span>(</span><span>fun</span><span>),</span><span> </span><span>n</span><span> </span><span>=</span><span> </span><span>25L</span><span>)</span><span>
</span><span>random.des</span><span>$</span><span>y</span><span> </span><span>=</span><span> </span><span>apply</span><span>(</span><span>random.des</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>fun</span><span>)</span><span>
</span><span>random.des</span><span>[</span><span>which.min</span><span>(</span><span>random.des</span><span>$</span><span>y</span><span>),]</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##           x1         x2          y
## 20 -1.784371 -0.9802194 -0.1063019</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>autoplot</span><span>(</span><span>fun</span><span>,</span><span> </span><span>render.levels</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>render.contours</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span>+</span><span> </span><span>geom_point</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>random.des</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk randomSearch" src="https://mlr-org.github.io/figures/2017-03-13-First_release_of_mlrMBO_the_toolbox_for_Bayesian_Black_Box_Optimization/randomSearch-1.svg"></p>
<p>With the random search you could always be lucky but in average the optimum is not reached if smarter optimization strategies work well.</p>
<h4>A fair comarison</h4>
<p>… for stochastic optimization algorithms can only be achieved by repeating the runs.<br>
<strong>mlrMBO</strong> is stochastic as the initial design is generated randomly and the fit of the Kriging surrogate is also not deterministic.<br>
Furthermore we should include other optimization strategies like a genetic algorithm and direct competitors like <code>rBayesOpt</code>.<br>
An extensive benchmark is available in <a target="_blank" rel="nofollow" href="https://arxiv.org/abs/1703.03373">our <strong>mlrMBO</strong> paper</a>.<br>
The examples here are just meant to demonstrate the package.</p>
<h3>Engage</h3>
<p>If you want to contribute to <a target="_blank" rel="nofollow" href="https://github.com/mlr-org/mlrMBO"><strong>mlrMBO</strong></a> we ware always open to suggestions and pull requests on github.<br>
You are also invited to fork the repository and build and extend your own optimizer based on our toolbox.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://mlr-org.github.io/First_release_of_mlrMBO_the_toolbox_for_Bayesian_Black_Box_Optimization/"> mlr-org</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Vuf4-N5AAyU"></a></div>