<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/use-mlrmbo-to-optimize-via-command-line/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://mlr-org.github.io/Use_mlrMBO_to_optimize_via_command_line/"> mlr-org</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Many people who want to apply Bayesian optimization want to use it to optimize an algorithm that is not implemented in <strong>R</strong> but runs on the command line as a shell script or an executable.</p>
<p>We recently published <a target="_blank" rel="nofollow" href="http://mlr-org.github.io/mlrMBO/"><strong>mlrMBO</strong></a> on CRAN.<br>
As a normal package it normally operates inside of R, but with this post I want to demonstrate how <strong>mlrMBO</strong> can be used to optimize an external application.<br>
At the same time I will highlight some issues you can likely run into.</p>
<p><span></span></p>
<p>First of all we need a bash script that we want to optimize.<br>
This tutorial will only run on Unix systems (Linux, OSX etc.) but should also be informative for windows users.<br>
The following code will write a tiny bash script that uses <code>bc</code> to calculate $sin(x_1-1) + (x_1^2 + x_2^2)$ and write the result “hidden” in a sentence (<code>The result is 12.34!</code>) in a <code>result.txt</code> text file.</p>
<h3>The bash script</h3>
<figure>
<pre><code data-lang="r"><span># write bash script
</span><span>lines</span><span> </span><span>=</span><span> </span><span>'#!/bin/bash
fun ()
{
x1=$1
x2=$2
command=&quot;(s($x1-1) + ($x1^2 + $x2^2))&quot;
result=$(bc -l &lt;&lt;&lt; $command)
}
echo &quot;Start calculation.&quot;
fun $1 $2
echo &quot;The result is $result!&quot; > &quot;result.txt&quot;
echo &quot;Finish calculation.&quot;
'</span><span>
</span><span>writeLines</span><span>(</span><span>lines</span><span>,</span><span> </span><span>&quot;fun.sh&quot;</span><span>)</span><span>
</span><span># make it executable:
</span><span>system</span><span>(</span><span>&quot;chmod +x fun.sh&quot;</span><span>)</span></code></pre>
</figure>
<h3>Running the script from R</h3>
<p>Now we need a R function that starts the script, reads the result from the text file and returns it.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>stringi</span><span>)</span><span>
</span><span>runScript</span><span> </span><span>=</span><span> </span><span>function</span><span>(</span><span>x</span><span>)</span><span> </span><span>{</span><span>
</span><span>command</span><span> </span><span>=</span><span> </span><span>sprintf</span><span>(</span><span>&quot;./fun.sh %f %f&quot;</span><span>,</span><span> </span><span>x</span><span>[[</span><span>'x1'</span><span>]],</span><span> </span><span>x</span><span>[[</span><span>'x2'</span><span>]])</span><span>
</span><span>error.code</span><span> </span><span>=</span><span> </span><span>system</span><span>(</span><span>command</span><span>)</span><span>
</span><span>if</span><span> </span><span>(</span><span>error.code</span><span> </span><span>!=</span><span> </span><span>0</span><span>)</span><span> </span><span>{</span><span>
</span><span>stop</span><span>(</span><span>&quot;Simulation had error.code != 0!&quot;</span><span>)</span><span>
</span><span>}</span><span>
</span><span>result</span><span> </span><span>=</span><span> </span><span>readLines</span><span>(</span><span>&quot;result.txt&quot;</span><span>)</span><span>
</span><span># the pattern matches 12 as well as 12.34 and .34
</span><span> </span><span># the ?: makes the decimals a non-capturing group.
</span><span> </span><span>result</span><span> </span><span>=</span><span> </span><span>stri_match_first_regex</span><span>(</span><span>result</span><span>,</span><span> </span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;\\d*(?:\\.\\d+)?(?=\\!)&quot;</span><span>)</span><span>
</span><span>as.numeric</span><span>(</span><span>result</span><span>)</span><span>
</span><span>}</span></code></pre>
</figure>
<p>This function uses <code>stringi</code> and <em>regular expressions</em> to match the result within the sentence.<br>
Depending on the output different strategies to read the result make sense.<br>
XML files can usually be accessed with <code>XML::xmlParse</code>, <code>XML::getNodeSet</code>, <code>XML::xmlAttrs</code> etc. using <code>XPath</code> queries.<br>
Sometimes the good old <code>read.table()</code> is also sufficient.<br>
If, for example, the output is written in a file like this:</p>
<figure>
<pre><code data-lang="r"><span>value1</span><span> </span><span>=</span><span> </span><span>23.45</span><span>
</span><span>value2</span><span> </span><span>=</span><span> </span><span>13.82</span></code></pre>
</figure>
<p>You can easily use <code>source()</code> like that:</p>
<figure>
<pre><code data-lang="r"><span>EV</span><span> </span><span>=</span><span> </span><span>new.env</span><span>()</span><span>
</span><span>eval</span><span>(</span><span>expr</span><span> </span><span>=</span><span> </span><span>{</span><span>a</span><span> </span><span>=</span><span> </span><span>1</span><span>},</span><span> </span><span>envir</span><span> </span><span>=</span><span> </span><span>EV</span><span>)</span><span>
</span><span>as.list</span><span>(</span><span>EV</span><span>)</span><span>
</span><span>source</span><span>(</span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;result.txt&quot;</span><span>,</span><span> </span><span>local</span><span> </span><span>=</span><span> </span><span>EV</span><span>)</span><span>
</span><span>res</span><span> </span><span>=</span><span> </span><span>as.list</span><span>(</span><span>EV</span><span>)</span><span>
</span><span>rm</span><span>(</span><span>EV</span><span>)</span></code></pre>
</figure>
<p>which will return a list with the entries <code>$value1</code> and <code>$value2</code>.</p>
<h3>Define bounds, wrap function.</h3>
<p>To evaluate the function from within <strong>mlrMBO</strong> it has to be wrapped in <strong>smoof</strong> function.<br>
The smoof function also contains information about the bounds and scales of the domain of the objective function defined in a <em>ParameterSet</em>.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>mlrMBO</span><span>)</span><span>
</span><span># Defining the bounds of the parameters:
</span><span>par.set</span><span> </span><span>=</span><span> </span><span>makeParamSet</span><span>(</span><span>
</span><span>makeNumericParam</span><span>(</span><span>&quot;x1&quot;</span><span>,</span><span> </span><span>lower</span><span> </span><span>=</span><span> </span><span>-3</span><span>,</span><span> </span><span>upper</span><span> </span><span>=</span><span> </span><span>3</span><span>),</span><span>
</span><span>makeNumericParam</span><span>(</span><span>&quot;x2&quot;</span><span>,</span><span> </span><span>lower</span><span> </span><span>=</span><span> </span><span>-2.5</span><span>,</span><span> </span><span>upper</span><span> </span><span>=</span><span> </span><span>2.5</span><span>)</span><span>
</span><span>)</span><span>
</span><span># Wrapping everything in a smoof function:
</span><span>fn</span><span> </span><span>=</span><span> </span><span>makeSingleObjectiveFunction</span><span>(</span><span>
</span><span>id</span><span> </span><span>=</span><span> </span><span>&quot;fun.sh&quot;</span><span>,</span><span> </span><span>fn</span><span> </span><span>=</span><span> </span><span>runScript</span><span>,</span><span>
</span><span>par.set</span><span> </span><span>=</span><span> </span><span>par.set</span><span>,</span><span>
</span><span>has.simple.signature</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>
</span><span>)</span><span>
</span><span># let's see if the function is working
</span><span>des</span><span> </span><span>=</span><span> </span><span>generateGridDesign</span><span>(</span><span>par.set</span><span>,</span><span> </span><span>resolution</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span>
</span><span>des</span><span>$</span><span>y</span><span> </span><span>=</span><span> </span><span>apply</span><span>(</span><span>des</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>fn</span><span>)</span><span>
</span><span>des</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##   x1   x2         y
## 1 -3 -2.5 16.006802
## 2  0 -2.5  5.408529
## 3  3 -2.5 16.159297
## 4 -3  0.0  9.756802
## 5  0  0.0  0.841471
## 6  3  0.0  9.909297
## 7 -3  2.5 16.006802
## 8  0  2.5  5.408529
## 9  3  2.5 16.159297</code></pre>
</figure>
<p>If you run this locally, you will see that the console output generated by our shell script directly appears in the R-console.<br>
This can be helpful but also annoying.</p>
<h3>Redirecting output</h3>
<p>If a lot of output is generated during a single call of <code>system()</code> it might even crash R.<br>
To avoid that I suggest to redirect the output into a file.<br>
This way no output is lost and the R console does not get flooded.<br>
We can simply achieve that by replacing the <code>command</code> in the function <code>runScript</code> from above with the following code:</p>
<figure>
<pre><code data-lang="r"><span> </span><span># console output file output_1490030005_1.1_2.4.txt
</span><span> </span><span>output_file</span><span> </span><span>=</span><span> </span><span>sprintf</span><span>(</span><span>&quot;output_%i_%.1f_%.1f.txt&quot;</span><span>,</span><span> </span><span>as.integer</span><span>(</span><span>Sys.time</span><span>()),</span><span> </span><span>x</span><span>[[</span><span>'x1'</span><span>]],</span><span> </span><span>x</span><span>[[</span><span>'x2'</span><span>]])</span><span>
</span><span># redirect output with ./fun.sh 1.1 2.4 > output.txt
</span><span> </span><span># alternative: ./fun.sh 1.1 2.4 > /dev/null to drop it
</span><span> </span><span>command</span><span> </span><span>=</span><span> </span><span>sprintf</span><span>(</span><span>&quot;./fun.sh %f %f > %s&quot;</span><span>,</span><span> </span><span>x</span><span>[[</span><span>'x1'</span><span>]],</span><span> </span><span>x</span><span>[[</span><span>'x2'</span><span>]],</span><span> </span><span>output_file</span><span>)</span></code></pre>
</figure>
<h3>Start the Optimization</h3>
<p>Now everything is set so we can proceed with the usual MBO setup:</p>
<figure>
<pre><code data-lang="r"><span>ctrl</span><span> </span><span>=</span><span> </span><span>makeMBOControl</span><span>()</span><span>
</span><span>ctrl</span><span> </span><span>=</span><span> </span><span>setMBOControlInfill</span><span>(</span><span>ctrl</span><span>,</span><span> </span><span>crit</span><span> </span><span>=</span><span> </span><span>crit.ei</span><span>)</span><span>
</span><span>ctrl</span><span> </span><span>=</span><span> </span><span>setMBOControlTermination</span><span>(</span><span>ctrl</span><span>,</span><span> </span><span>iters</span><span> </span><span>=</span><span> </span><span>10</span><span>)</span><span>
</span><span>configureMlr</span><span>(</span><span>show.info</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>show.learner.output</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>run</span><span> </span><span>=</span><span> </span><span>mbo</span><span>(</span><span>fun</span><span> </span><span>=</span><span> </span><span>fn</span><span>,</span><span> </span><span>control</span><span> </span><span>=</span><span> </span><span>ctrl</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Computing y column(s) for design. Not provided.</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x1=-1.58; x2=-1.64 : y = 4.65 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x1=-0.251; x2=0.0593 : y = 0.883 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x1=1.04; x2=2.05 : y = 5.3 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x1=-2.39; x2=-0.345 : y = 6.07 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x1=0.608; x2=-0.742 : y = 0.538 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x1=2.85; x2=0.9 : y = 9.87 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x1=2.07; x2=-2.11 : y = 9.58 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 0: x1=-0.967; x2=1.25 : y = 1.58 : 0.0 secs : initdesign</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 1: x1=0.301; x2=-2.08 : y = 3.79 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 2: x1=0.375; x2=-0.191 : y = 0.408 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 3: x1=0.189; x2=-0.586 : y = 0.346 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 4: x1=0.397; x2=-0.516 : y = 0.144 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 5: x1=-0.702; x2=-0.45 : y = 0.296 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 6: x1=-0.476; x2=-0.536 : y = 0.481 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 7: x1=-0.982; x2=-0.258 : y = 0.115 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 8: x1=-0.94; x2=0.0274 : y = 0.0493 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 9: x1=-1.09; x2=0.238 : y = 0.364 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [mbo] 10: x1=-0.897; x2=-0.102 : y = 0.132 : 0.0 secs : infill_ei</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span># The resulting optimal configuration:
</span><span>run</span><span>$</span><span>x</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## $x1
## [1] -0.9395412
## 
## $x2
## [1] 0.02737539</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span># The best reached value:
</span><span>run</span><span>$</span><span>y</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [1] 0.04929388</code></pre>
</figure>
<h3>Execute the R script from a shell</h3>
<p>Also you might not want to bothered having to start <em>R</em> and run this script manually so what I would recommend is saving all above as an R-script plus some lines that write the output in a JSON file like this:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>jsonlite</span><span>)</span><span>
</span><span>write_json</span><span>(</span><span>run</span><span>[</span><span>c</span><span>(</span><span>&quot;x&quot;</span><span>,</span><span>&quot;y&quot;</span><span>)],</span><span> </span><span>&quot;mbo_res.json&quot;</span><span>)</span></code></pre>
</figure>
<p>Let’s assume we saved all of that above as an R-script under the name <a target="_blank" rel="nofollow" href="https://gist.github.com/jakob-r/6be022d49e135c7905fd4c097bc3d376"><code>runMBO.R</code> (actually it is available as a gist)</a>.</p>
<p>Then you can simply run it from the command line:</p>
<figure>
<pre><code data-lang="bash">Rscript runMBO.R </code></pre>
</figure>
<p>As an extra the script in the gist also contains a simple handler for command line arguments.<br>
In this case you can define the number of optimization iterations and the maximal allowed time in seconds for the optimization.<br>
You can also define the seed to make runs reproducible:</p>
<figure>
<pre><code data-lang="bash">Rscript runMBO.R <span>iters</span><span>=</span>20 <span>time</span><span>=</span>10 <span>seed</span><span>=</span>3</code></pre>
</figure>
<p>If you want to build a more advanced command line interface you might want to have a <a target="_blank" rel="nofollow" href="https://www.slideshare.net/EdwindeJonge1/docopt-user2014">look</a> <a target="_blank" rel="nofollow" href="https://github.com/docopt/docopt.R">at</a> <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=docopt">docopt</a>.</p>
<h3>Clean up</h3>
<p>To clean up all the files generated by this script you can run:</p>
<figure>
<pre><code data-lang="r"><span>file.remove</span><span>(</span><span>&quot;result.txt&quot;</span><span>)</span><span>
</span><span>file.remove</span><span>(</span><span>&quot;fun.sh&quot;</span><span>)</span><span>
</span><span>file.remove</span><span>(</span><span>&quot;mbo_res.json&quot;</span><span>)</span><span>
</span><span>output.files</span><span> </span><span>=</span><span> </span><span>list.files</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;output_\\d+_[0-9_.-]+\\.txt&quot;</span><span>)</span><span>
</span><span>file.remove</span><span>(</span><span>output.files</span><span>)</span></code></pre>
</figure>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://mlr-org.github.io/Use_mlrMBO_to_optimize_via_command_line/"> mlr-org</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/ApoQpJZQxqI"></a></div>