<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/parallel-benchmarking-with-openml-and-mlr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://mlr-org.github.io/Parallel_benchmarking_with_OpenML_and_mlr/"> mlr-org</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>With this post I want to show you how to benchmark several learners (or learners with different parameter settings) using several data sets in a structured and parallelized fashion.<br>
For this we want to use <a target="_blank" rel="nofollow" href="https://mllg.github.io/batchtools/"><code>batchtools</code></a>.</p>
<p>The data that we will use here is stored on the open machine learning platform <a target="_blank" rel="nofollow" href="https://www.openml.org/">openml.org</a> and we can download it together with information on what to do with it in form of a task.</p>
<p><span></span></p>
<p>If you have a small project and don’t need to parallelize, you might want to just look at the previous blog post called <a target="_blank" rel="nofollow" href="http://mlr-org.github.io/mlr-loves-OpenML/">mlr loves OpenML</a>.</p>
<p>The following packages are needed for this:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>&quot;OpenML&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>&quot;mlr&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>&quot;batchtools&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>&quot;ggplot2&quot;</span><span>)</span></code></pre>
</figure>
<p>Now we download five OpenML-tasks from OpenML:</p>
<figure>
<pre><code data-lang="r"><span>set.seed</span><span>(</span><span>2017</span><span>)</span><span>
</span><span>## get useful tasks
</span><span>task_infos</span><span> </span><span>=</span><span> </span><span>listOMLTasks</span><span>(</span><span>tag</span><span> </span><span>=</span><span> </span><span>&quot;study_14&quot;</span><span>)</span><span>
</span><span>## take a sample of 5 tasks from these
</span><span>task_ids</span><span> </span><span>=</span><span> </span><span>sample</span><span>(</span><span>task_infos</span><span>$</span><span>task.id</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>5</span><span>)</span><span>
</span><span>tasks</span><span> </span><span>=</span><span> </span><span>lapply</span><span>(</span><span>task_ids</span><span>,</span><span> </span><span>getOMLTask</span><span>)</span></code></pre>
</figure>
<p>In a next step we need to create the so called registry.<br>
What this basically does is to create a folder with a certain subfolder structure.</p>
<figure>
<pre><code data-lang="r"><span>## create the experiment registry
</span><span>reg</span><span> </span><span>=</span><span> </span><span>makeExperimentRegistry</span><span>(</span><span>
</span><span>file.dir</span><span> </span><span>=</span><span> </span><span>&quot;parallel_benchmarking_blogpost&quot;</span><span>,</span><span>
</span><span>packages</span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;mlr&quot;</span><span>,</span><span> </span><span>&quot;OpenML&quot;</span><span>,</span><span> </span><span>&quot;party&quot;</span><span>),</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>123</span><span>)</span><span>
</span><span>names</span><span>(</span><span>reg</span><span>)</span><span>
</span><span>reg</span><span>$</span><span>cluster.functions</span><span>
</span><span>## allow for parallel computing, for other options see ?makeClusterFunctions
</span><span>reg</span><span>$</span><span>cluster.functions</span><span> </span><span>=</span><span> </span><span>makeClusterFunctionsMulticore</span><span>()</span></code></pre>
</figure>
<p>Now you should have a new folder in your working directory with the name <code>parallel_benchmarking_blogpost</code> and the following subfolders / files:</p>
<pre><code>parallel_benchmarking_blogpost/
├── algorithms
├── exports
├── external
├── jobs
├── logs
├── problems
├── registry.rds
├── results
└── updates
</code></pre>
<p>In the next step we get to the interesting point.<br>
We need to define…</p>
<ul>
<li>the <strong>problems</strong>, which in our case are simply the OpenML tasks we downloaded.</li>
<li>the <strong>algorithm</strong>, which with mlr and OpenML is quite simply achieved using <code>makeLearner</code> and <code>runTaskMlr</code>.<br>
We do not have to save the run results (result of applying the learner to the task), but we can directly upload it to OpenML where the results are automatically evaluated.</li>
<li>the machine learning <strong>experiment</strong>, i.e. in our case which parameters do we want to set for which learner.<br>
As an example here, we will look at the <em>ctree</em> algorithm from the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=party"><em>party</em></a> package and see whether Bonferroni correction (correction for multiple testing) helps getting better predictions and also we want to check whether we need a tree that has more than two leaf nodes (<code>stump = FALSE</code>) or if a small tree is enough (<code>stump = TRUE</code>).</li>
</ul>
<figure>
<pre><code data-lang="r"><span>## add the problem, in our case the tasks from OpenML
</span><span>for</span><span>(</span><span>task</span><span> </span><span>in</span><span> </span><span>tasks</span><span>)</span><span> </span><span>{</span><span>
</span><span>addProblem</span><span>(</span><span>name</span><span> </span><span>=</span><span> </span><span>paste</span><span>(</span><span>&quot;omltask&quot;</span><span>,</span><span> </span><span>task</span><span>$</span><span>task.id</span><span>,</span><span> </span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot;_&quot;</span><span>),</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>task</span><span>)</span><span>
</span><span>}</span><span>
</span><span>##' Function that takes the task (data) and the learner, runs the learner on
##' the task, uploads the run and returns the run ID.
##'
##' @param job required argument for addAlgorithm
##' @param instance required argument for addAlgorithm
##' @param data the task
##' @param learner the string that defines the learner, see listLearners()
</span><span>runTask_uploadRun</span><span> </span><span>=</span><span> </span><span>function</span><span>(</span><span>job</span><span>,</span><span> </span><span>instance</span><span>,</span><span> </span><span>data</span><span>,</span><span> </span><span>learner</span><span>,</span><span> </span><span>...</span><span>)</span><span> </span><span>{</span><span>
</span><span>learner</span><span> </span><span>=</span><span> </span><span>makeLearner</span><span>(</span><span>learner</span><span>,</span><span> </span><span>par.vals</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>...</span><span>))</span><span>
</span><span>run</span><span> </span><span>=</span><span> </span><span>runTaskMlr</span><span>(</span><span>data</span><span>,</span><span> </span><span>learner</span><span>)</span><span>
</span><span>run_id</span><span> </span><span>=</span><span> </span><span>uploadOMLRun</span><span>(</span><span>run</span><span>,</span><span> </span><span>tag</span><span> </span><span>=</span><span> </span><span>&quot;test&quot;</span><span>,</span><span> </span><span>confirm.upload</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>return</span><span>(</span><span>run_id</span><span>)</span><span>
</span><span>}</span><span>
</span><span>## add the algorithm
</span><span>addAlgorithm</span><span>(</span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;mlr&quot;</span><span>,</span><span> </span><span>fun</span><span> </span><span>=</span><span> </span><span>runTask_uploadRun</span><span>)</span><span>
</span><span>## what versions of the algorithm do we want to compute
</span><span>algo.design</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>mlr</span><span> </span><span>=</span><span> </span><span>expand.grid</span><span>(</span><span>
</span><span>learner</span><span> </span><span>=</span><span> </span><span>&quot;classif.ctree&quot;</span><span>,</span><span>
</span><span>testtype</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Bonferroni&quot;</span><span>,</span><span> </span><span>&quot;Univariate&quot;</span><span>),</span><span>
</span><span>stump</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>FALSE</span><span>,</span><span> </span><span>TRUE</span><span>),</span><span>
</span><span>stringsAsFactors</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>algo.design</span><span>$</span><span>mlr</span><span>
</span><span>addExperiments</span><span>(</span><span>algo.designs</span><span> </span><span>=</span><span> </span><span>algo.design</span><span>,</span><span> </span><span>repls</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>## get an overview of what we will submit
</span><span>summarizeExperiments</span><span>()</span></code></pre>
</figure>
<p>Now we can simply run our experiment:</p>
<figure>
<pre><code data-lang="r"><span>submitJobs</span><span>()</span></code></pre>
</figure>
<p>While your job is running, you can check the progress using <code>getStatus()</code>.<br>
As soon as <code>getStatus()</code> tells us that all our runs are done, we can collect the results of our experiment from OpenML.<br>
To be able to do this we need to collect the run IDs from the uploaded runs we did during the experiment.<br>
Also we want to add the info of the parameters used (<code>getJobPars()</code>).</p>
<figure>
<pre><code data-lang="r"><span>results0</span><span> </span><span>=</span><span> </span><span>reduceResultsDataTable</span><span>()</span><span>
</span><span>job.pars</span><span> </span><span>=</span><span> </span><span>getJobPars</span><span>()</span><span>
</span><span>results</span><span> </span><span>=</span><span> </span><span>cbind</span><span>(</span><span>run.id</span><span> </span><span>=</span><span> </span><span>results0</span><span>$</span><span>V</span><span>1</span><span>,</span><span> </span><span>job.pars</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>results</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##      run.id job.id       problem algorithm       learner   testtype
##  1: 1852889      1 omltask_34536       mlr classif.ctree Bonferroni
##  2: 1852882      2 omltask_34536       mlr classif.ctree Univariate
##  3: 1852888      3 omltask_34536       mlr classif.ctree Bonferroni
##  4: 1852885      4 omltask_34536       mlr classif.ctree Univariate
##  5: 1852883      5  omltask_3918       mlr classif.ctree Bonferroni
##  6: 1852884      6  omltask_3918       mlr classif.ctree Univariate
##  7: 1852886      7  omltask_3918       mlr classif.ctree Bonferroni
##  8: 1852887      8  omltask_3918       mlr classif.ctree Univariate
##  9: 1852895      9  omltask_3891       mlr classif.ctree Bonferroni
## 10: 1852897     10  omltask_3891       mlr classif.ctree Univariate
## 11: 1852890     11  omltask_3891       mlr classif.ctree Bonferroni
## 12: 1852891     12  omltask_3891       mlr classif.ctree Univariate
## 13: 1852892     13  omltask_2074       mlr classif.ctree Bonferroni
## 14: 1852896     14  omltask_2074       mlr classif.ctree Univariate
## 15: 1852893     15  omltask_2074       mlr classif.ctree Bonferroni
## 16: 1852894     16  omltask_2074       mlr classif.ctree Univariate
## 17: 1852900     17  omltask_9976       mlr classif.ctree Bonferroni
## 18: 1852901     18  omltask_9976       mlr classif.ctree Univariate
## 19: 1852898     19  omltask_9976       mlr classif.ctree Bonferroni
## 20: 1852899     20  omltask_9976       mlr classif.ctree Univariate
##     stump
##  1: FALSE
##  2: FALSE
##  3:  TRUE
##  4:  TRUE
##  5: FALSE
##  6: FALSE
##  7:  TRUE
##  8:  TRUE
##  9: FALSE
## 10: FALSE
## 11:  TRUE
## 12:  TRUE
## 13: FALSE
## 14: FALSE
## 15:  TRUE
## 16:  TRUE
## 17: FALSE
## 18: FALSE
## 19:  TRUE
## 20:  TRUE</code></pre>
</figure>
<p>With the run ID information we can now grab the evaluations from OpenML and plot for example the parameter settings against the predictive accuracy.</p>
<figure>
<pre><code data-lang="r"><span>run.evals0</span><span> </span><span>=</span><span> </span><span>listOMLRunEvaluations</span><span>(</span><span>run.id</span><span> </span><span>=</span><span> </span><span>results</span><span>$</span><span>run.id</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Downloading from 'http://www.openml.org/api/v1/json/evaluation/list/run/1852889,1852882,1852888,1852885,1852883,1852884,1852886,1852887,1852895,1852897,1852890,1852891,1852892,1852896,1852893,1852894,1852900,1852901,1852898,1852899' to '&lt;mem>'.</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>run.evals</span><span> </span><span>=</span><span> </span><span>merge</span><span>(</span><span>results</span><span>,</span><span> </span><span>run.evals0</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;run.id&quot;</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>run.evals</span><span>,</span><span> </span><span>aes</span><span>(</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>interaction</span><span>(</span><span>testtype</span><span>,</span><span> </span><span>stump</span><span>),</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>predictive.accuracy</span><span>,</span><span>
</span><span>group</span><span> </span><span>=</span><span> </span><span>data.name</span><span>,</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>interaction</span><span>(</span><span>task.id</span><span>,</span><span> </span><span>data.name</span><span>)))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>()</span><span> </span><span>+</span><span> </span><span>geom_line</span><span>()</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-12" src="https://mlr-org.github.io/figures/2017-03-22-Parallel_benchmarking_with_OpenML_and_mlr/unnamed-chunk-12-1.svg"></p>
<p>We see that the only data set where a stump is good enough is the pc1 data set.<br>
For the madelon data set Bonferroni correction helps.<br>
For the others it does not seem to matter.<br>
You can check out the results online by going to the task websites (e.g. for task 9976 for the madelon data set go to <a target="_blank" rel="nofollow" href="https://www.openml.org/t/9976">openml.org/t/9976</a>) or the run websites (e.g. <a target="_blank" rel="nofollow" href="https://www.openml.org/r/1852889">openml.org/r/1852889</a>).</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://mlr-org.github.io/Parallel_benchmarking_with_OpenML_and_mlr/"> mlr-org</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Okkmxfm-dbg"></a></div>