<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/choosing-which-method-to-use-for-optimizing-functions/"></div></div><div><a data-url="https://www.r-bloggers.com/choosing-which-method-to-use-for-optimizing-functions/" data-counturl="https://www.r-bloggers.com/choosing-which-method-to-use-for-optimizing-functions/" data-count="horizontal" data-text="Choosing which method to use for “optimizing” functions" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://nashjc.wordpress.com/2016/11/20/choosing-which-method-to-use-for-optimizating-functions/"> Blog: John C. Nash</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em><strong>A decision tree for function minimization</strong></em></p>
<p>What R calls “optimization” is more generally known as function minimization. The tools in the <strong>stats </strong>package function <em>optim() </em>are all essentially function mimizers, as are those in the package <strong>optimrx </strong>found at https://r-forge.r-project.org/projects/optimizer/. <strong>optimrx </strong>tries to make it easier for users to call quite a few methods without having to learn many different sets of command syntax. It uses the same syntax as <em>optim()<strong>, </strong></em>even down to parameter scaling, by changing the <strong>method=”name”</strong> argument.</p>
<p>There are, however, a number of methods. Which is best for your purpose? This posting presents a form of decision tree to try to help. It isn’t the last word, and you should not take it as a set of hard rules. But it does reflect a lot of experience and I have tried to give the reasoning behind some of the suggestions.</p>
<p>We assume there is a function we wish to minimize or maximize. Call it <em>fn(x) </em>where <em>x</em> is parameter vector of length <em>n</em>; we will let e<em>data</em> be exogenous data — those numbers or factors that are needed to compute the function but which we will not be adjusting. So we should really write <em>fn(x, edata)</em>, but will often be lazy, just as R allows us to be with exogenous data.</p>
<p>Function <em>fn(x)</em> is assumed:</p>
<ul>
<li>computable</li>
<li>to have an initial starting set of parameters <em>x0</em> that is feasible (which I won’t define just now)</li>
<li>to have more or less 1 local minimum (though we often try to solve problems with  multiple minima present if these minima are believed to be sufficiently separated).</li>
</ul>
<p>Note: if <em>fn()</em> is a sum of squares, then packages <strong>nlmrt</strong> or <strong>minpack.lm </strong>are preferred. The <strong>stats</strong> function <em>nls()</em> may also be useful. I plan to write more about nonlinear least squares in a later post. If you are desperate, then get in touch with me!</p>
<p>Let us proceed in a stepwise fashion.</p>
<p><em><strong>Step 0: Presets.</strong></em><br>
If we are maximizing , create   <em>fn(x, edata) &lt;- -fn.orig(x, edata) .    </em>That is, we<br>
minimize the negative of the function we want to maximize.</p>
<p>Let us set some logical variables to define our problem:</p>
<ul>
<li><em>haveg</em> = TRUE if there are analytic gradients of <em>fn(x)</em> available</li>
<li><em>bounds</em> = TRUE if there are lower and/or upper bounds on the parameters <em>x</em></li>
<li><em>masks</em> = TRUE if some parameters are fixed — currently only packages <strong>Rvmmin </strong> and <strong>Rcgmin</strong> support masks well.</li>
</ul>
<p><em><strong>Step 1: Check function  </strong></em>(This is always a REALLY good idea.)</p>
<ul>
<li>compute <em>fn(x, edata)</em> for at least <em>x0 </em>and verify the value is correct</li>
<li><em>haveg </em>TRUE: compute <em>g(x0)</em> and compare it with <em>grad(fn, x0, edata)</em> from package <strong>numDeriv</strong></li>
</ul>
<p><em><strong>Step2: Running a method</strong></em></p>
<ul>
<li><em>bounds</em> TRUE,
<ul>
<li><em>haveg</em> FALSE:  try <em>nmkb()</em> from package <strong>dfOptim</strong>, but note that the transfinite function approach to bounds in this method cannot have <em>x0 </em>— in fact any initial parameter — on a bound.  <em>bobyqa() </em>from package <strong>minqa</strong> may be much more efficient, but can behave poorly. <em>hjkb()<strong> </strong></em>from package  <strong>dfOptim</strong> is very easy to understand, but sometimes stalls and generally inefficient.</li>
<li><em>haveg</em> TRUE:
<ul>
<li><em>n &lt; 50 </em>(i.e., small)<em>: </em>packages <strong>Rvmmin </strong>or <strong>nlminb</strong></li>
<li><em>n large</em>: packages <strong>Rtnmin</strong>, <strong>Rcgmin</strong>, or method <strong>L-BFGS-B </strong>(from <em>optim() </em>in <strong>stats</strong>). Package <strong>lbfgsb3 </strong>is an implementation of the 2011 FORTRAN version of the same algorithm and in theory may be preferred, but <em>L-BFGS-B()</em> called via <em>optim() </em>or <em>optimr() </em>is likely more efficient. The three approaches mentioned can also be applied to small <em>n </em>problems, and sometimes are as efficient as the methods recommended for small <em>n</em>. <strong>Rtnmin</strong> and LBFGS variants are in the same general class of methods, but have many differences in implementation approach. Generally I have found that the relatively newer <strong>Rtnmin</strong> may be faster, but less reliable than,  <strong>Rcgmin </strong>though the former seems to work better when the function has more “nonlinearity” (which for now I will not define).  <strong>Rvmmin</strong> is, by design, very aggressive in trying to find a minimum, so will “try too hard” in many cases. As such it should not be criticised for slowness when it is intended to favour reliability over speed.</li>
</ul>
</li>
</ul>
</li>
<li><em>bounds</em> FALSE:
<ul>
<li><em>haveg </em>FALSE: Methods <em>nmk()</em> from <strong>dfOptim</strong>,  or <em>uobyqa()</em> from <strong>minqa </strong>are indicated (but <em>bobyqa()</em> from <strong>minqa </strong>may be better as it is more up-to-date than <em>uobyqa();</em> we can use very loose values for the bounds on <em>x</em>)</li>
<li><em>haveg</em> TRUE:
<ul>
<li><em>n </em>small: packages <strong>Rvmmin </strong>or <strong>ucminf </strong>, or function <em>nlm() </em>from <strong>stats.</strong></li>
<li><em>n</em> large: method <strong>L-BFGS-B</strong> or packages <strong>Rcgmin </strong>and <strong>Rtnmin</strong></li>
</ul>
</li>
</ul>
</li>
</ul>
<p>In the above I have not mentioned function <em>spg()</em> from package <strong>BB</strong>. My experience is that occasionally this can be very efficient, but more often slower than other methods. However it has a particular advantage of allowing the application of quite complicated constraints.</p>
<p> </p>
<p> </p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/nashjc.wordpress.com/55/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/nashjc.wordpress.com/55/" border="0"></a> <img alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/choosing-which-method-to-use-for-optimizing-functions/"></div></div><div><a data-url="https://www.r-bloggers.com/choosing-which-method-to-use-for-optimizing-functions/" data-counturl="https://www.r-bloggers.com/choosing-which-method-to-use-for-optimizing-functions/" data-count="vertical" data-text="Choosing which method to use for “optimizing” functions" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://nashjc.wordpress.com/2016/11/20/choosing-which-method-to-use-for-optimizating-functions/"> Blog: John C. Nash</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/RY23UO-I2Jo"></a></div>