<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/progress-bar-overhead-comparisons/"></div></div><div><a data-url="https://www.r-bloggers.com/progress-bar-overhead-comparisons/" data-counturl="https://www.r-bloggers.com/progress-bar-overhead-comparisons/" data-count="horizontal" data-text="Progress bar overhead comparisons" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://peter.solymos.org/code/2016/10/15/progress-bar-overhead-comparisons.html"> Peter Solymos - R related posts</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>As a testament to my obsession with progress bars in R, here is<br>
a quick investigation about the overhead cost of<br>
drawing a progress bar during computations in R.<br>
I compared several approaches including<br>
my <strong>pbapply</strong> and Hadley Wickham’s <strong>plyr</strong>.</p>
<p>Let’s compare the good old <code>lapply</code> function from base R,<br>
a custom-made variant called <code>lapply_pb</code> that was<br>
proposed <a target="_blank" href="http://ryouready.wordpress.com/2010/01/11/progress-bars-in-r-part-ii-a-wrapper-for-apply-functions/">here</a>, <code>l_ply</code> from the <strong>plyr</strong> package,<br>
and finally <code>pblapply</code> from the <strong>pbapply</strong> package:</p>
<pre><code>library(pbapply)
library(plyr)

lapply_pb &lt;- function(X, FUN, ...) {
    env &lt;- environment()
    pb_Total &lt;- length(X)
    counter &lt;- 0
    pb &lt;- txtProgressBar(min = 0, max = pb_Total, style = 3)
    wrapper &lt;- function(...){
        curVal &lt;- get(&quot;counter&quot;, envir = env)
        assign(&quot;counter&quot;, curVal +1 ,envir = env)
        setTxtProgressBar(get(&quot;pb&quot;, envir = env), curVal + 1)
        FUN(...)
    }
    res &lt;- lapply(X, wrapper, ...)
    close(pb)
    res
}

f &lt;- function(n, type = &quot;lapply&quot;, s = 0.1) {
    i &lt;- seq_len(n)
    out &lt;- switch(type, 
        &quot;lapply&quot; = system.time(lapply(i, function(i) Sys.sleep(s))), 
        &quot;lapply_pb&quot; = system.time(lapply_pb(i, function(i) Sys.sleep(s))), 
        &quot;l_ply&quot; = system.time(l_ply(i, function(i) 
            Sys.sleep(s), .progress=&quot;text&quot;)), 
        &quot;pblapply&quot; = system.time(pblapply(i, function(i) Sys.sleep(s))))
    unname(out[&quot;elapsed&quot;] - (n * s))
}
</code></pre>
<p>Use the function <code>f</code> to run all four variants. The expected run time<br>
is <code>n * s</code> (number of iterations x sleep duration),<br>
therefore we can calculate the overhead from the<br>
return objects as elapsed minus expected. Let’s get some numbers<br>
for a variety of <code>n</code> values and replicated <code>B</code> times<br>
to smooth out the variation:</p>
<pre><code>n &lt;- c(10, 100, 1000)
s &lt;- 0.01
B &lt;- 10

x1 &lt;- replicate(B, sapply(n, f, type = &quot;lapply&quot;, s = s))
x2 &lt;- replicate(B, sapply(n, f, type = &quot;lapply_pb&quot;, s = s))
x3 &lt;- replicate(B, sapply(n, f, type = &quot;l_ply&quot;, s = s))
x4 &lt;- replicate(B, sapply(n, f, type = &quot;pblapply&quot;, s = s))

m &lt;- cbind(
    lapply = rowMeans(x1),
    lapply_pb = rowMeans(x2),
    l_ply = rowMeans(x3),
    pblapply = rowMeans(x4))

op &lt;- par(mfrow=c(1, 2))
matplot(n, m, type = &quot;l&quot;, lty = 1, lwd = 3,
    ylab = &quot;Overhead (sec)&quot;, xlab = &quot;# iterations&quot;)
legend(&quot;topleft&quot;, bty = &quot;n&quot;, col = 1:4, lwd = 3, text.col = 1:4,
    legend = colnames(m))
matplot(n, m / n, type = &quot;l&quot;, lty = 1, lwd = 3,
    ylab = &quot;Overhead / # iterations (sec)&quot;, xlab = &quot;# iterations&quot;)
par(op)
dev.off()
</code></pre>
<p><img alt="" src="https://i2.wp.com/peter.solymos.org/images/2016/10/15/pb-overhead.png?w=456" data-recalc-dims="1"></p>
<p>The plot tells us that the overhead increases linearly<br>
with the number of iterations when using <code>lapply</code><br>
without progress bar.<br>
All other three approaches show similar patterns to each other<br>
and the overhead is constant: lines are<br>
parallel above 100 iterations after an initial increase.<br>
The per iteration overhead is decreasing, approaching<br>
the <code>lapply</code> line. Note that all the differences are tiny<br>
and there is no practical consequence<br>
for choosing one approach over the other in terms of processing times.<br>
This is good news and another argument for using progress bar<br>
because its usefulness far outweighs the minimal<br>
(&lt;2 seconds here for 1000 iterations)<br>
overhead cost.</p>
<p>As always, suggestions and feature requests are welcome.<br>
Leave a comment or visit the <a target="_blank" href="https://github.com/psolymos/pbapply/issues">GitHub repo</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/progress-bar-overhead-comparisons/"></div></div><div><a data-url="https://www.r-bloggers.com/progress-bar-overhead-comparisons/" data-counturl="https://www.r-bloggers.com/progress-bar-overhead-comparisons/" data-count="vertical" data-text="Progress bar overhead comparisons" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://peter.solymos.org/code/2016/10/15/progress-bar-overhead-comparisons.html"> Peter Solymos - R related posts</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/NTjqR--osJQ"></a></div>