<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/tracking-progress-in-r/"></div></div><div><a data-url="https://www.r-bloggers.com/tracking-progress-in-r/" data-counturl="https://www.r-bloggers.com/tracking-progress-in-r/" data-count="horizontal" data-text="Tracking progress in R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://kevinkuang.net/tracking-progress-in-r-ad97998c359f?source=rss----a1ff9aea4bf1--r_programming"> R Programming in Bioinformatics Playground on Medium</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Admit it or not, we human beings become anxious and impatient when it comes to wait. Especially when we are blindfolded — that is, we are unaware of how long we have to suffer the endless wait. As pointed out by Brad Allan Myers, arguably the designer of progress bar in 1980s, being able to track the progress during the waiting can significantly improve the user experience (Myers, 1985).</p>
<figure><img alt="" src="https://i1.wp.com/d262ilb51hltx0.cloudfront.net/max/277/1*u63Gazq8LZ-p9T5f2mXGqQ.png?w=456&ssl=1" data-recalc-dims="1"><br>
<figcaption>A typical progress bar by Simeon87 (Own screenshot by uploader (cropped)) [GPL (<a target="_blank" rel="nofollow" href="http://www.gnu.org/licenses/gpl.html">http://www.gnu.org/licenses/gpl.html</a>)], via Wikimedia Commons</figcaption>
</figure>
<p>As an R programmer in bioinformatic research, often my codes are not designed for the general public, but it is important to make sure that my users, namely my fellow colleagues and researchers, are as happy as possible. However, tracking process in R can be tricky. In this article, I am going to present you some approaches and my solution (<a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=pbmcapply"><em>pbmcapply</em></a>).</p>
<h4>Print</h4>
<p>The easiest way to handle progress tracking in R is to periodically print the percentage of completion to the output, which is the screen by default, or write it to your log file located somewhere on the disk. Needless to say, this is probably the least elegant way to solve the problem, but many people are still following this path nowadays.</p>
<h4>Pbapply</h4>
<p>An better (and still easy) solution is to adopt a package named <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/pbapply/index.html"><em>pbapply</em></a>. According to <a target="_blank" rel="nofollow" href="https://github.com/psolymos/pbapply">its dev page</a>, the package has been very popular — 90k downloads. The package is easy to use. Whenever you are about to call the apply function, use the <em>pbapply</em> version of it. For example:</p>
<pre># Some numbers we are going to work with<br>nums &lt;- 1:10</pre>
<pre># Let's call the lapply to get the sqare root of these numbers<br>sqrt &lt;- sapply(nums, sqrt)</pre>
<pre># Now let's track the process using pbapply package<br>sqrt &lt;- pbsapply(nums, sqrt)</pre>
<p>While the numbers are processed, a progress bar will be printed to the output and refreshed repeatedly.</p>
<figure><img alt="" src="https://i1.wp.com/d262ilb51hltx0.cloudfront.net/max/522/1*l1Vsq4UgEaCXNfDQObEOGQ.jpeg?w=456&ssl=1" data-recalc-dims="1"><br>
<figcaption>A progress bar generated by pbapply. A user is able to get the estimated count-down time together with the current progress in the form of a progress bar.</figcaption>
</figure>
<p>Although <em>pbapply</em> is a great tool and I use it frequently, it failed to track the progress of the paralleled version of apply —<em> mcapply </em>— until recently. In September, the author of <em>pbapply</em> updated his package with support to snow-type clusters and multicore-type forking. However, his approach relies on splitting the elements into fractions and applies <em>mcapply</em> to them sequentially. One of the greatest caveat of this approach is that if the number of elements is significantly higher than the number of cores, a lot of <em>mcapply</em> calls will be executed. <em>Mcapply</em> calles, which is built upon the <em>fork()</em> function in Unix/Linux, is very expensive: forking into lots of child processes is time consuming and creates memory overhead.</p>
<figure><img alt="" src="https://i2.wp.com/d262ilb51hltx0.cloudfront.net/max/640/1*3a0frs1C7di7SV5MCqCXdg.gif?w=456&ssl=1" data-recalc-dims="1"><br>
<figcaption>Notice that <em>pbapply</em> generates lots of child processes while pbmcapply reuses them as much as possible. Four cores were allocated for pbapply/pbmcapply. R code used for benchmark can be downloaded <a target="_blank" rel="nofollow" href="https://gist.github.com/kvnkuang/884ac81f5e7d0ff3e262216a230bdc61">here</a>.</figcaption>
</figure>
<h4>Pbmcapply</h4>
<p><em>Pbmcapply</em> is my own solution to address this problem. Available as a <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=pbmcapply">CRAN</a> package, it can be easily incorporated into your code:</p>
<pre># Install pbmcapply<br>install.packages(pbmcapply)</pre>
<p>As you might have realized by its name, I was inspired by the <em>pbapply </em>package. Unlike <em>pbapply</em>, my solution does not rely on executing multiple <em>mcapply</em> calls. Instead, <em>pbmcapply</em> takes advantages of a package named <em>future</em>.</p>
<figure><img alt="" src="https://i2.wp.com/d262ilb51hltx0.cloudfront.net/max/1024/1*QOwcI8dNrqq-_DHXYzvhiA.jpeg?w=456&ssl=1" data-recalc-dims="1"><br>
<figcaption>The flowchart of pbmcapply. FutureCall() is executed in a separate process which will then be forked into defined amount of child processes. Child processes will update their progress frequently with the progressMonitor. Once received the updates, progressMonitor will print the progress to the output.</figcaption>
</figure>
<p>In Computer Science, future refers to an object that will hold values later. It allows the program to execute some code as a future and, without waiting for the return, proceed to the next step. In <em>pbmcapply</em>, <em>mcapply</em> will be wrapped into a future. The future will update the main program with its progress periodically and the main program will maintain a progress bar to display the updates.</p>
<p>Because the overhead was minimal and non-linear in <em>pbmcapply</em>, a dramatic increase of performance is seen when the number of elements to iterate over is significantly bigger than the number of CPU cores. Single-thread and multi-threaded apply functions from the R base are used as a reference. It is obvious that even with <em>pbmcapply</em>, the performance is affected due to time required to set up the monitor process.</p>
<figure><img alt="" src="https://i0.wp.com/d262ilb51hltx0.cloudfront.net/max/1024/1*5RTHtoliPSmAB927nLLLtA.png?w=456&ssl=1" data-recalc-dims="1"><br>
<figcaption>Performance between pbapply and pbmcapply. R code used for the benchmark can be downloaded from <a target="_blank" rel="nofollow" href="https://gist.github.com/kvnkuang/dfb7c441ccb141f7e2aa38c85b53c11b">here</a>. Left panel shows the overhead generated when each package was called. Right panel displays the time elapsed for the call to return.</figcaption>
</figure>
<p>Everything comes at a price. When enjoying the convenience of interactive progress tracking, please keep in mind that it slightly slows down the program.</p>
<h4>Conclusion</h4>
<p>Like always, one shoe doesn’t fit all. If performance is your top priority (e.g. when running a program on a cluster), a better way to track progress might be <strong><em>print</em></strong>. On the other hand, if letting the program run for an extra second sounds reasonable, you are more than welcome to check either my solution (<a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=pbmcapply"><em>pbmcapply</em></a>) or <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=pbapply%5D"><em>pbapply</em></a> in order to get a more user-friendly way to track the progress.</p>
<h4>Reference</h4>
<p>Myers, B. A. (1985). The importance of percent-done progress indicators for computer-human interfaces. In <em>ACM SIGCHI Bulletin</em> (Vol. 16, №4, pp. 11–17). ACM.</p>
<p><img height="1" width="1" src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad97998c359f"><br>
<hr>
<p><a target="_blank" rel="nofollow" href="https://kevinkuang.net/tracking-progress-in-r-ad97998c359f">Tracking progress in R</a> was originally published in <a target="_blank" rel="nofollow" href="https://kevinkuang.net/">Bioinformatics Playground</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/tracking-progress-in-r/"></div></div><div><a data-url="https://www.r-bloggers.com/tracking-progress-in-r/" data-counturl="https://www.r-bloggers.com/tracking-progress-in-r/" data-count="vertical" data-text="Tracking progress in R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://kevinkuang.net/tracking-progress-in-r-ad97998c359f?source=rss----a1ff9aea4bf1--r_programming"> R Programming in Bioinformatics Playground on Medium</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/lD35hdVMlyo"></a></div>