<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/11-much-faster-package-re-installation-via-caching/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://dirk.eddelbuettel.com/blog/2017/11/27#011_faster_package_installation_one"> Thinking inside the box </a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Welcome to the eleventh post in the <em>rarely rued R rants</em> series, or <span><em>R</em><sup>4</sup></span> for short. Time clearly flies as it has been three months since out last post on <a rel="nofollow" href="http://dirk.eddelbuettel.com/blog/2017/08/20#010_stripping_shared_libraries" target="_blank">significantly reducing library size via stripping</a>. I had been meaning to post on today’s topic for quite some time, but somehow <em>something</em> (working on a paper, releasing a package, …) got in the way.</p>
<p>Just a few days ago <a rel="nofollow" href="http://www.mas.ncl.ac.uk/~ncsg3/" target="_blank">Colin</a> (of <a rel="nofollow" href="https://www.amazon.com/Efficient-R-Programming-Colin-Gillespie/dp/1491950781/" target="_blank">Efficient R Programming</a> fame) posted about <a rel="nofollow" href="http://blog.jumpingrivers.com/posts/2017/speed_package_installation/" target="_blank">speed(ing up) package installation</a>. His recommendation? Remember that we (usually) have multiple cores and using several of them via <code>options(Ncpus = XX)</code>. It is an excellent point, and it bears repeating.</p>
<p>But it turns I have not one but two salient recommendations too. Today covers the first, we should hopefully get pretty soon to the second. Both have one thing in common: <em>you will be fastest if you avoid doing the work in the first place</em>.</p>
<h3>What?</h3>
<p>One truly outstanding tool for this in the context of the installation of <em>compiled</em> packages is <a rel="nofollow" href="https://ccache.samba.org/" target="_blank">ccache</a>. It is actually a pretty old tool that has been out for well over a decade, and it comes from the folks that gave the Samba filesystem.</p>
<p>What does it do? Well, it nutshell, it &quot;hashes&quot; a checksum of a source file once the preprocessor has operated on it and stores the resulting object file. In the case of rebuild <em>with unchanged code</em> you get the object code back pretty much immediately. The idea is very similar to memoisation (as implemented in R for example in the excellent little <a rel="nofollow" href="https://cran.r-project.org/package=memoise" target="_blank">memoise</a> package by Hadley, Jim, Kirill and Daniel. The idea is the same: if you have to do something even moderately expensive a few times, do it once and then recall it the other times.</p>
<p>This happens (at least to me) more often that not in package development. Maybe you change just one of several source files. Maybe you just change the R code, the Rd documentation or a test file—yet still need a full reinstallation. In all these cases, <code>ccache</code> can help <em>tremdendously</em> as illustrated below.</p>
<h3>How?</h3>
<p>Because essentially all our access to compilation happens through R, we need to set this in a file read by R. I use <code>~/.R/Makevars</code> for this and have something like these lines on my machines:</p>
<pre><code>VER=
CCACHE=ccache
CC=$(CCACHE) gcc$(VER)
CXX=$(CCACHE) g++$(VER)
CXX11=$(CCACHE) g++$(VER)
CXX14=$(CCACHE) g++$(VER)
FC=$(CCACHE) gfortran$(VER)
F77=$(CCACHE) gfortran$(VER)</code></pre>
<p>That way, when R calls the compiler(s) it will prefix with <code>ccache</code>. And <code>ccache</code> will then speed up.</p>
<p>There is an additional issue due to R use. Often we install from a <code>.tar.gz</code>. These will be freshly unpackaged, and hence have &quot;new&quot; timestamps. This would usually lead <code>ccache</code> to skip to file (fear of &quot;false positives&quot;) so we have to override this. Similarly, the tarball is usually unpackage in a temporary directory with an ephemeral name, creating a unique path. That too needs to be overwritten. So in my <code>~/.ccache/ccache.conf</code> I have this:</p>
<pre><code>max_size = 5.0G
# important for R CMD INSTALL *.tar.gz as tarballs are expanded freshly -> fresh ctime
sloppiness = include_file_ctime
# also important as the (temp.) directory name will differ
hash_dir = false</code></pre>
<h3>Show Me</h3>
<p>A quick illustration will round out the post. Some packages are meatier than others. More C++ with more templates usually means longer build times. Below is a quick chart comparing times for a few such packages (ie RQuantLib, dplyr, rstan) as well as igraph (&quot;merely&quot; a large C package) and lme4 as well as Rcpp. The worst among theseis still my own <a rel="nofollow" href="http://dirk.eddelbuettel.com/code/rquantlib.html" target="_blank">RQuantLib</a> package wrapping (still just parts of) the genormous and Boost-heavy <a rel="nofollow" href="http://quantlib.org/index.shtml" target="_blank">QuantLib</a> library.</p>
<div>
<img src="https://i2.wp.com/dirk.eddelbuettel.com/blog/2017/11/27/ccache_impact.png?w=456" data-recalc-dims="1">
</div>
<p>Pretty dramatic gains. Best of all, we can of course <em>combine</em> these with other methods such as Colin’s use of multiple CPUs, or even a simple <code>MAKE=make -j4</code> to have multiple compilation units being considered in parallel. So maybe we all get to spend less time on social media and other timewasters as we spend less time waiting for our builds. Or maybe that is too much to hope for…</p>
<p>
This post by <a rel="nofollow" href="http://dirk.eddelbuettel.com/" target="_blank">Dirk Eddelbuettel</a> originated on his <a rel="nofollow" href="http://dirk.eddelbuettel.com/blog/" target="_blank">Thinking inside the box</a> blog. Please report excessive re-aggregation in third-party for-profit settings.</p>
<p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://dirk.eddelbuettel.com/blog/2017/11/27#011_faster_package_installation_one"> Thinking inside the box </a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/R82bBynMVBo" height="1"></a></div>