<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/14-finding-binary-deb-files-for-cran-packages/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://dirk.eddelbuettel.com/blog/2017/12/22#014_finding_binary_deb_packages"> Thinking inside the box </a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Welcome to the fourteenth post in the <em>rationally rambling R rants</em> series, or <span><em>R</em><sup>4</sup></span> for short. The last two posts were concerned with faster installation. First, we showed how <a rel="nofollow" href="http://dirk.eddelbuettel.com/blog/2017/11/27#011_faster_package_installation_one" target="_blank">ccache can speed up (re-)installation</a>. This was followed by a second post on <a rel="nofollow" href="http://dirk.eddelbuettel.com/blog/2017/12/13#013_faster_package_installation_two" target="_blank">faster installation via binaries</a>.</p>
<p>This last post immediately sparked some follow-up. Replying to <a rel="nofollow" href="https://twitter.com/eddelbuettel/status/941339379137679360" target="_blank">my tweet</a> about it, <a rel="nofollow" href="https://twitter.com/revodavid?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor" target="_blank">David Smith</a> wondered how to combine binary and source installation (<em>tl;dr</em>: it is hard as you need to combine two package managers). Just this week, <a rel="nofollow" href="https://github.com/maxogden" target="_blank">Max Ogden</a> wondered <a rel="nofollow" href="https://twitter.com/denormalize/status/943226601835134976" target="_blank">how to install CRAN packages as binaries on Linux</a>, and <a rel="nofollow" href="https://github.com/nuest" target="_blank">Daniel Nuest</a> poked me <a rel="nofollow" href="https://github.com/o2r-project/containerit/issues/107" target="_blank">on GitHub as part of his excellent containerit project</a> as installation of binaries would of course also make Docker container builds much faster. (<em>tl;dr</em>: Oh yes, see below!)</p>
<p>So can one? Sure. We have a tool. But first the basics.</p>
<h4>The Basics</h4>
<p>Packages for a particular distribution are indexed by a packages file for that distribution. This is not unlike <a rel="nofollow" href="https://cran.r-project.org/" target="_blank">CRAN</a> using top-level <code>PACKAGES*</code> files. So in principle you could just fetch those packages files, parse and index them, and then search them. In practice that is a lot of work as Debian and Ubuntu now have several tens of thousands of packages.</p>
<p>So it is better to use the distro tool. In my use case on <code>.deb</code>-based distros, this is <code>apt-cache</code>. Here is a quick example for the (Ubuntu 17.04) server on which I type this:</p>
<div>
<pre><code>$ <span>sudo</span> apt-get update -qq            ## suppress stdout display
$ <span>apt-cache</span> search r-cran- <span>|</span> <span>wc</span> -l
<span>419</span>
$</code></pre>
</div>
<p>So a very vanilla Ubuntu installation has &quot;merely&quot; 400+ binary CRAN packages. Nothing to write home about (yet) — but read on.</p>
<h4>cran2deb4ubuntu, or c2d4u for short</h4>
<p>A decade ago, I was involved in two projects to turn all of CRAN into .deb binaries. We had a first ad-hoc predecessor project, and then (much better) a ‘version 2’ thanks to the <a rel="nofollow" href="https://wiki.debian.org/SummerOfCode2008/cran2deb" target="_blank">excellent Google Summer of Code work</a> by <a rel="nofollow" href="http://www.gatsby.ucl.ac.uk/~ucgtcbl/" target="_blank">Charles Blundell</a> (and mentored by me). I ran with that for a while and carried at the peak about 2500 binaries or so. And then my controlling db died, just as I visited CRAN to show it off. Very sad. <a rel="nofollow" href="https://www.donarmstrong.com/" target="_blank">Don Armstrong</a> ran with the code and rebuilt it on better foundations and had for quite some time all of CRAN and BioC built (peaking at maybe 7k package). Then his RAID died. The surviving effort is the one by <a rel="nofollow" href="http://behrend.psu.edu/person/michael-rutter" target="_blank">Michael Rutter</a> who always leaned on the <a rel="nofollow" href="http://launchpad.net/" target="_blank">Lauchpad</a> PPA system to build his packages. And those <a rel="nofollow" href="https://launchpad.net/~marutter/+archive/ubuntu/c2d4u" target="_blank">still exist</a> and provide a core of over 10k packages (but across different Ubuntu flavours, see below).</p>
<h4>Using cran2deb4ubuntu</h4>
<p>In order to access <a rel="nofollow" href="https://launchpad.net/~marutter/+archive/ubuntu/c2d4u" target="_blank">c2d4u</a> you need an Ubuntu system. For example my <a rel="nofollow" href="https://github.com/eddelbuettel/r-travis/blob/master/run.sh" target="_blank">Travis runner script</a> does</p>
<div>
<pre><code><span># Add marutter's c2d4u repository, (and rrutter for CRAN builds too)</span>
<span>sudo</span> add-apt-repository -y <span>&quot;ppa:marutter/rrutter&quot;</span>
<span>sudo</span> add-apt-repository -y <span>&quot;ppa:marutter/c2d4u&quot;</span></code></pre>
</div>
<p>After that one can query <code>apt-cache</code> as above, but take advantage of a much larger pool with over 3500 packages (see below). The <code>add-apt-repository</code> command does the Right Thing (TM) in terms of both getting the archive key, and adding the <code>apt</code> source entry to the config directory.</p>
<h4>How about from R? Sure, via RcppAPT</h4>
<p>Now, all this command-line business is nice. But can we do all this programmatically from R? Sort of.</p>
<p>The <a rel="nofollow" href="http://dirk.eddelbuettel.com/code/rcpp.apt.html" target="_blank">RcppAPT package</a> interface the libapt library, and provides access to a few functions. I used this feature when I argued (unsuccessfully, as it turned out) for a particular <a rel="nofollow" href="http://eddelbuettel.github.io/rcppapt/binnmuAfterR340.html" target="_blank">issue concerning Debian and R upgrades</a>. But that is water under the bridge now, and the main point is that &quot;yes we can&quot;.</p>
<h4>In Docker: r-apt</h4>
<p>Building on <a rel="nofollow" href="http://dirk.eddelbuettel.com/code/rcpp.apt.html" target="_blank">RcppAPT</a>, within the <a rel="nofollow" href="https://www.rocker-project.org/" target="_blank">Rocker Project</a> we built on top of this by proving a particular class of containers for different Ubuntu releases which all contain i) <a rel="nofollow" href="http://dirk.eddelbuettel.com/code/rcpp.apt.html" target="_blank">RcppAPT</a> and ii) the required <code>apt</code> source entry for Michael’s repos.</p>
<p>So now we can do this</p>
<div>
<pre><code>$ <span>docker</span> run --rm -ti rocker/r-apt:xenial /bin/bash -c <span>'apt-get update -qq; apt-cache search r-cran- | wc -l'</span>
<span>3525</span>
$</code></pre>
</div>
<p>This fires up the corresponding Docker container for the <code>xenial</code> (ie 16.04 LTS) release, updates the <code>apt</code> indices and then searches for <code>r-cran-*</code> packages. And it seems we have a little over 3500 packages. Not bad at all (especially once you realize that this skews strongly towards the more popular packages).</p>
<h4>Example: An rstan container</h4>
<p>A little while a ago a seemingly very frustrated user came to <a rel="nofollow" href="http://www.carlboettiger.info/" target="_blank">Carl</a> and myself and claimed that out <a rel="nofollow" href="https://www.rocker-project.org/" target="_blank">Rocker Project</a> sucketh because building <code>rstan</code> was all but impossible. I don’t have the time, space or inclination to go into details, but he was just plain wrong. You do need to know a little about C++, package building, and more to do this from scratch. Plus, there was a <a rel="nofollow" href="https://github.com/stan-dev/rstan/issues/441" target="_blank">long-standing issue with rstan and newer Boost</a> (which also included several workarounds).</p>
<p>Be that as it may, it serves as nice example here. So the first question: is <code>rstan</code> packaged?</p>
<div>
<pre><code>$ <span>docker</span> run --rm -ti rocker/r-apt:xenial /bin/bash -c <span>'apt-get update -qq; apt-cache show r-cran-rstan'</span>
<span>Package</span>: r-cran-rstan
<span>Source</span>: rstan
<span>Priority</span>: optional
<span>Section</span>: gnu-r
<span>Installed-Size</span>: 5110
<span>Maintainer</span>: cran2deb4ubuntu <span></span>cran2deb4ubuntu@gmail.com<span>></span>
<span>Architecture</span>: amd64
<span>Version</span>: 2.16.2-1cran1ppa0
<span>Depends</span>: pandoc, r-base-core, r-cran-ggplot2, r-cran-stanheaders, r-cran-inline, r-cran-gridextra, r-cran-rcpp,\
   r-cran-rcppeigen, r-cran-bh, libc6 (<span>></span>= 2.14), <span>libgcc1</span> (<span>></span>= 1:4.0), <span>libstdc++6</span> (<span>></span>= 5.2)
<span>Filename</span>: pool/main/r/rstan/r-cran-rstan_2.16.2-1cran1ppa0_amd64.deb
<span>Size</span>: 1481562
<span>MD5sum</span>: 60fe7cfc3e8813a822e477df24b37ccf
<span>SHA1</span>: 75bbab1a4193a5731ed105842725768587b4ec22
<span>SHA256</span>: 08816ea0e62b93511a43850c315880628419f2b817a83f92d8a28f5beb871fe2
<span>Description</span>: GNU R package <span>&quot;R Interface to Stan&quot;</span>
<span>Description-md5</span>: c9fc74a96bfde57f97f9d7c16a218fe5

$</code></pre>
</div>
<p>It would seem so. With that, the following <em>very minimal</em> Dockerfile is all we need:</p>
<div>
<pre><code><span>## Emacs, make this -*- mode: sh; -*-</span>

<span>## Start from xenial</span>
<span>FROM</span> rocker/r-apt:xenial

<span>## This handle reaches Carl and Dirk</span>
<span>MAINTAINER</span> <span>&quot;Carl Boettiger and Dirk Eddelbuettel&quot;</span> rocker-maintainers@eddelbuettel.com

<span>## Update and install rstan</span>
<span>RUN</span> apt-get update <span>&amp;&amp;</span> <span>apt-get</span> install -y --no-install-recommends r-cran-rstan

<span>## Make R the default</span>
<span>CMD</span> [<span>&quot;R&quot;</span>]</code></pre>
</div>
<p>In essence, it executes one command: install <code>rstan</code> but from <em>binary</em> taking care of all dependencies. And lo and behold, it works as advertised:</p>
<pre><code>$ docker run --rm -ti rocker/rstan:local Rscript -e 'library(rstan)'
Loading required package: ggplot2
Loading required package: StanHeaders
rstan (Version 2.16.2, packaged: 2017-07-03 09:24:58 UTC, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
rstan_options(auto_write = TRUE)
options(mc.cores = parallel::detectCores())
$</code></pre>
<p>So there: installing from binary works, takes care of dependencies, is easy and as an added bonus even faster. What’s not too like?</p>
<p>(And yes, a few of us are working on a system to have more packages available as binaries, but it may take another moment…)</p>
<p>
This post by <a rel="nofollow" href="http://dirk.eddelbuettel.com/" target="_blank">Dirk Eddelbuettel</a> originated on his <a rel="nofollow" href="http://dirk.eddelbuettel.com/blog/" target="_blank">Thinking inside the box</a> blog. Please report excessive re-aggregation in third-party for-profit settings.</p>
<p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://dirk.eddelbuettel.com/blog/2017/12/22#014_finding_binary_deb_packages"> Thinking inside the box </a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/lCqDgV0ALjo" height="1"></a></div>