<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/17-dependencies/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://dirk.eddelbuettel.com/blog/2018/02/28#017_dependencies"> Thinking inside the box </a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<br>
<blockquote>
<p>Dependencies are invitations for other people to break your package.<br>
— Josh Ulrich, private communication</p>
</blockquote>
<p>Welcome to the seventeenth post in the <em>relentlessly random R ravings</em> series of posts, or <span><em>R</em><sup>4</sup></span> for short.</p>
<p>Dependencies. A truly loaded topic.</p>
<p>As R users, we are spoiled. Early in the history of R, Kurt Hornik and Friedrich Leisch built support for <em>packages</em> right into R, and started the Comprehensive R Archive Network (CRAN). And R and CRAN had a fantastic run with. Roughly twenty years later, we are looking at over 12,000 packages which can (generally) be installed with absolute ease and no suprises. No other (relevant) open source language has anything of comparable rigour and quality. This is a big deal.</p>
<p>And coding practices evolved and changed to play to this advantage. Packages are a near-unanimous recommendation, use of the <code>install.packages()</code> and <code>update.packages()</code> tooling is nearly universal, and most R users learned to their advantage to group code into interdependent packages. Obvious advantages are versioning and snap-shotting, attached documentation in the form of help pages and vignettes, unit testing, and of course continuous integration as a side effect of the package build system.</p>
<p>But the notion of ‘oh, let me just build another package and add it to the pool of packages’ can get carried away. A recent example I had was the work on the <a rel="nofollow" href="http://dirk.eddelbuettel.com/code/prrd.html" target="_blank">prrd</a> package for <em>parallel recursive dependency testing</em> — coincidentally, created entirely to allow for easier <em>voluntary</em> tests I do on reverse dependencies for the packages I maintain. It uses a job queue for which I relied on the <a rel="nofollow" href="https://github.com/r-lib/liteq/" target="_blank">liteq</a> package by <a rel="nofollow" href="https://github.com/gaborcsardi" target="_blank">Gabor</a> which does the job: enqueue jobs, and reliably dequeue them (also in a parallel fashion) and more. It looks light enough:</p>
<div>
<pre><code>R<span>></span><span> </span>tools<span>::</span><span>package_dependencies</span>(<span>package=</span><span>&quot;liteq&quot;</span>, <span>recursive=</span><span>FALSE</span>, <span>db=</span>AP)<span>$</span>liteq
[<span>1</span>] <span>&quot;assertthat&quot;</span> <span>&quot;DBI&quot;</span>        <span>&quot;rappdirs&quot;</span>   <span>&quot;RSQLite&quot;</span>   
R<span>></span><span> </span></code></pre>
</div>
<p>Two dependencies because it uses an internal SQLite database, one for internal tooling and one for configuration.</p>
<p>All good then? Not so fast. The devil here is the very innocuous and versatile <a rel="nofollow" href="http://rsqlite.r-dbi.org/" target="_blank">RSQLite</a> package because when we look at fully <em>recursive</em> dependencies all hell breaks loose:</p>
<div>
<pre><code>R<span>></span><span> </span>tools<span>::</span><span>package_dependencies</span>(<span>package=</span><span>&quot;liteq&quot;</span>, <span>recursive=</span><span>TRUE</span>, <span>db=</span>AP)<span>$</span>liteq
 [<span>1</span>] <span>&quot;assertthat&quot;</span> <span>&quot;DBI&quot;</span>        <span>&quot;rappdirs&quot;</span>   <span>&quot;RSQLite&quot;</span>    <span>&quot;tools&quot;</span>     
 [<span>6</span>] <span>&quot;methods&quot;</span>    <span>&quot;bit64&quot;</span>      <span>&quot;blob&quot;</span>       <span>&quot;memoise&quot;</span>    <span>&quot;pkgconfig&quot;</span> 
[<span>11</span>] <span>&quot;Rcpp&quot;</span>       <span>&quot;BH&quot;</span>         <span>&quot;plogr&quot;</span>      <span>&quot;bit&quot;</span>        <span>&quot;utils&quot;</span>     
[<span>16</span>] <span>&quot;stats&quot;</span>      <span>&quot;tibble&quot;</span>     <span>&quot;digest&quot;</span>     <span>&quot;cli&quot;</span>        <span>&quot;crayon&quot;</span>    
[<span>21</span>] <span>&quot;pillar&quot;</span>     <span>&quot;rlang&quot;</span>      <span>&quot;grDevices&quot;</span>  <span>&quot;utf8&quot;</span>      
R<span>></span>
R<span>></span><span> </span>tools<span>::</span><span>package_dependencies</span>(<span>package=</span><span>&quot;RSQLite&quot;</span>, <span>recursive=</span><span>TRUE</span>, <span>db=</span>AP)<span>$</span>RSQLite
 [<span>1</span>] <span>&quot;bit64&quot;</span>      <span>&quot;blob&quot;</span>       <span>&quot;DBI&quot;</span>        <span>&quot;memoise&quot;</span>    <span>&quot;methods&quot;</span>   
 [<span>6</span>] <span>&quot;pkgconfig&quot;</span>  <span>&quot;Rcpp&quot;</span>       <span>&quot;BH&quot;</span>         <span>&quot;plogr&quot;</span>      <span>&quot;bit&quot;</span>       
[<span>11</span>] <span>&quot;utils&quot;</span>      <span>&quot;stats&quot;</span>      <span>&quot;tibble&quot;</span>     <span>&quot;digest&quot;</span>     <span>&quot;cli&quot;</span>       
[<span>16</span>] <span>&quot;crayon&quot;</span>     <span>&quot;pillar&quot;</span>     <span>&quot;rlang&quot;</span>      <span>&quot;assertthat&quot;</span> <span>&quot;grDevices&quot;</span> 
[<span>21</span>] <span>&quot;utf8&quot;</span>       <span>&quot;tools&quot;</span>     
R<span>></span><span> </span></code></pre>
</div>
<p>Now we went from four to twenty-four, due to the twenty-two dependencies pulled in by <a rel="nofollow" href="http://rsqlite.r-dbi.org/" target="_blank">RSQLite</a>.</p>
<p>There, my dear friend, lies madness. The moment one of these packages breaks we get potential side effects. And this is no laughing matter. Here is a tweet from <a rel="nofollow" href="https://kieranhealy.org/" target="_blank">Kieran</a> posted days before a book deadline of his when he was forced to roll a CRAN package back because it broke his entire setup. (The original tweet has by now been deleted; why people do that to their entire tweet histories is somewhat I fail to comprehened too; in any case the screenshot is from a private discussion I had with a few like-minded folks over slack.)</p>
<div>
<img src="https://i1.wp.com/dirk.eddelbuettel.com/blog/2018/02/28/tweet_kieran_healey_20180106.png?w=456" data-recalc-dims="1">
</div>
<p>That illustrates the quote by Josh at the top. As I too have &quot;production code&quot; (well, <a rel="nofollow" href="http://dirk.eddelbuettel.com/cranberries/" target="_blank">CRANberries</a> for one relies on it), I was interested to see if we could easily amend RSQLite. And yes, we can. A quick <a rel="nofollow" href="https://github.com/eddelbuettel/RSQLite" target="_blank">fork and few commits later</a>, we have something we could call ‘RSQLighter’ as it reduces the dependencies quite a bit:</p>
<div>
<pre><code>R<span>></span><span> </span>IP &lt;-<span> </span><span>installed.packages</span>()   <span># using my installed mod'ed version</span>
R<span>></span><span> </span>tools<span>::</span><span>package_dependencies</span>(<span>package=</span><span>&quot;RSQLite&quot;</span>, <span>recursive=</span><span>TRUE</span>, <span>db=</span>IP)<span>$</span>RSQLite
 [<span>1</span>] <span>&quot;bit64&quot;</span>     <span>&quot;DBI&quot;</span>       <span>&quot;methods&quot;</span>   <span>&quot;Rcpp&quot;</span>      <span>&quot;BH&quot;</span>        <span>&quot;bit&quot;</span>      
 [<span>7</span>] <span>&quot;utils&quot;</span>     <span>&quot;stats&quot;</span>     <span>&quot;grDevices&quot;</span> <span>&quot;graphics&quot;</span> 
R<span>></span></code></pre>
</div>
<p>That is less than half. I have not proceeded with the fork because I do not believe in needlessly splitting codebases. But this could be a viable candidate for an alternate or shadow repository with more minimal and hence more robust dependencies. Or, as Josh calls, the <a rel="nofollow" href="http://www.tinyverse.org/" target="_blank"><em>tinyverse</em></a>.</p>
<p>Another maddening aspect of dependencies is the ruthless application of what we could jokingly call <a rel="nofollow" href="https://en.wikipedia.org/wiki/Metcalfe%27s_law" target="_blank">Metcalf’s Law</a>: the likelihood of breakage does of course increase with the number edges in the dependency graph. A nice illustration is this post by <a rel="nofollow" href="https://github.com/jennybc" target="_blank">Jenny</a> trying to rationalize why one of the 87 (as of today) tidyverse packages has now state &quot;ORPHANED&quot; at CRAN:</p>
<div>
<img src="https://i1.wp.com/dirk.eddelbuettel.com/blog/2018/02/28/issue_comment_jenny_bryan_20180127.png?w=456" data-recalc-dims="1">
</div>
<p><em>An invitation for other people to break your code.</em> Well put indeed. Or to put rocks up your path.</p>
<p>But things are not all that dire. Most folks appear to understand the issue, some even do something about it. The <a rel="nofollow" href="http://dbi.r-dbi.org/" target="_blank">DBI</a> and <a rel="nofollow" href="https://github.com/r-dbi/RMySQL" target="_blank">RMySQL</a> packages have saner <em>strict</em> dependencies, maybe one day things will improve for <a rel="nofollow" href="https://github.com/r-dbi/RMariaDB" target="_blank">RMariaDB</a> and <a rel="nofollow" href="http://rsqlite.r-dbi.org/" target="_blank">RSQLite</a> too:</p>
<div>
<pre><code>R<span>></span><span> </span>tools<span>::</span><span>package_dependencies</span>(<span>package=</span><span>c</span>(<span>&quot;DBI&quot;</span>, <span>&quot;RMySQL&quot;</span>, <span>&quot;RMariaDB&quot;</span>), <span>recursive=</span><span>TRUE</span>, <span>db=</span>AP)
<span>$</span>DBI
[<span>1</span>] <span>&quot;methods&quot;</span>

<span>$</span>RMySQL
[<span>1</span>] <span>&quot;DBI&quot;</span>     <span>&quot;methods&quot;</span>

<span>$</span>RMariaDB
 [<span>1</span>] <span>&quot;bit64&quot;</span>     <span>&quot;DBI&quot;</span>       <span>&quot;hms&quot;</span>       <span>&quot;methods&quot;</span>   <span>&quot;Rcpp&quot;</span>      <span>&quot;BH&quot;</span>       
 [<span>7</span>] <span>&quot;plogr&quot;</span>     <span>&quot;bit&quot;</span>       <span>&quot;utils&quot;</span>     <span>&quot;stats&quot;</span>     <span>&quot;pkgconfig&quot;</span> <span>&quot;rlang&quot;</span>    

R<span>></span><span> </span></code></pre>
</div>
<p>And to be clear, I do not believe in giving up and using everything via docker, or virtualenvs, or packrat, or … A well-honed dependency system is <em>wonderful</em> and the right resource to get code deployed and updated. But it required buy-in from everyone involved, and an understanding of the possible trade-offs. I think we can, and will, do better going forward.</p>
<p>Or else, there will always be the <a rel="nofollow" href="http://www.tinyverse.org/" target="_blank"><em>tinyverse</em></a> …</p>
<p>
This post by <a rel="nofollow" href="http://dirk.eddelbuettel.com/" target="_blank">Dirk Eddelbuettel</a> originated on his <a rel="nofollow" href="http://dirk.eddelbuettel.com/blog/" target="_blank">Thinking inside the box</a> blog. Please report excessive re-aggregation in third-party for-profit settings.</p>
<p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://dirk.eddelbuettel.com/blog/2018/02/28#017_dependencies"> Thinking inside the box </a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/hBz2d2QBK1Q" height="1"></a></div>