<p>I am participating in a session at <a target="_blank" rel="nofollow" href="https://user2017.brussels/">userR 2017</a> this coming July focusing on discovering and learning about R packages. How do R users find packages that meet their needs? Can we make this process easier? As somebody who is relatively new to the R world compared to many, this is a topic that resonates with me and I am happy to be part of the discussion. I am working on this session with <a target="_blank" rel="nofollow" href="http://web5.uottawa.ca/www2/mcs-smc/media/experts-details-iframe-93.html">John Nash</a> and <a target="_blank" rel="nofollow" href="https://www.r-pkg.org/maint/spencer.graves@prodsyse.com">Spencer Graves</a>, and we hope that some useful discussion and results come out of the session.</p>
<p>In preparation for this session, I wanted to look at the distribution of R packages by date, number of version, etc. There have been <a target="_blank" rel="nofollow" href="http://blog.revolutionanalytics.com/2017/01/cran-10000.html">some great plots</a> that came out around the time when CRAN passed the 10,000 package mark but most of the code to make those scripts involve packages and idioms I am less familiar with, so here is an rvest and tidyverse centered version of those analyses!</p>
<h2>Scraping CRAN</h2>
<p>The first thing we need to do is get all the packages that are currently available on CRAN. Let’s use <a target="_blank" rel="nofollow" href="https://github.com/hadley/rvest">rvest</a> to scrape the <a target="_blank" rel="nofollow" href="https://cran.rstudio.com/src/contrib">page that lists all the packages currently on CRAN</a>. It also has some other directories besides packages so we can use <code>filter</code> to remove the things that don’t look like R packages.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>rvest</span><span>)</span><span>
</span><span>library</span><span>(</span><span>stringr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>lubridate</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>cran_link</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>...</span><span>)</span><span> </span><span>{</span><span>
</span><span>file.path</span><span>(</span><span>&quot;https://cran.rstudio.com/src/contrib&quot;</span><span>,</span><span> </span><span>...</span><span>)</span><span>
</span><span>}</span><span>
</span><span>pkgs_raw</span><span> </span><span>&lt;-</span><span> </span><span>read_html</span><span>(</span><span>cran_link</span><span>())</span><span> </span><span>%>%</span><span> </span><span>html_nodes</span><span>(</span><span>&quot;table&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>.</span><span>[[</span><span>1</span><span>]]</span><span> </span><span>%>%</span><span>
</span><span>html_table</span><span>()</span><span>
</span><span>pkgs_raw</span><span> </span><span>&lt;-</span><span> </span><span>pkgs_raw</span><span>[,</span><span>-1</span><span>]</span><span>
</span><span>pkgs</span><span> </span><span>&lt;-</span><span> </span><span>pkgs_raw</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>Size</span><span> </span><span>!=</span><span> </span><span>&quot;-&quot;</span><span>,</span><span>
</span><span>str_detect</span><span>(</span><span>Name</span><span>,</span><span> </span><span>&quot;tar.gz$&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Date</span><span> </span><span>=</span><span> </span><span>dmy_hm</span><span>(</span><span>`Last modified`</span><span>),</span><span>
</span><span>Name</span><span> </span><span>=</span><span> </span><span>str_extract</span><span>(</span><span>Name</span><span>,</span><span> </span><span>&quot;^[^_]+(?=_)&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>-</span><span>Size</span><span>,</span><span> </span><span>-</span><span>Description</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as_tibble</span><span>()</span><span>
</span><span>pkgs</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 10,211 × 3
##            Name   `Last modified`                Date
##           &lt;chr>             &lt;chr>              &lt;dttm>
## 1            A3 16-Aug-2015 21:05 2015-08-16 21:05:00
## 2       ABC.RAP 20-Oct-2016 08:52 2016-10-20 08:52:00
## 3   ABCanalysis 23-Aug-2016 12:57 2016-08-23 12:57:00
## 4      ABCoptim 17-Nov-2016 09:04 2016-11-17 09:04:00
## 5         ABCp2 04-Feb-2016 10:27 2016-02-04 10:27:00
## 6  ABHgenotypeR 04-Feb-2016 10:27 2016-02-04 10:27:00
## 7           ACA 10-Mar-2016 16:55 2016-03-10 16:55:00
## 8        ACCLMA 29-Oct-2012 12:13 2012-10-29 12:13:00
## 9           ACD 31-Oct-2013 19:59 2013-10-31 19:59:00
## 10         ACDm 16-Jul-2016 10:19 2016-07-16 10:19:00
## # ... with 10,201 more rows</code></pre>
</figure>
<p>So that’s currently available packages!</p>
<p>Now let’s turn to the <a target="_blank" rel="nofollow" href="https://cran.rstudio.com/src/contrib/Archive/">archive</a>. Let’s do a similar operation.</p>
<figure>
<pre><code data-lang="r"><span>archives_raw</span><span> </span><span>&lt;-</span><span> </span><span>read_html</span><span>(</span><span>cran_link</span><span>(</span><span>&quot;Archive&quot;</span><span>))</span><span> </span><span>%>%</span><span> </span><span>html_nodes</span><span>(</span><span>&quot;table&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>.</span><span>[[</span><span>1</span><span>]]</span><span> </span><span>%>%</span><span>
</span><span>html_table</span><span>()</span><span>
</span><span>archives_raw</span><span> </span><span>&lt;-</span><span> </span><span>archives_raw</span><span>[,</span><span>-1</span><span>]</span><span>
</span><span>archives_processed</span><span> </span><span>&lt;-</span><span> </span><span>archives_raw</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>str_detect</span><span>(</span><span>Name</span><span>,</span><span> </span><span>&quot;/$&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Date</span><span> </span><span>=</span><span> </span><span>dmy_hm</span><span>(</span><span>`Last modified`</span><span>),</span><span>
</span><span>Name</span><span> </span><span>=</span><span> </span><span>str_sub</span><span>(</span><span>Name</span><span>,</span><span> </span><span>end</span><span> </span><span>=</span><span> </span><span>-2</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>-</span><span>Size</span><span>,</span><span> </span><span>-</span><span>Description</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as_tibble</span><span>()</span><span>
</span><span>archives_processed</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 8,897 × 3
##            Name   `Last modified`                Date
##           &lt;chr>             &lt;chr>              &lt;dttm>
## 1            A3 16-Aug-2015 21:05 2015-08-16 21:05:00
## 2   ABCExtremes 19-Jun-2015 11:26 2015-06-19 11:26:00
## 3   ABCanalysis 23-Aug-2016 12:57 2016-08-23 12:57:00
## 4      ABCoptim 17-Nov-2016 09:04 2016-11-17 09:04:00
## 5         ABCp2 01-Jul-2015 06:12 2015-07-01 06:12:00
## 6  ABHgenotypeR 04-Feb-2016 10:27 2016-02-04 10:27:00
## 7           ACD 31-Oct-2013 19:59 2013-10-31 19:59:00
## 8          ACDm 16-Jul-2016 10:19 2016-07-16 10:19:00
## 9          ACEt 18-Nov-2016 21:19 2016-11-18 21:19:00
## 10         ACNE 27-Oct-2015 07:09 2015-10-27 07:09:00
## # ... with 8,887 more rows</code></pre>
</figure>
<p>That is good, but now we need to get more detailed information for packages that have been archived at least once to get the date they originally were released and how many versions they have had.</p>
<h2>Visiting every page in the archive</h2>
<p>Let’s set up a function for scraping an individual page for a package and apply that to every page in the archive. This step takes A WHILE because it queries a web page for every package in the CRAN archive. I’ve set this up with <code>map</code> from purrr; it is one of my favorite ways to organize tasks these days.</p>
<figure>
<pre><code data-lang="r"><span>read_page</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>name</span><span>)</span><span> </span><span>{</span><span>
</span><span>message</span><span>(</span><span>name</span><span>)</span><span>
</span><span>read_html</span><span>(</span><span>cran_link</span><span>(</span><span>&quot;Archive&quot;</span><span>,</span><span> </span><span>name</span><span>))</span><span> </span><span>%>%</span><span> </span><span>html_nodes</span><span>(</span><span>&quot;td&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>html_text</span><span>()</span><span>
</span><span>}</span><span>
</span><span>archives_scraped</span><span> </span><span>&lt;-</span><span> </span><span>archives_processed</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>page</span><span> </span><span>=</span><span> </span><span>map</span><span>(</span><span>Name</span><span>,</span><span> </span><span>read_page</span><span>))</span></code></pre>
</figure>
<p>What do these pages look like?</p>
<figure>
<pre><code data-lang="r"><span>archives_scraped</span><span>$</span><span>page</span><span>[</span><span>8457</span><span>]</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [[1]]
##  [1] &quot;&quot;                      &quot;Parent Directory&quot;      &quot; &quot;                     &quot;  - &quot;                 
##  [5] &quot; &quot;                     &quot;&quot;                      &quot;tidytext_0.1.0.tar.gz&quot; &quot;28-Apr-2016 09:50  &quot;  
##  [9] &quot;1.4M&quot;                  &quot; &quot;                     &quot;&quot;                      &quot;tidytext_0.1.1.tar.gz&quot;
## [13] &quot;25-Jun-2016 17:07  &quot;   &quot;1.5M&quot;                  &quot; &quot;</code></pre>
</figure>
<p>This is exactly what we need: the dates that the packages were released and how many times they have been released. Let’s use <code>mutate</code> and <code>map</code> again to extract these values.</p>
<figure>
<pre><code data-lang="r"><span>archives</span><span> </span><span>&lt;-</span><span> </span><span>archives_scraped</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Date</span><span> </span><span>=</span><span> </span><span>dmy_hm</span><span>(</span><span>map_chr</span><span>(</span><span>page</span><span>,</span><span> </span><span>~</span><span> </span><span>.</span><span>[</span><span>8</span><span>])),</span><span>
</span><span>ArchivedVersions</span><span> </span><span>=</span><span> </span><span>map_dbl</span><span>(</span><span>page</span><span>,</span><span> </span><span>~</span><span> </span><span>length</span><span>(</span><span>.</span><span>)</span><span> </span><span>/</span><span> </span><span>5</span><span> </span><span>-</span><span> </span><span>1</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>-</span><span>page</span><span>)</span><span>
</span><span>archives</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 8,897 × 4
##            Name   `Last modified`                Date ArchivedVersions
##           &lt;chr>             &lt;chr>              &lt;dttm>            &lt;dbl>
## 1            A3 16-Aug-2015 21:05 2013-02-07 09:00:00                2
## 2   ABCExtremes 19-Jun-2015 11:26 2013-05-15 08:45:00                1
## 3   ABCanalysis 23-Aug-2016 12:57 2015-04-20 15:40:00                5
## 4      ABCoptim 17-Nov-2016 09:04 2013-11-05 17:00:00                2
## 5         ABCp2 01-Jul-2015 06:12 2013-04-10 15:04:00                2
## 6  ABHgenotypeR 04-Feb-2016 10:27 2016-01-21 07:26:00                1
## 7           ACD 31-Oct-2013 19:59 2012-11-27 06:43:00                2
## 8          ACDm 16-Jul-2016 10:19 2015-07-16 12:24:00                2
## 9          ACEt 18-Nov-2016 21:19 2016-02-14 17:48:00                7
## 10         ACNE 27-Oct-2015 07:09 2011-09-16 17:43:00                4
## # ... with 8,887 more rows</code></pre>
</figure>
<h2>Putting it together</h2>
<p>Not it’s time to join the data from the currently available packages and the archives.</p>
<ul>
<li>Packages that are in <code>archives</code> but not <code>pkgs</code> are no longer on CRAN.</li>
<li>Packages that are in <code>pkgs</code> but not <code>archives</code> only have one CRAN release.</li>
<li>Packages that are in both dataframes have had more than one CRAN release.</li>
</ul>
<p>Sounds like a good time to use <code>anti_join</code> and <code>inner_join</code>.</p>
<figure>
<pre><code data-lang="r"><span>all_pkgs</span><span> </span><span>&lt;-</span><span> </span><span>bind_rows</span><span>(</span><span>archives</span><span> </span><span>%>%</span><span> </span><span>anti_join</span><span>(</span><span>pkgs</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;Name&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>mutate</span><span>(</span><span>Archived</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>),</span><span>
</span><span>pkgs</span><span> </span><span>%>%</span><span> </span><span>anti_join</span><span>(</span><span>archives</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;Name&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>mutate</span><span>(</span><span>ArchivedVersions</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span>
</span><span>Archived</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>),</span><span>
</span><span>archives</span><span> </span><span>%>%</span><span>
</span><span>semi_join</span><span>(</span><span>pkgs</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;Name&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Archived</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Versions</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>Archived</span><span>,</span><span> </span><span>ArchivedVersions</span><span>,</span><span> </span><span>ArchivedVersions</span><span> </span><span>+</span><span> </span><span>1</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>Name</span><span>)</span><span>
</span><span>all_pkgs</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 11,489 × 6
##           Name   `Last modified`                Date ArchivedVersions Archived Versions
##          &lt;chr>             &lt;chr>              &lt;dttm>            &lt;dbl>    &lt;lgl>    &lt;dbl>
## 1           A3 16-Aug-2015 21:05 2013-02-07 09:00:00                2    FALSE        3
## 2         aaMI 30-Jul-2010 12:17 2005-06-24 15:55:00                2     TRUE        2
## 3       abbyyR 20-Jun-2016 15:32 2015-06-12 04:43:00                7    FALSE        8
## 4          abc 05-May-2015 09:34 2010-10-05 08:45:00               10    FALSE       11
## 5     abc.data 05-May-2015 09:34 2015-05-05 09:34:00                0    FALSE        1
## 6      ABC.RAP 20-Oct-2016 08:52 2016-10-20 08:52:00                0    FALSE        1
## 7  ABCanalysis 23-Aug-2016 12:57 2015-04-20 15:40:00                5    FALSE        6
## 8     abcdeFBA 15-Sep-2012 13:13 2011-11-05 10:48:00                3    FALSE        4
## 9  ABCExtremes 19-Jun-2015 11:26 2013-05-15 08:45:00                1     TRUE        1
## 10    ABCoptim 17-Nov-2016 09:04 2013-11-05 17:00:00                2    FALSE        3
## # ... with 11,479 more rows</code></pre>
</figure>
<h2>Plotting results</h2>
<p>Let’s look at some results now.</p>
<figure>
<pre><code data-lang="r"><span>all_pkgs</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>!</span><span>Archived</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>Date</span><span> </span><span>=</span><span> </span><span>floor_date</span><span>(</span><span>Date</span><span>,</span><span> </span><span>unit</span><span> </span><span>=</span><span> </span><span>&quot;month&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>NewPackages</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>TotalPackages</span><span> </span><span>=</span><span> </span><span>cumsum</span><span>(</span><span>NewPackages</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>Date</span><span>,</span><span> </span><span>TotalPackages</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>1.5</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;midnightblue&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Number of available packages&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;How many packages are available on CRAN?&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Only packages that are still available&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i2.wp.com/juliasilge.com/figs/2017-03-06-Scraping-CRAN/unnamed-chunk-8-1.png?w=456" data-recalc-dims="1"></p>
<p>There we go! That is similar to the results we all saw going around when CRAN passed 10,000 packages, which is good.</p>
<p>What about the number of archived vs. available packages?</p>
<figure>
<pre><code data-lang="r"><span>all_pkgs</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>Archived</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>stat</span><span> </span><span>=</span><span> </span><span>&quot;count&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;midnightblue&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_x_discrete</span><span>(</span><span>labels</span><span>=</span><span>c</span><span>(</span><span>&quot;Still available&quot;</span><span>,</span><span> </span><span>&quot;Archived, no longer available&quot;</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Number of packages&quot;</span><span>,</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;How many packages are no longer available on CRAN?&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;About 10% of total packages are no longer available&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i1.wp.com/juliasilge.com/figs/2017-03-06-Scraping-CRAN/unnamed-chunk-9-1.png?w=456" data-recalc-dims="1"></p>
<p>And lastly, let’s look at the distribution of number of releases for each package.</p>
<figure>
<pre><code data-lang="r"><span>all_pkgs</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>Versions</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>binwidth</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;midnightblue&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Number of packages&quot;</span><span>,</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Number of versions on CRAN&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;How many versions do CRAN packages have?&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;About 25% of packages are on their first version&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i0.wp.com/juliasilge.com/figs/2017-03-06-Scraping-CRAN/unnamed-chunk-10-1.png?w=456" data-recalc-dims="1"></p>
<h2>The End</h2>
<p>It is pretty ironic that I worked on this code and wrote this post because I wanted to do an analysis using different packages than the ones used in the original scripts shared. That is exactly part of the challenge facing all of us as R users now that there is such a diversity of tools out there! I hope that our session at useR this summer provides some clarity and perspective for attendees on these types of issues. The R Markdown file used to make this blog post is available <a target="_blank" rel="nofollow" href="https://github.com/juliasilge/juliasilge.github.io/blob/master/_R/2017-03-06-Scraping-CRAN.Rmd">here</a>. I am very happy to hear feedback or questions!</p>
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/y8NNrmji4jk">