<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/exploring-the-cran-social-network/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.pieceofk.fr/?p=431"> R_EN – Piece of K</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>A few months ago, I published <a rel="nofollow" href="http://www.pieceofk.fr/?p=408" target="_blank">a post</a> where I was trying to map the dependencies relationships between R packages. Today I want to do something similar with package contributors. My idea is to reconstruct a social graph where each node would be a person (presumably a developer), and two persons would be connected by an edge if they have collaborated on the same package. Thus I would be able to explore the CRAN social network!</p>
<p>This post is also a bit special for me. It’s the very first time I’m using <code>dplyr</code> and the <a rel="nofollow" href="https://www.tidyverse.org/" target="_blank">tidyverse</a>. I used to write my code in base R but the amazing work of <a rel="nofollow" href="https://twitter.com/thomasp85" target="_blank">Thomas Lin Pedersen</a> around <a rel="nofollow" href="https://github.com/thomasp85/tidygraph" target="_blank"><code>tiygraph</code></a> and <a rel="nofollow" href="https://github.com/thomasp85/ggraph" target="_blank"><code>ggraph</code></a> convinced me to take the plunge. And it was a lot of fun!</p>
<p>First, we load the packages and the data. In the first version of the project I used the XML file of each package that I retrieved from CRAN using wget and then read and parsed with <code>xml2</code>. Since <code>R 3.4.0</code>, things are much, much simpler with the function <code>CRAN_package_db</code>.</p>
<pre>
library(tidyverse)
library(stringr)
library(igraph)
library(tidygraph)
library(ggraph)
library(magrittr)

pdb &lt;- tools::CRAN_package_db()
aut &lt;- pdb$Author
</pre>
<p>That’s all we need to get the full list of contributors. But it comes in a very messy way. R provides <a rel="nofollow" href="https://journal.r-project.org/archive/2012/RJ-2012-009/index.html" target="_blank">a structured system to describe persons</a> which should be used in the <code>Authors@R</code> field of the DESCRIPTION file, but most people use a simple character string.</p>
<h1>The big cleanup</h1>
<p>I combined a series of regular expressions and string manipulations to extract every name. It was the first time I was using <code>stringr</code> and I have to say, I don’t regret <code>gsub</code>, <code>grepl</code>, <em>etc</em>. I just miss the support for some PCRE functionalities…</p>
<pre>
aut &lt;- aut %>%
  str_replace_all(&quot;\\(([^)]+)\\)&quot;, &quot;&quot;) %>%
  str_replace_all(&quot;\\[([^]]+)\\]&quot;, &quot;&quot;) %>%
  str_replace_all(&quot;&lt;([^>]+)>&quot;, &quot;&quot;) %>%
  str_replace_all(&quot;\n&quot;, &quot; &quot;) %>%
  str_replace_all(&quot;[Cc]ontribution.* from|[Cc]ontribution.* by|[Cc]ontributors&quot;, &quot; &quot;) %>%
  str_replace_all(&quot;\\(|\\)|\\[|\\]&quot;, &quot; &quot;) %>%
  iconv(to = &quot;ASCII//TRANSLIT&quot;) %>%
  str_replace_all(&quot;'$|^'&quot;, &quot;&quot;) %>%
  gsub(&quot;([A-Z])([A-Z]{1,})&quot;, &quot;\\1\\L\\2&quot;, ., perl = TRUE) %>%
  gsub(&quot;\\b([A-Z]{1}) \\b&quot;, &quot;\\1\\. &quot;, .) %>%
  map(str_split, &quot;,|;|&amp;| \\. |--|(?&lt;=[a-z])\\.| [Aa]nd | [Ww]ith | [Bb]y &quot;, simplify = TRUE) %>%
  map(str_replace_all, &quot;[[:space:]]+&quot;, &quot; &quot;) %>%
  map(str_replace_all, &quot; $|^ | \\.&quot;, &quot;&quot;) %>%
  map(function(x) x[str_length(x) != 0]) %>%
  set_names(pdb$Package) %>%
  extract(map_lgl(., function(x) length(x) > 1))

aut_list &lt;- aut %>%
  unlist() %>%
  dplyr::as_data_frame() %>%
  count(value) %>% 
  rename(Name = value, Package = n) 
</pre>
<p>At this point we have a list of 13668 (unique) names, and this number is increasing almost every day. The R community is huge! Of course there are still some errors. And we can find surprising contributors:</p>
<pre>
aut_list$Name[4922]
</pre>
<pre>
# [1] &quot;Her Majesty the Queen in Right of Canada&quot;
</pre>
<h1>The Social Network</h1>
<p>The next step is to produce a two-column matrix describing all the connections of the network (edge list). An edge list can be turned into a graph object with the <code>igraph</code> package. Finally we can convert the igraph graph into a tidy graph so we can use the API provided by the <code>tidygraph</code> package. For example to filter nodes/edges or select only the main component.</p>
<pre>
edge_list &lt;- aut %>%
  map(combn, m = 2) %>%
  do.call(&quot;cbind&quot;, .) %>%
  t() %>%
  dplyr::as_data_frame() %>%
  arrange(V1, V2) %>%
  count(V1, V2)

g &lt;- edge_list %>%
  select(V1, V2) %>%
  as.matrix() %>%
  graph.edgelist(directed = FALSE) %>% 
  as_tbl_graph() %>%
  activate(&quot;edges&quot;) %>%
  mutate(Weight = edge_list$n) %>%
  activate(&quot;nodes&quot;) %>%
  rename(Name = name) %>% 
  mutate(Component = group_components()) %>%
  filter(Component == names(table(Component))[which.max(table(Component))])
</pre>
<p>The author of <code>tidygraph</code>, Thomas, also developed the <code>ggraph</code> package, an implementation of the grammar of graphics for relational data. Using <code>ggraph</code>, it is very easy to visualize our graph.</p>
<pre>
ggraph(g, layout = 'lgl') +
  geom_edge_fan(alpha = 0.1) +
  theme_graph()
</pre>
<p><a rel="nofollow" href="http://www.pieceofk.fr/?attachment_id=511" target="_blank"><img sizes="(max-width: 598px) 100vw, 598px" src="https://i0.wp.com/www.pieceofk.fr/wp-content/uploads/2017/09/Rplot.png?w=450" alt="R cran network" class="wp-image-511" data-recalc-dims="1"></a></p>
<p>This is it! This hairy ink stain is the main connected component of the CRAN social network, representing collaborations between 6758 developers!</p>
<p>We can look for who’s the « most important » node in the graph according to the Google’s <a rel="nofollow" href="https://en.wikipedia.org/wiki/PageRank" target="_blank">PageRank algorithm</a>, although it’s not straightforward what could mean a high PR here…</p>
<pre>
g %>%
  mutate(Page_rank = centrality_pagerank()) %>% 
  top_n(10, Page_rank) %>% 
  as_tibble() %>% 
  ggplot() +
    geom_col(aes(forcats::fct_reorder(Name, Page_rank), Page_rank)) +
    coord_flip() + theme_minimal() +
    labs(title = &quot;Top 10 contributors based on Page Rank&quot;, x = &quot;&quot;)
</pre>
<p><a rel="nofollow" href="http://www.pieceofk.fr/?attachment_id=500" target="_blank"><img sizes="(max-width: 700px) 100vw, 700px" src="https://i1.wp.com/www.pieceofk.fr/wp-content/uploads/2017/09/Rplot02.png?w=450" alt="Barplot_top10_pagerank" class="wp-image-500" data-recalc-dims="1"></a></p>
<h1>Focus on the core network</h1>
<p>The complete graph makes a nice painting to hang on the wall of your living room. But hard to say anything about it. So, we will reduce the data and focus on the contributors involved in more than 4 packages.</p>
<pre>
g &lt;- g %>% 
  left_join(aut_list) %>% 
  filter(Package > 4) %>% 
  mutate(Component = group_components()) %>%
  filter(Component == names(table(Component))[which.max(table(Component))])

ggraph(g, layout = 'lgl') +
  geom_edge_fan(alpha = 0.1) +
  theme_graph()
</pre>
<p><a rel="nofollow" href="http://www.pieceofk.fr/?attachment_id=499" target="_blank"><img sizes="(max-width: 632px) 100vw, 632px" src="https://i0.wp.com/www.pieceofk.fr/wp-content/uploads/2017/09/Rplot03.png?w=450" alt="CRAN network zoom" class="wp-image-499" data-recalc-dims="1"></a></p>
<p>Nice! It seems like there are two distinct clusters in the middle. Let’s see if we can identify them using a community detection algorithm.</p>
<pre>
g &lt;- mutate(g, Community = group_edge_betweenness(),
            Degree = centrality_degree())

filter(g, Community == names(sort(table(Community), decr = TRUE))[1]) %>%
  select(Name, Package) %>%
  arrange(desc(Package)) %>%
  top_n(10, Package) %>%
  as_tibble() %>% 
  knitr::kable(format = &quot;html&quot;, caption = &quot;Cluster 1&quot;)
filter(g, Community == names(sort(table(Community), decr = TRUE))[2]) %>%
  select(Name, Package) %>%
  arrange(desc(Package)) %>%
  top_n(10, Package) %>%
  as_tibble() %>% 
  knitr::kable(format = &quot;html&quot;, caption = &quot;Cluster 2&quot;)
</pre>
<table>
<caption>Cluster 1</caption>
<thead>
<tr>
<th> Name </th>
<th> Package </th>
</tr>
</thead>
<tbody>
<tr>
<td> Kurt Hornik </td>
<td> 56 </td>
</tr>
<tr>
<td> Martin Maechler </td>
<td> 46 </td>
</tr>
<tr>
<td> Achim Zeileis </td>
<td> 44 </td>
</tr>
<tr>
<td> Dirk Eddelbuettel </td>
<td> 41 </td>
</tr>
<tr>
<td> Romain Francois </td>
<td> 27 </td>
</tr>
<tr>
<td> Ben Bolker </td>
<td> 25 </td>
</tr>
<tr>
<td> Brian Ripley </td>
<td> 25 </td>
</tr>
<tr>
<td> Torsten Hothorn </td>
<td> 24 </td>
</tr>
<tr>
<td> Roger Bivand </td>
<td> 23 </td>
</tr>
<tr>
<td> Douglas Bates </td>
<td> 23 </td>
</tr>
</tbody>
</table>
<table>
<caption>Cluster 2</caption>
<thead>
<tr>
<th> Name </th>
<th> Package </th>
</tr>
</thead>
<tbody>
<tr>
<td> Hadley Wickham </td>
<td> 95 </td>
</tr>
<tr>
<td> Rstudio </td>
<td> 92 </td>
</tr>
<tr>
<td> Scott Chamberlain </td>
<td> 44 </td>
</tr>
<tr>
<td> Inc </td>
<td> 36 </td>
</tr>
<tr>
<td> Yihui Xie </td>
<td> 34 </td>
</tr>
<tr>
<td> Jeroen Ooms </td>
<td> 33 </td>
</tr>
<tr>
<td> Jj Allaire </td>
<td> 29 </td>
</tr>
<tr>
<td> R. Core Team </td>
<td> 27 </td>
</tr>
<tr>
<td> Bob Rudis </td>
<td> 26 </td>
</tr>
<tr>
<td> Gabor Csardi </td>
<td> 22 </td>
</tr>
<tr>
<td> Oliver Keyes </td>
<td> 22 </td>
</tr>
<tr>
<td> Winston Chang </td>
<td> 22 </td>
</tr>
</tbody>
</table>
<p>If you know a bit R and its community, you will certainly recognize some names. The challenge is to interpret this classification. It is not a good thing to put people in boxes, but this is mandatory to make nice and colorful visualizations! So, I give it a try. For me, the first group include people of the early days who contributed to major packages which constitute the heart of R. The second group is more related to the second generation, associated with Rstudio products and who have been particularly prolific in the last years. Of course the two groups are strongly connected.</p>
<p>How can we label these two groups? After long consideration, I chose to go for « The Ancients » and « The Moderns », without any value judgment! This is a reference to <a rel="nofollow" href="https://en.wikipedia.org/wiki/Quarrel_of_the_Ancients_and_the_Moderns" target="_blank">an old debate in French literature</a>. This dubious comparison would certainly be more appropriate to designate the ongoing debate base vs. tidyverse, but… not today!</p>
<p>Let’s see how we can visualize these two groups.</p>
<pre>
g &lt;- g %>% 
  mutate(Community = case_when(Community == names(sort(table(Community),
                                                       decr = TRUE))[1] ~ &quot;The Ancients&quot;,
                               Community == names(sort(table(Community),
                                                       decr = TRUE))[2] ~ &quot;The Moderns&quot;,
                               Community %in% names(sort(table(Community),
                                                       decr = TRUE))[-1:-2] ~ &quot;Unclassified&quot;)) %>% 
  mutate(Community = factor(Community))

g &lt;- g %>%
  filter(Degree > 5) %>% 
  mutate(Degree = centrality_degree())

ggraph(g, layout = 'lgl') +
  geom_edge_fan(alpha = 0.1) +
  geom_node_point(aes(color = Community, size = Package)) +
  theme_graph() +
  scale_color_manual(breaks = c(&quot;The Ancients&quot;, &quot;The Moderns&quot;),
                     values=c(&quot;#F8766D&quot;, &quot;#00BFC4&quot;, &quot;#969696&quot;))
</pre>
<p><a rel="nofollow" href="http://www.pieceofk.fr/?attachment_id=498" target="_blank"><img sizes="(max-width: 740px) 100vw, 740px" src="https://i1.wp.com/www.pieceofk.fr/wp-content/uploads/2017/09/Rplot04.png?w=450" alt="CRAN network communities" class="wp-image-498" data-recalc-dims="1"></a></p>
<p><code>ggraph</code> really does a great job! I really like the result.</p>
<p>BTW, the R community is not really divided…</p>
<pre>
g %>% activate(&quot;edges&quot;) %>% 
  mutate(Community_from = .N()$Community[from],
         Community_to = .N()$Community[to]) %>% 
  filter((Community_from == &quot;The Ancients&quot; &amp; Community_to == &quot;The Moderns&quot;)|
           Community_from == &quot;The Moderns&quot; &amp; Community_to == &quot;The Ancients&quot;) %>% 
  as_tibble() %>% 
  nrow()
</pre>
<p>… there are 254 edges connecting the the Ancients and the Moderns! But I think this is a nice illustration of how social graphs keeps the imprint of history and past events. Next step would be to look at the graph dynamics over time. Maybe an idea for a future post…</p>
<p><a rel="nofollow" data-provider="facebook" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D431&t=Exploring%20the%20CRAN%20social%20network&%23038;s=100&%23038;p&%23091;url&%23093;=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D431&%23038;p&%23091;images&%23093;&%23091;0&%23093;=http%3A%2F%2Fwww.pieceofk.fr%2Fwp-content%2Fuploads%2F2017%2F09%2FRplot.png&%23038;p&%23091;title&%23093;=Exploring%20the%20CRAN%20social%20network" title="Share on Facebook" target="_blank"><img src="https://i0.wp.com/www.pieceofk.fr/wp-content/plugins/social-media-feather/synved-social/image/social/regular/32x32/facebook.png?resize=16%2C16" alt="Facebook" title="Share on Facebook" data-recalc-dims="1"></a><a rel="nofollow" data-provider="twitter" href="https://twitter.com/intent/tweet?url=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D431&text=Hey%20check%20this%20out" title="Share on Twitter" target="_blank"><img src="https://i0.wp.com/www.pieceofk.fr/wp-content/plugins/social-media-feather/synved-social/image/social/regular/32x32/twitter.png?resize=16%2C16" alt="twitter" title="Share on Twitter" data-recalc-dims="1"></a></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.pieceofk.fr/?p=431"> R_EN – Piece of K</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/VOvTn3i3JEU" height="1"></a></div>