<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/how-much-work-is-onboarding/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/05/03/onboarding-is-work/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Our <a rel="nofollow" href="https://github.com/ropensci/onboarding/" target="_blank">onboarding process</a>, that<br>
ensures that packages contributed by the community undergo a<br>
transparent, constructive, non adversarial and open review process,<br>
involves a lot of work from many actors: authors, reviewers and editors;<br>
but <em>how much work</em>? Managing the effort involved in the peer-review<br>
process is a major part of ensuring its sustainability and quality. In<br>
this post, we’ll take a look at the effort put in by participants in the<br>
review process, and also learn something about exploring GitHub data<br>
along the way.</p>
<h3>Work done by authors</h3>
<p>We can try to quantify the development work of authors by looking at the<br>
number of lines deleted and added in the git repo of the package before,<br>
during, and after review. By number of lines we mean number of lines in<br>
any file in the repo so it can be lines of actual code and lines of docs<br>
(in the README for instance). To do so, we link information from the<br>
package git repos (information about sizes and dates of commits) and<br>
information from the onboarding issue threads (start and end dates of<br>
the review process, as measured by open/close dates).</p>
<pre><code>library(&quot;ggplot2&quot;)
library(&quot;magrittr&quot;)

commits &lt;- readr::read_csv(&quot;output/gitsum_reports.csv&quot;)
issues &lt;- readr::read_csv(&quot;data/clean_data.csv&quot;)
issues &lt;- dplyr::group_by(issues, package) %>%
  dplyr::summarise(opened = min(created_at),
                   closed = max(closed_at))
commits &lt;- dplyr::left_join(commits, issues, by = &quot;package&quot;)

custom_transf &lt;- function(x){
  pof &lt;- log(x)
  pof[x == 0] &lt;- 0
  pof
}

custom_exp &lt;- function(x){
  pof &lt;- exp(x)
  pof[x &lt; 0] &lt;- - exp(-x)
  pof
}


plot_commits &lt;- function(package_name, commits){
  message(package_name)
  example &lt;- dplyr::filter(commits, package == package_name, !is_merge)

  p &lt;- ggplot(example)   +
    geom_segment(aes(x = datetime, xend = datetime,
                     y = 0, yend = custom_transf(total_insertions)),
                 col = &quot;salmon&quot;, size = 1) +
    geom_segment(aes(x = datetime, xend = datetime,
                     y = 0, yend = - custom_transf(total_deletions)), 
                 col = &quot;blue&quot;, size = 1) +
    ylab(&quot;lines&quot;) +
    annotate(&quot;segment&quot;, 
             x = min(example$datetime),
             xend = max(example$datetime),
             y = 0, yend = 0,
             col = &quot;grey50&quot;,
             size = 0.2) +
    scale_y_continuous(breaks= seq(min(-custom_transf(example$total_deletions)),
                                   to = max(custom_transf(example$total_insertions)),
                                   by = 5), 
                       labels= round(custom_exp(seq(min(-custom_transf(example$total_deletions)),
                                         to = max(custom_transf(example$total_insertions)),
                                         by = 5))))+
    annotate(&quot;rect&quot;,
             xmin = example$opened[1], xmax = example$closed[1],
             ymin = - custom_transf(max(example$total_deletions)), 
             ymax = custom_transf(max(example$total_insertions)),
             fill = &quot;grey80&quot;, alpha = 0.6,
             col = &quot;grey10&quot;,
             size = 0.2) +
    ggtitle(package_name) +
    hrbrthemes::theme_ipsum() 
  print(p)
}
</code></pre>
<p>Let’s have a look at a few packages. The x-axis represents time, and the<br>
grey box the onboarding period (from issue opening i.e. submission to<br>
issue closing i.e. approval). The salmon lines above the y-axis are the<br>
number of lines <em>added</em> in one commit while the blue lines below the<br>
y-axis are the number of lines <em>deleted</em> in one commit.</p>
<pre><code>plot_commits(&quot;opencage&quot;, commits = commits)
</code></pre>
<p><img alt="commits plot of the opencage
package" src="https://i2.wp.com/ropensci.org/img/blog-images/2018-05-03-onboarding-is-work/unnamed-chunk-2-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<pre><code>plot_commits(&quot;charlatan&quot;, commits = commits)
</code></pre>
<p><img alt="commits plot of the charlatan
package" src="https://i1.wp.com/ropensci.org/img/blog-images/2018-05-03-onboarding-is-work/unnamed-chunk-3-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>When looking at all onboarded repos, we don’t see a general pattern to<br>
commit histories. Commits have diffent sizes, and the activity frequency<br>
is highly variable. While we see the updates to packages that occur<br>
during review, there is just as often change again after onboarding, as<br>
packages continue to improve and be maintained. This highlights an<br>
important aspect of our peer review, which is that acceptance is not the<br>
end of the process. While we aim for the process to ensure high quality<br>
at acceptance, packages are not static moving forward. Not all<br>
deficiencies are detected by peer reviewers, and further improvements<br>
and maintenance will continue. This is why many of our standards are<br>
about maintainability, and our process is in part an approach to onboard<br>
authors into a community of dedicated and supportive package<br>
maintainers.</p>
<p>We do try to onboard mature packages, i.e. that are not <em>drafts</em>.<br>
Furthermore, the absence of general patterns in the previous figures<br>
could be due to different age at submission. How old are packages at<br>
submission?</p>
<pre><code>age &lt;- commits %>%
  dplyr::group_by(package) %>%
  dplyr::summarise(age = difftime(min(opened), min(datetime), units = &quot;weeks&quot;)) %>%
  dplyr::filter(age > 0)

library(hrbrthemes)
ggplot(age) +
  geom_dotplot(aes(age), binwidth = 5) +
  xlab(&quot;Age (weeks)&quot;) +
  scale_y_continuous(NULL, breaks = NULL) +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16)
</code></pre>
<p><img alt="age of packages at
submission" src="https://i1.wp.com/ropensci.org/img/blog-images/2018-05-03-onboarding-is-work/unnamed-chunk-4-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>On this dotplot, the x-axis represents age at submission and each dot is<br>
a package falling in that age bin. We say <em>apparent</em> age because the<br>
first commit might happen a long time after the package was created,<br>
although often one makes an initial commit not long after having started<br>
to work. We needed to filter positive age because in one case the GitHub<br>
repo of the onboarded package was apparently deleted and re-created<br>
without history after approval. Such things happen, thankfully the<br>
package wasn’t lost, just its history!</p>
<p>Many packages are submitted while still very young which might indicate<br>
rapid development. It might also mean some authors viewed onboarding as<br>
a part of development, i.e. authors knew they intended to submit quite<br>
rapidly. For instance, Maëlle developed <code>opencage</code> in just a few days,<br>
then submitted it to get it ready for wider use by the community.</p>
<h3>Work done by reviewers</h3>
<p>How much effort is put in by reviewers in this process? Reviewer time<br>
and effort is one of our most precious resources (you can read more<br>
about reviewer motivations and perspectives in other blog posts by <a rel="nofollow" href="https://ropensci.org/blog/2017/08/22/first-package-review/" target="_blank">Mara<br>
Averick</a>,<br>
<a rel="nofollow" href="https://ropensci.org/blog/2017/09/08/first-review-experiences/" target="_blank">Verena<br>
Haunschmid</a>,<br>
<a rel="nofollow" href="https://ropensci.org/blog/2018/03/13/ode-to-testing/" target="_blank">Charles T. Gray</a><br>
and <a rel="nofollow" href="https://ropensci.org/blog/2018/04/06/peer-review-value/" target="_blank">Miles<br>
McBain</a>). We<br>
measure this effort by asking reviewers to self-report the hours spent<br>
reviewing. We do so to understand the amount of effort we are asking<br>
reviewers for, so we can let new reviewers know what to expect, and so<br>
hopefully in the future we can measure the success of efforts to<br>
automate some reviewer tasks.</p>
<pre><code># airtable data
# this is our private database of who's reviewed what
# and of reviewers' areas of expertise
airtable &lt;- airtabler::airtable(&quot;appZIB8hgtvjoV99D&quot;, &quot;Reviews&quot;)
airtable &lt;- airtable$Reviews$select_all()
</code></pre>
<p>We have self-reported reviewing times for 136 of 184 reviews.</p>
<pre><code>ggplot(airtable) +
  geom_boxplot(aes(y = review_hours, x = &quot;&quot;)) +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16) +
  theme(axis.title.x=element_blank()) +
  ylab(&quot;Reviewing time (hours)&quot;)
</code></pre>
<p><img alt="hours spent
reviewing" src="https://i1.wp.com/ropensci.org/img/blog-images/2018-05-03-onboarding-is-work/unnamed-chunk-6-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>It appears to take our reviewers a similar amount of time to review R<br>
packages as <a rel="nofollow" href="http://publishingresearchconsortium.com/index.php/112-prc-projects/research-reports/peer-review-in-scholarly-journals-research-report/142-peer-review-in-scholarly-journals-perspective-of-the-scholarly-community-an-international-study" target="_blank">scientists take to review a manuscript (5 hour median and 9<br>
hour<br>
mean)</a>.</p>
<p>One potential question is whether reviewer time is affected by the size<br>
of the package reviewed as measured by for instance number of exports<br>
(classes and functions).</p>
<pre><code>get_namespace &lt;- function(package_name){
  message(package_name)
  local_path_archive &lt;- paste0(getwd(), &quot;/repos_at_submission/&quot;, package_name)
  if(length(fs::dir_ls(local_path_archive)) != 0){
    ns &lt;- devtools::parse_ns_file(local_path_archive)
    tibble::tibble(package = package_name, 
                   exports = length(ns$exports) +
                     length(ns$exportClasses))
  }else{
    return(NULL)
  }
}

packages &lt;- fs::dir_ls(&quot;repos_at_submission&quot;) 
packages &lt;- stringr::str_replace_all(packages, &quot;repos_at_submission\\/&quot;, &quot;&quot;)
purrr::map_df(packages, get_namespace) %>%
  readr::write_csv(&quot;output/namespace.csv&quot;)
</code></pre>
<p>Interestingly, we find no relationship between the reviewing time and<br>
number of exports:</p>
<pre><code>namespace_ro &lt;- readr::read_csv(&quot;output/namespace.csv&quot;)
namespace_ro &lt;- dplyr::left_join(namespace_ro, airtable, by = &quot;package&quot;)
ggplot(namespace_ro) +
  geom_point(aes(exports, review_hours)) +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16) +
  ylab(&quot;Reviewing time (hours)&quot;) +
  xlab(&quot;No. of exports&quot;)
</code></pre>
<p><img alt="" src="https://i2.wp.com/ropensci.org/img/blog-images/2018-05-03-onboarding-is-work/scatterplot-size-vs-reviewing-time-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>There are a few potential explanations to this that might be fruitful to<br>
explore. For instance, does this mean that there’s only so much time,<br>
and so larger packages get less scrutiny per line of code or per export?<br>
Or does review time just depend more on the reviewer than the package?</p>
<h3>Work done by editors</h3>
<p>Editors manage the review process, performing initial package checks,<br>
identifying and contacting reviewers, and then moderating and cajoling<br>
the process forward. Our best measure for editor effort is the number of<br>
packages handled by an editor in a given time frame, which we can track<br>
from the assignments on GitHub issues since the start of onboarding.<br>
Looking at this over time, we can see how editor workloads changed in<br>
response to growing number of assignments and how we have attempted to<br>
manage this by expanding our editorial board:</p>
<pre><code>library(tidyverse)
library(gh)
library(lubridate)

issues &lt;- gh(&quot;/repos/ropensci/onboarding/issues?state=all&amp;labels=package&quot;, .limit=1000)

edits = map_df(issues,
~data_frame(url = .$html_url,
editor = .$assignee$login %||% NA_character_,
opened = as.Date(.$created_at))) %>%
filter(!is.na(editor)) %>%
mutate(quarter = paste(year(opened), quarter(opened), sep=&quot;Q&quot;),
half =  paste(year(opened), if_else(quarter(opened) &lt;= 2, 1, 2), sep=&quot;H&quot;),
year = year(opened))

edits %>%
group_by(editor, half) %>%
summarize(n_assigned = n()) %>%
{ full_join(., crossing(editor = unique(.$editor),  #can't get expand() to work.
half = unique(.$half))) } %>%
mutate(n_assigned = coalesce(n_assigned, 0L)) %>%
ggplot(aes(x=half, y=n_assigned, fill=editor)) +
geom_col(position=&quot;dodge&quot;) +
geom_hline(yintercept = c(3, 6)) +
xlab(&quot;Half / Year&quot;) + ylab(&quot;No. Issues Handled&quot;)+
    hrbrthemes::theme_ipsum() +
  theme(legend.position = &quot;bottom&quot;) +
  viridis::scale_fill_viridis(discrete = TRUE)
</code></pre>
<p><img alt="number of assignments per editor per half a
year" src="https://i1.wp.com/ropensci.org/img/blog-images/2018-05-03-onboarding-is-work/unnamed-chunk-7-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>As an side-note, the same data allows to visualize the increase in the<br>
number of submissions.</p>
<pre><code>edits %>%
group_by(half) %>%
summarize(n_submissions = n()) %>%
ggplot(aes(x=half, y=n_submissions)) +
geom_col(position=&quot;dodge&quot;) +
xlab(&quot;Half / Year&quot;) + ylab(&quot;No. Submissions&quot;)+
    hrbrthemes::theme_ipsum() 
</code></pre>
<p><img alt="number of submissions per half a
year" src="https://i1.wp.com/ropensci.org/img/blog-images/2018-05-03-onboarding-is-work/unnamed-chunk-8-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h3>Outlook: decreasing work by automation</h3>
<p>As mentioned <a rel="nofollow" href="https://ropensci.org/blog/2018/04/17/author-survey/" target="_blank">in this blog post about the authors and reviewers<br>
survey</a>, we’re in<br>
the process of trying to maximize automation of all than can be<br>
automated in order to reduce and simplify work for everyone involved.<br>
This way, humans can focus on what they’re best at. Our current<br>
automation efforts include two packages in development: one for package<br>
authors, <a rel="nofollow" href="https://github.com/ropenscilabs/rodev" target="_blank"><code>rodev</code></a>, by rOpenSci<br>
staff, and one for package reviewers,<br>
<a rel="nofollow" href="https://github.com/ropenscilabs/pkgreviewr" target="_blank"><code>pkgreviewr</code></a>, by community<br>
member (and guest editor!) <a rel="nofollow" href="http://www.annakrystalli.com/" target="_blank">Anna<br>
Krystalli</a>. We also are working on<br>
automating most of the checks that use <code>goodpractice::gp</code> so they are<br>
run on CI infrastructure automatically on submission, rather than<br>
requiring editors to do so locally.</p>
<p>If you liked this data exploration, stay tuned for the third and final<br>
post of this series, about the social weather of onboarding as<br>
characterized by a tidy text analysis of onboarding threads! And don’t<br>
forget to check out <a rel="nofollow" href="https://ropensci.org/blog/2018/04/26/rectangling-onboarding/" target="_blank">the first post of the series, about data<br>
collection</a>,<br>
if you haven’t read it yet.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://ropensci.org/blog/2018/05/03/onboarding-is-work/"> rOpenSci - open tools for open science</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/gaspyIXsnF8" height="1"></a></div>