<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/the-2018-author-and-reviewer-survey/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/04/17/author-survey/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>rOpenSci’s package review system (aka<br>
<a rel="nofollow" href="https://github.com/ropensci/onboarding/" target="_blank">onboarding</a>) is one of our key<br>
activities to <a rel="nofollow" href="https://ropensci.org/blog/2017/09/01/nf-softwarereview/" target="_blank">improve quality and sustainability of scientific R<br>
packages</a>. The<br>
editorial team are constantly working towards improving the experience<br>
for both authors and reviewers. After our first year, we <a rel="nofollow" href="https://ropensci.org/blog/2016/03/28/software-review/" target="_blank">surveyed<br>
authors and<br>
reviewers</a> who<br>
participated in our onboarding process to help us better understand<br>
what’s working well and where there is room for improvement. At the end<br>
of last year, we did so again, re-designing our survey so as to better<br>
track participant opinions year-to-year. In this post we summarize the<br>
45 responses that we received and what we’re doing to address your<br>
feedback.</p>
<h3>Section 1: Satisfaction with the review and onboarding process</h3>
<p><img alt="" src="https://i1.wp.com/i.imgur.com/x4ICORq.png?w=456&ssl=1" data-recalc-dims="1"><img alt="" src="https://i0.wp.com/i.imgur.com/pUF0kS0.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>We’re glad to see high levels of overall satisfaction with the<br>
onboarding process. We primarily ask this question to establish a<br>
baseline to compare year-to-year, and hope that we can maintain high<br>
satisfaction as we continue to grow.</p>
<h3>Section 2: Guidelines for authors and reviewers</h3>
<p><img alt="" src="https://i2.wp.com/i.imgur.com/Vx0HtQ9.png?w=456&ssl=1" data-recalc-dims="1"><img alt="" src="https://i0.wp.com/i.imgur.com/N6rYblm.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>We provide<br>
<a rel="nofollow" href="https://github.com/ropensci/onboarding/blob/master/packaging_guide.md" target="_blank">packaging</a><br>
and<br>
<a rel="nofollow" href="https://github.com/ropensci/onboarding/blob/master/reviewing_guide.md" target="_blank">reviewing</a><br>
guides for our authors and reviewers to consult before and during the<br>
peer-review process. While both were highly rated, enthusiasm was<br>
greater for the packaging guide than the reviewing guide.</p>
<p>For the packaging guide, authors gave feedback that they could be better<br>
organized, including a suggestion that the collection of markdown files<br>
be converted to a bookdown website for easier navigation. We thought<br>
this was a great suggestion, and have begun the process of porting all<br>
our documentation for packaging and reviewing, into a common bookdown<br>
book. Editor Maëlle Salmon is leading this work, and you can find the<br>
in-progress book, <em>rOpenSci Packages: Development, Maintenance, and Peer<br>
Review</em>, <a rel="nofollow" href="https://github.com/ropenscilabs/dev_guide/" target="_blank">here</a>.</p>
<p>We also got feedback from authors that they wanted guidance on how best<br>
to acknowledge the efforts of reviewers. This was one of our drivers to<br>
get reviewers recognized as MARC author type in package DESCRIPTION<br>
files, which you can read more about in <a rel="nofollow" href="https://ropensci.org/blog/2018/03/16/thanking-reviewers-in-metadata/" target="_blank">in this previous blog<br>
post</a>.</p>
<p>For the reviewing guide, reviewers suggested greater structure,<br>
including separation between code and UI reviewing, more advice and<br>
examples on how to approach review, better guidance on expectations for<br>
timelines and collaboration between multiple reviewers. We aim to fold<br>
all these suggestions into the book. There are <a rel="nofollow" href="https://ropensci.org/tags/reviewer/" target="_blank">great posts about how to<br>
tackle a peer review on our blog</a><br>
that we will incorporate.</p>
<h3>Section 3: The review process</h3>
<p><img alt="" src="https://i2.wp.com/i.imgur.com/zNSsW2V.png?w=456&ssl=1" data-recalc-dims="1"><img alt="" src="https://i2.wp.com/i.imgur.com/noxbYih.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This was perhaps the most controversial topic in the responses. Both<br>
authors and reviewers were conflicted about the best time to submit for<br>
review. Submitting early in development could allow reviewers to provide<br>
concrete feedback on software design and usability, but it is generally<br>
hard to review incomplete software. On the other extreme, software that<br>
is mature and already on CRAN may not be able to implement reviewer<br>
suggestions, especially if it would result in breaking changes for<br>
users. One suggestion was to consider packages in both stages and<br>
provide short (less than one hour of review) for early stage packages<br>
and reserve the more detailed reviews for the mature packages.</p>
<p>This has been a perennial tension in the review process. A similar split<br>
came up in our previous survey, and Miles McBain recently wrote about it<br>
in his <a rel="nofollow" href="https://milesmcbain.xyz/ropensci-onboarding2/" target="_blank">blog post about reviewing<br>
<strong>bowerbird</strong></a>. As we<br>
don’t see a strong net pull in either direction, we don’t plan to change<br>
submission requirements, and we’re worried that a multi-step process<br>
would make too much work for reviewers. One useful suggestion, though,<br>
is for authors of early-development packages to open discussions on our<br>
<a rel="nofollow" href="http://discuss.ropensci.org/" target="_blank">forum</a> to solicit early input.</p>
<p>Some respondents noted the challenge of providing an objective review<br>
under a fully open system, and one suggestion was to consider a single<br>
blinded system in order to allow reviewers (especially more early career<br>
folks) to provide critical comments without fear of retaliation.</p>
<h4>Automated checking</h4>
<p><img alt="" src="https://i2.wp.com/i.imgur.com/boRl02k.png?w=456&ssl=1" data-recalc-dims="1"><img alt="" src="https://i1.wp.com/i.imgur.com/NGtsJpX.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>We received specific feedback about our use of the <strong>goodpractice</strong><br>
package to provide automated checks of some package quality issues.<br>
Respondents suggested that authors should pre-run these checks before<br>
submitting, and also to provide better guidance as to which outputs can<br>
be ignored. We’re adding<br>
<a rel="nofollow" href="https://github.com/ropenscilabs/dev_guide/issues/12" target="_blank">both</a><br>
<a rel="nofollow" href="https://github.com/ropenscilabs/dev_guide/issues/11" target="_blank">suggestions</a> to<br>
the new bookdown book.</p>
<p>There were also suggestions for improvement of <strong>goodpractice</strong> itself.<br>
<strong>goodpractice</strong> was created by <a rel="nofollow" href="http://gaborcsardi.org/" target="_blank">Gábor Csárdi</a><br>
and is maintained by <a rel="nofollow" href="https://www.mango-solutions.com/" target="_blank">Mango<br>
Solutions</a>. Thankfully Mango data<br>
scientist <a rel="nofollow" href="https://rladies.org/united-kingdom-rladies/name/hannah-frick/" target="_blank">Hannah<br>
Frick</a><br>
has undertaken a project to <a rel="nofollow" href="https://github.com/MangoTheCat/goodpractice/commits/master" target="_blank">improve the package and its<br>
documentation</a><br>
in recent weeks, and is incorporating suggestions we’ve passed on from<br>
the rOpenSci survey.</p>
<p>As <strong>goodpractice</strong> allows users to ignore some checks and add custom<br>
checks, we also aim this year to have a specialized set of rOpenSci<br>
checks that will align better align with our packaging guide. Finally,<br>
we hope to have these checks run automatically upon submission, reducing<br>
effort for both authors and editors.</p>
<h3>Section 4: Value of software review</h3>
<p><img alt="" src="https://i1.wp.com/i.imgur.com/TFSA7RZ.png?w=456&ssl=1" data-recalc-dims="1"><img alt="" src="https://i1.wp.com/i.imgur.com/GME7Kny.png?w=456&ssl=1" data-recalc-dims="1"><img alt="" src="https://i0.wp.com/i.imgur.com/WPNhiLS.png?w=456&ssl=1" data-recalc-dims="1"><img alt="" src="https://i0.wp.com/i.imgur.com/plB1hoL.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Package authors are given the option to submit directly to the Journal<br>
of Open Source Software (JOSS) for publication.</p>
<p><img alt="" src="https://i1.wp.com/i.imgur.com/gE6PNRf.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h4>The good and the bad with onboarding</h4>
<p>We asked authors to tell us some of the best and worst attributes of<br>
participating in onboarding.</p>
<p><em>Aspects of onboarding that were challenging?</em></p>
<ul>
<li>Deciding when to submit (see above)</li>
<li>Package review is quite time consuming.</li>
<li>Keeping up with ongoing revisions to the package during submission.</li>
<li>Long time lag between review and revision.</li>
</ul>
<p><em>Aspects of onboarding that were great</em></p>
<ul>
<li>The opportunity to network and share ideas.</li>
<li>The openness and transparency of the entire review process.</li>
<li>Learning about software design, package development, and improving<br>
one’s own coding skills.</li>
<li>The opportunity to have two thorough code reviews which are quite<br>
difficult to get elsewhere.</li>
<li>The collegial and non-adversarial nature of the process.</li>
</ul>
<p><img alt="" src="https://i1.wp.com/i.imgur.com/YtgnBO0.png?w=456&ssl=1" data-recalc-dims="1"><img alt="" src="https://i1.wp.com/i.imgur.com/2cBNR2z.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h4>Did blogging post-acceptance have any impact?</h4>
<p>At the completion of each review, the editors invite package authors to<br>
submit either a long-form blog post aimed at a general reader, or a<br>
shorter technical note. Most authors who wrote posts reported greater<br>
visibility for their package and some also enjoyed reflecting on the<br>
development process.</p>
<h4>Upcoming improvements to onboarding</h4>
<p>We’re grateful to all the authors and reviewers that took this survey.<br>
The responses have helped us improve our workflow quite a bit and we are<br>
excited to roll out several improvements in the coming weeks to months.<br>
Here are a few things to look forward to:</p>
<ul>
<li>A <a rel="nofollow" href="https://github.com/ropenscilabs/dev_guide" target="_blank">much improved (bookdown)<br>
guide</a> for both package<br>
development and package reviewing.</li>
<li>Improvements to <code>goodpractice</code> thanks to Mango The Cat’s <a rel="nofollow" href="https://rladies.org/united-kingdom-rladies/name/hannah-frick/" target="_blank">Hannah<br>
Frick</a><br>
and also a <code>goodpractice</code> API down the line.</li>
<li>A new and improved reviewer sign up form that will make it much<br>
easier for editors to match reviewers to submissions.</li>
<li>Suggestions are always welcome even if you missed this round of the<br>
survey. Chime in in the <a rel="nofollow" href="https://github.com/ropenscilabs/dev_guide/issues" target="_blank">issue tracker of our in-progress<br>
guide</a> or in <a rel="nofollow" href="https://github.com/ropensci/onboarding-meta/issues" target="_blank">this<br>
other issue<br>
tracker</a> where<br>
we keep track of more general questions &amp; suggestions about<br>
onboarding.</li>
</ul>
<p>We hope to see your package or review expertise on the onboarding repo<br>
soon!</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://ropensci.org/blog/2018/04/17/author-survey/"> rOpenSci - open tools for open science</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/Hr5RINcmfOA" height="1"></a></div>