Our onboarding process
ensures that packages contributed by the community undergo a
transparent, constructive, non adversarial and open review process.
Before even submitting my first R package to rOpenSci onboarding system
in December 2015, I spent a fair amount of time reading through previous
issue threads in order to assess whether onboarding was a friendly place
for me: a newbie, very motivated to learn more but a newbie nonetheless.
I soon got the feeling that yes, onboarding would help me make my
package better without ever making me feel inadequate.
More recently, I read Anne Gentle’s
essay
in Open Advice where she mentions the concept of the social weather of
an online community. By listening before I even posted, I was simply
trying to get a good idea of the social weather of onboarding – as a
side-note, how wonderful is it that onboarding’s being open makes it
possible to listen?!
In the meantime, I’ve now only submitted and reviewed a few packages but
also become an associate
editor. In
general, when one of us editors talks about onboarding, we like to use
examples illustrating the system in a good light: often excerpts from
review processes, or quotes of tweets by authors or reviewers. Would
there be a more quantitative way for us to support our promotion
efforts? In this blog post, I shall explore how a
tidytext analysis of review processes
(via their GitHub threads) might help us characterize the social weather
of onboarding.
Preparing the data
A note about text mining
In this blog post, I’ll leverage the tidytext package, with the help
of its accompanying book “Tidy text
mining”. The authors, Julia Silge and
David Robinson, actually met and started working on this project, at
rOpenSci unconf in 2016!
The “tidy text” of the analysis
I’ve described in the first post of this
series
how I got all onboarding issue threads. Now, I’ll explain how I cleaned
their text. Why does it need to be cleaned? Well, this text contains
elements that I wished to remove: code chunks, as well as parts from our
submission and review
templates.
My biggest worry was the removal of templates from issues. I was already
picturing my spending hours writing regular expressions to remove these
lines… and then I realized that the word “lines” was the key! I could go
split all issue comments into lines, which is called tokenization in
proper text mining vocabulary, and then remove duplicates! This way, I
didn’t even have to worry about the templates having changed a bit over
time, since each version was used at least twice. A tricky part that
remained was the removal of code chunks since I only wanted to keep
human conversation. In theory, it was easy: code chunks are lines
located between two lines containing “```”… I’m still not sure I
solved this in the easiest possible way.
library(&quot;magrittr&quot;)
threads %
widyr::pairwise_cor(word, line_id, sort = TRUE)
For instance, what often goes in the same line as vignette?
dplyr::filter(word_cors, item1 == &quot;vignette&quot;)
## # A tibble: 853 x 3
## item1 item2 correlation
## ## 1 vignette readme 0.176 ## 2 vignette vignettes 0.174 ## 3 vignette chunk 0.145 ## 4 vignette eval 0.120 ## 5 vignette examples 0.108 ## 6 vignette overview 0.0933
## 7 vignette building 0.0914
## 8 vignette link 0.0863
## 9 vignette maps 0.0840
## 10 vignette package 0.0831
## # ... with 843 more rows
Now let’s plot the network of these relationships between words, using
the igraph package by Gábor Csárdi and Támas
Nepusz and ggraph package by
Thomas Lin Pedersen.
library(&quot;igraph&quot;)
library(&quot;ggraph&quot;)
set.seed(2016)
word_cors %__%
dplyr::filter(correlation __ .35) %__%
graph_from_data_frame() %__%
ggraph(layout = &quot;fr&quot;) +
geom_edge_link(aes(edge_alpha = correlation), show.legend = FALSE) +
geom_node_point(color = &quot;lightblue&quot;, size = 5) +
geom_node_text(aes(label = name), repel = TRUE) +
theme_void()
This figure gives a good sample of things discussed in reviews. Despite
our efforts filtering words specific to issues, some of them remain very
specific, such as country/city/location that are very frequent in
ropenaq review.
How positive is onboarding?
Using sentiment analysis, we can look at how positive comments are.
sentiments %__%
dplyr::group_by(role) %__%
skimr::skim(sentiment)
## Skim summary statistics
## n obs: 11553 ## n variables: 6 ## group variables: role ## ## Variable type: numeric ## role variable missing complete n mean sd min p25
## author sentiment 0 4823 4823 0.07 0.21 -1.2 0
## community_manager sentiment 0 97 97 0.13 0.21 -0.41 0
## editor sentiment 0 1521 1521 0.13 0.22 -1.63 0
## other sentiment 0 344 344 0.073 0.2 -0.6 0
## reviewer sentiment 0 4768 4768 0.073 0.21 -1 0
## median p75 max hist
## 0 0.17 1.84 ## 0.071 0.23 1 ## 0.075 0.25 1.13 ## 0 0.2 0.81 ## 0 0.17 1.73 summary(sentiments$sentiment)
## Min. 1st Qu. Median Mean 3rd Qu. Max. ## -1.63200 0.00000 0.00000 0.07961 0.18353 1.84223
sentiments %__%
dplyr::filter(!role %in% c(&quot;other&quot;, &quot;community_manager&quot;)) %__%
ggplot(aes(role, sentiment)) +
geom_boxplot(fill = &quot;salmon&quot;) +
hrbrthemes::theme_ipsum(base_size = 16,
axis_title_size = 16,
strip_text_size = 16)
These boxplots seem to indicate that lines are generally positive
(positive mean, zero 25th-quantile), although it’d be better to be able
to compare them with text from traditional review processes of
scientific manuscripts in order to get a better feeling for the meaning
of these numbers.
On these boxplots we also see that we do get lines with a negative
sentiment value… about what? Here are the most common words in negative
lines.
sentiments %__%
dplyr::filter(sentiment __ 0) %__%
tidytext::unnest_tokens(word, line) %__%
dplyr::filter(!word %in% stopwords) %__%
dplyr::count(word, sort = TRUE) %__%
dplyr::mutate(word = reorder(word, n)) %__%
dplyr::filter(n __ 100) %__%
ggplot() +
geom_lollipop(aes(word, n),
size = 2, col = &quot;salmon&quot;) +
hrbrthemes::theme_ipsum(base_size = 16,
axis_title_size = 16) +
coord_flip()
And looking at a sample…
sentiments %__%
dplyr::arrange(sentiment) %__%
dplyr::select(line, sentiment) %__%
head(n = 15) %__%
knitr::kable()
line
sentiment
@ultinomics no more things, although do make sure to add more examples - perhaps open an issue ropenscilabs/gtfsr/issues to remind yourself to do that,
-1.6320000
not sure what you mean, but i’ll use different object names to avoid any confusion (ropenscilabs/mregions#24)
-1.2029767
error in .local(.object, …) :
-1.0000000
error:
-1.0000000
#### miscellaneous
-1.0000000
error: command failed (1)
-0.8660254
- get_plate_size_from_number_of_columns: maybe throwing an error makes more sense than returning a string indicating an error
-0.7855844
this code returns an error, which is good, but it would be useful to return a more clear error. filtering on a non-existant species results in a 0 “length” onekp object (ok), but then the download_* functions return a curl error due to a misspecified url.
-0.7437258
0 errors | 0 warnings | 0 notes
-0.7216878
once i get to use this package more, i’m sure i’ll have more comments/issues but for the moment i just want to get this review done so it isn’t a blocker.
-0.7212489
- i now realize i’ve pasted the spelling mistakes without thinking too much about us vs. uk english, sorry.
-0.7071068
minor issues:
-0.7071068
## minor issues
-0.7071068
replicates issue
-0.7071068
visualization issue
-0.7071068
It seems that negative lines are mostly people discussing bugs and
problems in code, and GitHub issues, and trying to solve them. The kind
of negative lines we’re happy to see in our process, since once solved,
they mean the software got more robust!
Last but not least, I mentioned our using particular cases as examples
of how happy everyone seems to be in the process. To find such examples,
we rely on memory, but what about picking heart-warming lines using
their sentiment score?
sentiments %__%
dplyr::arrange(- sentiment) %__%
dplyr::select(line, sentiment) %__%
head(n = 15) %__%
knitr::kable()
line
sentiment
absolutely - it’s really important to ensure it really has been solved!
1.842234
overall, really easy to use and really nicely done.
1.733333
this package is a great and lightweight addition to working with rdf and linked data in r. coming after my review of the codemetar package which introduced me to linked data, i found this a great learning experience into a topic i’ve become really interested in but am still quite novice in so i hope my feedback helps to appreciate that particular pov.
1.463226
i am very grateful for your approval and i very much look forward to collaborating with you and the ropensci community.
1.256935
thank you very much for the constructive thoughts.
1.237437
thanks for the approval, all in all a very helpful and educational process!
1.217567
- really good use of helper functions
1.139013
- i believe the utf note is handled correctly and this is just a snafu in goodpractice, but i will seek a reviewer with related expertise in ensuring that all unicode is handled properly.
1.132201
seem more unified and consistent.
1.126978
very much appreciated!
1.125833
- well organized, readable code
1.100000
- wow very extensive testing! well done, very thorough
1.100000
- i’m delighted that you find my work interesting and i’m very keen to help, contribute and collaborate in any capacity.
1.084493
thank you very much for your thorough and thoughtful review, @batpigandme ! this is great feedback, and i think that visdat will be much improved because of these reviews.
1.083653
great, thank you very much for accepting this package. i am very grateful about the reviews, which were very helpful to improve this package!
1.074281
As you can imagine, these sentences make the whole team very happy! And
we hope they’ll encourage you to contribute to rOpenSci onboarding.
Outlook
This first try at text analysis of onboarding issue threads is quite
promising: we were able to retrieve text and to use natural language
processing to extract most common words and bigrams, and sentiment. This
allowed us to describe the social weather of onboarding: we could see
that this system is about software, and that negative sentiment was
often due to bugs being discussed and solved; and we could extract the
most positive lines where volunteers praised the review system or the
piece of software under review.
One could expand this analysis with a study of emoji use, in the text
using an emoji dictionary as in this blog
post and around the
text (so-called emoji reactions, present in the API and used in e.g
ghrecipes::get_issues_thumbs).
Another aspect of social weather is maybe the timeliness that’s expected
or implemented at the different parts of the process, but it’d be
covered by other data such as the labelling history of the issues, which
could get extracted from GitHub V4 API as well.
This is the final post of the “Our package review system in review”
series. The first post presented data collection from
GitHub,
the second post aimed at quantifying the work represented by
onboarding.
The posts motivated us to keep using data to illustrate and assess the
system. Brace yourself for more onboarding data analyses in the future!