Many R package authors (including myself) lump a collection of small, useful functions into some type of utils.R file and usually do not export the functions since they are (generally) designed to work on package internals rather than expose their functionality via the exported package API. Just like Batman’s utility belt, which can be customized for any mission, any set of utilities in a given R package will also likely be different from those in other packages.
I thought it would be neat to take a look at:
- just how many packages have one or more
util*.Rfiles and what the most common file names are for them; - utility function naming preferences — specifically snake-case, camel-case or dot-case
- what the most common “utility” functions names are across the packages
- coding style — specifically compare ratios of white space, full-line comments to code size
for all the published packages on CRAN.
There are many more questions one can ask and then use this corpus to answer, so we’ll close out the post with a link to it so any intrepid readers can do just that, especially since reproducing the first bit of this post would require a local CRAN mirror (which most folks — rightly so — do not have handy).
Acquiring and Transforming the Data We Need
Since I have local CRAN mirror, it’s just a matter of iterating through all the package tar.gz files in src/contrib and grepping through a tar listing of each for a pattern like "R/util.*$. That pattern isn’t perfect but it’s quick and we’ll be able to filter out any files it catches that don’t belong. I chose to use a small bash script for this but it’s possible to do this with R as well (an exercise left to the reader). The resultant data file looks a bit like the output from an ls -l (linux-ish) directory listing:
-rw-r--r-- 0 hornik users 1658 Jun 5 2016 AHR/R/util.R
-rw-r--r-- 0 ligges users 12609 Dec 13 2016 ALA4R/R/utilities_internal.R
-rw-r--r-- 0 hornik users 0 Feb 24 2017 AWR.Kinesis/R/utils.R
-rw-r--r-- 0 ligges users 4127 Aug 30 2017 AlphaVantageClient/R/utils.R
-rw-r--r-- 0 ligges users 121 Jan 19 2017 AmyloGram/R/utils.R
-rw-r--r-- 0 ligges users 2873 Jan 17 23:04 DT/R/utils.R
-rw-r--r-- 0 ligges users 3055 Jan 17 2017 cleanr/inst/source/R/utils.R
drwxr-xr-x 0 ligges users 0 Sep 24 2017 JGR/java/org/rosuda/JGR/util/
I made sure to show a few examples of where a better search pattern would have helped ensure lines like the three at the bottom of that listings aren’t included. But, we all often have to deal with imperfect data, so we’ll make sure to deal with that during the ingestion & cleanup process.
library(stringi)
library(hrbrthemes)
library(archive) # devtools::install_github("jimhester", "archive")
library(tidyverse)
# I ran readr::type_convert() once and it returns this column type spec. By using it
# for subsequent conversions, we'll gain reproducibility and data format change
# detection capabilities "for free"
cols(
permsissions = col_character(),
links = col_integer(),
owner = col_character(),
group = col_character(),
size = col_integer(),
month = col_character(),
day = col_integer(),
year_hr = col_character(),
path = col_character()
) -> tar_cols
# Now, we parse the tar verbose ('ls -l') listing
stri_read_lines("~/Data/pkutils.txt") %>% # stringi was loaded so might as well use it
stri_split_regex(" +", 9, simplify = TRUE) %>% # split input into 9 columns
as_data_frame() %>% # ^^ returns a matrix but data frames are more useful for our work
set_names(names(tar_cols$cols)) %>% # column names are useful and we can use our colspec for it
type_convert(col_types = tar_cols) %>% # see comment block before cols()
mutate(day = sprintf("%02d", day)) %>% # now we'll work on getting the date pieces to be a Date
mutate(year_hr = case_when( # the year_hr field can be either %Y or %H:%M depending on file 'recency'
stri_detect_fixed(year_hr, ":") &
(month %in% c("Jan", "Feb", "Mar", "Apr")) ~ "2018", # if %H:%M but 'starter' months it's 2018
stri_detect_fixed(year_hr, ":") &
(month %in% c("Dec", "Nov", "Oct", "Sep", "Aug", "Jul", "Jun")) ~ "2017", # %H:%M & 'end' months
TRUE ~ year_hr # already in %Y format
)) %>%
mutate(date= lubridate::mdy(sprintf("%s %s, %s", month, day, year_hr))) %>% # get a Date
mutate(pkg = stri_match_first_regex(path, "^(.*)/R/")[,2]) %>% # extract package name (stri_extract is also usable here)
mutate(fil = basename(path)) %>% # extrafct just the file name
filter(!is.na(pkg)) %>% # handle one type of wrongly included file
filter(!stri_detect_fixed(pkg, "/")) %>% # ande another
filter(!is.na(path)) -> xdf # and another; but we're done so we close with an assignment
glimpse(xdf)
## Observations: 1,746
## Variables: 12
## $ permsissions <chr> "-rw-r--r--", "-rw-r--r--", "-rw-r--r--", "-rw-r-...
## $ links <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
## $ owner <chr> "hornik", "ligges", "hornik", "ligges", "ligges",...
## $ group <chr> "users", "users", "users", "users", "users", "her...
## $ size <int> 1658, 12609, 0, 4127, 121, 52, 36977, 34198, 3676...
## $ month <chr> "Jun", "Dec", "Feb", "Aug", "Jan", "Aug", "Jan", ...
## $ day <chr> "05", "13", "24", "30", "19", "10", "06", "10", "...
## $ year_hr <chr> "2016", "2016", "2017", "2017", "2017", "2017", "...
## $ path <chr> "AHR/R/util.R", "ALA4R/R/utilities_internal.R", "...
## $ date <date> 2016-06-05, 2016-12-13, 2017-02-24, 2017-08-30, ...
## $ pkg <chr> "AHR", "ALA4R", "AWR.Kinesis", "AlphaVantageClien...
## $ fil <chr> "util.R", "utilities_internal.R", "utils.R", "uti...To the analysis!
Finding the Utility of ‘util’s
A careful look at the glimpse() listing shows we have 1,745 files that begin with util, but how many packages have at least one util files?
nrow(distinct(xdf, pkg))
## [1] 1397That’s roughly 10% of CRAN, but doesn’t mean other packages do not have “utility belt” functions since other authors may have just been more creative or deliberate with their file naming conventions.
Readers with keen eyes may have noticed we spent some deliberate CPU cycles to get a Date column. Part of that was to show how to do that (mostly as an example for folks new to R) but we also did it to ask temporal questions, such as “Are package ‘utility belts’ a “new” thing?”. The data suggests that utility belts are products/attributes of more recently published or updated packages:
distinct(xdf, pkg, date) %>%
mutate(yr = as.integer(lubridate::year(date))) %>%
count(yr) %>%
complete(yr, fill=list(n=0)) %>%
ggplot(aes(yr, n)) +
geom_col(fill="lightslategray", width=0.65) +
labs(
x = NULL, y = "Package count",
title = "Recently published or updated packages tend to have more 'util'\nthan older/less actively-maintained ones",
subtitle = "Count of packages (by year) with 'util's"
) +
theme_ipsum_rc(grid="Y")
We could answer this more completely by going through the CRAN archives for all these packages, but for now we’ll just see which packages might have helped set this trend going:
distinct(xdf, pkg, date) %>%
arrange(date) %>%
print(n=20)
## # A tibble: 1,540 x 2
## date pkg
## 1 1980-01-01 bsts
## 2 2006-06-28 evdbayes
## 3 2006-11-29 hexView
## 4 2006-12-17 StatDataML
## 5 2007-10-05 tpr
## 6 2007-11-07 seqinr
## 7 2007-11-26 registry
## 8 2008-07-25 ramps
## 9 2008-10-23 RobAStBase
## 10 2009-02-23 vcd
## 11 2009-06-26 ttutils
## 12 2009-07-03 histogram
## 13 2009-11-27 polynom
## 14 2009-11-27 tau
## 15 2010-01-05 itertools
## 16 2010-01-22 tableplot
## 17 2010-06-09 rbugs
## 18 2011-03-17 playwith
## 19 2011-05-11 marelac
## 20 2011-10-11 timeSeries
## # ... with 1,520 more rowsGoing back to our corpus, what are the most common names for these utility belt files?
## count(xdf, fil, sort=TRUE) %>%
## mutate(pct = scales::percent(n/sum(n))) %>%
## print(n=20)
## # A tibble: 409 x 3
## fil n pct
## 1 utils.R 865 49.5%
## 2 utilities.R 145 8.3%
## 3 util.R 134 7.7%
## 4 utils.r 68 3.9%
## 5 utility.R 47 2.7%
## 6 utility_functions.R 25 1.4%
## 7 util.r 16 0.9%
## 8 utilities.r 14 0.8%
## 9 utils-pipe.R 9 0.5%
## 10 utilityFunctions.R 6 0.3%
## 11 utils-format.r 3 0.2%
## 12 util_functions.R 2 0.1%
## 13 util_rescale.R 2 0.1%
## 14 util-aux.R 2 0.1%
## 15 util-checkparam.R 2 0.1%
## 16 util-startarg.R 2 0.1%
## 17 utilcmst.R 2 0.1%
## 18 utilhot.R 2 0.1%
## 19 utilities_internal.R 2 0.1%
## 20 utility-functions.R 2 0.1%
## # ... with 389 more rowsOver 50% of other CRAN packages are as “un-creative” as I am when it comes to naming these files.
Let’s see how packed these belts are:
ggplot(xdf, aes(x="", size)) +
ggbeeswarm::geom_quasirandom(
fill="lightslategray", color="white",
alpha=1/2, stroke=0.25, size=3, shape=21
) +
geom_boxplot(fill="#00000000", outlier.colour = "#00000000") +
geom_text(
data=data_frame(), aes(x=-Inf, y=median(xdf$size), label="Median:\n2,717"),
hjust = 0, family = font_rc, size = 3, color = "lightslateblue"
) +
scale_y_comma(
name = "File size", trans="log10", limits=c(NA, 200000),
breaks = c(10, 100, 1000, 10000, 100000)
) +
labs(
x = NULL,
title = "Most 'util' files are between 1K and 10K in size",
caption = "Note y-axis log10 scale"
) +
theme_ipsum_rc(grid="Y")
We’ll need to do a bit more data collection to answer the last two questions.
Focus on Functions
To examine function names and source code statistics, we’ll need to read in the contents of each file and parse them. Let’s do that first bit with some help from the archive package which will help us open up these compressed tar files and pull out the file(s) we need from them vs have to code this up more manually.
Again, this code is only reproducible if you have CRAN handy, but soon (promise!) you’ll have a file you can work with for the remainder of the post:
extract_source <- function(pkg, fil, .pb = NULL) {
if (!is.null(.pb)) .pb$tick()$print()
list.files(
path = "/cran/src/contrib", # my path to local CRAN
pattern = sprintf("^%s_.*gz", pkg), # rough pattern for the package archive filename
recursive = FALSE,
full.names = TRUE
) -> tgt
con <- archive_read(tgt[1], fil)
src <- readLines(con, warn = FALSE)
close(con)
paste0(src, collapse="\n")
}
pb <- progress_estimated(nrow(xdf))
xdf <- mutate(xdf, file_src = map2_chr(pkg, path, extract_source, .pb=pb))That (on-drive) ~10MB data frame is in https://rud.is/dl/utility-belt.rds. The rest of the post builds off of it so you can start coding along at home now.
Let’s extract the function names:
# we'll use these two functions to help test whether bits
# of our parsed code are, indeed, functions.
#
# Alternately: "I heard you liked functions so I made
# functions to help you find functions"
#
# we could have used `rlang` helpers here, but I had these
# handy from pre-`rlang` days.
is_assign <- function(x) {
as.character(x) %in% c('<-', '=', '<<-', 'assign')
}
is_func <- function(x) {
is.call(x) &&
is_assign(x[[1]]) &&
is.call(x[[3]]) &&
(x[[3]][[1]] == quote(`function`))
}
read_rds("~/Data/utility-belt.rds") %>% # I have this file in ~/Data; change this for your location
mutate(parsed = map(file_src, ~parse(text = .x, keep.source = TRUE))) %>% # parse each file
mutate(func_names = map(parsed, ~{ # go through parsed file
keep(.x, is_func) %>% # and only keep functions
map(~as.character(.x[[2]])) %>% # extract the function name
flatten_chr() # return a character vector
})) -> xdfWith those handy, we can see if there are any commonalities across all these packages:
select(xdf, pkg, fil, func_names) %>%
unnest() %>%
count(func_names, sort=TRUE) %>%
print(n=20)
## func_names n
## 1 %||% 84
## 2 compact 19
## 3 isFALSE 19
## 4 assertthat::on_failure 16
## 5 is_windows 16
## 6 trim 14
## 7 .on Load 13 # (IRL there's no space here but the WP input sanitizer hates this word due to js abuse
## 8 names2 12
## 9 dots 11
## 10 is_string 11
## 11 vlapply 11
## 12 .onAttach 10
## 13 error.bars 10
## 14 normalize 10
## 15 vcapply 10
## 16 cat0 9
## 17 collapse 9
## 18 err 9
## 19 getmin 9
## 20 is_dir 9
## # ... with 1.252e+04 more rowsWe can also see if there are common case conventions:
select(xdf, pkg, fil, func_names) %>%
unnest() %>%
mutate(is_camel = (!stri_detect_fixed(func_names, "_")) &
(!stri_detect_regex(func_names, "[[:alpha:]]\\.[[:alpha:]]")) &
(stri_detect_regex(func_names, "[A-Z]"))) %>%
mutate(is_dotcase = stri_detect_regex(func_names, "[[:alpha:]]\\.[[:alpha:]]")) %>%
mutate(is_snake = stri_detect_fixed(func_names, "_") &
(!stri_detect_regex(func_names, "[[:alpha:]]\\.[[:alpha:]]"))) -> case_hunt
count(case_hunt, is_camel, is_dotcase, is_snake) %>%
mutate(pct = scales::percent(n/sum(n))) %>%
mutate(description = c(
"one-'word' names",
"snake_case",
"dot.case",
"camelCase"
)) %>%
arrange(n) %>%
mutate(description = factor(description, description)) %>%
ggplot(aes(description, n)) +
geom_col(fill="lightslategray", width=0.65) +
geom_label(aes(y = n, label=pct), label.size=0, family=font_rc, nudge_y=150) +
scale_y_comma("Number of functions") +
labs(
x=NULL,
title = "dot.case does not seem to be en-vogue for utility belt functions"
) +
theme_ipsum_rc(grid="Y")
I had a hunch that isX…()/is_x…() could be likely names for utility belt functions, so let’s normalize the function names to snake_case and see if that’s true:
select(xdf, pkg, fil, func_names) %>%
unnest() %>%
filter(stri_detect_regex(func_names, "^(\\.is|is)")) %>%
mutate(func_names = snakecase::to_snake_case(func_names)) %>%
count(func_names, sort=TRUE)
## # A tibble: 547 x 2
## func_names n
## 1 is_false 24
## 2 is_windows 19
## 3 is_string 18
## 4 is_empty 13
## 5 is_dir 11
## 6 is_formula 11
## 7 is_installed 11
## 8 is_linux 9
## 9 is_na 9
## 10 is_error 8
## # ... with 537 more rowsOnly 5% (819) out of 14,123 extracted function names are is_; not overwhelming, but a respectable slice.
There are more questions we could ask of function names and styles, but we’ll leave some work for y’all to do on your own.
Let’s head over to the final rooftop exercise.
Code, Comment & Blank Line Density
Since we have the raw source, we can also take a look at coding style. There are many questions we could ask here and more than a few packages we could draw on to help answer them. For now, we’ll just take a look at the mean ratios of comments and blank lines to code across the packages in this utility belt corpus and give you the opportunity to tease out other interesting tidbits such as “what base R and other package functions are most often used in utility belt functions?” or “are package authors using evil = for assignment or proper <-?”.
xdf %>%
mutate(
num_lines = stri_count_fixed(xdf$file_src, "\n"),
num_blank_lines = stri_count_regex(xdf$file_src, "^[[:space:]]*$", opts_regex = stri_opts_regex(multiline=TRUE)),
num_whole_line_comments = lengths(cmnt_df$comments),
comment_density = num_whole_line_comments / (num_lines - num_blank_lines - num_whole_line_comments),
blank_density = num_blank_lines / (num_lines - num_whole_line_comments)
) %>%
select(-permsissions, -links, -owner, -group, month, -day, -year_hr) -> xdf
# now compute mean ratios
group_by(xdf, pkg) %>%
summarise(
`Comment-to-code Ratio` = mean(comment_density),
`Blank lines-to-code Ratio` = mean(blank_density)
) %>%
ungroup() %>%
filter(!is.infinite(`Comment-to-code Ratio`)) %>%
filter(!is.nan(`Comment-to-code Ratio`)) %>%
filter(!is.infinite(`Blank lines-to-code Ratio`)) %>%
filter(!is.nan(`Blank lines-to-code Ratio`)) %>%
gather(measure, value, -pkg) -> code_ratios
# we want to label the median values
group_by(code_ratios, measure) %>%
summarise(median = median(value)) -> code_ratio_meds
ggplot(code_ratios, aes(measure, value, group=measure)) +
ggbeeswarm::geom_quasirandom(
fill="lightslategray", color="#2b2b2b", alpha=1/2,
stroke=0.25, size=3, shape=21
) +
geom_boxplot(fill="#00000000", outlier.colour = "#00000000") +
geom_label(
data = code_ratio_meds,
aes(-Inf, c(0.3, 5), label=sprintf("Median:\n%s", round(median, 2)), group=measure),
family = font_rc, size=3, color="lightslateblue", hjust = 0, label.size=0
) +
scale_y_continuous() +
labs(
x = NULL, y = NULL,
caption = "Note free y scale"
) +
facet_wrap(~measure, scales="free") +
theme_ipsum_rc(grid="Y", strip_text_face = "bold") +
theme(axis.text.x=element_blank())
FIN
You can find the (on-drive) ~10MB data frame is at: https://rud.is/dl/utility-belt.rds.
All the above code in this gist: https://gist.github.com/hrbrmstr/33d29bb39eaa7f2f1e95308038f85b59.
If you do your own ‘utility belt’ analyses, drop a note in the comments with a link to your findings!
