rud.is

macOS R users who tend to work on the bleeding edge likely noticed some downtime at <mac.r-project.org> this past weekend. Part of the issue was an SSL/TLS certificate expiration situation. Moving forward, we can monitor this with R using the super spiffy {openssl} and {pushoverr} packages whilst also generating a daily report with {rmarkdown} and {DT}.

The Basic Process

The {openssl} package has a handy function — download_ssl_cert() — which will, by default, hit a given host on the standard HTTPS port (443/TCP) and grab the site certificate and issuer. We’ll grab the “validity end” field and convert that to a date to use for comparison.

To get the target list of sites to check I used Rapid7’s FDNS data set and a glance at a few certificate transparency logs to put together a current list of “r-project” domains that have been known to have SSL certs. This process could be made more dynamic, but things don’t change that quickly in r-project domain land.

Finally, we use the {DT} package to build a pretty HTML table and the {pushoverr} package to send notifications at normal priority for certs expiring within a week and critical priority for certs that have expired (the package has excellent documentation which will guide you through setting up a Pushover account).

I put this all in a plain R script named r-project-ssl-notify.R that’s then called from a Linux CRON job which runs:

/usr/bin/Rscript -e 'rmarkdown::render(input="PATH_TO/r-project-ssl-notify.R", output_file="PATH_TO/r-project-cert-status/index.html", quiet=TRUE)'

once a day at 0930 ET to make this status page and also fire off any notifications which I have going to my watch and phone (I did a test send by expanding the delta to 14 days):

Here’s the contents of

#' ---
#' title: "r-project SSL/TLS Certificate Status"
#' date: "`r format(Sys.time(), '%Y-%m-%d')`"
#' output:
#'   html_document:
#'     keep_md: false
#'     theme: simplex
#'     highlight: monochrome
#' ---
#+ init, include=FALSE
knitr::opts_chunk$set(
  message = FALSE, 
  warning = FALSE, 
  echo = FALSE, 
  collapse=TRUE
)

#+ libs
library(DT)
library(openssl)
library(pushoverr)
library(tidyverse)

# Setup -----------------------------------------------------------------------------------------------------------

# This env config file contains two lines:
#
# PUSHOVER_USER=YOUR_PUSHOVER_USER_STRING
# PUSHOVER_APP=YOUR_PUSHOVER_APP_KEY
#
# See the {pushoverr} package for how to setup your Pushover account
readRenviron("~/jobs/conf/r-project-ssl-notify.env")


# Check certs -----------------------------------------------------------------------------------------------------

# r-project.org domains retrieved from Rapid7's FDNS data set
# (https://opendata.rapid7.com/sonar.fdns_v2/) and cert transparency logs

#+ work
c(
  "beta.r-project.org", "bugs.r-project.org", "cloud.r-project.org", 
  "cran-archive.r-project.org", "cran.at.r-project.org", "cran.ch.r-project.org", 
  "cran.es.r-project.org", "cran.r-project.org", "cran.uk.r-project.org", 
  "cran.us.r-project.org", "developer.r-project.org", "ess.r-project.org", 
  "ftp.cran.r-project.org", "journal.r-project.org", "lists.r-forge.r-project.org", 
  "mac.r-project.org", "r-project.org", "svn.r-project.org", "translation.r-project.org", 
  "user2011.r-project.org", "user2014.r-project.org", "user2016.r-project.org", 
  "user2018.r-project.org", "user2019.r-project.org", "user2020.r-project.org", 
  "user2020muc.r-project.org", "win-builder.r-project.org", "www.cran.r-project.org", 
  "www.r-project.org", "www.user2019.fr"
) -> r_doms

# grab each cert

r_certs <- map(r_doms, openssl::download_ssl_cert)

# make a nice table
tibble(
  dom = r_doms,
  expires = map_chr(r_certs, ~.x[[1]][["validity"]][[2]]) %>% # this gets us the "validity end"
    as.Date(format = "%b %d %H:%M:%S %Y", tz = "GMT"),        # and converts it to a date object
  delta = as.numeric(expires - Sys.Date(), "days")            # this computes the delta from the day this script was called
) %>% 
  arrange(expires) -> r_certs_expir

# Status page generation ------------------------------------------------------------------------------------------

# output nice table  
DT::datatable(r_certs_expir, list(pageLength = nrow(r_certs_expir))) # if the # of r-proj doms gets too large we'll cap this for pagination

# Notifications ---------------------------------------------------------------------------------------------------

# See if we need to notify abt things expiring within 1 week
# REMOVE THIS or edit the delta max if you want less noise
one_week <- filter(r_certs_expir, between(delta, 1, 7))
if (nrow(one_week) > 0) {
  pushover_normal(
    title = "There are r-project SSL Certs Expiring Within 1 Week", 
    message = "Check which ones: https://rud.is/r-project-cert-status"
  )
}

# See if we have expired certs
expired <- filter(r_certs_expir, delta <= 0)
if (nrow(expired) > 0) {
  pushover_critical(
    title = "There are expired r-project SSL Certs!", 
    message = "Check which ones: https://rud.is/r-project-cert-status"
  )
}

FIN

With just a tiny bit of R code we have the ability to monitor expiring SSL certs via a diminutive status page and alerts to any/all devices at our disposal.

UPDATE 2020-02-11 Apple now supports downloading transactions as CSV or OFX! (via MacObserver).

I saw this CNBC article on an in-theory browser client-side-only conversion utility for taking Apple Card PDF statements and turning them into CSV files.

Since I (a) never trust any browser or site and (b) the article indicated that there is a $5 fee to avoid the “single random transaction removal”, I felt compelled to throw together an R script to do this for at least folks who are capable of setting up R so that all processing is guaranteed to be local.

FWIW the site does appear to do what it says on the tin (all processing is, indeed, local). That doesn’t mean one of your extensions isn’t spying on you, nor does it mean that the site could not turn evil someday (one its own or via an attacker compromise).

read_apple_card_statement <- function(path) {

  require(stringi)
  require(pdftools)
  require(tidyverse)

  # make sure the file exists
  path <- path.expand(path[1])
  if (!file.exists(path)) stop("File '", path, "' not found.", call.=FALSE)

  pdf_text(path) %>% # read it in
    stri_split_lines() %>% # turn \n to a separate character vector element
    unlist() %>% # flatten it
    stri_trim_both() %>% # get rid of leading/trailing spaces
    keep(stri_detect_regex, "^([[:digit:]]{2}/[[:digit:]]{2}/[[:digit:]]{4})") %>% # find lines that start with a date
    map_df(~{
      rec <- as.list(unlist(stri_split_regex(.x, "[[:space:]]{3,}"))) # find the columns
      if (stri_detect_fixed(.x, "%")) { # lines with a `%` in them distinguish charges from payments
        rec <- set_names(rec, c("date", "description", "daily_cash_pct", "daily_cash_amt", "amt")) # ones with charges have cash back columns
      } else {
        rec <- set_names(rec, c("date", "description", "amt")) # ones w/o % do not
      }
    }) %>%
    mutate(
      date = lubridate::mdy(date), # make dates dates
      amt = stri_replace_first_fixed(amt, "$", "") %>% parse_number(), # dollars to numbers
      daily_cash_pct = parse_number(daily_cash_pct)/100, # % to numbers
      daily_cash_amt = parse_number(daily_cash_amt) # dollars to numbners
    )

}

list.files("~/Downloads", pattern = "Apple Card Statement", full.names = TRUE) %>% 
  map_df(read_apple_card_statement)

You can send the PDF statements from the Apple Card app to your Mac via Air Drop and it will put them into ~/Downloads. I recommend putting them somewhere else since you’ve likely given all sorts of applications access to ~/Downloads when prompted to on Catalina (yay security theatre). Wherever you put them, you can read them individually with read_apple_card_statment() or you can then just list_files() and bind all the individual statements together:

list.files("~/WhereYouPutAppleCardStatements", pattern = "Apple Card Statement", full.names = TRUE) %>% 
  map_df(read_apple_card_statement)

FIN

Be very wary of what you put your trust into online. Just because a site is benign one day does not mean it won’t be malicious (deliberately or otherwise) the next. Also, lobby Apple to provide data in more useful formats, especially since it provides applications like Numbers for free with their operating system.

Before we start wrapping foreign language code we need to make sure that basic R packages can be created. If you’ve followed along from the previous post you have everything you need to get started here. Just to make sure, you should be able to fire up a new RStudio session and execute the following R code and see similar output. If not, you’ll need to go through the steps and resources outlined there before continuing.

pkgbuild::check_build_tools()
### Your system is ready to build packages!

Also: the {bookdown} version should now always match the blog post (apart from some verbiage changes to denote it’s in a book vs a series of blog posts). You can refer to it at any time via — https://rud.is/books/writing-frictionless-r-package-wrappers/.

Configuring {devtools}

We’re going to rely on the {devtools} package for many operations and the first thing you should do now is execute help("create", "devtools") in an RStudio R console to see the package documentation page where you’ll see guidance pointing you to devtools::use_description() that lists some R session options() that you can set to make your package development life much easier and quicker. Specifically, it lets you know that you can setup your ~/.Rprofile to include the certain options settings which will automatically fill in fields each time you create a new package vs you either specifying these fields manually in the package creation GUI or as parameters to devtools::create().

A good, minimal setup would be something like:

options(
  usethis.description = list(
    `Authors@R` = 'person("Some", "One", email = "someone@example.com", role = c("aut", "cre"),
                          comment = c(ORCID = "YOUR-ORCID-ID"))',
    License = "MIT + file LICENSE"
  )
)

NOTE: If you do not have an “ORCID” you really should get one (they’re free!) by heading over to
— https://orcid.org/ — and filling in some basic information.

Take a moment to edit your ~/.Rprofile. If you’re not sure about how to do that there is an excellent chapter in Efficient R Programming¹ which walks you through the process.

Once you’ve added or verified these new options() settings, restart your R session.

Creating A Package

We’re almost ready to create and build a basic R package. All R packages live in a package directory and I highly suggest creating a packages directory right off your home directory (e.g. “~/packages“) or someplace where you’ll be able to keep them all organized and accessible. The rest of these posts will assume you’re using “~/packages” as the

With {devtools} now pre-configured, use the RStudio R Console pane to execute the following code which should produce similar output and open up a new RStudio session with the new package directory:

devtools::create("~/packages/myfirstpackage") 
## ✔ Creating '/Users/someuser/packages/myfirstpackage/'
## ✔ Setting active project to '/Users/someuser/packages/myfirstpackage'
## ✔ Creating 'R/'
## ✔ Writing 'DESCRIPTION'
## Package: myfirstpackage
## Title: What the Package Does (One Line, Title Case)
## Version: 0.0.0.9000
## Authors@R (parsed):
##     * Bob Rudis <bob@rud.is> [aut, cre] (<https://orcid.org/0000-0001-5670-2640>)
## Description: What the package does (one paragraph).
## License: MIT + file LICENSE
## Encoding: UTF-8
## LazyData: true
## ✔ Writing 'NAMESPACE'
## ✔ Writing 'myfirstpackage.Rproj'
## ✔ Adding '.Rproj.user' to '.gitignore'
## ✔ Adding '^myfirstpackage\\.Rproj$', '^\\.Rproj\\.user$' to '.Rbuildignore'
## ✔ Opening '/Users/someuser/packages/myfirstpackage/' in new RStudio session
## ✔ Setting active project to '<no active project>'

The directory structure will look like this:

.
├── DESCRIPTION
├── NAMESPACE
├── R/
└── myfirstpackage.Rproj

At this point we still do not have a “perfect” R package. To prove this, use the R console to run devtools::check() and — after some rather verbose output — you’ll see the following lines at the end:

> checking DESCRIPTION meta-information ... WARNING
  Invalid license file pointers: LICENSE

0 errors ✓ | 1 warning x | 0 notes ✓

Since we’re saying that our package will be using the MIT license, we need to ensure there’s an associated LICENSE file which we can do by executing usethis::use_mit_license() which will create the necessary files and ensure the License field in the DESCRIPTION file is formatted properly.

If you run devtools::check() again, now, your final line should report:

## 0 errors ✓ | 0 warnings ✓ | 0 notes ✓

and the package directory tree should look like this:

├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
├── R/
└── myfirstpackage.Rproj

Rounding Out The Corners

While we have a minimum viable package there are a few other steps we should take during this setup phase. First we’ll setup our package to use {roxygen2}² for documenting functions, declaring NAMESPACE imports, and other helper-features that will be introduced in later posts. We can do this via usethis::use_roxygen_md():

usethis::use_roxygen_md()
## ✔ Setting Roxygen field in DESCRIPTION to 'list(markdown = TRUE)'
## ✔ Setting RoxygenNote field in DESCRIPTION to '7.0.2'
## ● Run `devtools::document()`

We won’t run devtools::document() just yet, though. Before we do that we’ll also create an R file where we can store top-level package introduction/meta-information:

usethis::use_package_doc()
## ✔ Writing 'R/myfirstpackage-package.R'

Now, our directory tree should look like:

.
├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
├── R
│   └── myfirstpackage-package.R
└── myfirstpackage.Rproj

Now, run devtools::document() which will translate the {roxygen2} comments into a properly-formatted R documentation file and regenerate the NAMESPACE file (as we’ll be managing package imports and exports via {roxygen2} comments). The directory tree will now look like:

.
├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
├── R
│   └── myfirstpackage-package.R
├── man
│   └── myfirstpackage-package.Rd
└── myfirstpackage.Rproj

and, we can now re-run devtools::check() to make sure we have the three “0’s” we’re aiming for each time we check our package for errors.

Passing The Test

We’re going to want to write and use tests to ensure our package works properly. There are many R package testing frameworks available. To ease the introduction into this process, we’ll use one of the frameworks that came along for the ride when you installed the various packages outlined in the previous post: {testthat}³. Setting up {testthat} is also pretty painless thanks to the {usethis} package we’ve been taking advantage of quite a bit so far. We’ll create the {testthat} overall infrastructure then add a placeholder test script since devtools::check() will complain about no tests being available if we do not have at least a single script it can execute during the test phase of the package checking process.

usethis::use_testthat()
## ✔ Adding 'testthat' to Suggests field in DESCRIPTION
## ✔ Creating 'tests/testthat/'
## ✔ Writing 'tests/testthat.R'
## ● Call `use_test()` to initialize a basic test file and open it for editing.

usethis::use_test("placeholder")
## ✔ Increasing 'testthat' version to '>= 2.1.0' in DESCRIPTION
## ✔ Writing 'tests/testthat/test-placeholder.R'
## ● Modify 'tests/testthat/test-placeholder.R'

The directory tree will now look like this:

.
├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
├── R
│   └── myfirstpackage-package.R
├── man
│   └── myfirstpackage-package.Rd
├── myfirstpackage.Rproj
└── tests
    ├── testthat
    │   └── test-placeholder.R
    └── testthat.R

Run devtools::check() one more time to make sure we’ve got those precious 3 “0’s” one last time.

Getting Things Under Control

We’re almost done! One final step is to turn this directory into a git-managed directory so we can work a bit more safely and eventually share our development work with a broader audience. Provided you followed the outline in the previous post, setting up git is as straightforward as one {usethis} function call:

usethis::use_git()
## ✔ Setting active project to '/Users/someuser/packages/myfirstpackage'
## ✔ Initialising Git repo
## ✔ Adding '.Rhistory', '.RData' to '.gitignore'
## There are 10 uncommitted files:
## * '.gitignore'
## * '.Rbuildignore'
## * 'DESCRIPTION'
## * 'LICENSE'
## * 'LICENSE.md'
## * 'man/'
## * 'myfirstpackage.Rproj'
## * 'NAMESPACE'
## * 'R/'
## * 'tests/'
## Is it ok to commit them?
## 
## 1: For sure
## 2: Negative
## 3: Not now
## 
## Selection: 1
## ✔ Adding files
## ✔ Commit with message 'Initial commit'
## ● A restart of RStudio is required to activate the Git pane
## Restart now?
## 
## 1: Negative
## 2: Not now
## 3: Yup
## 
## Selection: 3

RStudio should have been restarted (so it can add a “Git” pane in case you want to use the GUI to manage git) and the directory tree will now have a .git/ subdirectory that you should (almost) never touch by hand.

The last thing to do is to “vaccinate” your git setup so you don’t leak sensitive or unnecessary files when you (eventually) share your creation with the world:

usethis::git_vaccinate()
## ✔ Adding '.Rproj.user', '.Rhistory', '.Rdata' to '/Users/someuser/.gitignore'

We now have a basic, working R package that is devoid of any real functionality other than that of getting us familiar with the package setup and validation processes. We’ll be building upon this experience in most of the coming posts.

Quick Reference

After ensuring you’ve got the recommended options() in place, here are the steps to setup a new package:

# in any RStudio R Console session
devtools::create("~/packages/THE-PACKAGE-NAME")

# in the newly created package RStudio R Console session:
usethis::use_mit_license()       # need a LICENSE file
usethis::use_roxygen_md()        # use {roxygen2} for documentation and configuration
usethis::use_package_doc()       # setup a package-level manual page
usethis::use_testthat()          # setup testing infrastructure
usethis::use_test("placeholder") # setup a placeholder test file
devtools::document()             # Let {roxygen2} create NAMESPACE entries, build manual pages (and, more later on)
devtools::check()                # looking for the three "0's" that tell us we're ready to roll!
usethis::use_git()               # put the directory under git version control
usethis::git_vaccinate()         # Prevent leaking credentials and other unnecessary filesystem cruft

Rather than re-type devtools::document() (et al) whenever you need to run {roxygen2} or build/check a package you can use RStudio keyboard shortcuts that are designed to seamlessly integrate with the {devtools} ecosystem:

We’ll refer to these operations as “install” (or “build”), “load all”, “test”, “check”, and “document” from now on so you can choose to use the console or the shortcuts as you prefer.

Exercises

Our package may be kinda, well, useless for the moment but that doesn’t mean you can’t show it some love and get some practice in at the same time while things are still relatively straightforward.

• Modify the Title, Version, and Description fields of the DESCRIPTION file and refine them as needed until package checks pass.
• Deliberately mangle parts of the DESCRIPTION file to see what errors or warnings you receive during the package check process.
• Read up on {roxygen2} and add some Sections to it formatted with markdown and/or LaTeX. Re-“document” the package and see how your changes look.
• Edit the test-placeholder.R file and change the placeholder test it created so it fails and then re-check the package to see what warnings or errors show up.
• After you’ve made (valid, working) modifications to any/all of the above and package checks pass, use either the git command line tools or the RStudio Git pane to add your updates to the git tree. Use the resources linked to in the previous post if you need a refresher on how to do that.
• Re-run through all the steps with a brand new package name just to make sure you’re comfortable with the package creation process.

Up Next

In the next installment in the series we will start wrapping by creating a basic wrapper that just calls out to the operating system shell to run commands.