rud.is

UPDATE 2020-02-11 Apple now supports downloading transactions as CSV or OFX! (via MacObserver).

I saw this CNBC article on an in-theory browser client-side-only conversion utility for taking Apple Card PDF statements and turning them into CSV files.

Since I (a) never trust any browser or site and (b) the article indicated that there is a $5 fee to avoid the “single random transaction removal”, I felt compelled to throw together an R script to do this for at least folks who are capable of setting up R so that all processing is guaranteed to be local.

FWIW the site does appear to do what it says on the tin (all processing is, indeed, local). That doesn’t mean one of your extensions isn’t spying on you, nor does it mean that the site could not turn evil someday (one its own or via an attacker compromise).

read_apple_card_statement <- function(path) {

  require(stringi)
  require(pdftools)
  require(tidyverse)

  # make sure the file exists
  path <- path.expand(path[1])
  if (!file.exists(path)) stop("File '", path, "' not found.", call.=FALSE)

  pdf_text(path) %>% # read it in
    stri_split_lines() %>% # turn \n to a separate character vector element
    unlist() %>% # flatten it
    stri_trim_both() %>% # get rid of leading/trailing spaces
    keep(stri_detect_regex, "^([[:digit:]]{2}/[[:digit:]]{2}/[[:digit:]]{4})") %>% # find lines that start with a date
    map_df(~{
      rec <- as.list(unlist(stri_split_regex(.x, "[[:space:]]{3,}"))) # find the columns
      if (stri_detect_fixed(.x, "%")) { # lines with a `%` in them distinguish charges from payments
        rec <- set_names(rec, c("date", "description", "daily_cash_pct", "daily_cash_amt", "amt")) # ones with charges have cash back columns
      } else {
        rec <- set_names(rec, c("date", "description", "amt")) # ones w/o % do not
      }
    }) %>%
    mutate(
      date = lubridate::mdy(date), # make dates dates
      amt = stri_replace_first_fixed(amt, "$", "") %>% parse_number(), # dollars to numbers
      daily_cash_pct = parse_number(daily_cash_pct)/100, # % to numbers
      daily_cash_amt = parse_number(daily_cash_amt) # dollars to numbners
    )

}

list.files("~/Downloads", pattern = "Apple Card Statement", full.names = TRUE) %>% 
  map_df(read_apple_card_statement)

You can send the PDF statements from the Apple Card app to your Mac via Air Drop and it will put them into ~/Downloads. I recommend putting them somewhere else since you’ve likely given all sorts of applications access to ~/Downloads when prompted to on Catalina (yay security theatre). Wherever you put them, you can read them individually with read_apple_card_statment() or you can then just list_files() and bind all the individual statements together:

list.files("~/WhereYouPutAppleCardStatements", pattern = "Apple Card Statement", full.names = TRUE) %>% 
  map_df(read_apple_card_statement)

FIN

Be very wary of what you put your trust into online. Just because a site is benign one day does not mean it won’t be malicious (deliberately or otherwise) the next. Also, lobby Apple to provide data in more useful formats, especially since it provides applications like Numbers for free with their operating system.

Before we start wrapping foreign language code we need to make sure that basic R packages can be created. If you’ve followed along from the previous post you have everything you need to get started here. Just to make sure, you should be able to fire up a new RStudio session and execute the following R code and see similar output. If not, you’ll need to go through the steps and resources outlined there before continuing.

pkgbuild::check_build_tools()
### Your system is ready to build packages!

Also: the {bookdown} version should now always match the blog post (apart from some verbiage changes to denote it’s in a book vs a series of blog posts). You can refer to it at any time via — https://rud.is/books/writing-frictionless-r-package-wrappers/.

Configuring {devtools}

We’re going to rely on the {devtools} package for many operations and the first thing you should do now is execute help("create", "devtools") in an RStudio R console to see the package documentation page where you’ll see guidance pointing you to devtools::use_description() that lists some R session options() that you can set to make your package development life much easier and quicker. Specifically, it lets you know that you can setup your ~/.Rprofile to include the certain options settings which will automatically fill in fields each time you create a new package vs you either specifying these fields manually in the package creation GUI or as parameters to devtools::create().

A good, minimal setup would be something like:

options(
  usethis.description = list(
    `Authors@R` = 'person("Some", "One", email = "someone@example.com", role = c("aut", "cre"),
                          comment = c(ORCID = "YOUR-ORCID-ID"))',
    License = "MIT + file LICENSE"
  )
)

NOTE: If you do not have an “ORCID” you really should get one (they’re free!) by heading over to
— https://orcid.org/ — and filling in some basic information.

Take a moment to edit your ~/.Rprofile. If you’re not sure about how to do that there is an excellent chapter in Efficient R Programming¹ which walks you through the process.

Once you’ve added or verified these new options() settings, restart your R session.

Creating A Package

We’re almost ready to create and build a basic R package. All R packages live in a package directory and I highly suggest creating a packages directory right off your home directory (e.g. “~/packages“) or someplace where you’ll be able to keep them all organized and accessible. The rest of these posts will assume you’re using “~/packages” as the

With {devtools} now pre-configured, use the RStudio R Console pane to execute the following code which should produce similar output and open up a new RStudio session with the new package directory:

devtools::create("~/packages/myfirstpackage") 
## ✔ Creating '/Users/someuser/packages/myfirstpackage/'
## ✔ Setting active project to '/Users/someuser/packages/myfirstpackage'
## ✔ Creating 'R/'
## ✔ Writing 'DESCRIPTION'
## Package: myfirstpackage
## Title: What the Package Does (One Line, Title Case)
## Version: 0.0.0.9000
## Authors@R (parsed):
##     * Bob Rudis <bob@rud.is> [aut, cre] (<https://orcid.org/0000-0001-5670-2640>)
## Description: What the package does (one paragraph).
## License: MIT + file LICENSE
## Encoding: UTF-8
## LazyData: true
## ✔ Writing 'NAMESPACE'
## ✔ Writing 'myfirstpackage.Rproj'
## ✔ Adding '.Rproj.user' to '.gitignore'
## ✔ Adding '^myfirstpackage\\.Rproj$', '^\\.Rproj\\.user$' to '.Rbuildignore'
## ✔ Opening '/Users/someuser/packages/myfirstpackage/' in new RStudio session
## ✔ Setting active project to '<no active project>'

The directory structure will look like this:

.
├── DESCRIPTION
├── NAMESPACE
├── R/
└── myfirstpackage.Rproj

At this point we still do not have a “perfect” R package. To prove this, use the R console to run devtools::check() and — after some rather verbose output — you’ll see the following lines at the end:

> checking DESCRIPTION meta-information ... WARNING
  Invalid license file pointers: LICENSE

0 errors ✓ | 1 warning x | 0 notes ✓

Since we’re saying that our package will be using the MIT license, we need to ensure there’s an associated LICENSE file which we can do by executing usethis::use_mit_license() which will create the necessary files and ensure the License field in the DESCRIPTION file is formatted properly.

If you run devtools::check() again, now, your final line should report:

## 0 errors ✓ | 0 warnings ✓ | 0 notes ✓

and the package directory tree should look like this:

├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
├── R/
└── myfirstpackage.Rproj

Rounding Out The Corners

While we have a minimum viable package there are a few other steps we should take during this setup phase. First we’ll setup our package to use {roxygen2}² for documenting functions, declaring NAMESPACE imports, and other helper-features that will be introduced in later posts. We can do this via usethis::use_roxygen_md():

usethis::use_roxygen_md()
## ✔ Setting Roxygen field in DESCRIPTION to 'list(markdown = TRUE)'
## ✔ Setting RoxygenNote field in DESCRIPTION to '7.0.2'
## ● Run `devtools::document()`

We won’t run devtools::document() just yet, though. Before we do that we’ll also create an R file where we can store top-level package introduction/meta-information:

usethis::use_package_doc()
## ✔ Writing 'R/myfirstpackage-package.R'

Now, our directory tree should look like:

.
├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
├── R
│   └── myfirstpackage-package.R
└── myfirstpackage.Rproj

Now, run devtools::document() which will translate the {roxygen2} comments into a properly-formatted R documentation file and regenerate the NAMESPACE file (as we’ll be managing package imports and exports via {roxygen2} comments). The directory tree will now look like:

.
├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
├── R
│   └── myfirstpackage-package.R
├── man
│   └── myfirstpackage-package.Rd
└── myfirstpackage.Rproj

and, we can now re-run devtools::check() to make sure we have the three “0’s” we’re aiming for each time we check our package for errors.

Passing The Test

We’re going to want to write and use tests to ensure our package works properly. There are many R package testing frameworks available. To ease the introduction into this process, we’ll use one of the frameworks that came along for the ride when you installed the various packages outlined in the previous post: {testthat}³. Setting up {testthat} is also pretty painless thanks to the {usethis} package we’ve been taking advantage of quite a bit so far. We’ll create the {testthat} overall infrastructure then add a placeholder test script since devtools::check() will complain about no tests being available if we do not have at least a single script it can execute during the test phase of the package checking process.

usethis::use_testthat()
## ✔ Adding 'testthat' to Suggests field in DESCRIPTION
## ✔ Creating 'tests/testthat/'
## ✔ Writing 'tests/testthat.R'
## ● Call `use_test()` to initialize a basic test file and open it for editing.

usethis::use_test("placeholder")
## ✔ Increasing 'testthat' version to '>= 2.1.0' in DESCRIPTION
## ✔ Writing 'tests/testthat/test-placeholder.R'
## ● Modify 'tests/testthat/test-placeholder.R'

The directory tree will now look like this:

.
├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
├── R
│   └── myfirstpackage-package.R
├── man
│   └── myfirstpackage-package.Rd
├── myfirstpackage.Rproj
└── tests
    ├── testthat
    │   └── test-placeholder.R
    └── testthat.R

Run devtools::check() one more time to make sure we’ve got those precious 3 “0’s” one last time.

Getting Things Under Control

We’re almost done! One final step is to turn this directory into a git-managed directory so we can work a bit more safely and eventually share our development work with a broader audience. Provided you followed the outline in the previous post, setting up git is as straightforward as one {usethis} function call:

usethis::use_git()
## ✔ Setting active project to '/Users/someuser/packages/myfirstpackage'
## ✔ Initialising Git repo
## ✔ Adding '.Rhistory', '.RData' to '.gitignore'
## There are 10 uncommitted files:
## * '.gitignore'
## * '.Rbuildignore'
## * 'DESCRIPTION'
## * 'LICENSE'
## * 'LICENSE.md'
## * 'man/'
## * 'myfirstpackage.Rproj'
## * 'NAMESPACE'
## * 'R/'
## * 'tests/'
## Is it ok to commit them?
## 
## 1: For sure
## 2: Negative
## 3: Not now
## 
## Selection: 1
## ✔ Adding files
## ✔ Commit with message 'Initial commit'
## ● A restart of RStudio is required to activate the Git pane
## Restart now?
## 
## 1: Negative
## 2: Not now
## 3: Yup
## 
## Selection: 3

RStudio should have been restarted (so it can add a “Git” pane in case you want to use the GUI to manage git) and the directory tree will now have a .git/ subdirectory that you should (almost) never touch by hand.

The last thing to do is to “vaccinate” your git setup so you don’t leak sensitive or unnecessary files when you (eventually) share your creation with the world:

usethis::git_vaccinate()
## ✔ Adding '.Rproj.user', '.Rhistory', '.Rdata' to '/Users/someuser/.gitignore'

We now have a basic, working R package that is devoid of any real functionality other than that of getting us familiar with the package setup and validation processes. We’ll be building upon this experience in most of the coming posts.

Quick Reference

After ensuring you’ve got the recommended options() in place, here are the steps to setup a new package:

# in any RStudio R Console session
devtools::create("~/packages/THE-PACKAGE-NAME")

# in the newly created package RStudio R Console session:
usethis::use_mit_license()       # need a LICENSE file
usethis::use_roxygen_md()        # use {roxygen2} for documentation and configuration
usethis::use_package_doc()       # setup a package-level manual page
usethis::use_testthat()          # setup testing infrastructure
usethis::use_test("placeholder") # setup a placeholder test file
devtools::document()             # Let {roxygen2} create NAMESPACE entries, build manual pages (and, more later on)
devtools::check()                # looking for the three "0's" that tell us we're ready to roll!
usethis::use_git()               # put the directory under git version control
usethis::git_vaccinate()         # Prevent leaking credentials and other unnecessary filesystem cruft

Rather than re-type devtools::document() (et al) whenever you need to run {roxygen2} or build/check a package you can use RStudio keyboard shortcuts that are designed to seamlessly integrate with the {devtools} ecosystem:

We’ll refer to these operations as “install” (or “build”), “load all”, “test”, “check”, and “document” from now on so you can choose to use the console or the shortcuts as you prefer.

Exercises

Our package may be kinda, well, useless for the moment but that doesn’t mean you can’t show it some love and get some practice in at the same time while things are still relatively straightforward.

• Modify the Title, Version, and Description fields of the DESCRIPTION file and refine them as needed until package checks pass.
• Deliberately mangle parts of the DESCRIPTION file to see what errors or warnings you receive during the package check process.
• Read up on {roxygen2} and add some Sections to it formatted with markdown and/or LaTeX. Re-“document” the package and see how your changes look.
• Edit the test-placeholder.R file and change the placeholder test it created so it fails and then re-check the package to see what warnings or errors show up.
• After you’ve made (valid, working) modifications to any/all of the above and package checks pass, use either the git command line tools or the RStudio Git pane to add your updates to the git tree. Use the resources linked to in the previous post if you need a refresher on how to do that.
• Re-run through all the steps with a brand new package name just to make sure you’re comfortable with the package creation process.

Up Next

In the next installment in the series we will start wrapping by creating a basic wrapper that just calls out to the operating system shell to run commands.

The R language and RStudio IDE are a powerful combination for “getting stuff done”, and one aspect of R itself that makes it especially useful is the ability to use it with other programming languages via a robust foreign language interface capability¹. The term “foreign language” refers to another programming language such as C, C++, Fortran, Java, Python, Rust, etc. A common way of referring the this idiom of using functionality written in another programming language from directly within R is “wrapping” since we’re putting an R “shell” around the code from the other language. Another term you may see used is “extending” (hence the title of the “Writing R Extensions” R manual).

While R supports using this this extension mechanism from any R script leaving tiny trails of R and other language source and binary files all across your filesystem is not exactly the best way to keep these components organized and creates other challenges when you come across the need to use them in other projects or share them with others. Thankfully, the R Core team, along with many individual contributors over the years, has made it pretty straightforward to incorporate this extension capability into R packages which are much easier (honest!) to organize and share.

The goal of this blog series — which will have a {bookdown} book companion along with some screencasts — is to help you get up to speed using R and RStudio to write R packages that wrap code from many different languages to help you “get stuff done” with as little friction as possible.

Base Requirements

It is assumed that readers are familiar with the R programming language, RStudio IDE, and are comfortable installing and using packages. Since this work is about extending R with other programming languages, you should also have some knowledge of one or more of the target languages being covered.

To follow along with the series you’ll need to ensure you have the necessary components installed along the way. Rather than overwhelm you with all of them up front, each new section will introduce requirements specific to the language or situation being covered. However, there are some fundamentals you’ll need to ensure are available.

• An R² environment, preferably R 3.6.x which is what was used for this series.
• RStudio³, as we’ll be using many of the features provided in it to help reduce development friction
• The {pkgbuild}⁴ package installed

Once you’ve gotten through those steps, you should fire up RStudio and run:

pkgbuild::check_build_tools(debug = TRUE)

which will help you make sure your particular system is ready to build packages.

After performing the build tool check and/or installation of the necessary core tools, you will then need to install the {devtools}⁵ package, which will help ensure that the remaining core packages required are installed.

We’re also going to use the git⁶ source code version control system. The git ecosystem is not “GitHub”, which is just a public (or, potentially somewhat private) place to house source code repositories, just like other hosted services such as Bitbucket, GitLab, or SourceHut. You can use the excellent “Happy Git with R”⁷ resource to help ensure you’re source code control environment is also ready to use.

Supplemental References

It may be helpful to create a browser bookmark folder for supplemental reference material that will be referred to from time-to-time across the sections (we’ll be adding to this list in each chapter, too):

• Writing R Extensions (https://cran.r-project.org/doc/manuals/r-release/R-exts.html)
• Advanced R (http://adv-r.had.co.nz/)
• R Packaged (http://r-pkgs.had.co.nz/)

Up Next

If you’ve been a user of “development versions” of R packages or have authored R packages you likely made quick work of this first installment. Those new to creating packages with R, those who tend to only use fully-baked CRAN versions of R packages, and/or those who have not worked with git before likely had to do quite a bit of work to get down to this point (if this describes you, you definitely deserve both a break and kudos for getting this far!).

In the next installment we’ll make sure the package building infrastructure is ready to roll by creating a basic R package that we’ll use as a building block for future work.

I had need to present a wall-of-text to show off a giant list of SSL certificate alternate names and needed the entire list to fit on one slide (not really for reading in full, but to show just how many there were in a way that a simple count would not really convey).

Keynote, PowerPoint, and gslides all let you make tables or draw boxes but I really didn’t want to waste time fiddling as much as I’d need to with those tools just for this one slide.

Thankfully, I remembered that HTML5 <div> elements can be styled with a column-count attribute and we can use {htmltools} to make quick work of this task.

To show it off, first we’ll need some words, so let’s make some using stringi::stri_rand_lipsum():

library(stringi)
library(htmltools)
library(tidyverse)

set.seed(201912)

stri_rand_lipsum(5) %>%
  stri_paste(collapse = " ") %>%
  stri_split_boundaries() %>%
  flatten_chr() %>%
  stri_trim_both() -> words

head(words)
## [1] "Lorem"  "ipsum"  "dolor"  "sit"    "amet,"  "sapien"

length(words)
## [1] 514

Now, we’ll make a function — columnize() — that we can reuse in the future and have it take in a character vector, the column count we want and some CSS styling, then use some {htmltools} tag functions to make quick work of this task:

columnize <- function(words, ncol = 5,
                      style = "p { font-family:'Roboto Condensed';font-size:12pt;line-height:12.5pt;padding:0;margin:0}") {

  tagList(
    tags$style(style[1]),
    tags$div(
      words %>%
        map(tags$p) %>%
        tagList(),
      style = sprintf("column-count:%d", as.integer(ncol[1]))
    )
  )

}

In this function we turn the style param into a <style> section in the generated HTML, then turn words into <p> tags wrapped in <div>.

This function can be used in a R Markdown code block (set block parameters to results='markup') to have the columns appear automagically in the resultant HTML document output. You can also use it in standalone fashion by using html_print() on the results:

html_print(
  columnize(words, 10)
)

The above is an image just for easier blog display purposes. You can test out a working example from a spun R script over at https://rud.is/rpubs/columnize.html that has some different column count examples. Grow and shrink the browser width to see how the columns shrink and grow with it.

FIN

Hopefully this helps others save time and effort like it did for me today. You can experiment with making the columnize() function more robust by having it work with all the other column-formatting properties:

• column-count: Specifies the number of columns an element should be divided into
• column-fill: Specifies how to fill columns
• column-gap: Specifies the gap between the columns
• column-rule: A shorthand property for setting all the column-rule-* properties
• column-rule-color: Specifies the color of the rule between columns
• column-rule-style: Specifies the style of the rule between columns
• column-rule-width: Specifies the width of the rule between columns
• column-span: Specifies how many columns an element should span across
• column-width: Specifies a suggested, optimal width for the columns
• columns: A shorthand property for setting column-width and column-count

You can find out more about these properties (and play with some examples) over at https://www.w3schools.com/css/css3_multiple_columns.asp.

POST-FIN

I robustified the function a bit:

#' Make a responsive columnar text div
#'
#' @param words character vector of text to present in a columnar div
#' @param div_id tag `id` attribute to assign to the `<div>` (which can help you style it with the `style` param).
#' @param div_class tag `class` attribute to assign to the `<div>` (which can help you style it with the `style` param)
#' @param ncol number of columns
#' @param width  specifies the column width; one of "`auto`" (the default) which leaves it up to the
#'        browser implementation, a _length_ CSS size value that specifies the width of the columns.
#'        The number of columns will be the minimum number of columns needed to show all the content
#'        across the element., "`initial`" or "`inherit`" (see `fill` for descriptions of those).
#' @param fill how to fill columns, balanced or not. One of "`balance`", "`auto`", "`initial`", "`inherit`".
#'        Defaults to "`balance`" which fills each column with about the same amount of content, but will not
#'        allow the columns to be taller than the height (so, columns might be shorter than the height as the
#'        browser distributes the content evenly horizontally). "`auto`" fills each column until it reaches
#'        the height, and do this until it runs out of content (so, this value will not necessarily fill all
#'        the columns nor fill them evenly). "`initial`" sets this property to its default value; and
#'        "`inherit`" inherits this property from its parent element.
#' @param gap either a textual value (e.g. "`10px`") for the spacing gap between columns, or "`normal`"
#'        (the default) which uses a `1em` gap on most browsers, "`initial`" or "`inherit`" (see `fill` for descriptions
#'        of those).
#' @param rule_color specifies the CSS color value of the rule between columns; also can be "`initial`" or "`inherit`"
#'        (see `fill` for descriptions of those).
#' @param rule_style specifies the style of the rule between columns; valid values are "`none`" (the default) for no
#'        rule, "`hidden`", "`dotted`", "`dashed`", "`solid`", "`double`", "`groove`" for 3D grooved rule, "`ridge`"
#'        for a 3D ridged rule, "`inset`" for a 3D inset rule, "`outset`" for a 3D outset rule, "`initial`" or
#'        "`inherit`" (see `fill` for descriptions of those).
#' @param rule_width specifies the width of the rule between columns; one of "`medium`" (the default), "`thin`", "`thick`",
#'        a _length_ CSS size value, `initial`" or "`inherit`" (see `fill` for descriptions of those).
#' @param span specifies how many columns an element should span across; one of "`none`" (the default) so the element spans
#'        across one column, "`all`" (spans across all columns), "`initial`" or "`inherit`"
#'        (see `fill` for descriptions of those).
#' @param style CSS style properties (complete text spec) that will be put into an `{htmltools}` `tags$style()` call that
#'        will come along for the ride with the `<div>`; useful for specifying `<p>` properties for each item of the
#'        `words` vector
#' @note No validation is done on inputs
#' @export
#' @examples
#' columnize(state.name, ncol = 3, rule_color = "black", rule_width = "0.5px")
columnize <- function(words,
                      div_id = NULL,
                      div_class = NULL,
                      ncol = 5,
                      width = "auto",
                      fill = "balance",
                      gap = "normal",
                      rule_color = "initial",
                      rule_style = "none",
                      rule_width = "medium",
                      span = "none",
                      style = "p {font-family:'Roboto Condensed';font-size:12pt;line-height:12.5pt;padding:0;margin:0}") {

  tagList(
    tagList(
     do.call(tags$style, as.list(style)),
    ),
    tags$div(
      id = div_id,
      class = div_class,
      words %>%
        map(tags$p) %>%
        tagList(),
      style = sprintf(
        paste0(c(
          "column-count:%s",
          "column-fill: %s",
          "column-gap: %s",
          "column-rule-color: %s",
          "column-rule-style: %s",
          "column-rule-width: %s",
          "column-span: %s",
          "column-width: %s"
        ), collapse = ";"),
        ncol, fill, gap, rule_color, rule_style, rule_width, span, width
      )
    )
  )

}

So now you can do something like:

columnize(
  div_id = "states",
  words = state.name, 
  ncol = 3, 
  rule_color = "black", 
  rule_style = "solid", 
  rule_width = "2px",
  style = c(
    "#states { width: 50%; text-align: center };\np {font-family:'Roboto Condensed'}",
    "p { font-family: 'sans-serif'}"
  )
) %>% 
  htmltools::html_print()

and get: