rud.is

My {cdcfluview} package started tossing erros on CRAN just over a week ago when the CDC added an extra parameter to one of the hidden API endpoints that the package wraps. After a fairly hectic set of days since said NOTE came, I had time this morning to poke at a fix. There are alot of tests, so after successful debugging session I was awaiting CRAN checks on various remotes as well as README builds and figured I’d keep up some practice with another, nascent, package of mine, {swiftr}, which makes it dead simple to build R functions from Swift code, in similar fashion to what Rcpp::cppFunction() does for C/C++ code.

macOS comes with a full set of machine learning/AI libraries/frameworks that definitely have “batteries included” (i.e. you can almost just make one function call to get 90-95% what you want without even training new models). One of which is text extraction from Apple’s computer Vision framework. I thought it’d be a fun and quick “wait mode” distraction to wrap the VNRecognizeTextRequest() function and use it from R.

To show how capable the default model is, I pulled a semi-complex random image from DDG’s image search:

Let’s build the function (you need to be on macOS for this; exposition inine):

library(swiftr) # github.com/hrbrmstr/swiftr

swift_function(
  code = '
import Foundation
import CoreImage
import Cocoa
import Vision

@_cdecl ("detect_text")
public func detect_text(path: SEXP) -> SEXP {

   // turn R string into Swift String so we can use it
   let fileName = String(cString: R_CHAR(STRING_ELT(path, 0)))

   var res: String = ""
   var out: SEXP = R_NilValue

  // get image into the right format
  if let ciImage = CIImage(contentsOf: URL(fileURLWithPath:fileName)) {

    let context = CIContext(options: nil)
    if let img = context.createCGImage(ciImage, from: ciImage.extent) {

      // setup comptuer vision request
      let requestHandler = VNImageRequestHandler(cgImage: img)

      // start recognition
      let request = VNRecognizeTextRequest()
      do {
        try requestHandler.perform([request])

        // if we have results
        if let observations = request.results as? [VNRecognizedTextObservation] {

          // paste them together
          let recognizedStrings = observations.compactMap { observation in
            observation.topCandidates(1).first?.string
          }
          res = recognizedStrings.joined(separator: "\\n")
        }
      } catch {
        debugPrint("\\(error)")
      }
    }
  }

  res.withCString { cstr in out = Rf_mkString(cstr) }

  return(out)
}
')

The detect_text() is now available in R, so let’s see how it performs on that image of signs:

detect_text(path.expand("~/Data/signs.jpeg")) %>% 
  stringi::stri_split_lines() %>% 
  unlist()
##  [1] "BEWILDERED" "UNCLEAR"    "nAZEU"      "UNCERTAIN"  "VISA"       "INSURE"    
##  [7] "ATED"       "MUDDLED"    "LOsT"       "DISTRACTED" "PERPLEXED"  "CONFUSED"  
## [13] "PUZZLED" 

It works super-fast and gets far more correct than I would have expected.

Toy examples aside, it also works pretty well (as one would expect) on “real” text images, such as this example from the Tesseract test suite:

detect_text(path.expand("~/Data/tesseract/news.3B/0/8200_006.3B.tif")) %>% 
  stringi::stri_split_lines() %>% 
  unlist()
##  [1] "Tobacco chiefs still refuse to see the truth abou"                           
##  [2] "even of America's least conscionable"                                        
##  [3] "The tobacco industry would like to promote"                                  
##  [4] "men sat together in Washington last"                                         
##  [5] "under the conditions they are used.'"                                        
##  [6] "week to do what they do best: blow"                                          
##  [7] "the specter of prohibition."                                                 
##  [8] "panel\" of toxicologists as \"not hazardous"                                 
##  [9] "smoke at the truth about cigarettes."                                        
## [10] "'If cigarettes are too dangerous to be sold,"                                
## [11] "then ban them. Some smokers will obey the"                                   
## [12] "People not paid by the tobacco companies"                                    
## [13] "aren't so sure. The list includes several"                                   
## [14] "The CEOs of the nation's largest tobacco"                                    
## [15] "firms told congressional panel that nicotine"                                
## [16] "law, but many will not. People will be selling"                              
## [17] "iS not addictive, that they are unconvinced"                                 
## [18] "cigarettes out of the trunks of cars, cigarettes"                            
## [19] "substances the government does not allow in"                                 
## [20] "foods or classifies as potentially toxic. They"                              
## [21] "that smoking causes lung cancer or any other"                                
## [22] "made by who knows who, made of who knows include ammonia, a pesticide called"
## [23] "illness, and that smoking is no more harmful"                                
## [24] "what,\" said James Johnston of R.J. Reynolds."                               
## [25] "than drinking coffee or eating Twinkies."                                    
## [26] "It's a ruse. He knows cigarettes are not"                                    
## [27] "methoprene, and ethyl furoate, which has"                                    
## [28] "They said these things with straight taces."                                 
## [29] "going to be banned, at leasi not in his lifetime."                           
## [30] "caused liver damage in rats."                                                
## [31] "The list \"begs a number of important"                                       
## [32] "They said them in the face of massive"                                       
## [33] "STEVE WILSON"                                                                
## [34] "What he really fears are new taxes, stronger"                                
## [35] "questions about the safety of these additives,\""                            
## [36] "scientific evidence that smoking is responsible"                             
## [37] "anti-smoking campaigns, further smoking"                                     
## [38] "said a joint statement from the American"                                    
## [39] "for more than 400,000 deaths every year."                                    
## [40] "restrictions, limits on secondhand smoke and"                                
## [41] "Rep. Henry Waxman, D-Calif., put that"                                       
## [42] "Republic Columnist"                                                          
## [43] "Lung, Cancer and Heart associations. The"                                    
## [44] "limits on tar and nicotine."                                                 
## [45] "statement added that substances safe to eat"                                 
## [46] "frightful statistic another way:"                                            
## [47] "Collectively, these steps can accelerate the"                                
## [48] "\"Imagine our nation's outrage if two fully"                                 
## [49] "He and the others played dumb for the"                                       
## [50] "current 5 percent annual decline in cigarette"                               
## [51] "aren't necessarily safe to inhale."                                          
## [52] "The 50-page list can be obtained free by"                                    
## [53] "loaded jumbo jets crashed each day, killing all"                             
## [54] "entire six hours, but really didn't matter."                                 
## [55] "use and turn the tobacco business from highly"                               
## [56] "calling 1-800-852-8749."                                                     
## [57] "aboard. That's the same number of Americans"                                 
## [58] "The game i nearly over, and the tobacco"                                     
## [59] "profitable to depressed."                                                    
## [60] "Johnson's comment about cigarettes \"made"                                   
## [61] "Here are just the 44 ingredients that start"                                 
## [62] "that cigarettes kill every 24 hours.'"                                       
## [63] "executives know it."                                                         
## [64] "with the letter \"A\":"                                                      
## [65] "The CEOs were not impressed."                                                
## [66] "The hearing marked a turning point in the"                                   
## [67] "of who knows what\" was comical."                                            
## [68] "Acetanisole, acetic acid, acetoin,"                                          
## [69] "\"We have looked at the data."                                               
## [70] "It does"                                                                     
## [71] "nation's growing aversion to cigarettes. No"                                 
## [72] "The day before the hearing, the tobacco"                                     
## [73] "acetophenone,6-acetoxydihydrotheaspirane,"                                   
## [74] "not convince me that smoking causes death,\""                                
## [75] "2-acetyl-3-ethylpyrazine, 2-acetyl-5-"                                       
## [76] "said Andrew Tisch of the Lorillard Tobacco"                                  
## [77] "longer hamstrung by tobacco-state seniority"                                 
## [78] "companies released a long-secret list of 599"                                
## [79] "Co."                                                                         
## [80] "and the deep-pocketed tobacco lobby,"                                        
## [81] "methylfuran, acetylpyrazine, 2-acetylpyridine,"                              
## [82] "Congress is taking aim at cigarette makers."                                 
## [83] "additives used in cigarettes. The companies"                                 
## [84] "said all are certified by an \"independent"                                  
## [85] "3-acetylpyridine, 2-acetylthiazole, aconitic"   

(You can compare that on your own with the Tesseract results.)

FIN

{cdcfluview} checks are done, and the fixed functions are back on CRAN! Just in time to close out this post.

If you’re on macOS, definitely check out the various ML/AI frameworks Apple has to offer via Swift and have some fun playing with integrating them into R (or build some small, command line utilities if you want to keep Swift and R apart).

The last post showed how to work with the macOS mdls command line XML output, but with {swiftr} we can avoid the command line round trip by bridging the low-level Spotlight API (which mdls uses) directly in R via Swift.

If you’ve already played with {swiftr} before but were somewhat annoyed at various boilerplate elements you’ve had to drag along with you every time you used swift_function() you’ll be pleased that I’ve added some SEXP conversion helpers to the {swiftr} package, so there’s less cruft when using swift_function().

Let’s add an R↔Swift bridge function to retrieve all available Spotlight attributes for a macOS file:

library(swiftr)

swift_function('

  // Add an extension to URL which will retrieve the spotlight 
  // attributes as an array of Swift Strings
  extension URL {

  var mdAttributes: [String]? {

    get {
      guard isFileURL else { return nil }
      let item = MDItemCreateWithURL(kCFAllocatorDefault, self as CFURL)
      let attrs = MDItemCopyAttributeNames(item)!
      return(attrs as? [String])
    }

  }

}

@_cdecl ("file_attrs")
public func file_attrs(path: SEXP) -> SEXP {

  // Grab the attributres
  let outAttr = URL(fileURLWithPath: String(path)!).mdAttributes!

  // send them to R
  return(outAttr.SEXP!)

}
')

And, then try it out:

fil <-  "/Applications/RStudio.app"

file_attrs(fil)
##  [1] "kMDItemContentTypeTree"                 "kMDItemContentType"                    
##  [3] "kMDItemPhysicalSize"                    "kMDItemCopyright"                      
##  [5] "kMDItemAppStoreCategory"                "kMDItemKind"                           
##  [7] "kMDItemDateAdded_Ranking"               "kMDItemDocumentIdentifier"             
##  [9] "kMDItemContentCreationDate"             "kMDItemAlternateNames"                 
## [11] "kMDItemContentModificationDate_Ranking" "kMDItemDateAdded"                      
## [13] "kMDItemContentCreationDate_Ranking"     "kMDItemContentModificationDate"        
## [15] "kMDItemExecutableArchitectures"         "kMDItemAppStoreCategoryType"           
## [17] "kMDItemVersion"                         "kMDItemCFBundleIdentifier"             
## [19] "kMDItemInterestingDate_Ranking"         "kMDItemDisplayName"                    
## [21] "_kMDItemDisplayNameWithExtensions"      "kMDItemLogicalSize"                    
## [23] "kMDItemUsedDates"                       "kMDItemLastUsedDate"                   
## [25] "kMDItemLastUsedDate_Ranking"            "kMDItemUseCount"                       
## [27] "kMDItemFSName"                          "kMDItemFSSize"                         
## [29] "kMDItemFSCreationDate"                  "kMDItemFSContentChangeDate"            
## [31] "kMDItemFSOwnerUserID"                   "kMDItemFSOwnerGroupID"                 
## [33] "kMDItemFSNodeCount"                     "kMDItemFSInvisible"                    
## [35] "kMDItemFSTypeCode"                      "kMDItemFSCreatorCode"                  
## [37] "kMDItemFSFinderFlags"                   "kMDItemFSHasCustomIcon"                
## [39] "kMDItemFSIsExtensionHidden"             "kMDItemFSIsStationery"                 
## [41] "kMDItemFSLabel"   

No system() (et al.) round trip!

Now, lets make R↔Swift bridge function to retrieve the value of an attribute.

Before we do that, let me be up-front that relying on debugDescription (which makes a string representation of a Swift object) is a terrible hack that I’m using just to make the example as short as possible. We should do far more error checking and then further check the type of the object coming from the Spotlight API call and return an R-compatible version of that type. This mdAttr() method will almost certainly break depending on the item being returned.

swift_function('
extension URL {

  // Add an extension to URL which will retrieve the spotlight 
  // attribute value as a String. This will almost certainly die 
  // under various value conditions.

  func mdAttr(_ attr: String) -> String? {
    guard isFileURL else { return nil }
    let item = MDItemCreateWithURL(kCFAllocatorDefault, self as CFURL)
    return(MDItemCopyAttribute(item, attr as CFString).debugDescription!)
  }

}

@_cdecl ("file_attr")
public func file_attr(path: SEXP, attr: SEXP) -> SEXP {

  // file path as Swift String
  let xPath = String(cString: R_CHAR(Rf_asChar(path)))

  // attribute we want as a Swift String
  let xAttr = String(cString: R_CHAR(Rf_asChar(attr)))

  // the Swift debug string value of the attribute
  let outAttr = URL(fileURLWithPath: xPath).mdAttr(xAttr)

  // returned as an R string
  return(Rf_mkString(outAttr))
}
')

And try this out on some carefully selected attributes:

file_attr(fil, "kMDItemDisplayName")
## [1] "RStudio.app"

file_attr(fil, "kMDItemAppStoreCategory")
## [1] "Developer Tools"

file_attr(fil, "kMDItemVersion")
## [1] "1.4.1651"

Note that if we try to get fancy and retrieve an attribute value that is something like an array of strings, it doesn’t work so well:

file_attr(fil, "kMDItemExecutableArchitectures")
## [1] "<__NSSingleObjectArrayI 0x7fe1f6d19bf0>(\nx86_64\n)\n"

Again, ideally, we’d make a small package wrapper vs use swift_function() for this in production, but I wanted to show how straightforward it can be to get access to some fun and potentially powerful features of macOS right in R with just a tiny bit of Swift glue code.

Also, I hadn’t tried {swiftr} on the M1 Mini before and it seems I need to poke a bit to see what needs doing to get it to work properly in the arm64 RStudio rsession.

UPDATE (2021-04-14 a bit later)

It dawned on me that a minor tweak to the Swift mdAttr() function would make the method more resilient (but still hacky):

  func mdAttr(_ attr: String) -> String {
    guard isFileURL else { return "" }
    let item = MDItemCreateWithURL(kCFAllocatorDefault, self as CFURL)
    let x = MDItemCopyAttribute(item, attr as CFString)
    if (x == nil) {
      return("")
    } else {
      return("\(x!)")
    }
  }

Now we can (more) safely do something like this:

str(as.list(sapply(
  file_attrs(fil),
  function(attr) {
    file_attr(fil, attr)
  }
)), 1)
## List of 41
##  $ kMDItemContentTypeTree                : chr "(\n    \"com.apple.application-bundle\",\n    \"com.apple.application\",\n    \"public.executable\",\n    \"com"| __truncated__
##  $ kMDItemContentType                    : chr "com.apple.application-bundle"
##  $ kMDItemPhysicalSize                   : chr "767619072"
##  $ kMDItemCopyright                      : chr "RStudio 1.4.1651, © 2009-2021 RStudio, PBC"
##  $ kMDItemAppStoreCategory               : chr "Developer Tools"
##  $ kMDItemKind                           : chr "Application"
##  $ kMDItemDateAdded_Ranking              : chr "2021-04-09 00:00:00 +0000"
##  $ kMDItemDocumentIdentifier             : chr "0"
##  $ kMDItemContentCreationDate            : chr "2021-03-25 23:08:34 +0000"
##  $ kMDItemAlternateNames                 : chr "(\n    \"RStudio.app\"\n)"
##  $ kMDItemContentModificationDate_Ranking: chr "2021-03-25 00:00:00 +0000"
##  $ kMDItemDateAdded                      : chr "2021-04-09 13:25:11 +0000"
##  $ kMDItemContentCreationDate_Ranking    : chr "2021-03-25 00:00:00 +0000"
##  $ kMDItemContentModificationDate        : chr "2021-03-25 23:08:34 +0000"
##  $ kMDItemExecutableArchitectures        : chr "(\n    \"x86_64\"\n)"
##  $ kMDItemAppStoreCategoryType           : chr "public.app-category.developer-tools"
##  $ kMDItemVersion                        : chr "1.4.1651"
##  $ kMDItemCFBundleIdentifier             : chr "org.rstudio.RStudio"
##  $ kMDItemInterestingDate_Ranking        : chr "2021-04-15 00:00:00 +0000"
##  $ kMDItemDisplayName                    : chr "RStudio.app"
##  $ _kMDItemDisplayNameWithExtensions     : chr "RStudio.app"
##  $ kMDItemLogicalSize                    : chr "763253198"
##  $ kMDItemUsedDates                      : chr "(\n    \"2021-03-26 04:00:00 +0000\",\n    \"2021-03-30 04:00:00 +0000\",\n    \"2021-04-02 04:00:00 +0000\",\n"| __truncated__
##  $ kMDItemLastUsedDate                   : chr "2021-04-15 00:21:45 +0000"
##  $ kMDItemLastUsedDate_Ranking           : chr "2021-04-15 00:00:00 +0000"
##  $ kMDItemUseCount                       : chr "12"
##  $ kMDItemFSName                         : chr "RStudio.app"
##  $ kMDItemFSSize                         : chr "763253198"
##  $ kMDItemFSCreationDate                 : chr "2021-03-25 23:08:34 +0000"
##  $ kMDItemFSContentChangeDate            : chr "2021-03-25 23:08:34 +0000"
##  $ kMDItemFSOwnerUserID                  : chr "501"
##  $ kMDItemFSOwnerGroupID                 : chr "80"
##  $ kMDItemFSNodeCount                    : chr "1"
##  $ kMDItemFSInvisible                    : chr "0"
##  $ kMDItemFSTypeCode                     : chr "0"
##  $ kMDItemFSCreatorCode                  : chr "0"
##  $ kMDItemFSFinderFlags                  : chr "0"
##  $ kMDItemFSHasCustomIcon                : chr ""
##  $ kMDItemFSIsExtensionHidden            : chr "1"
##  $ kMDItemFSIsStationery                 : chr ""
##  $ kMDItemFSLabel                        : chr "0"

We’re still better off (in the long run) checking for and using proper types.

FIN

I hope to be able to carve out some more time in the not-too-distant-future for both {swiftr} and the in-progress guide on using Swift and R, but hopefully this post [re-]piqued interest in this topic for some R and/or Swift users.

The past two posts have (lightly) introduced how to use compiled Swift code in R, but they’ve involved a bunch of “scary” command line machinations and incantations.

One feature of {Rcpp} I’ve always 💙 is the cppFunction() (“r-lib” zealots have a similar cpp11::cpp_function()) which lets one experiment with C[++] code in R with as little friction as possible. To make it easier to start experimenting with Swift, I’ve built an extremely fragile swift_function() in {swiftr} that intends to replicate this functionality. Explaining it will be easier with an example.

Reading Property Lists in R With Swift

macOS relies heavily on property lists for many, many things. These can be plain text (XML) or binary files and there are command-line tools and Python libraries (usable via {reticulate}) that can read them along with the good ‘ol XML::readKeyValueDB(). We’re going to create a Swift function to read property lists and return JSON which we can use back in R via {jsonlite}.

This time around there’s no need to create extra files, just install {swiftr} and your favorite R IDE and enter the following (expository is after the code):

library(swiftr)

swift_function(
  code = '

func ignored() {
  print("""
this will be ignored by swift_function() but you could use private
functions as helpers for the main public Swift function which will be 
made available to R.
""")
}  

@_cdecl ("read_plist")
public func read_plist(path: SEXP) -> SEXP {

  var out: SEXP = R_NilValue

  do {
    // read in the raw plist
    let plistRaw = try Data(contentsOf: URL(fileURLWithPath: String(cString: R_CHAR(STRING_ELT(path, 0)))))

    // convert it to a PropertyList  
    let plist = try PropertyListSerialization.propertyList(from: plistRaw, options: [], format: nil) as! [String:Any]

    // serialize it to JSON
    let jsonData = try JSONSerialization.data(withJSONObject: plist , options: .prettyPrinted)

    // setup the JSON string return
    String(data: jsonData, encoding: .utf8)?.withCString { 
      cstr in out = Rf_mkString(cstr) 
    }

  } catch {
    debugPrint("\\(error)")
  }

  return(out)

}
')

This new swift_function() function — for the moment (the API is absolutely going to change) — is defined as:

swift_function(
  code,
  env = globalenv(),
  imports = c("Foundation"),
  cache_dir = tempdir()
)

where:

• code is a length 1 character vector of Swift code
• env is the environment to expose the function in (defaults to the global environment)
• imports is a character vector of any extra Swift frameworks that need to be imported
• cache_dir is where all the temporary files will be created and compiled dynlib will be stored. It defaults to a temporary directory so specify your own directory (that exists) if you want to keep the files around after you close the R session

Folks familiar with cppFunction() will notice some (on-purpose) similarities.

The function expects you to expose only one public Swift function which also (for the moment) needs to have the @_cdecl decorator before it. You can have as many other valid Swift helper functions as you like, but are restricted to one function that will be turned into an R function automagically.

In this example, swift_function() will see public func read_plist(path: SEXP) -> SEXP { and be able to identify

• the function name (read_plist)
• the number of parameters (they all need to be SEXP, for now)
• the names of the parameters

A complete source file with all the imports will be created and a pre-built bridging header (which comes along for the ride with {swiftr}) will be included in the compilation step and a dylib will be built and loaded into the R session. Finally, an R function that wraps a .Call() will be created and will have the function name of the Swift function as well as all the parameter names (if any).

In the case of our example, above, the built R function is:

function(path) {
  .Call("read_plist", path)
}

There’s a good chance you’re using RStudio, so we can test this with it’s property list, or you can substitute any other application’s property list (or any .plist you have) to test this out:

read_plist("/Applications/RStudio.app/Contents/Info.plist") %>% 
  jsonlite::fromJSON() %>% 
  str(1)
## List of 32
##  $ NSPrincipalClass                     : chr "NSApplication"
##  $ NSCameraUsageDescription             : chr "R wants to access the camera."
##  $ CFBundleIdentifier                   : chr "org.rstudio.RStudio"
##  $ CFBundleShortVersionString           : chr "1.4.1093-1"
##  $ NSBluetoothPeripheralUsageDescription: chr "R wants to access bluetooth."
##  $ NSRemindersUsageDescription          : chr "R wants to access the reminders."
##  $ NSAppleEventsUsageDescription        : chr "R wants to run AppleScript."
##  $ NSHighResolutionCapable              : logi TRUE
##  $ LSRequiresCarbon                     : logi TRUE
##  $ NSPhotoLibraryUsageDescription       : chr "R wants to access the photo library."
##  $ CFBundleGetInfoString                : chr "RStudio 1.4.1093-1, © 2009-2020 RStudio, PBC"
##  $ NSLocationWhenInUseUsageDescription  : chr "R wants to access location information."
##  $ CFBundleInfoDictionaryVersion        : chr "6.0"
##  $ NSSupportsAutomaticGraphicsSwitching : logi TRUE
##  $ CSResourcesFileMapped                : logi TRUE
##  $ CFBundleVersion                      : chr "1.4.1093-1"
##  $ OSAScriptingDefinition               : chr "RStudio.sdef"
##  $ CFBundleLongVersionString            : chr "1.4.1093-1"
##  $ CFBundlePackageType                  : chr "APPL"
##  $ NSContactsUsageDescription           : chr "R wants to access contacts."
##  $ NSCalendarsUsageDescription          : chr "R wants to access calendars."
##  $ NSMicrophoneUsageDescription         : chr "R wants to access the microphone."
##  $ CFBundleDocumentTypes                :'data.frame':  16 obs. of  8 variables:
##  $ NSPhotoLibraryAddUsageDescription    : chr "R wants write access to the photo library."
##  $ NSAppleScriptEnabled                 : logi TRUE
##  $ CFBundleExecutable                   : chr "RStudio"
##  $ CFBundleSignature                    : chr "Rstd"
##  $ NSHumanReadableCopyright             : chr "RStudio 1.4.1093-1, © 2009-2020 RStudio, PBC"
##  $ CFBundleName                         : chr "RStudio"
##  $ LSApplicationCategoryType            : chr "public.app-category.developer-tools"
##  $ CFBundleIconFile                     : chr "RStudio.icns"
##  $ CFBundleDevelopmentRegion            : chr "English"

FIN

A source_swift() function is on the horizon as is adding a ton of checks/validations to swift_function(). I’ll likely be adding some of the SEXP and R Swift utility functions I’ve demonstrated in the [unfinished] book to make it fairly painless to interface Swift and R code in this new and forthcoming function.

As usual, kick the tyres, submit feature requests and bugs in any forum that’s comfortable and stay strong, wear a 😷, and socially distanced when out and about.