rud.is

My {cdcfluview} package started tossing erros on CRAN just over a week ago when the CDC added an extra parameter to one of the hidden API endpoints that the package wraps. After a fairly hectic set of days since said NOTE came, I had time this morning to poke at a fix. There are alot of tests, so after successful debugging session I was awaiting CRAN checks on various remotes as well as README builds and figured I’d keep up some practice with another, nascent, package of mine, {swiftr}, which makes it dead simple to build R functions from Swift code, in similar fashion to what Rcpp::cppFunction() does for C/C++ code.

macOS comes with a full set of machine learning/AI libraries/frameworks that definitely have “batteries included” (i.e. you can almost just make one function call to get 90-95% what you want without even training new models). One of which is text extraction from Apple’s computer Vision framework. I thought it’d be a fun and quick “wait mode” distraction to wrap the VNRecognizeTextRequest() function and use it from R.

To show how capable the default model is, I pulled a semi-complex random image from DDG’s image search:

Let’s build the function (you need to be on macOS for this; exposition inine):

library(swiftr) # github.com/hrbrmstr/swiftr

swift_function(
  code = '
import Foundation
import CoreImage
import Cocoa
import Vision

@_cdecl ("detect_text")
public func detect_text(path: SEXP) -> SEXP {

   // turn R string into Swift String so we can use it
   let fileName = String(cString: R_CHAR(STRING_ELT(path, 0)))

   var res: String = ""
   var out: SEXP = R_NilValue

  // get image into the right format
  if let ciImage = CIImage(contentsOf: URL(fileURLWithPath:fileName)) {

    let context = CIContext(options: nil)
    if let img = context.createCGImage(ciImage, from: ciImage.extent) {

      // setup comptuer vision request
      let requestHandler = VNImageRequestHandler(cgImage: img)

      // start recognition
      let request = VNRecognizeTextRequest()
      do {
        try requestHandler.perform([request])

        // if we have results
        if let observations = request.results as? [VNRecognizedTextObservation] {

          // paste them together
          let recognizedStrings = observations.compactMap { observation in
            observation.topCandidates(1).first?.string
          }
          res = recognizedStrings.joined(separator: "\\n")
        }
      } catch {
        debugPrint("\\(error)")
      }
    }
  }

  res.withCString { cstr in out = Rf_mkString(cstr) }

  return(out)
}
')

The detect_text() is now available in R, so let’s see how it performs on that image of signs:

detect_text(path.expand("~/Data/signs.jpeg")) %>% 
  stringi::stri_split_lines() %>% 
  unlist()
##  [1] "BEWILDERED" "UNCLEAR"    "nAZEU"      "UNCERTAIN"  "VISA"       "INSURE"    
##  [7] "ATED"       "MUDDLED"    "LOsT"       "DISTRACTED" "PERPLEXED"  "CONFUSED"  
## [13] "PUZZLED" 

It works super-fast and gets far more correct than I would have expected.

Toy examples aside, it also works pretty well (as one would expect) on “real” text images, such as this example from the Tesseract test suite:

detect_text(path.expand("~/Data/tesseract/news.3B/0/8200_006.3B.tif")) %>% 
  stringi::stri_split_lines() %>% 
  unlist()
##  [1] "Tobacco chiefs still refuse to see the truth abou"                           
##  [2] "even of America's least conscionable"                                        
##  [3] "The tobacco industry would like to promote"                                  
##  [4] "men sat together in Washington last"                                         
##  [5] "under the conditions they are used.'"                                        
##  [6] "week to do what they do best: blow"                                          
##  [7] "the specter of prohibition."                                                 
##  [8] "panel\" of toxicologists as \"not hazardous"                                 
##  [9] "smoke at the truth about cigarettes."                                        
## [10] "'If cigarettes are too dangerous to be sold,"                                
## [11] "then ban them. Some smokers will obey the"                                   
## [12] "People not paid by the tobacco companies"                                    
## [13] "aren't so sure. The list includes several"                                   
## [14] "The CEOs of the nation's largest tobacco"                                    
## [15] "firms told congressional panel that nicotine"                                
## [16] "law, but many will not. People will be selling"                              
## [17] "iS not addictive, that they are unconvinced"                                 
## [18] "cigarettes out of the trunks of cars, cigarettes"                            
## [19] "substances the government does not allow in"                                 
## [20] "foods or classifies as potentially toxic. They"                              
## [21] "that smoking causes lung cancer or any other"                                
## [22] "made by who knows who, made of who knows include ammonia, a pesticide called"
## [23] "illness, and that smoking is no more harmful"                                
## [24] "what,\" said James Johnston of R.J. Reynolds."                               
## [25] "than drinking coffee or eating Twinkies."                                    
## [26] "It's a ruse. He knows cigarettes are not"                                    
## [27] "methoprene, and ethyl furoate, which has"                                    
## [28] "They said these things with straight taces."                                 
## [29] "going to be banned, at leasi not in his lifetime."                           
## [30] "caused liver damage in rats."                                                
## [31] "The list \"begs a number of important"                                       
## [32] "They said them in the face of massive"                                       
## [33] "STEVE WILSON"                                                                
## [34] "What he really fears are new taxes, stronger"                                
## [35] "questions about the safety of these additives,\""                            
## [36] "scientific evidence that smoking is responsible"                             
## [37] "anti-smoking campaigns, further smoking"                                     
## [38] "said a joint statement from the American"                                    
## [39] "for more than 400,000 deaths every year."                                    
## [40] "restrictions, limits on secondhand smoke and"                                
## [41] "Rep. Henry Waxman, D-Calif., put that"                                       
## [42] "Republic Columnist"                                                          
## [43] "Lung, Cancer and Heart associations. The"                                    
## [44] "limits on tar and nicotine."                                                 
## [45] "statement added that substances safe to eat"                                 
## [46] "frightful statistic another way:"                                            
## [47] "Collectively, these steps can accelerate the"                                
## [48] "\"Imagine our nation's outrage if two fully"                                 
## [49] "He and the others played dumb for the"                                       
## [50] "current 5 percent annual decline in cigarette"                               
## [51] "aren't necessarily safe to inhale."                                          
## [52] "The 50-page list can be obtained free by"                                    
## [53] "loaded jumbo jets crashed each day, killing all"                             
## [54] "entire six hours, but really didn't matter."                                 
## [55] "use and turn the tobacco business from highly"                               
## [56] "calling 1-800-852-8749."                                                     
## [57] "aboard. That's the same number of Americans"                                 
## [58] "The game i nearly over, and the tobacco"                                     
## [59] "profitable to depressed."                                                    
## [60] "Johnson's comment about cigarettes \"made"                                   
## [61] "Here are just the 44 ingredients that start"                                 
## [62] "that cigarettes kill every 24 hours.'"                                       
## [63] "executives know it."                                                         
## [64] "with the letter \"A\":"                                                      
## [65] "The CEOs were not impressed."                                                
## [66] "The hearing marked a turning point in the"                                   
## [67] "of who knows what\" was comical."                                            
## [68] "Acetanisole, acetic acid, acetoin,"                                          
## [69] "\"We have looked at the data."                                               
## [70] "It does"                                                                     
## [71] "nation's growing aversion to cigarettes. No"                                 
## [72] "The day before the hearing, the tobacco"                                     
## [73] "acetophenone,6-acetoxydihydrotheaspirane,"                                   
## [74] "not convince me that smoking causes death,\""                                
## [75] "2-acetyl-3-ethylpyrazine, 2-acetyl-5-"                                       
## [76] "said Andrew Tisch of the Lorillard Tobacco"                                  
## [77] "longer hamstrung by tobacco-state seniority"                                 
## [78] "companies released a long-secret list of 599"                                
## [79] "Co."                                                                         
## [80] "and the deep-pocketed tobacco lobby,"                                        
## [81] "methylfuran, acetylpyrazine, 2-acetylpyridine,"                              
## [82] "Congress is taking aim at cigarette makers."                                 
## [83] "additives used in cigarettes. The companies"                                 
## [84] "said all are certified by an \"independent"                                  
## [85] "3-acetylpyridine, 2-acetylthiazole, aconitic"   

(You can compare that on your own with the Tesseract results.)

FIN

{cdcfluview} checks are done, and the fixed functions are back on CRAN! Just in time to close out this post.

If you’re on macOS, definitely check out the various ML/AI frameworks Apple has to offer via Swift and have some fun playing with integrating them into R (or build some small, command line utilities if you want to keep Swift and R apart).

The last post showed how to work with the macOS mdls command line XML output, but with {swiftr} we can avoid the command line round trip by bridging the low-level Spotlight API (which mdls uses) directly in R via Swift.

If you’ve already played with {swiftr} before but were somewhat annoyed at various boilerplate elements you’ve had to drag along with you every time you used swift_function() you’ll be pleased that I’ve added some SEXP conversion helpers to the {swiftr} package, so there’s less cruft when using swift_function().

Let’s add an R↔Swift bridge function to retrieve all available Spotlight attributes for a macOS file:

library(swiftr)

swift_function('

  // Add an extension to URL which will retrieve the spotlight 
  // attributes as an array of Swift Strings
  extension URL {

  var mdAttributes: [String]? {

    get {
      guard isFileURL else { return nil }
      let item = MDItemCreateWithURL(kCFAllocatorDefault, self as CFURL)
      let attrs = MDItemCopyAttributeNames(item)!
      return(attrs as? [String])
    }

  }

}

@_cdecl ("file_attrs")
public func file_attrs(path: SEXP) -> SEXP {

  // Grab the attributres
  let outAttr = URL(fileURLWithPath: String(path)!).mdAttributes!

  // send them to R
  return(outAttr.SEXP!)

}
')

And, then try it out:

fil <-  "/Applications/RStudio.app"

file_attrs(fil)
##  [1] "kMDItemContentTypeTree"                 "kMDItemContentType"                    
##  [3] "kMDItemPhysicalSize"                    "kMDItemCopyright"                      
##  [5] "kMDItemAppStoreCategory"                "kMDItemKind"                           
##  [7] "kMDItemDateAdded_Ranking"               "kMDItemDocumentIdentifier"             
##  [9] "kMDItemContentCreationDate"             "kMDItemAlternateNames"                 
## [11] "kMDItemContentModificationDate_Ranking" "kMDItemDateAdded"                      
## [13] "kMDItemContentCreationDate_Ranking"     "kMDItemContentModificationDate"        
## [15] "kMDItemExecutableArchitectures"         "kMDItemAppStoreCategoryType"           
## [17] "kMDItemVersion"                         "kMDItemCFBundleIdentifier"             
## [19] "kMDItemInterestingDate_Ranking"         "kMDItemDisplayName"                    
## [21] "_kMDItemDisplayNameWithExtensions"      "kMDItemLogicalSize"                    
## [23] "kMDItemUsedDates"                       "kMDItemLastUsedDate"                   
## [25] "kMDItemLastUsedDate_Ranking"            "kMDItemUseCount"                       
## [27] "kMDItemFSName"                          "kMDItemFSSize"                         
## [29] "kMDItemFSCreationDate"                  "kMDItemFSContentChangeDate"            
## [31] "kMDItemFSOwnerUserID"                   "kMDItemFSOwnerGroupID"                 
## [33] "kMDItemFSNodeCount"                     "kMDItemFSInvisible"                    
## [35] "kMDItemFSTypeCode"                      "kMDItemFSCreatorCode"                  
## [37] "kMDItemFSFinderFlags"                   "kMDItemFSHasCustomIcon"                
## [39] "kMDItemFSIsExtensionHidden"             "kMDItemFSIsStationery"                 
## [41] "kMDItemFSLabel"   

No system() (et al.) round trip!

Now, lets make R↔Swift bridge function to retrieve the value of an attribute.

Before we do that, let me be up-front that relying on debugDescription (which makes a string representation of a Swift object) is a terrible hack that I’m using just to make the example as short as possible. We should do far more error checking and then further check the type of the object coming from the Spotlight API call and return an R-compatible version of that type. This mdAttr() method will almost certainly break depending on the item being returned.

swift_function('
extension URL {

  // Add an extension to URL which will retrieve the spotlight 
  // attribute value as a String. This will almost certainly die 
  // under various value conditions.

  func mdAttr(_ attr: String) -> String? {
    guard isFileURL else { return nil }
    let item = MDItemCreateWithURL(kCFAllocatorDefault, self as CFURL)
    return(MDItemCopyAttribute(item, attr as CFString).debugDescription!)
  }

}

@_cdecl ("file_attr")
public func file_attr(path: SEXP, attr: SEXP) -> SEXP {

  // file path as Swift String
  let xPath = String(cString: R_CHAR(Rf_asChar(path)))

  // attribute we want as a Swift String
  let xAttr = String(cString: R_CHAR(Rf_asChar(attr)))

  // the Swift debug string value of the attribute
  let outAttr = URL(fileURLWithPath: xPath).mdAttr(xAttr)

  // returned as an R string
  return(Rf_mkString(outAttr))
}
')

And try this out on some carefully selected attributes:

file_attr(fil, "kMDItemDisplayName")
## [1] "RStudio.app"

file_attr(fil, "kMDItemAppStoreCategory")
## [1] "Developer Tools"

file_attr(fil, "kMDItemVersion")
## [1] "1.4.1651"

Note that if we try to get fancy and retrieve an attribute value that is something like an array of strings, it doesn’t work so well:

file_attr(fil, "kMDItemExecutableArchitectures")
## [1] "<__NSSingleObjectArrayI 0x7fe1f6d19bf0>(\nx86_64\n)\n"

Again, ideally, we’d make a small package wrapper vs use swift_function() for this in production, but I wanted to show how straightforward it can be to get access to some fun and potentially powerful features of macOS right in R with just a tiny bit of Swift glue code.

Also, I hadn’t tried {swiftr} on the M1 Mini before and it seems I need to poke a bit to see what needs doing to get it to work properly in the arm64 RStudio rsession.

UPDATE (2021-04-14 a bit later)

It dawned on me that a minor tweak to the Swift mdAttr() function would make the method more resilient (but still hacky):

  func mdAttr(_ attr: String) -> String {
    guard isFileURL else { return "" }
    let item = MDItemCreateWithURL(kCFAllocatorDefault, self as CFURL)
    let x = MDItemCopyAttribute(item, attr as CFString)
    if (x == nil) {
      return("")
    } else {
      return("\(x!)")
    }
  }

Now we can (more) safely do something like this:

str(as.list(sapply(
  file_attrs(fil),
  function(attr) {
    file_attr(fil, attr)
  }
)), 1)
## List of 41
##  $ kMDItemContentTypeTree                : chr "(\n    \"com.apple.application-bundle\",\n    \"com.apple.application\",\n    \"public.executable\",\n    \"com"| __truncated__
##  $ kMDItemContentType                    : chr "com.apple.application-bundle"
##  $ kMDItemPhysicalSize                   : chr "767619072"
##  $ kMDItemCopyright                      : chr "RStudio 1.4.1651, © 2009-2021 RStudio, PBC"
##  $ kMDItemAppStoreCategory               : chr "Developer Tools"
##  $ kMDItemKind                           : chr "Application"
##  $ kMDItemDateAdded_Ranking              : chr "2021-04-09 00:00:00 +0000"
##  $ kMDItemDocumentIdentifier             : chr "0"
##  $ kMDItemContentCreationDate            : chr "2021-03-25 23:08:34 +0000"
##  $ kMDItemAlternateNames                 : chr "(\n    \"RStudio.app\"\n)"
##  $ kMDItemContentModificationDate_Ranking: chr "2021-03-25 00:00:00 +0000"
##  $ kMDItemDateAdded                      : chr "2021-04-09 13:25:11 +0000"
##  $ kMDItemContentCreationDate_Ranking    : chr "2021-03-25 00:00:00 +0000"
##  $ kMDItemContentModificationDate        : chr "2021-03-25 23:08:34 +0000"
##  $ kMDItemExecutableArchitectures        : chr "(\n    \"x86_64\"\n)"
##  $ kMDItemAppStoreCategoryType           : chr "public.app-category.developer-tools"
##  $ kMDItemVersion                        : chr "1.4.1651"
##  $ kMDItemCFBundleIdentifier             : chr "org.rstudio.RStudio"
##  $ kMDItemInterestingDate_Ranking        : chr "2021-04-15 00:00:00 +0000"
##  $ kMDItemDisplayName                    : chr "RStudio.app"
##  $ _kMDItemDisplayNameWithExtensions     : chr "RStudio.app"
##  $ kMDItemLogicalSize                    : chr "763253198"
##  $ kMDItemUsedDates                      : chr "(\n    \"2021-03-26 04:00:00 +0000\",\n    \"2021-03-30 04:00:00 +0000\",\n    \"2021-04-02 04:00:00 +0000\",\n"| __truncated__
##  $ kMDItemLastUsedDate                   : chr "2021-04-15 00:21:45 +0000"
##  $ kMDItemLastUsedDate_Ranking           : chr "2021-04-15 00:00:00 +0000"
##  $ kMDItemUseCount                       : chr "12"
##  $ kMDItemFSName                         : chr "RStudio.app"
##  $ kMDItemFSSize                         : chr "763253198"
##  $ kMDItemFSCreationDate                 : chr "2021-03-25 23:08:34 +0000"
##  $ kMDItemFSContentChangeDate            : chr "2021-03-25 23:08:34 +0000"
##  $ kMDItemFSOwnerUserID                  : chr "501"
##  $ kMDItemFSOwnerGroupID                 : chr "80"
##  $ kMDItemFSNodeCount                    : chr "1"
##  $ kMDItemFSInvisible                    : chr "0"
##  $ kMDItemFSTypeCode                     : chr "0"
##  $ kMDItemFSCreatorCode                  : chr "0"
##  $ kMDItemFSFinderFlags                  : chr "0"
##  $ kMDItemFSHasCustomIcon                : chr ""
##  $ kMDItemFSIsExtensionHidden            : chr "1"
##  $ kMDItemFSIsStationery                 : chr ""
##  $ kMDItemFSLabel                        : chr "0"

We’re still better off (in the long run) checking for and using proper types.

FIN

I hope to be able to carve out some more time in the not-too-distant-future for both {swiftr} and the in-progress guide on using Swift and R, but hopefully this post [re-]piqued interest in this topic for some R and/or Swift users.