<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2018/05/30/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Most modern operating systems keep secrets from you in many ways. One of these ways is by associating <a rel="nofollow" href="https://en.wikipedia.org/wiki/Extended_file_attributes" target="_blank">extended file attributes</a> with files. These attributes can serve useful purposes. For instance, macOS uses them to identify when files have passed through the <a rel="nofollow" href="https://support.apple.com/en-us/HT202491" target="_blank">Gatekeeper</a> or to store the URLs of files that were downloaded via Safari (though most other browsers add the <code>com.apple.metadata:kMDItemWhereFroms</code> attribute now, too).</p>
<p>Attributes are nothing more than a series of key/value pairs. They key must be a character value &amp; unique, and it’s fairly standard practice to keep the value component under 4K. Apart from that, you can put anything in the value: text, binary content, etc.</p>
<p>When you’re in a terminal session you can tell that a file has extended attributes by looking for an <code>@</code> sign near the permissions column:</p>
<pre>
$ cd ~/Downloads
$ ls -l
total 264856
-rw-r--r--@ 1 user  staff     169062 Nov 27  2017 1109.1968.pdf
-rw-r--r--@ 1 user  staff     171059 Nov 27  2017 1109.1968v1.pdf
-rw-r--r--@ 1 user  staff     291373 Apr 27 21:25 1804.09970.pdf
-rw-r--r--@ 1 user  staff    1150562 Apr 27 21:26 1804.09988.pdf
-rw-r--r--@ 1 user  staff     482953 May 11 12:00 1805.01554.pdf
-rw-r--r--@ 1 user  staff  125822222 May 14 16:34 RStudio-1.2.627.dmg
-rw-r--r--@ 1 user  staff    2727305 Dec 21 17:50 athena-ug.pdf
-rw-r--r--@ 1 user  staff      90181 Jan 11 15:55 bgptools-0.2.tar.gz
-rw-r--r--@ 1 user  staff    4683220 May 25 14:52 osquery-3.2.4.pkg
</pre>
<p>You can work with extended attributes from the terminal with the <code>xattr</code> command, but do you really want to go to the terminal every time you want to examine these secret settings (now that you know your OS is keeping secrets from you)?</p>
<p>I didn’t think so. Thus begat the <a rel="nofollow" href="https://github.com/hrbrmstr/xattrs" target="_blank"><code>xattrs</code><img alt="🔗" src="https://i2.wp.com/s.w.org/images/core/emoji/2.4/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> package</a>.</p>
<h3>Exploring Past Downloads</h3>
<p>Data scientists are (generally) inquisitive folk and tend to accumulate things. We grab papers, data, programs (etc.) and some of those actions are performed in browsers. Let’s use the <code>xattrs</code> package to rebuild a list of download URLs from the extended attributes on the files located in <code>~/Downloads</code> (if you’ve chosen a different default for your browsers, use that directory).</p>
<p>We’re not going to work with the entire package in this post (it’s really straightforward to use and has a README on the GitHub site along with extensive examples) but I’ll use one of the example files from the directory listing above to demonstrate a couple functions before we get to the main example.</p>
<p>First, let’s see what is hidden with the RStudio disk image:</p>
<pre><code>
library(xattrs)
library(reticulate) # not 100% necessary but you'll see why later
library(tidyverse) # we'll need this later

list_xattrs(&quot;~/Downloads/RStudio-1.2.627.dmg&quot;)
## [1] &quot;com.apple.diskimages.fsck&quot;            &quot;com.apple.diskimages.recentcksum&quot;    
## [3] &quot;com.apple.metadata:kMDItemWhereFroms&quot; &quot;com.apple.quarantine&quot;   
</code></pre>
<p>There are four keys we can poke at, but the one that will help transition us to a larger example is <code>com.apple.metadata:kMDItemWhereFroms</code>. This is the key Apple has standardized on to store the source URL of a downloaded item. Let’s take a look:</p>
<pre><code>
get_xattr_raw(&quot;~/Downloads/RStudio-1.2.627.dmg&quot;, &quot;com.apple.metadata:kMDItemWhereFroms&quot;)
##   [1] 62 70 6c 69 73 74 30 30 a2 01 02 5f 10 4c 68 74 74 70 73 3a 2f 2f 73 33 2e 61 6d 61
##  [29] 7a 6f 6e 61 77 73 2e 63 6f 6d 2f 72 73 74 75 64 69 6f 2d 69 64 65 2d 62 75 69 6c 64
##  [57] 2f 64 65 73 6b 74 6f 70 2f 6d 61 63 6f 73 2f 52 53 74 75 64 69 6f 2d 31 2e 32 2e 36
##  [85] 32 37 2e 64 6d 67 5f 10 2c 68 74 74 70 73 3a 2f 2f 64 61 69 6c 69 65 73 2e 72 73 74
## [113] 75 64 69 6f 2e 63 6f 6d 2f 72 73 74 75 64 69 6f 2f 6f 73 73 2f 6d 61 63 2f 08 0b 5a
## [141] 00 00 00 00 00 00 01 01 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00
## [169] 00 00 00 89
</code></pre>
<p>Why “raw”? Well, as noted above, the value component of these attributes can store anything and this one definitely has embedded nul[l]s (<code>0x00</code>) in it. We can try to read it as a string, though:</p>
<pre><code>
get_xattr(&quot;~/Downloads/RStudio-1.2.627.dmg&quot;, &quot;com.apple.metadata:kMDItemWhereFroms&quot;)
## [1] &quot;bplist00\xa2\001\002_\020Lhttps://s3.amazonaws.com/rstudio-ide-build/desktop/macos/RStudio-1.2.627.dmg_\020,https://dailies.rstudio.com/rstudio/oss/mac/\b\vZ&quot;
</code></pre>
<p>So, we can <em>kinda</em> figure out the URL but it’s definitely not pretty. The general practice of Safari (and other browsers) is to use a <a rel="nofollow" href="https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/PropertyLists/AboutPropertyLists/AboutPropertyLists.html" target="_blank">binary property list</a> to store metadata in the value component of an extended attribute (at least for these URL references).</p>
<p>There will eventually be a native <a rel="nofollow" href="https://github.com/ebarnard/rust-plist" target="_blank">Rust-backed</a> property list reading package for R, but we can work with that binary plist data in two ways: first, via the <code>read_bplist()</code> function that comes with the <code>xattrs</code> package and wraps Linux/BSD or macOS system utilities (which are super expensive since it also means writing out data to a file each time) <em>or</em> turn to Python which already has this capability. We’re going to use the latter.</p>
<p>I like to prime the Python setup with <code>invisible(py_config())</code> but that is not really necessary (I do it mostly b/c I have a wild number of Python — don’t judge — installs and use the <code>RETICULATE_PYTHON</code> env var for the one I use with R). You’ll need to install the <code>biplist</code> module via <code>pip3 install bipist</code> or <code>pip install bipist</code> depending on your setup. I highly recommended using Python 3.x vs 2.x, though.</p>
<pre><code>
biplist &lt;- import(&quot;biplist&quot;, as=&quot;biplist&quot;)

biplist$readPlistFromString(
  get_xattr_raw(
    &quot;~/Downloads/RStudio-1.2.627.dmg&quot;, &quot;com.apple.metadata:kMDItemWhereFroms&quot;
  )
)
## [1] &quot;https://s3.amazonaws.com/rstudio-ide-build/desktop/macos/RStudio-1.2.627.dmg&quot;
## [2] &quot;https://dailies.rstudio.com/rstudio/oss/mac/&quot; 
</code></pre>
<p>That's <em>much</em> better.</p>
<p>Let's work with metadata for the whole directory:</p>
<pre><code>
list.files(&quot;~/Downloads&quot;, full.names = TRUE) %>% 
  keep(has_xattrs) %>% 
  set_names(basename(.)) %>% 
  map_df(read_xattrs, .id=&quot;file&quot;) -> xdf

xdf
## # A tibble: 24 x 4
##    file            name                                  size contents   
##    </code></pre>